From 30fd9d3ecdd67524c0eb47e9e115b74e268a7a7f Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 20 Jul 2026 09:09:25 +0200 Subject: [PATCH] =?UTF-8?q?docs(audits):=20remote-ops=20remediation=20?= =?UTF-8?q?=E2=80=94=20F1=20mitigated,=20F2=20closed,=20F3=20fixed=20(v0.1?= =?UTF-8?q?49.0),=20Immich=20resolved;=20R-50..R-53=20minted?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../architecture/00-capability-map.md | 1 + .../AUDIT-vacation-remote-ops-2026-07-20.md | 150 ++++++++++++++++++ documentation/backlog/ROADMAP.md | 4 + 3 files changed, 155 insertions(+) diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index f9f8d7c..c374290 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -63,6 +63,7 @@ | Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill | | PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | hub v0.56.0 (`pbsdrheal/reconciler.go`, `RestageHostPBSSecret`, all §10 red-proofs); `SPIKE-pbsdr-selfheal-2026-07-15` (root cause) | Reconciler is **scoped to one host** (`PBSDRHEAL_ONLY_HOST`), not fleet-wide; already fired live hands-free on drill qm300 (07-15) — real-customer firing + fleet-wide widening pending | | Crash/power-loss mid-backup/mid-migration → self-heal on next run | controller, agent | **PROVEN-LIVE** | `CAMPAIGN-6D` P5-REST (SIGKILL mid-offbox → auto-restart ~15s, run marked failed not false-success, no stale lock); `CAMPAIGN-6E` B1-B3 | (Cited `CAMPAIGN-2` T-RBT-* legs were empty / auth-hollow — corrected.) Live mid-**migration** crash→self-heal is the weakest sub-claim (P5-REST is mid-backup) | +| Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent, controller, bootstrap | **PARTIAL** | `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — a real relocation of the demo box: guest + hub telemetry + WG/PBS + Cloudflare tunnel all survived untouched, but the **controller↔agent control plane did not** (agent binds a LAN literal → `bind: cannot assign requested address` → storage/PBS-backup/quiesce/restore-test/DR down until fixed). Mitigated for the window by pinning `vmbr0` static | **R-50** (island-bridge control plane, spike-first) is the durable fix. Related: **R-51** (dead-primary alerting) and **R-52** (boot desired-state reconciliation) — the same event left two apps `Exited` with no alarm and no recovery | | Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | | | **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3. **2026-07-19:** the C6 evidence attempt ran and found a **product gap instead of evidence** — `audits/DIAG-immich-restore-2026-07-19.md`. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING **by finding, not by absence of attempt** — the rehearsal system working, not failing. **2026-07-19: the blocking product gaps are CLOSED in controller v0.148.0** (R-43 + R-44 shipped), so this row is now blocked only on the evidence run itself, not on missing capability. It flips the moment the §9 acceptance produces screenshots + the outcome flash + a snapshot ID. **2026-07-19 round 2 — PARTIAL EVIDENCE ONLY, row NOT flipped** (`audits/DIAG-immich-restore-round2-2026-07-19.md`): a deliberate run from snapshot `49e7cb46` did recover all 11 assets (`status=active`, files resolve), but the operation **reported failure** and left immich reporting schema drift, because the replay aborted against the running app (H4). Photos back ≠ clean acceptance. Blocked on H4 closing in v0.149, then one clean run. Method note for R-3's script: deleting in an app's own UI usually means *trash*, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash **and** verify the app's *content*, not the file count | diff --git a/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md b/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md index 601ea4c..c811c29 100644 --- a/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md +++ b/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md @@ -285,3 +285,153 @@ Nothing else was mutated. The ssh alias needed **no** edit (already on the tailn 5. Re-verify: agent `active (running)`, dashboard banner clear, Tárhely renders, a PBS backup completes, Immich healthy and reachable. 6. Flip this document's status from **ACTIVE** to **CLOSED**. + +--- + +# Remediation — 2026-07-20 + +Executed the same day as the audit, on the relocated host. Mutations: one network file, one service +start, one app start, two backup runs (one of them the product's own scheduler), one code line. + +## R.1 Static-IP flip (F1 window mitigation) + +**State found had already moved.** By the time remediation started, the router's reservation had +taken effect and `vmbr0` held **`192.168.0.162` again — but via DHCP** (`dhclient.vmbr0.leases` +stamped 08:13 CEST). So the address existed; nothing had retried the agent since systemd gave up. + +The 1.1 duplicate-address check needed interpretation, not blind obedience: `ping -c1 192.168.0.162` +**answered**, because the host was answering *itself*. The genuine test is from another node — +ARP for `.162` seen from guest 9201 resolved to `68:1d:ef:5d:a6:64`, which **is** this host's vmbr0 +MAC. No duplicate; the STOP did not apply. + +Interfaces diff (`.bak-2026-07-20` → new) — one stanza, nothing else touched: + +```diff + auto vmbr0 +-iface vmbr0 inet dhcp ++iface vmbr0 inet static ++ address 192.168.0.162/24 ++ gateway 192.168.0.1 + bridge-ports enp1s0 + bridge-stp off + bridge-fd 0 +``` + +The gateway was taken from the live default route (`192.168.0.1`), not from the spec. Applied +detached: `setsid ifreload -a` → log contained only `Killed old client process` (the vmbr0 dhclient). +Post-flip: `inet 192.168.0.162/24 scope global vmbr0` (no `dynamic` flag), `default via 192.168.0.1 +… proto kernel onlink`, `curl https://gitea.dooplex.hu` → **200**, tailscale peer still +`active; direct`. Because the address was unchanged, connectivity never blipped. + +## R.2 Agent recovery (Scenario A) + +`systemctl reset-failed && systemctl start felhom-agent` — up at **08:40:11 CEST**: + +``` +active +LISTEN 192.168.0.162:8443 users:(("felhom-agent",pid=72859)) +level=INFO msg="local-api server listening" addr=192.168.0.162:8443 +wg-felhom latest handshake: 1 minute, 16 seconds ago +``` + +Controller side, `agentapi` errors **stopped entirely** (no `no route to host`, no +`channel not verified` in the following 5 min) — so this is not the "agent active but controller +still erroring" wrong outcome the scenario warns about. On an authenticated fetch of the real +dashboard the red **„A tárolókezelő ügynök nem elérhető"** banner is **gone**; the only +`alert-warning` strings left on the page are two modal "this cannot be undone" confirmations. + +> **Method note for anyone repeating this:** grep the fetched page with **ASCII-only** patterns. +> Accented patterns sent through the ssh→`pct exec`→shell layers get mangled and return a +> false `0`, which reads exactly like "the banner is gone" when it is not. The page content itself +> is fine — it is the *pattern* that gets corrupted. + +## R.3 F2 — the protection gap closed itself, then a manual run confirmed it + +Restoring the agent was sufficient: the quiesce loop noticed the overdue whole-guest backup and ran +it **unprompted**, which is the strongest possible evidence the seam is healthy end to end: + +``` +06:40:36Z [quiesce] backup due — quiescing 3 stack(s): [bookstack calibre-web immich] +06:40:45Z [quiesce] backup job backup-9201-1784529645204076480 started — polling to completion +06:40:55Z [quiesce] … snapshotted — resuming app early (8B.2) +06:42:52Z [quiesce] … done +``` + +A manual app-data run then followed via the real UI endpoint (`POST /api/backup/run`, authenticated +session + `X-CSRF-Token` read from the page's `` — the same call the „Teljes mentés" button +makes), started 06:45:26Z: + +``` +06:45:27Z DB dump: bookstack-db → bookstack-mariadb.sql (55.9 KB, 228ms, 41 tables) +06:45:28Z DB dump: immich-postgres → immich-postgres.sql (49.9 MB, 1.28s, 66 tables) +06:45:44Z Volume dump: calibre-web/…_config → 360.0 KB +06:45:55Z Volume dump: immich/…_ml_cache → 785.5 MB +06:45:58Z Volume dump: immich/…_postgres_data → 241.5 MB +06:45:59Z Volume dump: immich/…_redis_data → 6.6 MB +06:46:10Z App-data backup completed: 2 databases (50.0 MB total), 3 volume dump(s) (43.348s) +``` + +**T = 2026-07-20 06:46:10Z** (run completion; newest dump artifact 06:45:28Z). Every container the +run stopped for a safe volume dump was restarted — all 12 containers verified `Up (healthy)` +afterwards. + +## R.4 Immich (Scenario: app triage) + +Started via the controller's real endpoint (`POST /api/stacks/immich/start`) → all four containers +`Up (healthy)`. + +**Correction to §3.3 of the audit.** That section reported the URL probe returning 404 and treated +it as a live symptom. The probe used the **wrong hostname**: the traefik router rule is +``Host(`photos.demo-felhom.eu`)``, not `immich.demo-felhom.eu`. The container-down diagnosis stands +(and the 404 was genuinely traefik having no backend), but the specific probe URL in §3.3 was +invalid. Post-start, on the correct host: `https://photos.demo-felhom.eu` → **200** in 0.28 s. + +## R.5 F3 — dashboard backup card (controller v0.149.0) + +Handler-side one-liner, no template change: `data["BackupStatus"] = fullStatus.LastDBDump` in +`dashboardHandler`. Deployed to guest 9201 and verified on the **real, authenticated dashboard**: + +``` +Utolsó mentés: +2026-07-20 06:45 +Adatbázisok: +3 mentve +``` + +Was „Még nem futott" on the identical page before the deploy. (The controller restart cleared the +in-memory run state, so the card renders from the dumps on disk — newest artifact 06:45 — which is +the honest value and exactly what the fresh-box branch must NOT fabricate.) + +## R.6 Per-finding status after remediation + +| # | Was | Now | +|---|-----|-----| +| **F1** | CRITICAL, open | **MITIGATED for the window** — `vmbr0` static `192.168.0.162/24`; agent up. **Durable fix still open → R-50** (island-bridge control plane, spike-first). Still depends on the site's subnet being `192.168.0.0/24` and `.162` free | +| **F2** | HIGH, open | **CLOSED** — whole-guest backup ran (06:42:52Z) + manual app-data run completed (T = 06:46:10Z) | +| **F3** | HIGH, open | **FIXED in controller v0.149.0**, live-verified on the dashboard | +| **Immich** | ROOT-CAUSED, down | **RESOLVED** — healthy, `photos.demo-felhom.eu` → 200 | +| **F4** | MEDIUM, open | Roadmap **R-51** (constraint recorded: must NOT simply fold `unhealthy` into down) | +| **F5** | MEDIUM, open | Roadmap **R-52** (includes root-causing the `unless-stopped` non-resurrection) | +| **F6** | LOW, open | **Already tracked** — it is R-39's remaining item (b), the agent's PBS `.pw` read path. No duplicate item minted | +| **F7** | LOW, open | Roadmap **R-53** | + +Capability map: added a **PARTIAL** row — *"Box survives a site/network change (relocation, +different subnet, DHCP re-lease) with the control plane intact"* — citing this audit and pointing at +R-50/R-51/R-52. + +## R.7 Not done / out of scope + +No agent code, guest bootstrap config, golden state, nftables, tailscale, router, guest-9201 +network, or `resolv.conf` was touched. No restores or drills. F4/F5/F6/F7 diagnosed and ranked only. + +## R.8 Return-day checklist — amendments + +Supersedes items 1 and 4 above: + +1a. **Decide whether static `.162` stays when the box comes home.** *Recommendation: keep it.* It + removes the home router's DHCP/reservation from the control plane's dependency chain, which is + the same class of dependency that broke this window. Either way, R-50 is the real fix — the + static pin is a crutch, and it will fail at any site whose subnet is not `192.168.0.0/24`. +4a. `/etc/network/interfaces.bak-2026-07-20` on the host is the pre-flip DHCP config — delete it + once the decision in 1a is made and verified. +7. Note the deployed controller is **v0.149.0** (was 0.148.0 at audit time). diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index f46e63a..2c6c87b 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -58,6 +58,10 @@ | R-35 | **Config-apply should not end the customer's session.** The offsite config push bumped `config_version` 10→11 at 16:54:58 and the controller self-restarted (container `StartedAt` 16:54:59Z, back up 16:55:02); in-memory sessions died with it and **customer zero was force-logged-out mid-flow**. | S | idea | Direction: **hot-apply the offbox target** (no restart for a config the running process can adopt), or **persist sessions** across restart. The restart itself is by design — the collateral is not. Evidence `controller-log-full.txt` | | R-36 | **Post-RESET re-enroll leaves offsite "enabled but unprovisioned" — silently.** The hub knows the state and says nothing on the customer page. | S | **SHIPPED (hub v0.67.0, 2026-07-18)** | Both halves delivered. **(1) The warning:** the customer page now names the state and the fix — enabled-but-unprovisioned raises an amber banner saying provisioning is *Save*-triggered (press Save once, then verify), reusing the exact `enabled && type == ""` predicate the offsite re-issue handler already refuses on. **(2) The related sub-item, also done:** the self-bind link is now **auto-minted at customer creation AND at RESET completion**, so the console banner's „e-mailben kapott link" is already true instead of true-once-the-operator-remembers. Extracting the shared `mintAndSendSelfBindLink` core keeps the button and the auto-mint callers on the same F1/F2 honesty rules, and the auto-mint never fails the operation it rides on. **Gap found and closed while wiring it:** `PurgeCustomerResetDBState` does NOT clear `selfbind_tokens`, so a link minted BEFORE a reset would have stayed live across it — the skip paths now clear stale tokens, giving the invariant "after auto-mint, the only live link is one we just issued, or none". Tests assert the banner is ABSENT in all three nominal cases too; red-proofed. — Original analysis: Source-cited behaviour, confirmed live in the rehearsal: **provisioning is Save-triggered** (`configs.go` `applyOffsite`) — which also answers S6's open question — and the re-enroll auto-re-issue **correctly** skips unprovisioned targets (`handler.go`). So nothing is broken; the gap is that nobody is told. Direction: flash it on the customer page. **Interim: an R-3 step.** **Related sub-item:** auto-mint the **self-bind link on customer create/RESET**, so the console banner's „e-mailben kapott link" is always already true instead of true-once-the-operator-remembers | | R-27c | **Customer self-bind, slice 2 — console-passphrase bind.** Viktor's direction: bind using a passphrase shown on the box console, alongside (not instead of) the emailed capability link. | M | idea | **Security constraints from the session ruling, all load-bearing:** passphrase **issued at customer creation**; the global-lookup endpoint must be **spray-hardened** — per-appliance **and** per-IP caps, constant-time comparison, a **single generic failure** (no oracle), alerting on abuse; an **accent-free wordlist** (console keymaps are not Hungarian); the **web capability-link path is RETAINED**; **claim-by-email is RETAINED** as the delivery-channel proof. **Also under this item:** the self-bind email gains the **public universal-ISO download link + two-line instructions** (the DIY case). **Secret-bearing per-customer ISOs are ruled OUT.** Sibling of R-27b (second-box flow) — different axis, both build on the same `/bind/` page | +| R-50 | **[P2-HIGH] Island-bridge control plane — make controller↔agent independent of the LAN.** The agent's `localapi` binds a **LAN literal** (`listen_addr`) and the guest dials that same literal from `bootstrap.json`. Move both onto a **host-internal bridge with a fixed, private address** that no router, DHCP lease, or site move can invalidate, so the control plane is location-independent and golden-image-stable. | M | **idea — SPIKE-FIRST** | Origin: `audits/AUDIT-vacation-remote-ops-2026-07-20.md` (F1), where this failed for real. The demo box moved to a remote site, DHCP handed it `.147` instead of `.162`, and the agent then **could not start at all** — `bind: cannot assign requested address`, systemd gave up after 4 retries — taking storage, PBS backup, quiesce, restore-test and DR down for as long as nobody noticed. Mitigated for that window by pinning `vmbr0` static back to `.162`; that is a **window mitigation, not the fix** — it still depends on the site's subnet being `192.168.0.0/24` and free at that address. **Spike-first is mandatory:** validate end-to-end on the drill environment (agent bind + guest dial + TLS SAN/pin + reinstall/golden survival + the bootstrap-config migration for already-deployed boxes) BEFORE any production spec. Note the pin: the agent serves a leaf cert the controller pins, so the SAN set must cover the new address. Flips a future "box survives a site/network change" map row | +| R-51 | **Dead-primary alerting — a multi-container app whose MAIN container is dead must alert.** Aggregation currently classifies such a stack `unhealthy`, and `IsDownState` deliberately excludes `unhealthy`, so nothing fires. | S | idea | Origin: `AUDIT-vacation-remote-ops-2026-07-20.md` (F4). Observed live: `immich-server` was `Exited` for **18 h** with the app 100 % unreachable, and the box produced **no** dead-app banner and **no** `app_start_failed` hub event — while single-container Calibre-Web, down for the same reason, alerted correctly within 90 s. **Constraint (load-bearing): do NOT simply fold `unhealthy` into down.** That exclusion is deliberate (`stacks/manager.go` fix-3, `downstate_test.go`) and reverting it reintroduces the flapping it was added to stop. Direction: distinguish *member-container-exited* from *healthcheck-failing* in the aggregation, and treat a dead primary as down | +| R-52 | **Boot desired-state reconciliation — a `deployed: true` app should be running after boot.** The controller *reports* deployed-but-stopped apps (30 s `deadapp-check`) but never starts them, so an app that misses its boot start stays down until a human notices. | M | idea | Origin: `AUDIT-vacation-remote-ops-2026-07-20.md` (F5). Observed live: the pre-transport shutdown left `immich-server` and `calibre-web` `Exited`; **10 sibling containers came back and those two did not**, and they were still down ~18 h later. **Includes root-causing why `restart: unless-stopped` did not resurrect them** — both were stopped ~25 s before power-off, so Docker most likely recorded them as user-stopped; that hypothesis is untested because the guest journal is volatile and the controller's own logs were rotated by the container recreate. Direction: a bounded start-once reconciliation (N attempts, reusing the existing boot grace), never a restart loop. Pairs with R-51 — that one is the *alarm*, this one is the *recovery* | +| R-53 | **`app_export.html` substitutes the CSRF token where the domain belongs.** `var domain = '.{{$.CSRFToken}}'` (~L93) — a one-line copy-paste defect that yields a broken „open in browser" link and puts the session CSRF token into a URL. | XS | idea | Origin: `AUDIT-vacation-remote-ops-2026-07-20.md` (F7), found incidentally while tracing the CSRF flow for live validation. Fix is to use the domain value, cf. the neighbouring correct `csrfH()` helper. Low exploitability (same-origin, authed page) but a token in a URL is a token in history/logs | ## P3 — post-alpha