docs(audits): remote-ops remediation — F1 mitigated, F2 closed, F3 fixed (v0.149.0), Immich resolved; R-50..R-53 minted
This commit is contained in:
@@ -63,6 +63,7 @@
|
||||
| Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill |
|
||||
| PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | hub v0.56.0 (`pbsdrheal/reconciler.go`, `RestageHostPBSSecret`, all §10 red-proofs); `SPIKE-pbsdr-selfheal-2026-07-15` (root cause) | Reconciler is **scoped to one host** (`PBSDRHEAL_ONLY_HOST`), not fleet-wide; already fired live hands-free on drill qm300 (07-15) — real-customer firing + fleet-wide widening pending |
|
||||
| Crash/power-loss mid-backup/mid-migration → self-heal on next run | controller, agent | **PROVEN-LIVE** | `CAMPAIGN-6D` P5-REST (SIGKILL mid-offbox → auto-restart ~15s, run marked failed not false-success, no stale lock); `CAMPAIGN-6E` B1-B3 | (Cited `CAMPAIGN-2` T-RBT-* legs were empty / auth-hollow — corrected.) Live mid-**migration** crash→self-heal is the weakest sub-claim (P5-REST is mid-backup) |
|
||||
| Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent, controller, bootstrap | **PARTIAL** | `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — a real relocation of the demo box: guest + hub telemetry + WG/PBS + Cloudflare tunnel all survived untouched, but the **controller↔agent control plane did not** (agent binds a LAN literal → `bind: cannot assign requested address` → storage/PBS-backup/quiesce/restore-test/DR down until fixed). Mitigated for the window by pinning `vmbr0` static | **R-50** (island-bridge control plane, spike-first) is the durable fix. Related: **R-51** (dead-primary alerting) and **R-52** (boot desired-state reconciliation) — the same event left two apps `Exited` with no alarm and no recovery |
|
||||
| Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | |
|
||||
| **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3. **2026-07-19:** the C6 evidence attempt ran and found a **product gap instead of evidence** — `audits/DIAG-immich-restore-2026-07-19.md`. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING **by finding, not by absence of attempt** — the rehearsal system working, not failing. **2026-07-19: the blocking product gaps are CLOSED in controller v0.148.0** (R-43 + R-44 shipped), so this row is now blocked only on the evidence run itself, not on missing capability. It flips the moment the §9 acceptance produces screenshots + the outcome flash + a snapshot ID. **2026-07-19 round 2 — PARTIAL EVIDENCE ONLY, row NOT flipped** (`audits/DIAG-immich-restore-round2-2026-07-19.md`): a deliberate run from snapshot `49e7cb46` did recover all 11 assets (`status=active`, files resolve), but the operation **reported failure** and left immich reporting schema drift, because the replay aborted against the running app (H4). Photos back ≠ clean acceptance. Blocked on H4 closing in v0.149, then one clean run. Method note for R-3's script: deleting in an app's own UI usually means *trash*, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash **and** verify the app's *content*, not the file count |
|
||||
|
||||
|
||||
@@ -285,3 +285,153 @@ Nothing else was mutated. The ssh alias needed **no** edit (already on the tailn
|
||||
5. Re-verify: agent `active (running)`, dashboard banner clear, Tárhely renders, a PBS backup
|
||||
completes, Immich healthy and reachable.
|
||||
6. Flip this document's status from **ACTIVE** to **CLOSED**.
|
||||
|
||||
---
|
||||
|
||||
# Remediation — 2026-07-20
|
||||
|
||||
Executed the same day as the audit, on the relocated host. Mutations: one network file, one service
|
||||
start, one app start, two backup runs (one of them the product's own scheduler), one code line.
|
||||
|
||||
## R.1 Static-IP flip (F1 window mitigation)
|
||||
|
||||
**State found had already moved.** By the time remediation started, the router's reservation had
|
||||
taken effect and `vmbr0` held **`192.168.0.162` again — but via DHCP** (`dhclient.vmbr0.leases`
|
||||
stamped 08:13 CEST). So the address existed; nothing had retried the agent since systemd gave up.
|
||||
|
||||
The 1.1 duplicate-address check needed interpretation, not blind obedience: `ping -c1 192.168.0.162`
|
||||
**answered**, because the host was answering *itself*. The genuine test is from another node —
|
||||
ARP for `.162` seen from guest 9201 resolved to `68:1d:ef:5d:a6:64`, which **is** this host's vmbr0
|
||||
MAC. No duplicate; the STOP did not apply.
|
||||
|
||||
Interfaces diff (`.bak-2026-07-20` → new) — one stanza, nothing else touched:
|
||||
|
||||
```diff
|
||||
auto vmbr0
|
||||
-iface vmbr0 inet dhcp
|
||||
+iface vmbr0 inet static
|
||||
+ address 192.168.0.162/24
|
||||
+ gateway 192.168.0.1
|
||||
bridge-ports enp1s0
|
||||
bridge-stp off
|
||||
bridge-fd 0
|
||||
```
|
||||
|
||||
The gateway was taken from the live default route (`192.168.0.1`), not from the spec. Applied
|
||||
detached: `setsid ifreload -a` → log contained only `Killed old client process` (the vmbr0 dhclient).
|
||||
Post-flip: `inet 192.168.0.162/24 scope global vmbr0` (no `dynamic` flag), `default via 192.168.0.1
|
||||
… proto kernel onlink`, `curl https://gitea.dooplex.hu` → **200**, tailscale peer still
|
||||
`active; direct`. Because the address was unchanged, connectivity never blipped.
|
||||
|
||||
## R.2 Agent recovery (Scenario A)
|
||||
|
||||
`systemctl reset-failed && systemctl start felhom-agent` — up at **08:40:11 CEST**:
|
||||
|
||||
```
|
||||
active
|
||||
LISTEN 192.168.0.162:8443 users:(("felhom-agent",pid=72859))
|
||||
level=INFO msg="local-api server listening" addr=192.168.0.162:8443
|
||||
wg-felhom latest handshake: 1 minute, 16 seconds ago
|
||||
```
|
||||
|
||||
Controller side, `agentapi` errors **stopped entirely** (no `no route to host`, no
|
||||
`channel not verified` in the following 5 min) — so this is not the "agent active but controller
|
||||
still erroring" wrong outcome the scenario warns about. On an authenticated fetch of the real
|
||||
dashboard the red **„A tárolókezelő ügynök nem elérhető"** banner is **gone**; the only
|
||||
`alert-warning` strings left on the page are two modal "this cannot be undone" confirmations.
|
||||
|
||||
> **Method note for anyone repeating this:** grep the fetched page with **ASCII-only** patterns.
|
||||
> Accented patterns sent through the ssh→`pct exec`→shell layers get mangled and return a
|
||||
> false `0`, which reads exactly like "the banner is gone" when it is not. The page content itself
|
||||
> is fine — it is the *pattern* that gets corrupted.
|
||||
|
||||
## R.3 F2 — the protection gap closed itself, then a manual run confirmed it
|
||||
|
||||
Restoring the agent was sufficient: the quiesce loop noticed the overdue whole-guest backup and ran
|
||||
it **unprompted**, which is the strongest possible evidence the seam is healthy end to end:
|
||||
|
||||
```
|
||||
06:40:36Z [quiesce] backup due — quiescing 3 stack(s): [bookstack calibre-web immich]
|
||||
06:40:45Z [quiesce] backup job backup-9201-1784529645204076480 started — polling to completion
|
||||
06:40:55Z [quiesce] … snapshotted — resuming app early (8B.2)
|
||||
06:42:52Z [quiesce] … done
|
||||
```
|
||||
|
||||
A manual app-data run then followed via the real UI endpoint (`POST /api/backup/run`, authenticated
|
||||
session + `X-CSRF-Token` read from the page's `<meta>` — the same call the „Teljes mentés" button
|
||||
makes), started 06:45:26Z:
|
||||
|
||||
```
|
||||
06:45:27Z DB dump: bookstack-db → bookstack-mariadb.sql (55.9 KB, 228ms, 41 tables)
|
||||
06:45:28Z DB dump: immich-postgres → immich-postgres.sql (49.9 MB, 1.28s, 66 tables)
|
||||
06:45:44Z Volume dump: calibre-web/…_config → 360.0 KB
|
||||
06:45:55Z Volume dump: immich/…_ml_cache → 785.5 MB
|
||||
06:45:58Z Volume dump: immich/…_postgres_data → 241.5 MB
|
||||
06:45:59Z Volume dump: immich/…_redis_data → 6.6 MB
|
||||
06:46:10Z App-data backup completed: 2 databases (50.0 MB total), 3 volume dump(s) (43.348s)
|
||||
```
|
||||
|
||||
**T = 2026-07-20 06:46:10Z** (run completion; newest dump artifact 06:45:28Z). Every container the
|
||||
run stopped for a safe volume dump was restarted — all 12 containers verified `Up (healthy)`
|
||||
afterwards.
|
||||
|
||||
## R.4 Immich (Scenario: app triage)
|
||||
|
||||
Started via the controller's real endpoint (`POST /api/stacks/immich/start`) → all four containers
|
||||
`Up (healthy)`.
|
||||
|
||||
**Correction to §3.3 of the audit.** That section reported the URL probe returning 404 and treated
|
||||
it as a live symptom. The probe used the **wrong hostname**: the traefik router rule is
|
||||
``Host(`photos.demo-felhom.eu`)``, not `immich.demo-felhom.eu`. The container-down diagnosis stands
|
||||
(and the 404 was genuinely traefik having no backend), but the specific probe URL in §3.3 was
|
||||
invalid. Post-start, on the correct host: `https://photos.demo-felhom.eu` → **200** in 0.28 s.
|
||||
|
||||
## R.5 F3 — dashboard backup card (controller v0.149.0)
|
||||
|
||||
Handler-side one-liner, no template change: `data["BackupStatus"] = fullStatus.LastDBDump` in
|
||||
`dashboardHandler`. Deployed to guest 9201 and verified on the **real, authenticated dashboard**:
|
||||
|
||||
```
|
||||
<span class="backup-label">Utolsó mentés:</span>
|
||||
<span class="backup-value"><span>2026-07-20 06:45</span></span>
|
||||
<span class="backup-label">Adatbázisok:</span>
|
||||
<span class="backup-value">3 mentve</span>
|
||||
```
|
||||
|
||||
Was „Még nem futott" on the identical page before the deploy. (The controller restart cleared the
|
||||
in-memory run state, so the card renders from the dumps on disk — newest artifact 06:45 — which is
|
||||
the honest value and exactly what the fresh-box branch must NOT fabricate.)
|
||||
|
||||
## R.6 Per-finding status after remediation
|
||||
|
||||
| # | Was | Now |
|
||||
|---|-----|-----|
|
||||
| **F1** | CRITICAL, open | **MITIGATED for the window** — `vmbr0` static `192.168.0.162/24`; agent up. **Durable fix still open → R-50** (island-bridge control plane, spike-first). Still depends on the site's subnet being `192.168.0.0/24` and `.162` free |
|
||||
| **F2** | HIGH, open | **CLOSED** — whole-guest backup ran (06:42:52Z) + manual app-data run completed (T = 06:46:10Z) |
|
||||
| **F3** | HIGH, open | **FIXED in controller v0.149.0**, live-verified on the dashboard |
|
||||
| **Immich** | ROOT-CAUSED, down | **RESOLVED** — healthy, `photos.demo-felhom.eu` → 200 |
|
||||
| **F4** | MEDIUM, open | Roadmap **R-51** (constraint recorded: must NOT simply fold `unhealthy` into down) |
|
||||
| **F5** | MEDIUM, open | Roadmap **R-52** (includes root-causing the `unless-stopped` non-resurrection) |
|
||||
| **F6** | LOW, open | **Already tracked** — it is R-39's remaining item (b), the agent's PBS `.pw` read path. No duplicate item minted |
|
||||
| **F7** | LOW, open | Roadmap **R-53** |
|
||||
|
||||
Capability map: added a **PARTIAL** row — *"Box survives a site/network change (relocation,
|
||||
different subnet, DHCP re-lease) with the control plane intact"* — citing this audit and pointing at
|
||||
R-50/R-51/R-52.
|
||||
|
||||
## R.7 Not done / out of scope
|
||||
|
||||
No agent code, guest bootstrap config, golden state, nftables, tailscale, router, guest-9201
|
||||
network, or `resolv.conf` was touched. No restores or drills. F4/F5/F6/F7 diagnosed and ranked only.
|
||||
|
||||
## R.8 Return-day checklist — amendments
|
||||
|
||||
Supersedes items 1 and 4 above:
|
||||
|
||||
1a. **Decide whether static `.162` stays when the box comes home.** *Recommendation: keep it.* It
|
||||
removes the home router's DHCP/reservation from the control plane's dependency chain, which is
|
||||
the same class of dependency that broke this window. Either way, R-50 is the real fix — the
|
||||
static pin is a crutch, and it will fail at any site whose subnet is not `192.168.0.0/24`.
|
||||
4a. `/etc/network/interfaces.bak-2026-07-20` on the host is the pre-flip DHCP config — delete it
|
||||
once the decision in 1a is made and verified.
|
||||
7. Note the deployed controller is **v0.149.0** (was 0.148.0 at audit time).
|
||||
|
||||
@@ -58,6 +58,10 @@
|
||||
| R-35 | **Config-apply should not end the customer's session.** The offsite config push bumped `config_version` 10→11 at 16:54:58 and the controller self-restarted (container `StartedAt` 16:54:59Z, back up 16:55:02); in-memory sessions died with it and **customer zero was force-logged-out mid-flow**. | S | idea | Direction: **hot-apply the offbox target** (no restart for a config the running process can adopt), or **persist sessions** across restart. The restart itself is by design — the collateral is not. Evidence `controller-log-full.txt` |
|
||||
| R-36 | **Post-RESET re-enroll leaves offsite "enabled but unprovisioned" — silently.** The hub knows the state and says nothing on the customer page. | S | **SHIPPED (hub v0.67.0, 2026-07-18)** | Both halves delivered. **(1) The warning:** the customer page now names the state and the fix — enabled-but-unprovisioned raises an amber banner saying provisioning is *Save*-triggered (press Save once, then verify), reusing the exact `enabled && type == ""` predicate the offsite re-issue handler already refuses on. **(2) The related sub-item, also done:** the self-bind link is now **auto-minted at customer creation AND at RESET completion**, so the console banner's „e-mailben kapott link" is already true instead of true-once-the-operator-remembers. Extracting the shared `mintAndSendSelfBindLink` core keeps the button and the auto-mint callers on the same F1/F2 honesty rules, and the auto-mint never fails the operation it rides on. **Gap found and closed while wiring it:** `PurgeCustomerResetDBState` does NOT clear `selfbind_tokens`, so a link minted BEFORE a reset would have stayed live across it — the skip paths now clear stale tokens, giving the invariant "after auto-mint, the only live link is one we just issued, or none". Tests assert the banner is ABSENT in all three nominal cases too; red-proofed. — Original analysis: Source-cited behaviour, confirmed live in the rehearsal: **provisioning is Save-triggered** (`configs.go` `applyOffsite`) — which also answers S6's open question — and the re-enroll auto-re-issue **correctly** skips unprovisioned targets (`handler.go`). So nothing is broken; the gap is that nobody is told. Direction: flash it on the customer page. **Interim: an R-3 step.** **Related sub-item:** auto-mint the **self-bind link on customer create/RESET**, so the console banner's „e-mailben kapott link" is always already true instead of true-once-the-operator-remembers |
|
||||
| R-27c | **Customer self-bind, slice 2 — console-passphrase bind.** Viktor's direction: bind using a passphrase shown on the box console, alongside (not instead of) the emailed capability link. | M | idea | **Security constraints from the session ruling, all load-bearing:** passphrase **issued at customer creation**; the global-lookup endpoint must be **spray-hardened** — per-appliance **and** per-IP caps, constant-time comparison, a **single generic failure** (no oracle), alerting on abuse; an **accent-free wordlist** (console keymaps are not Hungarian); the **web capability-link path is RETAINED**; **claim-by-email is RETAINED** as the delivery-channel proof. **Also under this item:** the self-bind email gains the **public universal-ISO download link + two-line instructions** (the DIY case). **Secret-bearing per-customer ISOs are ruled OUT.** Sibling of R-27b (second-box flow) — different axis, both build on the same `/bind/` page |
|
||||
| R-50 | **[P2-HIGH] Island-bridge control plane — make controller↔agent independent of the LAN.** The agent's `localapi` binds a **LAN literal** (`listen_addr`) and the guest dials that same literal from `bootstrap.json`. Move both onto a **host-internal bridge with a fixed, private address** that no router, DHCP lease, or site move can invalidate, so the control plane is location-independent and golden-image-stable. | M | **idea — SPIKE-FIRST** | Origin: `audits/AUDIT-vacation-remote-ops-2026-07-20.md` (F1), where this failed for real. The demo box moved to a remote site, DHCP handed it `.147` instead of `.162`, and the agent then **could not start at all** — `bind: cannot assign requested address`, systemd gave up after 4 retries — taking storage, PBS backup, quiesce, restore-test and DR down for as long as nobody noticed. Mitigated for that window by pinning `vmbr0` static back to `.162`; that is a **window mitigation, not the fix** — it still depends on the site's subnet being `192.168.0.0/24` and free at that address. **Spike-first is mandatory:** validate end-to-end on the drill environment (agent bind + guest dial + TLS SAN/pin + reinstall/golden survival + the bootstrap-config migration for already-deployed boxes) BEFORE any production spec. Note the pin: the agent serves a leaf cert the controller pins, so the SAN set must cover the new address. Flips a future "box survives a site/network change" map row |
|
||||
| R-51 | **Dead-primary alerting — a multi-container app whose MAIN container is dead must alert.** Aggregation currently classifies such a stack `unhealthy`, and `IsDownState` deliberately excludes `unhealthy`, so nothing fires. | S | idea | Origin: `AUDIT-vacation-remote-ops-2026-07-20.md` (F4). Observed live: `immich-server` was `Exited` for **18 h** with the app 100 % unreachable, and the box produced **no** dead-app banner and **no** `app_start_failed` hub event — while single-container Calibre-Web, down for the same reason, alerted correctly within 90 s. **Constraint (load-bearing): do NOT simply fold `unhealthy` into down.** That exclusion is deliberate (`stacks/manager.go` fix-3, `downstate_test.go`) and reverting it reintroduces the flapping it was added to stop. Direction: distinguish *member-container-exited* from *healthcheck-failing* in the aggregation, and treat a dead primary as down |
|
||||
| R-52 | **Boot desired-state reconciliation — a `deployed: true` app should be running after boot.** The controller *reports* deployed-but-stopped apps (30 s `deadapp-check`) but never starts them, so an app that misses its boot start stays down until a human notices. | M | idea | Origin: `AUDIT-vacation-remote-ops-2026-07-20.md` (F5). Observed live: the pre-transport shutdown left `immich-server` and `calibre-web` `Exited`; **10 sibling containers came back and those two did not**, and they were still down ~18 h later. **Includes root-causing why `restart: unless-stopped` did not resurrect them** — both were stopped ~25 s before power-off, so Docker most likely recorded them as user-stopped; that hypothesis is untested because the guest journal is volatile and the controller's own logs were rotated by the container recreate. Direction: a bounded start-once reconciliation (N attempts, reusing the existing boot grace), never a restart loop. Pairs with R-51 — that one is the *alarm*, this one is the *recovery* |
|
||||
| R-53 | **`app_export.html` substitutes the CSRF token where the domain belongs.** `var domain = '<subdomain>.{{$.CSRFToken}}'` (~L93) — a one-line copy-paste defect that yields a broken „open in browser" link and puts the session CSRF token into a URL. | XS | idea | Origin: `AUDIT-vacation-remote-ops-2026-07-20.md` (F7), found incidentally while tracing the CSRF flow for live validation. Fix is to use the domain value, cf. the neighbouring correct `csrfH()` helper. Low exploitability (same-origin, authed page) but a token in a URL is a token in history/logs |
|
||||
|
||||
## P3 — post-alpha
|
||||
|
||||
|
||||
Reference in New Issue
Block a user