docs(audits): remote-ops remediation — F1 mitigated, F2 closed, F3 fixed (v0.149.0), Immich resolved; R-50..R-53 minted
This commit is contained in:
@@ -285,3 +285,153 @@ Nothing else was mutated. The ssh alias needed **no** edit (already on the tailn
|
||||
5. Re-verify: agent `active (running)`, dashboard banner clear, Tárhely renders, a PBS backup
|
||||
completes, Immich healthy and reachable.
|
||||
6. Flip this document's status from **ACTIVE** to **CLOSED**.
|
||||
|
||||
---
|
||||
|
||||
# Remediation — 2026-07-20
|
||||
|
||||
Executed the same day as the audit, on the relocated host. Mutations: one network file, one service
|
||||
start, one app start, two backup runs (one of them the product's own scheduler), one code line.
|
||||
|
||||
## R.1 Static-IP flip (F1 window mitigation)
|
||||
|
||||
**State found had already moved.** By the time remediation started, the router's reservation had
|
||||
taken effect and `vmbr0` held **`192.168.0.162` again — but via DHCP** (`dhclient.vmbr0.leases`
|
||||
stamped 08:13 CEST). So the address existed; nothing had retried the agent since systemd gave up.
|
||||
|
||||
The 1.1 duplicate-address check needed interpretation, not blind obedience: `ping -c1 192.168.0.162`
|
||||
**answered**, because the host was answering *itself*. The genuine test is from another node —
|
||||
ARP for `.162` seen from guest 9201 resolved to `68:1d:ef:5d:a6:64`, which **is** this host's vmbr0
|
||||
MAC. No duplicate; the STOP did not apply.
|
||||
|
||||
Interfaces diff (`.bak-2026-07-20` → new) — one stanza, nothing else touched:
|
||||
|
||||
```diff
|
||||
auto vmbr0
|
||||
-iface vmbr0 inet dhcp
|
||||
+iface vmbr0 inet static
|
||||
+ address 192.168.0.162/24
|
||||
+ gateway 192.168.0.1
|
||||
bridge-ports enp1s0
|
||||
bridge-stp off
|
||||
bridge-fd 0
|
||||
```
|
||||
|
||||
The gateway was taken from the live default route (`192.168.0.1`), not from the spec. Applied
|
||||
detached: `setsid ifreload -a` → log contained only `Killed old client process` (the vmbr0 dhclient).
|
||||
Post-flip: `inet 192.168.0.162/24 scope global vmbr0` (no `dynamic` flag), `default via 192.168.0.1
|
||||
… proto kernel onlink`, `curl https://gitea.dooplex.hu` → **200**, tailscale peer still
|
||||
`active; direct`. Because the address was unchanged, connectivity never blipped.
|
||||
|
||||
## R.2 Agent recovery (Scenario A)
|
||||
|
||||
`systemctl reset-failed && systemctl start felhom-agent` — up at **08:40:11 CEST**:
|
||||
|
||||
```
|
||||
active
|
||||
LISTEN 192.168.0.162:8443 users:(("felhom-agent",pid=72859))
|
||||
level=INFO msg="local-api server listening" addr=192.168.0.162:8443
|
||||
wg-felhom latest handshake: 1 minute, 16 seconds ago
|
||||
```
|
||||
|
||||
Controller side, `agentapi` errors **stopped entirely** (no `no route to host`, no
|
||||
`channel not verified` in the following 5 min) — so this is not the "agent active but controller
|
||||
still erroring" wrong outcome the scenario warns about. On an authenticated fetch of the real
|
||||
dashboard the red **„A tárolókezelő ügynök nem elérhető"** banner is **gone**; the only
|
||||
`alert-warning` strings left on the page are two modal "this cannot be undone" confirmations.
|
||||
|
||||
> **Method note for anyone repeating this:** grep the fetched page with **ASCII-only** patterns.
|
||||
> Accented patterns sent through the ssh→`pct exec`→shell layers get mangled and return a
|
||||
> false `0`, which reads exactly like "the banner is gone" when it is not. The page content itself
|
||||
> is fine — it is the *pattern* that gets corrupted.
|
||||
|
||||
## R.3 F2 — the protection gap closed itself, then a manual run confirmed it
|
||||
|
||||
Restoring the agent was sufficient: the quiesce loop noticed the overdue whole-guest backup and ran
|
||||
it **unprompted**, which is the strongest possible evidence the seam is healthy end to end:
|
||||
|
||||
```
|
||||
06:40:36Z [quiesce] backup due — quiescing 3 stack(s): [bookstack calibre-web immich]
|
||||
06:40:45Z [quiesce] backup job backup-9201-1784529645204076480 started — polling to completion
|
||||
06:40:55Z [quiesce] … snapshotted — resuming app early (8B.2)
|
||||
06:42:52Z [quiesce] … done
|
||||
```
|
||||
|
||||
A manual app-data run then followed via the real UI endpoint (`POST /api/backup/run`, authenticated
|
||||
session + `X-CSRF-Token` read from the page's `<meta>` — the same call the „Teljes mentés" button
|
||||
makes), started 06:45:26Z:
|
||||
|
||||
```
|
||||
06:45:27Z DB dump: bookstack-db → bookstack-mariadb.sql (55.9 KB, 228ms, 41 tables)
|
||||
06:45:28Z DB dump: immich-postgres → immich-postgres.sql (49.9 MB, 1.28s, 66 tables)
|
||||
06:45:44Z Volume dump: calibre-web/…_config → 360.0 KB
|
||||
06:45:55Z Volume dump: immich/…_ml_cache → 785.5 MB
|
||||
06:45:58Z Volume dump: immich/…_postgres_data → 241.5 MB
|
||||
06:45:59Z Volume dump: immich/…_redis_data → 6.6 MB
|
||||
06:46:10Z App-data backup completed: 2 databases (50.0 MB total), 3 volume dump(s) (43.348s)
|
||||
```
|
||||
|
||||
**T = 2026-07-20 06:46:10Z** (run completion; newest dump artifact 06:45:28Z). Every container the
|
||||
run stopped for a safe volume dump was restarted — all 12 containers verified `Up (healthy)`
|
||||
afterwards.
|
||||
|
||||
## R.4 Immich (Scenario: app triage)
|
||||
|
||||
Started via the controller's real endpoint (`POST /api/stacks/immich/start`) → all four containers
|
||||
`Up (healthy)`.
|
||||
|
||||
**Correction to §3.3 of the audit.** That section reported the URL probe returning 404 and treated
|
||||
it as a live symptom. The probe used the **wrong hostname**: the traefik router rule is
|
||||
``Host(`photos.demo-felhom.eu`)``, not `immich.demo-felhom.eu`. The container-down diagnosis stands
|
||||
(and the 404 was genuinely traefik having no backend), but the specific probe URL in §3.3 was
|
||||
invalid. Post-start, on the correct host: `https://photos.demo-felhom.eu` → **200** in 0.28 s.
|
||||
|
||||
## R.5 F3 — dashboard backup card (controller v0.149.0)
|
||||
|
||||
Handler-side one-liner, no template change: `data["BackupStatus"] = fullStatus.LastDBDump` in
|
||||
`dashboardHandler`. Deployed to guest 9201 and verified on the **real, authenticated dashboard**:
|
||||
|
||||
```
|
||||
<span class="backup-label">Utolsó mentés:</span>
|
||||
<span class="backup-value"><span>2026-07-20 06:45</span></span>
|
||||
<span class="backup-label">Adatbázisok:</span>
|
||||
<span class="backup-value">3 mentve</span>
|
||||
```
|
||||
|
||||
Was „Még nem futott" on the identical page before the deploy. (The controller restart cleared the
|
||||
in-memory run state, so the card renders from the dumps on disk — newest artifact 06:45 — which is
|
||||
the honest value and exactly what the fresh-box branch must NOT fabricate.)
|
||||
|
||||
## R.6 Per-finding status after remediation
|
||||
|
||||
| # | Was | Now |
|
||||
|---|-----|-----|
|
||||
| **F1** | CRITICAL, open | **MITIGATED for the window** — `vmbr0` static `192.168.0.162/24`; agent up. **Durable fix still open → R-50** (island-bridge control plane, spike-first). Still depends on the site's subnet being `192.168.0.0/24` and `.162` free |
|
||||
| **F2** | HIGH, open | **CLOSED** — whole-guest backup ran (06:42:52Z) + manual app-data run completed (T = 06:46:10Z) |
|
||||
| **F3** | HIGH, open | **FIXED in controller v0.149.0**, live-verified on the dashboard |
|
||||
| **Immich** | ROOT-CAUSED, down | **RESOLVED** — healthy, `photos.demo-felhom.eu` → 200 |
|
||||
| **F4** | MEDIUM, open | Roadmap **R-51** (constraint recorded: must NOT simply fold `unhealthy` into down) |
|
||||
| **F5** | MEDIUM, open | Roadmap **R-52** (includes root-causing the `unless-stopped` non-resurrection) |
|
||||
| **F6** | LOW, open | **Already tracked** — it is R-39's remaining item (b), the agent's PBS `.pw` read path. No duplicate item minted |
|
||||
| **F7** | LOW, open | Roadmap **R-53** |
|
||||
|
||||
Capability map: added a **PARTIAL** row — *"Box survives a site/network change (relocation,
|
||||
different subnet, DHCP re-lease) with the control plane intact"* — citing this audit and pointing at
|
||||
R-50/R-51/R-52.
|
||||
|
||||
## R.7 Not done / out of scope
|
||||
|
||||
No agent code, guest bootstrap config, golden state, nftables, tailscale, router, guest-9201
|
||||
network, or `resolv.conf` was touched. No restores or drills. F4/F5/F6/F7 diagnosed and ranked only.
|
||||
|
||||
## R.8 Return-day checklist — amendments
|
||||
|
||||
Supersedes items 1 and 4 above:
|
||||
|
||||
1a. **Decide whether static `.162` stays when the box comes home.** *Recommendation: keep it.* It
|
||||
removes the home router's DHCP/reservation from the control plane's dependency chain, which is
|
||||
the same class of dependency that broke this window. Either way, R-50 is the real fix — the
|
||||
static pin is a crutch, and it will fail at any site whose subnet is not `192.168.0.0/24`.
|
||||
4a. `/etc/network/interfaces.bak-2026-07-20` on the host is the pre-flip DHCP config — delete it
|
||||
once the decision in 1a is made and verified.
|
||||
7. Note the deployed controller is **v0.149.0** (was 0.148.0 at audit time).
|
||||
|
||||
Reference in New Issue
Block a user