admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

CAMPAIGN-3 audit: unattended night run — data plane solid; NAS automount re-arm plane broken (F10 CRITICAL, F9/F11/F7 HIGH); demo host DOWN after 4e reboot (HUMAN)

Browse Source 
30 PASS / 17 FAIL / 11 FINDING / 1 DISCREPANCY. Fix-6 ring numbers (6.5min horizon
under load, wiped per restart), backup-vs-NAS policy discovery, morning recovery
runbook. Evidence: 180:~/campaign3/.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
This commit is contained in:
authentik Default Admin
2026-07-12 04:29:48 +02:00
parent ff7a5d1c56
commit bed9bd2c5a
3 changed files with 117 additions and 94 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+11 -94
View File
@@ -2,102 +2,19 @@
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
## Hub v0.47.0 — UI reorganization (customer tabs, Host tab, stale-host removal, offsite multi-endpoint UI, button contrast) — 2026-07-11
## CAMPAIGN-3 — unattended "no mercy" night run (NAS · deploy · backup · chaos · observability) — 2026-07-11/12
### 1. Baseline
**Full report: [`documentation/audits/CAMPAIGN-3-2026-07-11.md`](documentation/audits/CAMPAIGN-3-2026-07-11.md).** Run 22:09 → 04:27 CEST against the demo box (controller 0.117.0 / agent 0.84.0), operator-unattended, real surfaces only, no code changes. Ledger: **30 PASS · 17 FAIL · 11 FINDING · 1 DISCREPANCY** (66+ scenario entries, evidence at `180:~/campaign3/`).
felhom.eu `main` @ `8e1a3f0d82b62ac84835ddf0aa5a6fd523afb034`, hub v0.46.0 (CHANGELOG top +
running footer + `manifests/hub.yaml` all agreed). Spec: the "Hub UI reorganization" TASK.
### Headlines
### 2. Files created / modified
- **⚠ HUMAN, first thing:** felhom-pve **did not return** from the planned 4e host reboot (23:30) — polled every 5 min until 04:27; no ping, no SSH (22/8822), controller 530. Needs physical power/console. Morning recovery runbook is in the report (§Box state) — after boot expect F10/F11 and heal with `reset-failed` + automount restart + `docker start` of the 4 NAS apps.
- **CRITICAL F10 + HIGH F11/F9 — the reboot/recovery plane around NAS automounts is broken:** `mount-start-limit-hit` is never re-armed by any heal path (once even blocked guest start → guest DOWN); guest reboot with an idle share leaves the autofs trigger unpropagated into the container — the post-start reassert **logs its own WARNING and then skips** the automount restart that provably heals ("skip-active" branch). Every such reboot = 4 NAS apps dead-at-boot. Reproduced on 3 of 3 guest reboots.
- **HIGH F7 — backup dumps are written in place (no tmp+rename):** a mid-backup NAS cut left a 0-byte tar *replacing* the last good 247M dump; in that window restore = empty volume. Next run self-heals; run-level `success:false` is the only signal.
- **The data plane held:** all 5 refusal categories ×2 correct + fast (2–5 s, retry=0), verify/rollback/single-flight/orphan flows clean, deploy-view truth holds, restore round-trips **byte-identical**, EIO same-second under outage, organic stub → badge + deploy-409 live-validated, hardlinks work on NFSv4.1.
- **Fix-6 answered with numbers:** ring cap horizon = ~55 min idle but **~6.5 min under load**; every restart/reboot wipes both rings — persistence, not just size, is the gap.
- **Policy discovery (docs):** tier-1 = volumes+config only (NAS media userdata excluded by design); NAS apps' tier-1 lands *on the NAS*, tier-2 is what gets it off; volume-only apps back up to sys_drive with blank drive label and **no tier-2 copy**.
- `hub/internal/web/templates/style.css` — `.data-table td a:not(.btn)` contrast fix + tab/summary-strip/badge CSS
- `hub/internal/web/templates/customer_unified.html` — 8 hash-tab panels + sticky summary strip + Events badge + Host tab (all sections preserved; hiding = JS-added body class only)
- `hub/internal/web/templates/host_detail_body.html` — NEW: shared `{{define "host_detail_body"}}` (identity/vitals/guests/storage/diagnostics/DR + the Deletable-gated danger zone)
- `hub/internal/web/templates/host_detail.html` — reduced to chrome + sub-template call
- `hub/internal/web/templates/offsite.html` — endpoint cards + add/edit/delete forms + Endpoint peer column + deferral note
- `hub/internal/web/hosts.go` — `hostDetailData` builder; `handleHostDeleteImpact` + `handleHostDelete`
- `hub/internal/web/configs.go` — `Hosts` view models on the customer page (via `ListHostsByCustomer`)
- `hub/internal/web/offsite.go` — multi-endpoint render + save/delete handlers + subnet/peer guards
- `hub/internal/web/server.go` — routes: `/hosts/{id}/delete-impact|/delete` (above the catch-all), `/offsite/endpoints[...]`
- `hub/internal/store/store.go` — `ListHostsByCustomer`, `HostArtifacts`/`CountHostArtifacts`, `DeleteHost` (one-tx cascade), `ErrHostEscrowPresent`
- `hub/internal/store/wg.go` — `ListWGEndpoints`, `DeleteWGEndpoint`; `SetWGEndpoint` comment updated (behavior untouched)
- Tests: `hub/internal/web/customer_tabs_test.go` (new), `hosts_delete_test.go` (new), `offsite_test.go` (amended deliberately), `hosts_test.go` (pin comment), `hub/internal/store/host_delete_test.go` (new), `wg_endpoints_test.go` (new), `host_test.go` (+`TestListHostsByCustomer`)
- Docs: `hub/CHANGELOG.md` (v0.47.0), `hub/README.md` (tab map, host lifecycle, offsite mgmt), `CONTEXT.md`, `REUSE.md` (new §2 table), `manifests/hub.yaml` (tag 0.47.0)
### Box state / cleanup
### 3. Commits on `main`
| Commit | Content |
|---|---|
| `9f29bf3` | Part 1+2: CSS contrast fix + customer hash tabs + tab render tests |
| `ae950e5` | Part 3: `host_detail_body` sub-template + Host tab + `ListHostsByCustomer` |
| `146d165` | ⚠ NOT mine — the parallel NFS session's docs commit whose `git add -A` swept my Part-4 WIP **mid-red-proof** (hosts.go landed with the RED-PROOF-2 mutation, escrow ack bypassed; everything else was in final correct state) |
| `068427a` | Part 4 completion: restored the escrow-ack gate line (the only delta vs. the intended state) |
| `0daddcd` | Part 5: offsite multi-endpoint UI + guards + tests |
| `95d71fe` | Part 6: CHANGELOG/README/CONTEXT/REUSE docs + `manifests/hub.yaml` → 0.47.0 |
| `77247f9` | this REPORT.md |
### 4. Red-proofs (all five ran: mutation → FAIL with the wrong value visible → revert → green)
1. **Online gate removed** (`handleHostDelete`) → `TestHostDelete_OnlineRefused` FAILED: `online delete = 303, want 409` (live host deleted). Reverted.
2. **Escrow-ack check dropped** (`deleteEscrow := true`) → `TestHostDelete_EscrowAckRequired` FAILED: `escrow-unacked delete = 303, want 409`. Reverted. (This exact mutation is what `146d165` accidentally shipped; `068427a` is the revert on main.)
3. **Endpoint-delete peers-in-subnet guard removed** → `TestOffsiteEndpointDelete_Guard` FAILED: `delete with peers = 303, want 409`. Reverted.
4. **Subnet-change guard removed** → `TestOffsiteEndpointSave_SubnetChangeGuard` FAILED: `subnet change with peers = 303, want 409`. Reverted.
5. **log_bundles scope widened** (`WHERE scope_id = ? OR 1=1`) → `TestDeleteHost_CascadeAndIsolation` FAILED: `customer-scoped log bundle count = 0, want 1` + unrelated-host rows deleted. Reverted.
### 5. Tests
`cd hub && go build ./... && go vet ./... && go test ./...` — **all green** at every commit.
Test functions in `hub/`: **259 → 280** (+21). Pinned invariants kept green UNMODIFIED:
hosts-list zero-`<button>`, host-detail exactly-2-buttons (now doubles as the delete-hidden-
for-online proof, comment added), SECRET-LEAK assertions, pubkey-title + empty-href offsite pins.
### 6. Deploy (GitOps — no `kubectl set image` used)
- Image built + pushed on 180: `gitea.dooplex.hu/admin/felhom-hub:0.47.0` (25M).
- Deploy = the `manifests/hub.yaml` bump in `95d71fe`; ArgoCD hard-refresh → deliberate sync (Application CR patch).
- Verify: `sync=Synced health=Healthy`; `deployment "hub" successfully rolled out`; deploy image `gitea.dooplex.hu/admin/felhom-hub:0.47.0`; startup log clean (all checkers initialized, `Listening on :8080`, 2026-07-11 21:41 CEST).
### 7. Live validation — ALL PASS (method: claude-in-chrome driving hub.felhom.eu with the operator's logged-in session, 2026-07-11 ~22:00)
1. **Button contrast (Scenario F) — PASS.** `/hosts/demo-felhom-01` Diagnostics View/Download:
computed style verified `color: rgb(10,18,32)` (--bg-0) on `background: rgb(46,168,245)`
(--blue-bright); screenshot shows readable labels. Caveat found first: the browser served a
CACHED pre-fix style.css (`max-age=3600`, no cache-bust param) — `fetch('/style.css')` cached
copy lacked `:not(.btn)`, `{cache:'reload'}` copy had it; a hard reload fixed it (see §9).
2. **Customer tabs — PASS.** `/customers/demo-felhom`: 8 tabs render, only the active panel
visible (`body.js-tabs` set); `#tab=events` deep-link activates Events; `location.reload()`
(the auto-refresh mechanism) preserves the hash and the Events panel stays active; Host tab
rendered the shared body for BOTH hosts then bound to the customer (demo-felhom-01 ONLINE +
dr-drill-host NO REPORT — the list design proven live) with cross-links; sticky strip pinned
at `top:0` at scrollY=2000 (`position: sticky` computed). Mobile: at a 390px container the
tab nav scrolls horizontally (scrollWidth 725 > clientWidth 342), does not wrap, and the body
never scrolls horizontally (window resize was blocked by the maximized window — verified via
container-width constraint instead).
3. **Sanctioned deletions — PASS.** `/hosts` before: 4 hosts. `dr-drill-host` → danger-zone card
(host NO REPORT), Remove → impact preview "0 guest row(s), 0 host report(s), 0 agent log
bundle(s). Host status: pending." + escrow checkbox (both drill hosts HAD escrow rows) →
ticked + retyped id → 303 to `/hosts`, host gone. Repeated for `dr-drill-host3`. `/hosts`
after: exactly `demo-felhom-01` (ONLINE) + `peti-felhom-86d37d` (DOWN) — untouched. Neither
drill host had a bound wg peer (impact preview showed none), so there was no `/offsite`
convergence to observe; the peer table was unchanged (still the 2 real peers).
4. **Offsite — PASS.** ep0 card renders (address, truncated pubkey w/ title, subnet
10.77.0.0/24, PBS 10.77.0.1:8007, **Peers in subnet: 2**), deferral note visible, peer table
shows the Endpoint column (ep0). Bad-CIDR probe through the real edit form (`epEdit('ep0')` +
subnet `10.99.0.0/240`, posted with the form's own CSRF payload) → **400 "Invalid tunnel
subnet (CIDR, e.g. 10.77.0.0/24) — nothing stored."**; re-fetch confirmed ep0's subnet intact
and the bad value absent. No real value changed; no second endpoint created.
### 8. NOT live-validated (by design)
- Multi-endpoint allocation/reconciler/desired-state — deferred arc, deliberately untouched (`GetWGEndpoint` semantics pinned by test).
- Host deletion of a once-online host WITH escrow on the live hub — no such junk host exists; covered by unit tests only.
### 9. Observations (documented, not acted on)
- **`/style.css` is served with `max-age=3600` and no cache-bust param** — after a deploy, browsers can render with the previous CSS for up to an hour (live-observed: the contrast fix looked unfixed until a hard reload). A `?v={{hubVersion}}` on the stylesheet link would eliminate this class of false-negative.
- **A never-reported host renders its CUSTOMER's storage targets/log bundles**: `hostDetailData` reads `GetLatestHostReportJSON(host.CustomerID)`, so dr-drill-host (no report ever) displayed demo-felhom-01's storage table and the customer-scoped controller bundle. Pre-existing `handleHostDetail` behavior (customer-keyed report lookup), merely surfaced by two hosts sharing a customer — worth a per-host report scope someday.
- `146d165` (parallel session) demonstrates that `git add -A` doc commits in a shared repo can ship another session's mid-mutation WIP — worth a convention note (stage explicit paths).
- `hostInstallVersion` in `web/configs.go` still says `1.12.0` while host-install is at v1.13.0 (display-only constant; out of scope here).
- Hub web sessions live in process memory — every deploy logs every operator out; a persisted session store would remove the "revalidate after each rollout" friction.
- The workspace-root CLAUDE.md hub-deploy wording predates GitOps (noted in CONTEXT.md).
DooPlex NAS restored **md5-identical to baseline** (exports + smb.conf; campaign user/share/dirs/creds removed); services never touched (exportfs-only rail held). Guest 9201 as last seen healthy (11 containers, dummy data + backups intact on NAS); `privatebin` (campaign temp) removal deferred to post-recovery. Hub untouched all night.