admin
bed9bd2c5a
30 PASS / 17 FAIL / 11 FINDING / 1 DISCREPANCY. Fix-6 ring numbers (6.5min horizon under load, wiped per restart), backup-vs-NAS policy discovery, morning recovery runbook. Evidence: 180:~/campaign3/. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
21 lines
3.0 KiB
Markdown
21 lines
3.0 KiB
Markdown
# felhom.eu — task reports
|
||
|
||
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
|
||
|
||
## CAMPAIGN-3 — unattended "no mercy" night run (NAS · deploy · backup · chaos · observability) — 2026-07-11/12
|
||
|
||
**Full report: [`documentation/audits/CAMPAIGN-3-2026-07-11.md`](documentation/audits/CAMPAIGN-3-2026-07-11.md).** Run 22:09 → 04:27 CEST against the demo box (controller 0.117.0 / agent 0.84.0), operator-unattended, real surfaces only, no code changes. Ledger: **30 PASS · 17 FAIL · 11 FINDING · 1 DISCREPANCY** (66+ scenario entries, evidence at `180:~/campaign3/`).
|
||
|
||
### Headlines
|
||
|
||
- **⚠ HUMAN, first thing:** felhom-pve **did not return** from the planned 4e host reboot (23:30) — polled every 5 min until 04:27; no ping, no SSH (22/8822), controller 530. Needs physical power/console. Morning recovery runbook is in the report (§Box state) — after boot expect F10/F11 and heal with `reset-failed` + automount restart + `docker start` of the 4 NAS apps.
|
||
- **CRITICAL F10 + HIGH F11/F9 — the reboot/recovery plane around NAS automounts is broken:** `mount-start-limit-hit` is never re-armed by any heal path (once even blocked guest start → guest DOWN); guest reboot with an idle share leaves the autofs trigger unpropagated into the container — the post-start reassert **logs its own WARNING and then skips** the automount restart that provably heals ("skip-active" branch). Every such reboot = 4 NAS apps dead-at-boot. Reproduced on 3 of 3 guest reboots.
|
||
- **HIGH F7 — backup dumps are written in place (no tmp+rename):** a mid-backup NAS cut left a 0-byte tar *replacing* the last good 247M dump; in that window restore = empty volume. Next run self-heals; run-level `success:false` is the only signal.
|
||
- **The data plane held:** all 5 refusal categories ×2 correct + fast (2–5 s, retry=0), verify/rollback/single-flight/orphan flows clean, deploy-view truth holds, restore round-trips **byte-identical**, EIO same-second under outage, organic stub → badge + deploy-409 live-validated, hardlinks work on NFSv4.1.
|
||
- **Fix-6 answered with numbers:** ring cap horizon = ~55 min idle but **~6.5 min under load**; every restart/reboot wipes both rings — persistence, not just size, is the gap.
|
||
- **Policy discovery (docs):** tier-1 = volumes+config only (NAS media userdata excluded by design); NAS apps' tier-1 lands *on the NAS*, tier-2 is what gets it off; volume-only apps back up to sys_drive with blank drive label and **no tier-2 copy**.
|
||
|
||
### Box state / cleanup
|
||
|
||
DooPlex NAS restored **md5-identical to baseline** (exports + smb.conf; campaign user/share/dirs/creds removed); services never touched (exportfs-only rail held). Guest 9201 as last seen healthy (11 containers, dummy data + backups intact on NAS); `privatebin` (campaign temp) removal deferred to post-recovery. Hub untouched all night.
|