felhom.eu/REPORT.md

felhom.eu — task reports

Overwrite this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in hub/CHANGELOG.md; the scripts history lives in scripts/CHANGELOG.md.

---

TASK-D — unattended resilience: docs, roadmap, capability map, riders (2026-07-21)

No hub code was touched (§12 of the brief). This repo's share is documentation, the two Part-5 riders, and the honest status of the three claims the task makes.

Implementation reports live in the repos that shipped: felhom-controller/REPORT.md (R-51, R-52, v0.156.0) and felhom-agent/REPORT.md (R-54, v0.92.1).

1. ROADMAP

• R-51 → SHIPPED (controller v0.156.0), and the row's diagnosis is corrected in place. It claimed aggregation classified a dead-primary stack unhealthy, making the deliberate unhealthy exclusion the suppressor. The source says otherwise: aggregateState's final branch returned StateRunning for any running/stopped mix ("report as running (partial)"), so the stack read running and IsDownState was never consulted about unhealthy at all. The constraint the row protects was therefore never in tension with the fix, and downstate_test.go ships untouched.
• R-52 → SHIPPED (controller v0.156.0), internal/bootrecon. Note recorded that P1 is not a blocker: the reconciliation is correct whether or not Docker recorded those two containers as user-stopped, and P1's experiment falls out of STOP-1's reboot leg for free.
• R-54 → NEW ROW, SHIPPED (agent v0.92.1). Origin: INCIDENT-guest-dhclient-killed-2026-07-20 §5, whose OPEN RISK it closes. The row carries the load-bearing design fact — liveness of the DHCP client is itself a probe, because the damage is timed and the address survives the cause by 1–2 hours — the live sudoers finding, and the explicit deferral of the static-guest leg to R-50, which is where the incident's own option 2 ("give the guest a static address") belongs.

2. Capability map

New row: "Box survives an unattended app or guest-network failure" — PROVEN-LIVE (2026-07-21).

It was written as IMPLEMENTED and upgraded only after the evidence existed, which is the order the strict enum demands. All three legs then ran operator-present in one session (§4b) and each is cited with timestamps in the row itself. The row also carries the gap the validation found (R-55) and the leg deliberately left out of scope (the static guest → R-50), so the badge does not imply more than was proven.

The existing "Box survives a site/network change" row (PARTIAL) already pointed at R-51/R-52 as related work; it stays PARTIAL — R-50 is still its durable fix.

3. Part-5 riders

• Hub build.sh deploy hint → GitOps wording. The script printed kubectl set image … and kubectl apply -f manifests/hub.yaml as the deploy instructions — both are reverted by the next ArgoCD sync, which is the worst failure shape: it appears to work, then silently disappears. Now prints the manifest-bump + hard-refresh + deliberate-sync sequence, and names the trap explicitly. Caveat worth knowing: /mnt/5_hdd/felhom.eu/build/felhom-hub/build.sh is NOT in any git repo — it is a DooPlex-local build-dir script. So this rider is a fix to the operative file only, and it is not versioned anywhere. Worth adopting into the repo as its own change.
• documentation/PROMPT-TEMPLATE.md §10 gains the seam-discipline row. Text generalised from the brief's §9 rule 6, plus what this session added to it: three shipped inert-seam defects in three days (controller v0.154.0, agent v0.91.0, agent v0.92.0's missing sudoers grant), all fully green; and the finding that a strings.Contains source assertion is not sufficient, because a commented-out call still contains the string — walk the AST.

4. What was verified, and how

Endpoint/host-level, no browser (none on DooPlex). Everything claimed live in the two implementation reports was read out of journald or the agent's own capability self-check on felhom-pve. No claim in any of the three reports rests on a test that only proves a seam.

4b. The STOP legs RAN — all three (2026-07-21, operator-present)

The ordering below was accepted, executed, and all three legs passed. Full evidence chains are in the two implementation reports and in the ROADMAP rows; the capability-map row is now PROVEN-LIVE. Headline: dead primary → degraded in 13 s with exactly one event; boot orphan → recovered in one attempt with zero alerts; dead dhclient → detected in 57 s on process liveness alone and healed 120 s after the kill, roughly 80 minutes before the outage would have started — cloudflared never dropped (Up 29 hours).

One real gap surfaced, and it is the opposite of what the leg was designed to prove → new R-55. STOP-1's R-52 leg was meant to show that a customer's deliberate Stop survives a reboot. It does not, for any drive-backed app: the boot bind gate (internal/web/intermediary.go) selects on Deployed && HDD_PATH alone and runs StopStack + StartStack on every match, never consulting whether the app was stopped on purpose. immich, stopped from the UI seconds earlier, came back running. R-52 is not implicated — its own gate behaved exactly as specified (immich, at zero containers, was never a candidate), and this behaviour predates it. Two consequences are worth an operator ruling: a customer's Stop is silently undone on drive-backed apps, and R-52's practical scope is narrower than the brief assumed, because the gate reaches every drive-backed app first.

P1 is answered, as a free by-product of that leg. bookstack carries restart=unless-stopped, the Docker daemon came up at ~10:53:15Z, and the container's StartedAt is 10:53:28.05Z — the exact moment bootrecon's StartStack returned. Docker did not resurrect it; the F5 hypothesis is confirmed.

5. The ordering question (resolved)

The brief's Phase A says do not hand-deploy the controller — the floor save at STOP-3 is what deploys 0.156.0, banking another single-fire self-update datapoint. But STOP-1 exercises the controller legs, which need 0.156.0 to be live. As written the two are in the wrong order.

Resolved as: STOP-3's controller floor save first (floor → 0.156.0), box self-updates — that IS the R-23 datapoint — then STOP-1 against the new version. Done, and the datapoint is banked: [selfupdate] Post-update startup: update successful (0.155.0 → 0.156.0) followed by Current version 0.156.0 is up to date — one swap, no re-fire. The agent half of STOP-3 (manifest → 0.92.1) is independent and can happen at any point. Flagged rather than assumed, because reordering an operator's STOP sequence is not CC's call.

6. Follow-ups this session surfaced (none actioned here)

1. felhom-controller/controller/.gitignore's bare controller entry also matches the directory cmd/controller/, so ripgrep silently skips main.go and new files there need git add -f. Both directions produce inert-seam mistakes. XS fix: anchor it as /controller.
2. The hub build.sh above is unversioned.
3. TestGenerateRecoveryCode_EntropyAndFormat (agent) flakes on hyphenated wordlist entries (drop-down → 11 tokens). Observed 3/8 this session — worse than the documented ~1/5, and it is a one-line fix in the generator or the assertion, not a mystery.