111 lines
7.4 KiB
Markdown
111 lines
7.4 KiB
Markdown
# felhom.eu — task reports
|
||
|
||
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
|
||
|
||
---
|
||
|
||
# TASK-D — unattended resilience: docs, roadmap, capability map, riders (2026-07-21)
|
||
|
||
**No hub code was touched** (§12 of the brief). This repo's share is documentation, the two Part-5
|
||
riders, and the honest status of the three claims the task makes.
|
||
|
||
Implementation reports live in the repos that shipped: `felhom-controller/REPORT.md` (R-51, R-52,
|
||
v0.156.0) and `felhom-agent/REPORT.md` (R-54, v0.92.1).
|
||
|
||
## 1. ROADMAP
|
||
|
||
- **R-51 → SHIPPED (controller v0.156.0)**, and **the row's diagnosis is corrected in place**. It
|
||
claimed aggregation classified a dead-primary stack `unhealthy`, making the deliberate `unhealthy`
|
||
exclusion the suppressor. The source says otherwise: `aggregateState`'s final branch returned
|
||
`StateRunning` for any running/stopped mix ("report as running (partial)"), so the stack read
|
||
**running** and `IsDownState` was never consulted about `unhealthy` at all. The constraint the row
|
||
protects was therefore never in tension with the fix, and `downstate_test.go` ships untouched.
|
||
- **R-52 → SHIPPED (controller v0.156.0)**, `internal/bootrecon`. Note recorded that **P1 is not a
|
||
blocker**: the reconciliation is correct whether or not Docker recorded those two containers as
|
||
user-stopped, and P1's experiment falls out of STOP-1's reboot leg for free.
|
||
- **R-54 → NEW ROW, SHIPPED (agent v0.92.1).** Origin: `INCIDENT-guest-dhclient-killed-2026-07-20`
|
||
§5, whose OPEN RISK it closes. The row carries the load-bearing design fact — *liveness of the
|
||
DHCP client is itself a probe, because the damage is timed and the address survives the cause by
|
||
1–2 hours* — the live sudoers finding, and the explicit deferral of the **static-guest** leg to
|
||
**R-50**, which is where the incident's own option 2 ("give the guest a static address") belongs.
|
||
|
||
## 2. Capability map
|
||
|
||
New row: **"Box survives an unattended app or guest-network failure"** — **PROVEN-LIVE (2026-07-21)**.
|
||
|
||
It was written as IMPLEMENTED and upgraded only after the evidence existed, which is the order the
|
||
strict enum demands. All three legs then ran operator-present in one session (§4b) and each is cited
|
||
with timestamps in the row itself. The row also carries the gap the validation found (**R-55**) and
|
||
the leg deliberately left out of scope (the static guest → **R-50**), so the badge does not imply
|
||
more than was proven.
|
||
|
||
The existing **"Box survives a site/network change"** row (PARTIAL) already pointed at R-51/R-52 as
|
||
related work; it stays PARTIAL — R-50 is still its durable fix.
|
||
|
||
## 3. Part-5 riders
|
||
|
||
- **Hub `build.sh` deploy hint → GitOps wording.** The script printed `kubectl set image …` and
|
||
`kubectl apply -f manifests/hub.yaml` as the deploy instructions — both are reverted by the next
|
||
ArgoCD sync, which is the worst failure shape: it appears to work, then silently disappears. Now
|
||
prints the manifest-bump + hard-refresh + deliberate-sync sequence, and names the trap explicitly.
|
||
**Caveat worth knowing: `/mnt/5_hdd/felhom.eu/build/felhom-hub/build.sh` is NOT in any git repo** —
|
||
it is a DooPlex-local build-dir script. So this rider is a fix to the operative file only, and it
|
||
is not versioned anywhere. Worth adopting into the repo as its own change.
|
||
- **`documentation/PROMPT-TEMPLATE.md` §10 gains the seam-discipline row.** Text generalised from
|
||
the brief's §9 rule 6, plus what this session added to it: three shipped inert-seam defects in
|
||
three days (controller v0.154.0, agent v0.91.0, agent v0.92.0's missing sudoers grant), all fully
|
||
green; and the finding that a `strings.Contains` source assertion is not sufficient, because a
|
||
commented-out call still contains the string — walk the AST.
|
||
|
||
## 4. What was verified, and how
|
||
|
||
Endpoint/host-level, no browser (none on DooPlex). Everything claimed live in the two implementation
|
||
reports was read out of journald or the agent's own capability self-check on felhom-pve. No claim in
|
||
any of the three reports rests on a test that only proves a seam.
|
||
|
||
## 4b. The STOP legs RAN — all three (2026-07-21, operator-present)
|
||
|
||
The ordering below was accepted, executed, and all three legs passed. Full evidence chains are in
|
||
the two implementation reports and in the ROADMAP rows; the capability-map row is now
|
||
**PROVEN-LIVE**. Headline: dead primary → `degraded` in **13 s** with exactly one event; boot orphan
|
||
→ recovered in **one attempt** with **zero** alerts; dead dhclient → detected in **57 s** on process
|
||
liveness alone and healed **120 s** after the kill, roughly 80 minutes before the outage would have
|
||
started — `cloudflared` never dropped (`Up 29 hours`).
|
||
|
||
**One real gap surfaced, and it is the opposite of what the leg was designed to prove → new R-55.**
|
||
STOP-1's R-52 leg was meant to show that a customer's deliberate Stop survives a reboot. It does
|
||
not, for any **drive-backed** app: the boot bind gate (`internal/web/intermediary.go`) selects on
|
||
`Deployed && HDD_PATH` alone and runs `StopStack` + `StartStack` on every match, never consulting
|
||
whether the app was stopped on purpose. immich, stopped from the UI seconds earlier, came back
|
||
running. **R-52 is not implicated** — its own gate behaved exactly as specified (immich, at zero
|
||
containers, was never a candidate), and this behaviour predates it. Two consequences are worth an
|
||
operator ruling: a customer's Stop is silently undone on drive-backed apps, and R-52's practical
|
||
scope is narrower than the brief assumed, because the gate reaches every drive-backed app first.
|
||
|
||
**P1 is answered, as a free by-product of that leg.** bookstack carries `restart=unless-stopped`,
|
||
the Docker daemon came up at ~10:53:15Z, and the container's `StartedAt` is `10:53:28.05Z` — the
|
||
exact moment `bootrecon`'s `StartStack` returned. Docker did not resurrect it; the F5 hypothesis is
|
||
confirmed.
|
||
|
||
## 5. The ordering question (resolved)
|
||
|
||
The brief's Phase A says **do not hand-deploy** the controller — the floor save at **STOP-3** is what
|
||
deploys 0.156.0, banking another single-fire self-update datapoint. But **STOP-1 exercises the
|
||
controller legs**, which need 0.156.0 to be live. As written the two are in the wrong order.
|
||
|
||
Resolved as: **STOP-3's controller floor save first** (floor → `0.156.0`), box self-updates — that
|
||
IS the R-23 datapoint — then STOP-1 against the new version. **Done, and the datapoint is banked:**
|
||
`[selfupdate] Post-update startup: update successful (0.155.0 → 0.156.0)` followed by
|
||
`Current version 0.156.0 is up to date` — one swap, no re-fire. The agent half of STOP-3 (manifest → 0.92.1) is independent and can happen at any
|
||
point. Flagged rather than assumed, because reordering an operator's STOP sequence is not CC's call.
|
||
|
||
## 6. Follow-ups this session surfaced (none actioned here)
|
||
|
||
1. `felhom-controller/controller/.gitignore`'s bare `controller` entry also matches the directory
|
||
`cmd/controller/`, so ripgrep silently skips `main.go` and new files there need `git add -f`.
|
||
Both directions produce inert-seam mistakes. XS fix: anchor it as `/controller`.
|
||
2. The hub `build.sh` above is unversioned.
|
||
3. `TestGenerateRecoveryCode_EntropyAndFormat` (agent) flakes on hyphenated wordlist entries
|
||
(`drop-down` → 11 tokens). Observed 3/8 this session — worse than the documented ~1/5, and it is
|
||
a one-line fix in the generator or the assertion, not a mystery.
|