From d5be67b91363f81b008e081729e401814f11090f Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 2 Aug 2026 23:57:07 +0200 Subject: [PATCH] REPORT: CI run ids and conclusions (all three commits green) --- REPORT.md | 380 ++++++++++++++++++++++++++---------------------------- 1 file changed, 184 insertions(+), 196 deletions(-) diff --git a/REPORT.md b/REPORT.md index 82bbbee..e14534b 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,247 +1,235 @@ -# REPORT — v0.190.0: boot recovery finished, and a regression v0.189.0 opened (R-157 A · R-170 · R-171) +# REPORT — v0.191.0 / .1 / .2: warn before the wall comes down (R-167 · R-158 · R-174) -**Overwritten** per the standing rule; the prior contents (v0.189.0 / R-166, same day) have their -durable record in `CHANGELOG.md`. +**Overwritten** per the standing rule; the prior contents (v0.190.0 / R-157 A · R-170 · R-171, same +day) have their durable record in `CHANGELOG.md`. -**Session:** 2026-08-02 · **Repos:** `felhom-controller` v0.189.0 → **v0.190.0**, `felhom.eu` (docs -only, no hub bump) · **Live target:** demo box `felhom-pve` guest **9201** · **No STOP point.** +**Session:** 2026-08-02 · **Repos:** `felhom-controller` v0.190.0 → **v0.191.2**, `felhom.eu` hub +v0.88.0 → **v0.89.0** + docs. `felhom-agent` untouched (Part 3 read its golden-build script and +changed nothing in it). --- -## 1. Baselines and the five timing constants — all matched §1 on arrival +## 1. Baselines — one drifted -| Repo | `main` @ arrival | Expected | Match | +| Repo | §1 said | Found on arrival | Match? | |---|---|---|---| -| `felhom-controller` | `344660942015da292e5bfdf98c457124b6f633a1` | `344660942015` | ✅ | -| `felhom.eu` | `5c97fbc397f2148ea4d5c98d4de06e3dc8b69a91` | `5c97fbc397f2` | ✅ | +| `felhom-controller` | `95eb5c2c1af4`, v0.190.0 | `95eb5c2c1af4`, v0.190.0 | **yes** | +| `felhom.eu` | `d5774d318941`, hub v0.87.0 | **`8ef92a3f`, hub v0.88.0** | **NO** | -| Constant | Value | Confirmed at | Match | -|---|---|---|---| -| `bootReconcileSettle` | `5 * time.Second` | `cmd/controller/main.go:1264` | ✅ | -| `DefaultAttempts` | `2` | `internal/bootrecon/bootrecon.go:46` | ✅ | -| `DefaultRetryDelay` | `30 * time.Second` | `internal/bootrecon/bootrecon.go:49` | ✅ | -| `deadAppBootGrace` | `90 * time.Second` | `cmd/controller/main.go:1238` | ✅ | -| drive-gate tick | `30 * time.Second` | `internal/web/intermediary.go:339` | ✅ | +**hub v0.88.0 had already shipped** (R-172, the WAL fix) between the task being written and this +session. Target corrected to **hub v0.89.0**. Highest register ID in use was **R-173**, not R-171. -No drift. Both trees clean before the build. +## 2. One decision taken with the operator before any code was written -## 2. Part 0's verdict — **CONFIRMED**, and the first attempt was a false negative +Reconnaissance found that **`disk_warning` / `disk_critical` were a complete customer pipeline with no +producer** — in the hub's `allowedEventTypes`, carrying Hungarian `customerMessages`, in +`settings.DefaultEnabledEvents`, with a UI checkbox (`event_disk_alerts`) — and `grep` across all four +repos found **zero emitters**. The only thing reacting to guest disk pressure was `healthcheck.go:338`, +folding it into a generic `health_degraded` at 90% for registered storage paths only. -**The sweep DOES start an app whose data drive is absent.** Observed on 9201, both attempts: +The task said to mint a new customer-facing type. Doing so would have put a near-duplicate beside an +inert pair — and §5's own reason for rejecting `backup_failed` (it is customer-enabled and carries +Hungarian copy) is the argument *for* reusing this one, because D-c routes the fill warning **to the +customer**. **Operator chose: wire the existing pair.** So only ONE new hub type was minted (the +operator one), and the sixth *built-but-never-wired* instance was closed rather than joined by a seventh. -``` -17:22:07 [gate] drive ABSENT /mnt/felhom-drives/hdd_1 — stopped+blocked 2 app(s): [calibre-web immich] -17:22:34 [bootrecon] Boot reconciliation: 1 boot-orphaned app(s) found: [calibre-web] — up to 2 attempt(s) -17:22:34 [stacks] Starting stack: calibre-web -17:22:35 [bootrecon] attempt 1/2: start "calibre-web" failed after 0.4s: exit code 1 -17:23:05 [bootrecon] attempt 2/2: start "calibre-web" failed after 0.4s: exit code 1 -17:23:05 [bootrecon] gave up after 2 attempt(s): recovered=[] still down=[calibre-web] (the dead-app alarm now owns these) -``` +## 3. Files changed and commits -All four preconditions were verified first: drive mounts **0**, containers **0**, -`desired_state: running`, `deployed: true`. +**6 modified, 4 added.** `controller/cmd/controller/main.go`, +`internal/backup/{appstop_marker,backup,recovery_unit}.go`, `internal/notify/notifier.go`, +`cmd/controller/appstop_wiring_test.go`, `CHANGELOG.md`, `REUSE.md` · **new:** +`internal/fillwatch/{fillwatch.go,fillwatch_test.go}`, +`internal/backup/{appstop_drivegate_test.go,recovery_unit_notify_test.go}` -**RUN 1 WAS A FALSE NEGATIVE AND IS REPORTED BECAUSE IT PRODUCED A MECHANISM.** Unmounting only the -parent bind is **healed by the agent within ~60 s** (`/dev/sdb` is still mounted at `/mnt/hdd_1`), so -the drive gate's startup reconcile re-attached and restarted the apps at `17:18:47` — **one second -before** the sweep looked at `17:18:48` and logged `no boot-orphaned apps`. Reporting that as a -disproof would have been wrong: it is a race that went the safe way. Run 2 held **both** mounts off -for the duration and reproduced the defect immediately. +| Commit on `main` | Subject | +|---|---| +| `cf48214` | v0.191.0 — warn before the wall comes down (R-167, R-158, R-174) | +| `5adae4d` | v0.191.1 — the fill check also runs at startup (R-167) | +| `9a3c485` | v0.191.2 — a quiet fill check now says so (R-167) | -**The confirmation moved the severity in both directions.** +## 4. Tests -- The **write** hazard did NOT materialise. Compose failed with - `mkdir /mnt/felhom-drives/hdd_1/userdata: permission denied`; `find` showed only the bare directory - and `df -h /` was unchanged at `25G / 28%` before and after. The mechanism: with the drive unbound - the mountpoint is **host-root-owned** (`uid=0 gid=0 mode=755`) and guest 9201 is an **unprivileged** - LXC, so container root (uid 100000) cannot create the subdirectory. **This protection is accidental** - — no code chose it, no test pinned it, and it depends on the guest staying unprivileged and the - mountpoint staying root-owned (when the drive IS bound, that same path shows `uid=100000`). One - `chown` away from gone, and its removal would be invisible until data landed on the wrong disk. -- The harm that **did** occur was not in the hypothesis and is real on every box: two wasted attempts, - a leftover `Created` container per attempt, and a **false dead-app alarm for an app the drive gate - is deliberately holding** — the noise class R-97/F-A1 exist to prevent. +**1157 → 1184 (+27).** Full suite green in both repos (`go build ./... && go vet ./... && go test +./...`); all **eight** controller gates and all **five** felhom.eu gates OK. Hub: 574 → **579**. -**A second asymmetry, found while restoring the box:** the API's own start path already refuses this — -`{"ok":false,"error":"A(z) /mnt/felhom-drives/hdd_1 tárhely jelenleg nem elérhető …"}` from -`startGatedByMissingDrive`. The rule existed; the sweep bypassed it by calling `Manager.StartStack` -directly. Full evidence: `felhom.eu/documentation/audits/DIAG-bootrecon-drive-absent-2026-08-02.md`. +### Red-proofs — each mutated, observed failing, restored -**Consequence:** Part 3 applied and was implemented **first**, before the window widened. - -## 3. Files modified / created; commits - -**Created:** `controller/internal/bootrecon/drivegate_test.go`, -`controller/cmd/controller/bootwindow_test.go`, `controller/internal/web/recreate_intent_test.go`, -`felhom.eu/documentation/audits/DIAG-bootrecon-drive-absent-2026-08-02.md` - -**Modified:** `bootrecon/bootrecon.go` (StartGate seam, held filter, `Result.HeldByDrive`), -`cmd/controller/main.go` (settle window, `sampleBootFleet`, `recordLateRecovery`, `bootDriveGate`, -wiring), `stacks/deploy.go` (`Manager.DriveLive`), `backup/appstop_marker.go` (`HeldStacks`), -`web/intermediary.go` (`shouldRecreateOnBoot` + `bootStack.desired` + counters), -`web/intermediary_test.go`, `cmd/controller/appstop_wiring_test.go`, `CHANGELOG.md`, `CONTEXT.md`, -`REUSE.md`, `controller/README.md`; `felhom.eu`: `CONTEXT.md` (S-13), `STATUS.md`, -`architecture/00-capability-map.md`, `architecture/02-controller-module-map.md`, -`backlog/OPEN-ITEMS.md`, `backlog/ROADMAP.md` - -| Repo | Commit | Subject | +| # | What was mutated | Result | |---|---|---| -| `felhom-controller` | `582135f` | v0.190.0 — the boot settle window, both gates on intent, and R-171 | -| `felhom-controller` | `dcc3363` | boot window: sample REFRESHES first — a cached fleet made 'settled' meaningless | -| `felhom-controller` | `4bad6e0` | CHANGELOG: the boot window's sample refreshes first | -| `felhom.eu` | `ad28699` | docs: R-157 A / R-170 / R-171 closed — boot recovery finished | +| A1 | the `errors.Is(err, ErrStartRefused)` branch removed from `Recover` | **4 tests FAIL** — refusals collapse into `Failed` and alarm | +| A2 | `SetStarter(gatedAppStopStarter{…})` → `SetStarter(stackMgr)` (the v0.189.0 wiring) | `TestMainWiresGatedAppStopStarter` **FAILS** | +| C | the `m.unitNotify(...)` call removed from the capture loop | **2 tests FAIL** — the seam-never-wired shape | +| E | **both** edge guards removed | **3 tests FAIL** — warns twice; the dead zone breaks | +| G | `recovery_unit_capture_failed` removed from `operatorOnlyEvents` | **2 tests FAIL**, one reading *"a customer was emailed the OPERATOR-ONLY …"* | +| H | clear thresholds edited into equality with warn | `TestThresholdsKeepTheirHysteresisGap` **FAILS** | +| J | `fillWatcher.SetNotify(` **commented out** (string still in the file) | `TestMainWiresTheFillWatcher` **FAILS** — proves AST, not substring | -Direct to `main`, no branches, no `Co-Authored-By`. Pre-push hook green; **`--no-verify` NOT used.** +**One red-proof was a FALSE GREEN on the first attempt, and it is reported rather than quietly +redone.** Red-proof E's first mutation removed only `if next == prev { continue }`; a second guard +(`next <= prev`) still caught the repeat, so every test passed and the "proof" proved nothing. Redone +removing both guards. This is exactly the trap `CLAUDE.md` records — a mutation or a `-run` filter +that yields a green and reads like a completed proof. -## 4. Tests and red-proofs +**One pre-existing test was TIGHTENED, not loosened.** `TestMainReportsTheInterruptedOperation` +asserted the guard was `if appStopRecovery != nil`. R-174 makes that insufficient — `Recover` now +returns non-nil for a refusal-only recovery — so the test now *requires* `Alarming()` in the condition. -**27/27 packages green; +25 test functions across 3 packages.** `go build ./... && go vet ./... && -go test ./...` → **rc=0**. Timing is tested by shrinking the window constants — **never by sleeping**. -Test runs and commits were never combined in one command. +## 5. Thresholds and cadence, with their justification -Every red-proof observed **FAIL**, then reverted with a passing control run. - -| # | Test | Mutation | Result | -|---|---|---|---| -| **A** | `TestBootWindow_LateSettlerIsSweptOnASettledFleet` | restored the single-sweep shape (deleted the sampling loop) | **FAIL** ✅ — the sweep was handed the `starting` frame | -| **B** | `TestBootWindow_BudgetEndsAForeverChangingFleet` | removed the `time.Since(started) < budget` condition | **FAIL** ✅ — *"did not terminate on a forever-changing fleet"* (a genuine hang; the fixture was changed to CYCLE first, because a fixture whose frames stop changing terminates by settling and would not have reached the hang) | -| **C** | `TestBootWindow_CustomerStoppedAppSurvivesEveryPass` | dropped the `DesiredStateStopped` branch from `isBootOrphan` | **FAIL** ✅ — *"started 2 time(s) by the widened window"* | -| **D** | `TestShouldRecreateOnBoot_IntentTable` + `_AgreesWithBootrecon` + `RecreateDriveBackedApps_Recorded` | restored `&& hasContainers` | **FAIL** ✅ (3 tests, incl. the cross-gate disagreement assertion) | -| **G** | `TestReconcile_DriveAbsentApp_IsNeverStarted` + `_UndeterminableDrive_` + `_HeldByAnyHolder_` | removed the start-gate check from `Run` | **FAIL** ✅ — *"an app whose data drive is ABSENT was started 1 time(s)"* | -| **H** | `TestMainWiresBootDriveGate` | **commented out** `SetDriveGate(...)` | **FAIL** ✅ — while `grep -c` still returned **1**, i.e. a `strings.Contains` test would have passed | -| **I** | `TestBootWindow_EverySampleRefreshesTheStatus` | removed `RefreshStatus()` from `sampleBootFleet` | **FAIL** ✅ — *"refreshed 0 time(s) for 4 samples"* | - -**One defect in my own fix, caught by my own test:** `TestBootWindow_CommonCaseFitsInsideTheDeadAppGrace` -rejected the first budget at `5 + 60 + 30 = 95 s > 90 s`. The budget is 50 s **because a test said -so**, and the code comment records that rather than presenting the number as taste. - -## 5. The window shape, and its justification against the 90 s grace - -| Constant | Value | Why | +| | value | why | |---|---|---| -| `bootReconcileSample` | 5 s | fine enough to see a container settling at T+40 s within one sample; ~12 cheap calls on a quiet boot | -| `bootReconcileStableFor` | 3 identical samples (15 s) | one sample cannot tell "settled" from "sampled between two docker events"; three spans a container exiting and its restart policy re-creating it | -| `bootReconcileBudget` | 50 s | **binding constraint:** `settle 5 + budget 50 + one DefaultRetryDelay 30 = 85 s` < `deadAppBootGrace 90 s`, so a successful recovery stays SILENT | +| warn | used ≥ **85%** OR free < **5 GiB** | a percentage alone lies at both ends of this fleet's real size range: 85% of a 20 G `mp1` leaves 3 G — less than one DB-backed app's unit (~2× its data, §7.5) — while 85% of a 4 TB drive leaves 600 G | +| critical | used ≥ **95%** OR free < **2 GiB** | below 2 GiB a volume tar of almost any real app fails: "the next backup will not complete", not "it is getting tight" | +| clear | used ≤ **75%** AND free ≥ **7 GiB** | both must hold; the gap to warn is the hysteresis dead zone, so a filesystem on the line does not flap. Pinned with a real margin, not merely an inequality | +| cadence | **daily 03:30** + **once 90 s after startup** | a fill is slow-moving, so a shorter interval buys no earlier warning; 03:30 precedes the nightly app-data legs, so a customer about to lose a backup hears about it with a night's margin. The startup run exists because neither `Daily` nor `Every` fires on registration — §7 | -Terminates on **whichever comes first** — settled, or budget — and the log says which, because -"settled and found nothing" and "ran out of time still churning" are different facts. Sampling is -read-only apart from the status refresh; exactly **one** sweep runs, on a settled fleet, with the -per-app attempt bound untouched. Sweeping per sample was rejected: the sweep's own `StartStack` -changes the fleet, so it would never observe a settled one. +**The live proof vindicated the two-term design:** the critical crossing fired on the **free-byte** +term (1.7 GB) while the disk was only **91%** used. A percentage-only rule would have missed it entirely. -**§8.3 chosen design: allow the window, report a late recovery.** `recordLateRecovery` emits a -`LATE RECOVERY` WARN naming the apps when a recovery lands past the grace. The grace was **not** -widened — that hides a late recovery rather than reporting it. +## 6. The exact Hungarian customer copy, for review as copy -## 6. §8.2, row by row +**`disk_warning`** (severity `warning`) — as rendered live on 9201: -| Stopped by | Sweep starts it? | How answered | -|---|---|---| -| the customer (`desired_state: stopped`) | **No** | already handled by `isBootOrphan`; **re-proven** — unit (`TestBootWindow_CustomerStoppedAppSurvivesEveryPass`, red-proofed) and live in **6/6** hard resets | -| a quiesce for a backup | **No** | **exclusion WIRED** — `bootDriveGate.MayStart` consults `quiesce.SuppressedStacks()`, the set the quiesce loop already publishes for this exact question | -| the backup app-stop guard | **No** | **exclusion WIRED** — new read-only `AppStopGuard.HeldStacks()`. Its `Recover` completes before this goroutine starts, so a marker seen here belongs to an operation running **now**, not a crashed one | -| the drive-absent gate | **No** | **exclusion WIRED** — Part 3; settings flags (same signal as the API gate) **plus** the live `Manager.DriveLive` mountpoint check, because the flags can lag inside the boot window | -| an interrupted boot | **Yes** | the point of the change | +> A(z) „Rendszer- és mentési terület” tároló 85% foglalt — 4,7 GB szabad hely maradt. Kérjük, +> szabadíts fel helyet, mielőtt megtelik: törölj felesleges fájlokat, vagy csatlakoztass új +> meghajtót. Ha megtelik, a biztonsági mentések meghiúsulnak. -All three holders answer **one** seam because they differ only in the reason string. `TestReconcile_HeldByAnyHolder_IsNeverStarted` covers all three; none is answered by "it probably won't happen". +**`disk_critical`** (severity `critical`) — as rendered live on 9201: -## 7. The hard-reset log — every run individually +> A(z) „Rendszer- és mentési terület” tároló kritikusan megtelt: 1,7 GB szabad hely maradt (91% +> foglalt). A biztonsági mentések és az alkalmazások írásai bármikor meghiúsulhatnak. Kérjük, +> mielőbb szabadíts fel helyet: törölj felesleges fájlokat, vagy csatlakoztass új meghajtót. -**Shipped build (v0.190.0 with the refresh fix), the six that count:** +Decimal **comma** throughout (`4,7 GB`); the storage is named by its **label**, never its path; both +end in an action. No emoji (asserted by a test; `emoji_gate` passes). -| Run | Window verdict | Apps back (immich · calibre-web · docmost) | `bookstack` (desired=stopped) | +## 7. Two defects found by the validation itself, both fixed and shipped + +1. **v0.191.1 — the check was reachable only on its daily schedule.** Neither `sched.Daily` nor + `sched.Every` fires on registration; both wait for their first tick. A box that BOOTS already over + the line would have stayed silent for up to 24 h — the R-100 shape, and the same gap the hub's own + checkers avoid by leaving already-breached keys unseeded at init. Now also runs once 90 s after + startup; safe because the check is edge-triggered against persisted state, so an already-warned + filesystem stays silent. +2. **v0.191.2 — a quiet run was unreadable as evidence.** After the customer had been warned, a + restart produced **zero** `fillwatch` lines — equally consistent with "ran and chose silence" and + "never ran". For an edge-triggered check **the quiet run is the healthy steady state**, so that + ambiguity is permanent, not rare. `Check` now logs a per-run summary (`checked N filesystem(s), M + unreadable/skipped, K notification(s); bands: …`), counting unreadable separately so a silently + unreadable drive cannot read as "all fine". + +## 8. Live evidence — three flows, guest 9201 on `felhom-pve` + +**Method: endpoint/log-level on real hardware**, exercising the production paths — browser automation +is not available on DooPlex. The hub's `notification_log` was read from a **WAL-aware** copy of +`/data/hub.db` (all three files — R-172's lesson); the copies were deleted afterwards. + +**Flow 1 — the guard refuses, keeps the marker, does not alarm.** Both `hdd_1` mounts held unmounted +against the agent's ~60 s heal, per the documented method: + +``` +[WARN] [appstop] refusing to restart "calibre-web" after an interrupted operation: + drive /mnt/felhom-drives/hdd_1 is not a live mountpoint +[WARN] [appstop] crash recovery: NOT restarting calibre-web — start refused by a deliberate + holder: … the marker is KEPT and the holder owns the restart +[WARN] [appstop] crash recovery: 1 app(s) were deliberately NOT restarted (drive absent) — + KEEPING the marker; this is the gate working, not a fault: [calibre-web] +[WARN] [appstop] …HELD… — not alarming: … restarted=[] held_by_drive=[calibre-web] +``` + +Marker retained **byte-identical**; `docker ps -a --filter name=calibre-web` → **0**; no +`backup_failed` event. **Scenario B, on a later boot with the drive live:** `restarted calibre-web +after the interrupted an app-data backup (volume dump)` and the marker was **cleared**. + +**Flow 2 — the operator event arrives; the customer is refused.** Two real capture failures: + +``` +Event pushed: recovery_unit_capture_failed (error) — Recovery unit capture FAILED for "calibre-web" + … /mnt/felhom-drives/hdd_1: 24.5/93.9 GB used (26%), 64.6 GB free. Error: … permission denied +``` + +Hub `notification_log` — the positive observable: + +``` +customer | recovery_unit_capture_failed | skipped | operator_only +operator | recovery_unit_capture_failed | sent | +``` + +**Flow 3 — the customer is warned, once, in Hungarian.** `/mnt/sys_drive` filled with `fallocate`: + +| run | state | fired | log | |---|---|---|---| -| F1 | settled after **10 s** | all 3 ✅ | 0 containers ✅ | -| F2 | settled after **40 s** | all 3 ✅ | 0 containers ✅ | -| F3 | settled after **10 s** | all 3 ✅ | 0 containers ✅ | -| F4 | settled after **10 s** | all 3 ✅ | 0 containers ✅ | -| F5 | settled after **15 s** | all 3 ✅ | 0 containers ✅ | -| F6 | settled after **15 s** | all 3 ✅ | 0 containers ✅ | +| 1 | 90% / 4.7 GB | `disk_warning` | `ok → warning … notifying the customer` | +| 2 | unchanged | **nothing** | (edge trigger held) | +| 3 | 91% / 1.7 GB | `disk_critical` | `warning → critical …` | +| 4 | file removed | **nothing** | `critical → ok … cleared silently, re-armed`; state emptied | -**6/6 for both the recovery and the customer-stop.** An earlier six on the pre-refresh build were -also 6/6 (settle times 50/30/20/50/25/10 s) but are **not** counted as the deliverable, because the -build changed afterwards. +Hub: `customer | disk_warning | sent` and `customer | disk_critical | sent`, Hungarian rendered. +**Exactly two events across four runs.** Run 2's silence is meaningful **only because run 3 proves the +startup check executes every boot** — needing that inference is what motivated v0.191.2. -**The settle times are themselves the evidence for mechanism A:** the window waited 10–50 s where the -old code decided at a fixed 5 s. Two pre-fix runs settled at 50 s — the full budget — i.e. the old -sweep would have decided 45 s too early. - -**The sharpest evidence is a same-app, same-box before/after**, because 6/6 clean runs only show -nothing regressed: +## 9. Deployed versions ``` -pre-fix 18:08:35 boot window: fleet settled after 25s — sweeping - 18:08:35 Boot reconciliation: no boot-orphaned apps (nothing to start) ← calibre-web had ZERO containers -post-fix 18:18:50 boot window: fleet settled after 10s — sweeping - 18:18:50 Boot reconciliation: 1 boot-orphaned app(s) found: [calibre-web] - 18:18:52 attempt 1/2: started "calibre-web" (took 1.7s) - 18:18:52 complete: 1 app(s) recovered in 1 attempt(s): [calibre-web] +$ ssh felhom-pve "pct exec 9201 -- docker ps --filter name=felhom-controller …" +gitea.dooplex.hu/admin/felhom-controller:0.191.2 | Up (healthy) ``` -That miss is what exposed the cache defect in §4 (red-proof I). +Hub: ArgoCD app `felhom` **Synced / Healthy** after a deliberate hard-refresh + sync (never +`kubectl set image`); `deploy/hub` rolled out; image `gitea.dooplex.hu/admin/felhom-hub:0.89.0`; +startup log clean. -## 8. Which gate reads what afterwards — both answer intent +## 10. Teardown -**R-170 proven live in a single reboot**, both halves, on drive-backed apps with **zero containers**: +`R167-FILLTEST.bin` deleted (`/mnt/sys_drive` back to 2.0 G used / 45 G free); the app-stop marker +cleared by a successful recovery; `/root/hold-unmount.sh` + `/tmp/hold.log` removed from `felhom-pve`; +the secret-bearing local `hub.db*` copies deleted. **All 15 containers on 9201 healthy.** -``` -18:20:11 [gate] live bind confirmed — recreating drive-backed app calibre-web (state=stopped) onto /mnt/felhom-drives/hdd_1 -18:20:12 [gate] 1 drive-backed app(s) left stopped on purpose (recorded Stop, or a legacy app.yaml at zero containers) -→ calibre-web desired=running containers=1 (recreated — pre-R-170 this was SKIPPED: hasContainers=false) -→ immich desired=stopped containers=0 (left alone) -``` +**One self-inflicted incident, disclosed in full.** Restoring the drive by hand I ran +`mount --bind /mnt/hdd_1 /mnt/felhom-drives/hdd_1`, **omitting the `felhom-data` segment** the agent +uses (`felhom-agent/internal/localapi/intermediary.go:15` — +`mount --bind /mnt//felhom-data /mnt/felhom-drives/`). The guest then saw the drive root +instead of the namespace root, which an unprivileged guest cannot write to — producing real +`permission denied` capture failures. They served as flow 2's evidence, but **they were caused by me, +not found by me, and the report says so.** Separately, a manual `docker compose up -d` from a stack +dir started containers **without the controller-injected env** (there is no `.env` on disk — the +controller injects at exec time), which recreated `immich-server` with a blank DB password and left it +in a `28P01` auth-failure restart loop. Both were repaired through production paths: the bind +corrected, then the stacks brought back by the controller's own `StartStack` and boot reconciler +(`[bootrecon] … 1 app(s) recovered in 1 attempt(s): [immich]`). **Lesson, learned twice in one +session: on a box the agent manages, restore state by letting the agent and controller do it.** -The boot sweep then logged `no boot-orphaned apps` — the two gates agreeing rather than fighting. -Structurally, agreement is pinned from **both sides** against one identical fixture table -(`TestBothBootGatesAgreeOnIntent` in `bootrecon`, `TestShouldRecreateOnBoot_AgreesWithBootrecon` in -`web`), because an import cycle prevents calling the two from one package. +## 11. Register rows -## 9. `OPEN-ITEMS.md` rows +**Opened:** R-174 (closed same session), R-175, R-176, R-177 — each ID established free by +`grep -ro "R-17n\b" documentation/ *.md` → 0 hits. **Closed:** R-158 (by R-167 — *no second row was +filed for the same wire*), R-167, R-174. **Updated and still open:** R-165 (gains the spike's M1-M5 +and the operator question). -- **R-157 → CLOSED** (both mechanisms), with the repeat count as evidence, not one clean boot. -- **R-170 → CLOSED**, with the live both-halves reboot. -- **R-171 → NEW and CLOSED the same session**, marked a regression from v0.189.0. **ID established - free before minting:** `grep -ro "R-171\b" documentation/ *.md` → **0 hits**. -- `ROADMAP.md` collapsed to match; `STATUS.md`'s power-cut line moved from "What's broken" to "What - works right now" with the repeat count, plus one dated bullet. +## 12. CI and `--no-verify` -## 10. Teardown and CI +**`--no-verify` was NOT used.** Every push ran `.githooks/pre-push` (gates `--fast`) and it passed. -**Provisioned nothing** — no VM, no guest, no scratch app, no storage. The apps used for validation -already existed and were **all restored**: `bookstack`, `immich`, `calibre-web`, `docmost` all -`desired=running` and running, 15 containers, controller `0.190.0` healthy. The drive is remounted -(`/dev/sdb` at both `/mnt/hdd_1` and `/mnt/felhom-drives/hdd_1`), and the temporary unmount-holder -loops are gone (`ps aux | grep -c '[u]mount …'` → **0**). The credential-bearing helper was deleted -from the guest (`/root/act.sh`, `/tmp/act.sh`) and from DooPlex. +**CI checked by PULL, matching `head_sha` to each commit** — CI emails only on failure, so a green +that was never looked at is an assumption. All three commits green: -**CI — every push checked by run id and run number, matching `head_sha` to the commit:** +| Commit | Task id | Run # | Conclusion | +|---|---|---|---| +| `cf48214` (v0.191.0) | 31 | 11 | **success** | +| `5adae4d` (v0.191.1) | 34 | 12 | **success** | +| `9a3c485` (v0.191.2) | 35 | 13 | **success** | -| Repo | Task id | Run number | `head_sha` | Status | -|---|---|---|---|---| -| `felhom-controller` | **21** | 6 | `582135f8` | **success** | -| `felhom-controller` | **22** | 7 | `dcc3363d` | **success** | -| `felhom-controller` | **23** | 8 | `4bad6e06` | **success** | -| `felhom-controller` | **24** | 9 | `e6311f9f` | **success** | -| `felhom.eu` | **25** | 12 | `ad286997` | **success** | +The `felhom.eu` side (hub v0.89.0, the manifest bump and the docs) is in that repo's `REPORT.md` §6 — +also all green. -Pulled from the run list and polled until each left `running` — not inferred from the absence of a -failure e-mail, which is the check `felhom.eu/CLAUDE.md` now names. +## 13. Observations — noticed, documented, NOT acted on -## 11. Observations — noticed, documented, NOT acted on - -1. **`Manager.StartStack` has no drive gate of its own**, and now has fourteen callers. The API path - guards it (`startGatedByMissingDrive`), the boot sweep now guards it (R-171), and the remaining - machine callers each decide for themselves. Pushing the check *into* `StartStack` was NOT done — - it would break the legitimate callers that must start an app on a drive the gate is mid-way - through re-attaching (the drive gate's own `Return` branch is one). Recorded because the next - person to add a caller needs to know the gate is per-caller. -2. **The agent heals an unmounted drive within ~60 s** while its block device is present. This is - correct behaviour and it is why the Part 0 diagnosis needed the drive held off, but it also means - **"unmount to simulate an absent drive" does not work on this platform** — worth knowing before - the next drive-absence test. -3. **The accidental permission protection** (§2) is untested and unowned. Not pinned by a test here - because a test asserting it would enshrine an accident; the fix removes the dependency on it - instead. If a privileged guest is ever shipped, this is the note to re-read. -4. **`recreateBootStaleApps` runs before the boot sweep and can fix the same app first**, which is - why several runs logged `no boot-orphaned apps` after the gate had already acted. Not a defect — - the gates agree — but it means the sweep's log alone under-reports how much recovery happened. +- **R-177** — no operator-triggerable "run this scheduler job now" path. It cost a controller restart + per observation here, and costs the same on a support call. +- **R-175** — `07-backup-architecture.md` §7.5 states one box's size bound as if it were the fleet's. +- **R-176** — two R-165 prerequisites are unmeasured (a pre-merge-archive restore-test; the in-place + migration rehearsal). +- **The drive gate's return branch failed to restart both apps** (`[WARN] [gate] restart calibre-web: + … exit code 1`) at a moment when a manual start succeeded seconds later. **Not filed as a row**: it + happened during my incorrect bind, so the likeliest cause is mine and the evidence is contaminated. + Worth a look if it recurs on a clean box.