From e6311f9fbcd36ec0c1635678b5aeaa1a6c305542 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 2 Aug 2026 20:34:48 +0200 Subject: [PATCH] docs: CONTEXT + REPORT for v0.190.0 (R-157 A / R-170 / R-171) --- CONTEXT.md | 53 +++++++ REPORT.md | 457 +++++++++++++++++++++++------------------------------ 2 files changed, 249 insertions(+), 261 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index 3eef678..8ab3bf0 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -7,6 +7,59 @@ > > Ask Claude Code: "Please update CONTEXT.md with what we did today" +Last updated: 2026-08-02 (v0.190.0 — R-157 A · R-170 · R-171: boot recovery finished) + +> **2026-08-02 — v0.190.0 (R-157 mechanism A · R-170 · R-171).** Three items, one live validation +> cycle, because all three are boot behaviour and all three are proven by hard-resetting the box. +> +> **DIAGNOSE BEFORE THEORISING — and the first diagnosis was a FALSE NEGATIVE.** A hole was reasoned +> out of the v0.189.0 diff (a drive-gate-stopped app has zero containers and `desired_state: running`, +> so it now reads as a boot orphan) and confirmed on hardware BEFORE any fix was written. **Attempt 1 +> produced `no boot-orphaned apps` and would have been reported as a disproof.** It was a race: +> unmounting only the parent bind is healed by the agent within ~60 s, so the drive gate's startup +> reconcile restarted the apps **one second before** the sweep looked. Holding the drive genuinely +> absent reproduced the defect immediately. **"It didn't happen this time" is not a mechanism.** +> +> **The confirmation moved the severity in BOTH directions.** The write hazard did not materialise — +> compose failed `mkdir …/userdata: permission denied` because the unbound mountpoint is +> host-root-owned and the guest is unprivileged. **That protection is ACCIDENTAL**: no code chose it, +> no test pinned it, and it is one `chown` or one privileged guest away from gone. But the harm that +> DID occur was not in the hypothesis and is real on every box: two wasted attempts and a **false +> dead-app alarm for an app the drive gate is deliberately holding**. +> +> **The fix already existed one path over.** `startGatedByMissingDrive` (the API) refuses a customer's +> start on an absent drive; the sweep bypassed it by calling `Manager.StartStack` directly. +> **`StartStack` HAS NO GATE OF ITS OWN** — carry this: every caller that is not the customer must +> decide for itself whether the app may run. New consumer-side `bootrecon.StartGate`, fail-safe +> (cannot determine ⇒ do not start). +> +> **Widening a window makes previously-unreachable overlaps reachable — a design input, not an +> afterthought.** The old T+5 s sweep never met a quiesce or an in-flight app-data operation; a 50 s +> window can. All three holders answer ONE seam because they differ only in the reason string. +> +> **A TEST REJECTED MY FIRST CONSTANT, and the comment says so.** `settle + budget + one retry` must +> fit inside `deadAppBootGrace`; 60 s gave 95 s against 90 s. The budget is 50 s **because a test said +> so** — recorded in the code rather than presented as taste. Widening the grace was rejected: it +> hides a late recovery instead of reporting one (`recordLateRecovery`). +> +> **THE FIX HAD ITS OWN DEFECT, FOUND LIVE AND NOT BY REVIEW.** The window sampled `GetStacks()` — the +> Manager's map, refreshed by the scheduler every **10 s** — every 5 s, so two identical samples could +> mean *the cache did not update*. Observed: a container removed ~5 s before the window closed was +> still in the sampled fleet and the sweep logged `no boot-orphaned apps` for an app that had none. +> `sampleBootFleet` now refreshes first. **Generalise: a settle detector is only as good as the +> freshness of what it samples — if the source is cached, refresh it, or you are watching the cache +> settle rather than the system.** +> +> **R-170:** `shouldRecreateOnBoot` reads intent with the identical three-way table; absent keeps the +> old `hasContainers` behaviour exactly; `presentStable` untouched and still load-bearing. Its comment +> argued at length FOR the count and was rewritten. Agreement pinned from BOTH sides against one +> fixture table (an import cycle prevents testing the two gates together). +> +> **Live: 6/6 hard resets** (every app back; the customer-stopped app down all six), settle times +> 10/40/10/10/15/15 s. Sharpest evidence: same app, same box — missed at 18:08:35, recovered at +> 18:18:50. R-170 proven in one reboot (calibre-web recreated, immich left stopped). 27/27 packages; +> 7 red-proofs. Detail: `REPORT.md`. + Last updated: 2026-08-02 (v0.189.0 — R-166 / D-b: the box stops guessing what the customer wanted) > **2026-08-02 — v0.189.0 (R-166, operator decision D-b).** When an app was not running the box had diff --git a/REPORT.md b/REPORT.md index 13a5170..4c9a03b 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,301 +1,236 @@ -# REPORT — v0.189.0: the box stops guessing what the customer wanted (R-166 / decision D-b) +# REPORT — v0.190.0: boot recovery finished, and a regression v0.189.0 opened (R-157 A · R-170 · R-171) -**Overwritten** per the standing rule. The prior contents (the R-168 CI workflow, same day) have -their durable record in `CHANGELOG.md`. +**Overwritten** per the standing rule; the prior contents (v0.189.0 / R-166, same day) have their +durable record in `CHANGELOG.md`. -**Session:** 2026-08-02 · **Repos touched:** `felhom-controller` (v0.188.0 → **v0.189.0**), -`felhom.eu` (docs only — **no hub change, no hub version bump**) · **Live target:** demo box -`felhom-pve` guest **9201** · **No STOP point; nothing destructive; no customer data touched.** +**Session:** 2026-08-02 · **Repos:** `felhom-controller` v0.189.0 → **v0.190.0**, `felhom.eu` (docs +only, no hub bump) · **Live target:** demo box `felhom-pve` guest **9201** · **No STOP point.** --- -## 1. Confirmed baselines — matched §1 exactly on arrival +## 1. Baselines and the five timing constants — all matched §1 on arrival -| Repo | `main` @ arrival | Expected (§1) | Match | +| Repo | `main` @ arrival | Expected | Match | |---|---|---|---| -| `felhom-controller` | `e7c44c0e0f91d21d92b1dafda3979bf382182c1f` | `e7c44c0e0f91` | ✅ | -| `felhom.eu` | `c04ea4f6c2b1554f7d5b4f9fbe7b0066c37b4ea1` | `c04ea4f6c2b1` | ✅ | +| `felhom-controller` | `344660942015da292e5bfdf98c457124b6f633a1` | `344660942015` | ✅ | +| `felhom.eu` | `5c97fbc397f2148ea4d5c98d4de06e3dc8b69a91` | `5c97fbc397f2` | ✅ | -Both trees clean (`git status --porcelain` empty, `HEAD == origin/main`) before the build. +| Constant | Value | Confirmed at | Match | +|---|---|---|---| +| `bootReconcileSettle` | `5 * time.Second` | `cmd/controller/main.go:1264` | ✅ | +| `DefaultAttempts` | `2` | `internal/bootrecon/bootrecon.go:46` | ✅ | +| `DefaultRetryDelay` | `30 * time.Second` | `internal/bootrecon/bootrecon.go:49` | ✅ | +| `deadAppBootGrace` | `90 * time.Second` | `cmd/controller/main.go:1238` | ✅ | +| drive-gate tick | `30 * time.Second` | `internal/web/intermediary.go:339` | ✅ | -## 2. The four §4.9 facts, re-confirmed at source — none had drifted +No drift. Both trees clean before the build. -| Fact | Confirmed at | Result | -|---|---|---| -| `AppConfig` has no desired-run field; `Deployed` means *installed* | `internal/stacks/deploy.go:97-107` — five fields: `Deployed`, `DeployedAt`, `Env`, `LockedFields`, `EmailEnabled` | **Held** | -| `StopStack` runs `compose down` (removes containers) | `internal/stacks/manager.go:1056` → `m.composeExec(dir, "down")` | **Held** | -| `DumpAppVolumesSafe` stops/dumps/starts with no `defer`, no marker, no journal | `internal/backup/backup.go:682-707` — plain statements, no `defer` anywhere | **Held.** Same shape at `internal/backup/offbox_reconstitute.go:279` and `internal/appexport/export.go:232` | -| `aggregateState` already walks every container; any unhealthy/mixed result wins | `internal/stacks/manager.go:~782` | **Held — D-b's every-container requirement was already met and was NOT re-implemented** | +## 2. Part 0's verdict — **CONFIRMED**, and the first attempt was a false negative -**One fact the spec did not list, and it mattered more than any of them:** `SaveAppConfig` -(`deploy.go:742`) rebuilt `AppConfig` from a **five-field struct literal**. Adding a sixth field -without noticing would have dropped `desired_state` on **every** save, across **nine** call sites — -see §12. +**The sweep DOES start an app whose data drive is absent.** Observed on 9201, both attempts: -## 3. Files created / modified +``` +17:22:07 [gate] drive ABSENT /mnt/felhom-drives/hdd_1 — stopped+blocked 2 app(s): [calibre-web immich] +17:22:34 [bootrecon] Boot reconciliation: 1 boot-orphaned app(s) found: [calibre-web] — up to 2 attempt(s) +17:22:34 [stacks] Starting stack: calibre-web +17:22:35 [bootrecon] attempt 1/2: start "calibre-web" failed after 0.4s: exit code 1 +17:23:05 [bootrecon] attempt 2/2: start "calibre-web" failed after 0.4s: exit code 1 +17:23:05 [bootrecon] gave up after 2 attempt(s): recovered=[] still down=[calibre-web] (the dead-app alarm now owns these) +``` -**Created** -- `controller/internal/stacks/desiredstate.go` — constants, `SetDesiredState`, `DesiredStateOf`, `BackfillDesiredState`, `isObservedUp` -- `controller/internal/backup/appstop_marker.go` — `AppStopGuard`, `AppStopMarker`, `AppStopRecovery`, reasons -- `controller/internal/stacks/desiredstate_test.go`, `controller/internal/bootrecon/desiredstate_test.go`, `controller/internal/backup/appstop_marker_test.go`, `controller/internal/api/desiredstate_intent_test.go`, `controller/cmd/controller/appstop_wiring_test.go` +All four preconditions were verified first: drive mounts **0**, containers **0**, +`desired_state: running`, `deployed: true`. -**Modified** -- `controller/internal/stacks/deploy.go` — `DesiredState` field; **`SaveAppConfig` → copy-and-overlay**; `DeployStack` records `running`; `UpdateOptionalConfig`'s redeploy branch records `running` -- `controller/internal/api/router.go` — `desiredStateForAction` + the intent write in `actionStack` -- `controller/internal/bootrecon/bootrecon.go` — `isBootOrphan` rewritten on intent; package doc updated -- `controller/internal/backup/backup.go` — `appStop` field, guard construction, `SetAppStopGuard`/`AppStopGuard`, `DumpAppVolumesSafe` bracketed -- `controller/internal/backup/offbox_reconstitute.go` — `Begin` + a `restartStack` closure covering all four bring-up paths -- `controller/internal/appexport/export.go` — consumer-side `appStopGuard` seam + `SetStopGuard` -- `controller/cmd/controller/main.go` — guard construction + `Recover` before the bootrecon goroutine, backfill, `NotifyBackupFailed` reporting, `exportStopGuard` adapter, `.fab` restore records `running` -- `CHANGELOG.md`, `CONTEXT.md`, `REUSE.md`, `controller/README.md` -- `felhom.eu`: `CONTEXT.md` (S-12), `CLAUDE.md`, `STATUS.md`, `documentation/architecture/00-capability-map.md`, `documentation/architecture/02-controller-module-map.md` (**new §0a — the S-1 contract**), `documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/ROADMAP.md` +**RUN 1 WAS A FALSE NEGATIVE AND IS REPORTED BECAUSE IT PRODUCED A MECHANISM.** Unmounting only the +parent bind is **healed by the agent within ~60 s** (`/dev/sdb` is still mounted at `/mnt/hdd_1`), so +the drive gate's startup reconcile re-attached and restarted the apps at `17:18:47` — **one second +before** the sweep looked at `17:18:48` and logged `no boot-orphaned apps`. Reporting that as a +disproof would have been wrong: it is a race that went the safe way. Run 2 held **both** mounts off +for the duration and reproduced the defect immediately. -## 4. Commits pushed to `main` +**The confirmation moved the severity in both directions.** + +- The **write** hazard did NOT materialise. Compose failed with + `mkdir /mnt/felhom-drives/hdd_1/userdata: permission denied`; `find` showed only the bare directory + and `df -h /` was unchanged at `25G / 28%` before and after. The mechanism: with the drive unbound + the mountpoint is **host-root-owned** (`uid=0 gid=0 mode=755`) and guest 9201 is an **unprivileged** + LXC, so container root (uid 100000) cannot create the subdirectory. **This protection is accidental** + — no code chose it, no test pinned it, and it depends on the guest staying unprivileged and the + mountpoint staying root-owned (when the drive IS bound, that same path shows `uid=100000`). One + `chown` away from gone, and its removal would be invisible until data landed on the wrong disk. +- The harm that **did** occur was not in the hypothesis and is real on every box: two wasted attempts, + a leftover `Created` container per attempt, and a **false dead-app alarm for an app the drive gate + is deliberately holding** — the noise class R-97/F-A1 exist to prevent. + +**A second asymmetry, found while restoring the box:** the API's own start path already refuses this — +`{"ok":false,"error":"A(z) /mnt/felhom-drives/hdd_1 tárhely jelenleg nem elérhető …"}` from +`startGatedByMissingDrive`. The rule existed; the sweep bypassed it by calling `Manager.StartStack` +directly. Full evidence: `felhom.eu/documentation/audits/DIAG-bootrecon-drive-absent-2026-08-02.md`. + +**Consequence:** Part 3 applied and was implemented **first**, before the window widened. + +## 3. Files modified / created; commits + +**Created:** `controller/internal/bootrecon/drivegate_test.go`, +`controller/cmd/controller/bootwindow_test.go`, `controller/internal/web/recreate_intent_test.go`, +`felhom.eu/documentation/audits/DIAG-bootrecon-drive-absent-2026-08-02.md` + +**Modified:** `bootrecon/bootrecon.go` (StartGate seam, held filter, `Result.HeldByDrive`), +`cmd/controller/main.go` (settle window, `sampleBootFleet`, `recordLateRecovery`, `bootDriveGate`, +wiring), `stacks/deploy.go` (`Manager.DriveLive`), `backup/appstop_marker.go` (`HeldStacks`), +`web/intermediary.go` (`shouldRecreateOnBoot` + `bootStack.desired` + counters), +`web/intermediary_test.go`, `cmd/controller/appstop_wiring_test.go`, `CHANGELOG.md`, `CONTEXT.md`, +`REUSE.md`, `controller/README.md`; `felhom.eu`: `CONTEXT.md` (S-13), `STATUS.md`, +`architecture/00-capability-map.md`, `architecture/02-controller-module-map.md`, +`backlog/OPEN-ITEMS.md`, `backlog/ROADMAP.md` | Repo | Commit | Subject | |---|---|---| -| `felhom-controller` | `dbcb306` | v0.189.0 — desired state + the app-stop crash marker (R-166 / D-b) | -| `felhom-controller` | `a8f7c61` | docs: CONTEXT + REPORT for v0.189.0 (R-166) | -| `felhom.eu` | `5c97fbc` | docs: R-166 SHIPPED — the desired/in-flight/observed split (D-b) | +| `felhom-controller` | `582135f` | v0.190.0 — the boot settle window, both gates on intent, and R-171 | +| `felhom-controller` | `dcc3363` | boot window: sample REFRESHES first — a cached fleet made 'settled' meaningless | +| `felhom-controller` | `4bad6e0` | CHANGELOG: the boot window's sample refreshes first | +| `felhom.eu` | *(docs commit, this session)* | docs: R-157 A / R-170 / R-171 closed | -Direct to `main`, no branches, no `Co-Authored-By`. The pre-push hook ran the gate entry point -(`--fast`) and passed; **`--no-verify` was NOT used.** +Direct to `main`, no branches, no `Co-Authored-By`. Pre-push hook green; **`--no-verify` NOT used.** -## 5. Tests and red-proofs +## 4. Tests and red-proofs -**27/27 packages green before and after; +37 test functions across 5 packages.** -Full `go build ./... && go vet ./... && go test ./...` → **rc=0**, run to completion twice. -Test run and commit were **never** combined in one command (standing rule 1). +**27/27 packages green; +25 test functions across 3 packages.** `go build ./... && go vet ./... && +go test ./...` → **rc=0**. Timing is tested by shrinking the window constants — **never by sleeping**. +Test runs and commits were never combined in one command. -Every red-proof below was **observed FAIL, then the mutation was reverted and a control run passed.** +Every red-proof observed **FAIL**, then reverted with a passing control run. -| # | Test | Mutation applied | Result | +| # | Test | Mutation | Result | |---|---|---|---| -| **B** | `TestReconcile_DesiredRunning_ZeroContainers_IsRecovered` + `TestIsBootOrphan_DecisionTable` | restored `len(s.Containers) > 0 && IsDownState(...)` in the `DesiredStateRunning` branch | **FAIL** ✅ | -| **C** | `TestReconcile_LegacyNoDesiredState_BehavesExactlyAsBefore` + `..._AppConfigPresentButFieldAbsent...` | made the `default:` (absent) branch return `len == 0 \|\| IsDownState(...)`, i.e. absent treated as running | **FAIL** ✅ — *"a LEGACY app with no recorded intent and zero containers was started 1 time(s)"* | -| **D** | `TestBackfillDesiredState_RunningIsRecorded_AmbiguousIsLeftAlone` + `..._NeverWritesStopped...` | deleted the `if !isObservedUp(s)` guard | **FAIL** ✅ — *"exited (state=exited) was backfilled to \"running\""* | -| **E** | `TestDumpAppVolumesSafe_MarkerCoversTheWholeStopStartWindow`, `..._Interrupted_RecoveryBringsTheAppBack`, `..._FailedRestartKeepsTheMarker` | deleted the production `m.appStop.Begin(...)` call from `DumpAppVolumesSafe` | **FAIL** ✅ (3 tests) | -| **H** | `TestSaveAppConfig_PreservesEveryKnownFieldAcrossLoadSave` + `TestSetDesiredState_PersistsStoppedToDisk` | restored the field-by-field `saveCfg := AppConfig{...}` literal | **FAIL** ✅ — *"desired_state was LOST across load→save"* | -| **§8.2** | `TestActionStack_RecordsIntentBeforeActing` | moved the intent write **below** the action switch | **FAIL** ✅ | -| **I (seam)** | `TestMainWiresAppStopRecovery` + `TestAppStopRecoveryPrecedesTheBootReconciler` | **commented out** `appStopGuard.Recover()` in `main.go` | **FAIL** ✅ — and `grep -c "appStopGuard.Recover()" main.go` still returned **1**, i.e. a `strings.Contains` test would have passed. The AST walk is what bites. | +| **A** | `TestBootWindow_LateSettlerIsSweptOnASettledFleet` | restored the single-sweep shape (deleted the sampling loop) | **FAIL** ✅ — the sweep was handed the `starting` frame | +| **B** | `TestBootWindow_BudgetEndsAForeverChangingFleet` | removed the `time.Since(started) < budget` condition | **FAIL** ✅ — *"did not terminate on a forever-changing fleet"* (a genuine hang; the fixture was changed to CYCLE first, because a fixture whose frames stop changing terminates by settling and would not have reached the hang) | +| **C** | `TestBootWindow_CustomerStoppedAppSurvivesEveryPass` | dropped the `DesiredStateStopped` branch from `isBootOrphan` | **FAIL** ✅ — *"started 2 time(s) by the widened window"* | +| **D** | `TestShouldRecreateOnBoot_IntentTable` + `_AgreesWithBootrecon` + `RecreateDriveBackedApps_Recorded` | restored `&& hasContainers` | **FAIL** ✅ (3 tests, incl. the cross-gate disagreement assertion) | +| **G** | `TestReconcile_DriveAbsentApp_IsNeverStarted` + `_UndeterminableDrive_` + `_HeldByAnyHolder_` | removed the start-gate check from `Run` | **FAIL** ✅ — *"an app whose data drive is ABSENT was started 1 time(s)"* | +| **H** | `TestMainWiresBootDriveGate` | **commented out** `SetDriveGate(...)` | **FAIL** ✅ — while `grep -c` still returned **1**, i.e. a `strings.Contains` test would have passed | +| **I** | `TestBootWindow_EverySampleRefreshesTheStatus` | removed `RefreshStatus()` from `sampleBootFleet` | **FAIL** ✅ — *"refreshed 0 time(s) for 4 samples"* | -### The hollow test I caught and fixed +**One defect in my own fix, caught by my own test:** `TestBootWindow_CommonCaseFitsInsideTheDeadAppGrace` +rejected the first budget at `5 + 60 + 30 = 95 s > 90 s`. The budget is 50 s **because a test said +so**, and the code comment records that rather than presenting the number as taste. -Scenario E's first version called `m.appStop.Begin(...)` itself and then recovered — and it -**PASSED under red-proof E** while its two siblings failed. It proved the marker type worked, not -that `DumpAppVolumesSafe` uses it. Rewritten to drive the real function with a simulated hard abort -(`GetDockerVolumes` panics, unwinding past the restart *statement* — which is the point, since a -`defer` is not crash-safety). It then failed the same red-proof correctly. +## 5. The window shape, and its justification against the 90 s grace -## 6. The §8.1 table — every row proven by test - -`TestIsBootOrphan_DecisionTable` (10 sub-tests, all green; the whole table fails under red-proof B): - -| `desired_state` | containers | state | Expected | Sub-test | -|---|---|---|---|---| -| `stopped` | 0 | stopped | not an orphan | `stopped/no containers` ✅ | -| `stopped` | 2 | exited | not an orphan | `stopped/down containers` ✅ | -| `stopped` | 2 | running | not an orphan | `stopped/running` ✅ | -| `running` | 0 | — | **ORPHAN** | `running/no containers` ✅ | -| `running` | 2 | exited | **ORPHAN** | `running/down containers` ✅ | -| `running` | 2 | degraded | **ORPHAN** | `running/degraded` ✅ | -| `running` | 2 | running | not an orphan | `running/up` ✅ | -| absent | 0 | — | **not an orphan (legacy)** | `absent/no containers` ✅ | -| absent | 2 | exited | **ORPHAN (legacy)** | `absent/down containers` ✅ | -| absent | 2 | running | not an orphan | `absent/up` ✅ | - -Plus `TestIsBootOrphan_ExistingGuardsSurviveTheRewrite` — `Protected`, `Deploying` and `!Deployed` -still refuse at the strongest desired state, because the rewrite reordered the terms and a reorder is -how a guard silently disappears. - -## 7. Build, deploy, gates - -``` -build.sh 0.189.0 --push → gitea.dooplex.hu/admin/felhom-controller:0.189.0 (145M) -guest 9201: docker pull → /etc/felhom-controller-image → systemctl restart felhom-controller-bootstrap -``` -Verified live: `gitea.dooplex.hu/admin/felhom-controller:0.189.0 | Up … (healthy)`. -`python3 controller/scripts/controller_gates.py` — **all 8 gates OK** (template-id, emoji, -native-confirm, offbox-rename, app-row-dedup, mojibake, docker-v, reuse-refs: 133 cited paths, -126 exact / 6 suffix / 1 cross-repo, 0 failures). - -## 8. Live validation on the demo box — method, and the observables - -**Method: endpoint-level** (`claude-in-chrome` is not available on DooPlex). The **exact request the -UI's „Leállítás"/„Indítás" buttons make** was driven — `layout.html:231`, -`POST /api/stacks//` with `X-CSRF-Token`, session-authed through traefik. No server -logic was skipped; only rendering. **The rendered click-through remains the operator's manual check.** - -Two traps hit, both already in project memory: curl's cookie jar drops `felhom_session` (handled by -hand, one cookie), and the controller is not published on `:8080` — it sits behind traefik on `:443` -with `Host: felhom.demo-felhom.eu`. The credential-bearing helper was deleted afterwards (§14). - -### Flow 0 — the backfill, on four real legacy app.yaml files - -``` -[INFO] [stacks] desired-state backfill: 4 app(s) recorded as running, 0 left unrecorded … -``` -`bookstack`, `immich`, `docmost`, `calibre-web` — all deployed, all observed up, all with **no** -`desired_state` before the upgrade, all converged to `desired_state: running` on disk. - -### Flow 1 — a customer's Stop survives a restart (Scenario A) - -``` -action_response={"ok":true,"message":"Stack calibre-web stop completed"} -[INFO] [stacks] desired state for calibre-web recorded as "stopped" (was "running") ← 16:45:51 -[INFO] [stacks] Stopping stack: calibre-web ← 16:45:51 -``` -The intent line **precedes** the stop line — §8.2 observed live, not only structurally. - -- `app.yaml` on disk: **`desired_state: stopped`** -- `docker ps -a --filter name=calibre` → **empty** (`compose down` removed the containers) -- controller restarted → `[bootrecon] Boot reconciliation: no boot-orphaned apps (nothing to start)` -- `docker ps -a` → **still empty. The app stayed down.** - -### Flow 2 — the power-cut shape is recovered (Scenario B — the point of the release) - -Started from the UI endpoint → `desired_state: running`, `calibre-web Up 8 seconds`. Then the R-157 -shape was created out-of-band: `docker rm -f calibre-web` → `containers_now: 0`, `app.yaml` untouched -at `running`. Controller restarted: - -``` -[INFO] [stacks] desired-state backfill: 0 app(s) recorded as running, 0 left unrecorded … -[INFO] [bootrecon] Boot reconciliation: 1 boot-orphaned app(s) found: [calibre-web] — up to 2 attempt(s) -[INFO] [bootrecon] Boot reconciliation attempt 1/2: started "calibre-web" (took 0.2s) -[INFO] [bootrecon] Boot reconciliation complete: 1 app(s) recovered in 1 attempt(s): [calibre-web] -``` -`calibre-web Up 56 seconds (healthy)`. **Before v0.189.0 this app was invisible to the reconciler.** - -### Flow 3 — a legacy app.yaml behaves exactly as before (Scenario C) - -`desired_state` deleted from `calibre-web/app.yaml` (`desired_state_lines: 0`), containers removed -(`containers: 0`), `deployed: true` kept. Controller restarted: - -``` -[INFO] [stacks] desired-state backfill: 0 app(s) recorded as running, 1 left unrecorded (state ambiguous — legacy boot behaviour retained) -[INFO] [bootrecon] Boot reconciliation: no boot-orphaned apps (nothing to start) -``` -App **not** started; `grep -c "^desired_state" app.yaml` → **0**, i.e. `stopped` was **never -inferred**. Byte-identical to pre-v0.189.0 behaviour, with a positive observable for both halves. - -**Not live-validated, deliberately, and stated rather than rounded up:** the app-stop marker's -recovery leg. It is unit-proven and red-proofed, but nobody killed the controller mid-backup on real -hardware. The capability map records it as **IMPLEMENTED, not PROVEN-LIVE**. - -## 9. Every customer-intent entry point found - -A census of all 14 `StartStack`/`StopStack` call sites was run (`grep -rn --include=*.go`, tests -excluded). **Four are the customer; ten are machines.** - -| # | Intent point | Records | Note | -|---|---|---|---| -| 1 | `internal/api/router.go` `actionStack` | start/restart/update → `running`; stop → `stopped` | the primary one, named in §1.3 | -| 2 | `internal/stacks/deploy.go` `DeployStack` | `running` | named in §1.3 | -| 3 | `internal/stacks/deploy.go` `UpdateOptionalConfig` (redeploy branch only) | `running` | **found, not assumed** — an API endpoint ending in `compose up -d`; the non-deployed branch starts nothing, so it expresses no opinion | -| 4 | `cmd/controller/main.go` `exportAdapter.SaveEncryptedAppConfig` | `running` | **found, not assumed** — the `.fab` import path, which then starts the app (`appexport/restore.go:461`) | - -**Confirmed: no machine-initiated caller writes desired state.** The machine callers — -`quiesce/quiesce.go:494`, `backup/backup.go:688`, `backup/restore.go:55`, `backup/restore_unit.go:252`, -`backup/tier2_restore.go:177`, `backup/offbox_reconstitute.go:279`, `appexport/export.go:232`, -`appexport/restore.go:366`, `stacks/migrate.go:707`, `web/intermediary.go:186,430`, -`web/storage_handlers.go:526`, `bootrecon/bootrecon.go:135` — were each read and none touch it. - -## 10. Marker coverage — all three sites, none left out - -| Site | Covered | How | +| Constant | Value | Why | |---|---|---| -| `backup/backup.go` `DumpAppVolumesSafe` | ✅ | `Begin` before the stop; `End` only on a successful restart; cleared on a **failed stop** (nothing was stopped, so nothing is owed) | -| `backup/offbox_reconstitute.go` | ✅ | `Begin` before the stop; one `restartStack` closure replaces all **four** bring-up paths, so the success path cannot silently skip the clear | -| `appexport/export.go` | ✅ | consumer-side two-method seam; `main.go`'s `exportStopGuard` supplies the reason, so `backup.ReasonAppExport`'s value exists in exactly one place | +| `bootReconcileSample` | 5 s | fine enough to see a container settling at T+40 s within one sample; ~12 cheap calls on a quiet boot | +| `bootReconcileStableFor` | 3 identical samples (15 s) | one sample cannot tell "settled" from "sampled between two docker events"; three spans a container exiting and its restart policy re-creating it | +| `bootReconcileBudget` | 50 s | **binding constraint:** `settle 5 + budget 50 + one DefaultRetryDelay 30 = 85 s` < `deadAppBootGrace 90 s`, so a successful recovery stays SILENT | -**Nothing was left out.** All three share **one** guard over **one** file: `main.go` builds it, -`backup.Manager.SetAppStopGuard` receives it, `Exporter.SetStopGuard` receives it. A second guard -over the same path would be one file with two owners — the shape this marker was kept out of -quiesce's file to avoid. +Terminates on **whichever comes first** — settled, or budget — and the log says which, because +"settled and found nothing" and "ran out of time still churning" are different facts. Sampling is +read-only apart from the status refresh; exactly **one** sweep runs, on a settled fleet, with the +per-app attempt bound untouched. Sweeping per sample was rejected: the sweep's own `StartStack` +changes the fleet, so it would never observe a settled one. -## 11. The operator event path (§2.4) — reachable, used, no new channel +**§8.3 chosen design: allow the window, report a late recovery.** `recordLateRecovery` emits a +`LATE RECOVERY` WARN naming the apps when a recovery lands past the grace. The grace was **not** +widened — that hides a late recovery rather than reporting it. -**Reachable, but not from where `Recover` runs.** `Recover` must complete before the boot-reconcile -goroutine (`main.go:~236`) and the notifier is not constructed until `~307`. Rather than wire a seam -that would fire after the fact — the "built but never wired" shape with four recorded instances here -— `Recover` **returns** an `*AppStopRecovery` and `main.go` reports it once the notifier exists. +## 6. §8.2, row by row -It rides the **existing** `backup_failed` event type (`notifier.NotifyBackupFailed`). A new type -would need the hub's `allowedEventTypes` + `customerMessages` pair changed — a wire change, and this -release ships **no hub change**. The call is guarded by `if appStopRecovery != nil` so a healthy boot -pages nobody; `TestMainReportsTheInterruptedOperation` asserts both the call and the guard. +| Stopped by | Sweep starts it? | How answered | +|---|---|---| +| the customer (`desired_state: stopped`) | **No** | already handled by `isBootOrphan`; **re-proven** — unit (`TestBootWindow_CustomerStoppedAppSurvivesEveryPass`, red-proofed) and live in **6/6** hard resets | +| a quiesce for a backup | **No** | **exclusion WIRED** — `bootDriveGate.MayStart` consults `quiesce.SuppressedStacks()`, the set the quiesce loop already publishes for this exact question | +| the backup app-stop guard | **No** | **exclusion WIRED** — new read-only `AppStopGuard.HeldStacks()`. Its `Recover` completes before this goroutine starts, so a marker seen here belongs to an operation running **now**, not a crashed one | +| the drive-absent gate | **No** | **exclusion WIRED** — Part 3; settings flags (same signal as the API gate) **plus** the live `Manager.DriveLive` mountpoint check, because the flags can lag inside the boot window | +| an interrupted boot | **Yes** | the point of the change | -## 12. Does `app.yaml` round-trip unknown keys? — MEASURED: **no** +All three holders answer **one** seam because they differ only in the reason string. `TestReconcile_HeldByAnyHolder_IsNeverStarted` covers all three; none is answered by "it probably won't happen". -`TestSaveAppConfig_UnknownYAMLKeysAreDropped` writes an `app.yaml` containing -`future_field: keep-me`, loads it, saves it, and reads the file back: **the key is gone.** The -round-trip goes through the struct, so `yaml.Unmarshal` discards unmodelled keys before -`SaveAppConfig` ever sees them. Pre-existing, not widened by R-166, and now stated in -`SaveAppConfig`'s own comment — it is why **every** writer must load-then-save rather than construct. +## 7. The hard-reset log — every run individually -**What WAS fixed is the more dangerous half.** `SaveAppConfig` rebuilt `AppConfig` from a five-field -struct literal — the **R-100 shape**, which v0.181.0 shipped two live instances of. `desired_state` -would have been dropped on every save across **nine** call sites, so a customer's Stop would have -been erased by the next unrelated `app.yaml` write (an email-toggle change, an optional-config edit, -the encryption migration). Replaced with copy-and-overlay (`saveCfg := *cfg`), safe by construction -and pinned by `TestSaveAppConfig_PreservesEveryKnownFieldAcrossLoadSave` + red-proof H. +**Shipped build (v0.190.0 with the refresh fix), the six that count:** -## 13. `OPEN-ITEMS.md` rows opened / closed / re-ranked - -- **R-166 → SHIPPED + PROVEN-LIVE.** Both blocking facts recorded with their answers: (a) the - crash-safe journal pattern **did** already exist (quiesce, migrate) but covered **none** of the - app-data path — `DumpAppVolumesSafe` had no marker, no journal and not even a `defer`; (b) the - SQLite store is reachable and **deliberately unused**, because `metrics.db` is optional by design - and operational state must not live in a store built to be dropped. -- **R-157 → mechanism B CLOSED, mechanism A remains** and is now the whole row (READY, S). A is a - pure timing fix (the sweep observes ~5 s after start and never re-checks). Recorded: **B's fix - makes A cost more**, since the sweep now has more it could legitimately recover. -- **R-170 → NEW.** `shouldRecreateOnBoot` (`internal/web/intermediary.go:131`) still ends in - `&& hasContainers` — R-157 mechanism B, one gate over, for drive-backed apps. **Established free:** - `grep -ro "R-170\b" documentation/ *.md` → **0 hits**. Not acted on (§16). -- `ROADMAP.md` gained matching R-166/R-170 rows and an updated R-157; `STATUS.md`'s "an app can stay - switched off and nothing says so" line was rewritten, not extended. - -## 14. Teardown - -**This task provisioned nothing** — no VM, no guest, no scratch app, no storage. The one app used for -validation (`calibre-web`) already existed and was **restored to its original state**: started -through the same UI endpoint, `desired_state: running` on disk, `calibre-web Up … (health: starting)`, -15 containers running on the box, controller healthy on 0.189.0. The credential-bearing helper script -was deleted from **both** the guest (`/tmp/act.sh` plus its scratch HTML) and DooPlex; `ls` confirms -`No such file or directory`. - -## 15. CI - -| Repo | Run ID | `head_sha` | Conclusion | +| Run | Window verdict | Apps back (immich · calibre-web · docmost) | `bookstack` (desired=stopped) | |---|---|---|---| -| `felhom-controller` | **17** | `dbcb306f` (the code push) | **success** | -| `felhom-controller` | **18** | `a8f7c61d` (CONTEXT + REPORT) | **success** | -| `felhom.eu` | **19** | `5c97fbc3` (the docs push) | **success** | +| F1 | settled after **10 s** | all 3 ✅ | 0 containers ✅ | +| F2 | settled after **40 s** | all 3 ✅ | 0 containers ✅ | +| F3 | settled after **10 s** | all 3 ✅ | 0 containers ✅ | +| F4 | settled after **10 s** | all 3 ✅ | 0 containers ✅ | +| F5 | settled after **15 s** | all 3 ✅ | 0 containers ✅ | +| F6 | settled after **15 s** | all 3 ✅ | 0 containers ✅ | -Checked by pulling the run list and matching `head_sha` to each commit, **not** by trusting the -absence of a failure email — which is exactly the check now added to `felhom.eu/CLAUDE.md`'s -end-of-session list (§N.6). Run 19 was polled until it left `running` rather than read once while -still in flight. +**6/6 for both the recovery and the customer-stop.** An earlier six on the pre-refresh build were +also 6/6 (settle times 50/30/20/50/25/10 s) but are **not** counted as the deliverable, because the +build changed afterwards. -## 16. Observations — noticed, documented, NOT acted on +**The settle times are themselves the evidence for mechanism A:** the window waited 10–50 s where the +old code decided at a fixed 5 s. Two pre-fix runs settled at 50 s — the full budget — i.e. the old +sweep would have decided 45 s too early. -1. **R-170 (filed).** The drive-backed boot gate still infers a Stop from a container count. Left - alone deliberately: the task scoped `bootrecon` and named this gate nowhere, and shipping two boot - behaviour changes under one live validation is one too many. Its consequence is **bounded, not - zero** — `bootrecon` now recovers the app on intent seconds later, so what is actually lost is the - gate's *recreate-against-the-rebound-drive* step, which matters when an app needs recreating - rather than merely starting. -2. **`app.yaml` cannot carry an unmodelled key** (§12). Pre-existing; documented in-code rather than - fixed, since fixing it means an inline catch-all map on a struct holding encrypted secrets. -3. **The `.fab` restore adapter constructs a fresh `AppConfig`** (`main.go`), so it does not benefit - from copy-and-overlay — it is a genuinely new record. Correct today; noted because any future - field added to `AppConfig` must be considered there explicitly. -4. **Part 3 (UI) — N/A: no user-visible string changed.** No template, funcmap, badge, copy or - notifier-message change. The v0.164.0 `StateStopped` suppression is untouched and still correct: - an app the customer stopped is still not an alarm, and it is now not a recovery candidate either — - for a *recorded* reason instead of an inferred one. -5. **`aggregateState` already satisfies D-b's every-container requirement** and was deliberately not - re-implemented. The backfill's `isObservedUp` is an allow-list of up-states rather than - `!IsDownState`, because that negation would call a crash-looping or unreadable stack "up" and seed - a durable record from it. +**The sharpest evidence is a same-app, same-box before/after**, because 6/6 clean runs only show +nothing regressed: + +``` +pre-fix 18:08:35 boot window: fleet settled after 25s — sweeping + 18:08:35 Boot reconciliation: no boot-orphaned apps (nothing to start) ← calibre-web had ZERO containers +post-fix 18:18:50 boot window: fleet settled after 10s — sweeping + 18:18:50 Boot reconciliation: 1 boot-orphaned app(s) found: [calibre-web] + 18:18:52 attempt 1/2: started "calibre-web" (took 1.7s) + 18:18:52 complete: 1 app(s) recovered in 1 attempt(s): [calibre-web] +``` + +That miss is what exposed the cache defect in §4 (red-proof I). + +## 8. Which gate reads what afterwards — both answer intent + +**R-170 proven live in a single reboot**, both halves, on drive-backed apps with **zero containers**: + +``` +18:20:11 [gate] live bind confirmed — recreating drive-backed app calibre-web (state=stopped) onto /mnt/felhom-drives/hdd_1 +18:20:12 [gate] 1 drive-backed app(s) left stopped on purpose (recorded Stop, or a legacy app.yaml at zero containers) +→ calibre-web desired=running containers=1 (recreated — pre-R-170 this was SKIPPED: hasContainers=false) +→ immich desired=stopped containers=0 (left alone) +``` + +The boot sweep then logged `no boot-orphaned apps` — the two gates agreeing rather than fighting. +Structurally, agreement is pinned from **both sides** against one identical fixture table +(`TestBothBootGatesAgreeOnIntent` in `bootrecon`, `TestShouldRecreateOnBoot_AgreesWithBootrecon` in +`web`), because an import cycle prevents calling the two from one package. + +## 9. `OPEN-ITEMS.md` rows + +- **R-157 → CLOSED** (both mechanisms), with the repeat count as evidence, not one clean boot. +- **R-170 → CLOSED**, with the live both-halves reboot. +- **R-171 → NEW and CLOSED the same session**, marked a regression from v0.189.0. **ID established + free before minting:** `grep -ro "R-171\b" documentation/ *.md` → **0 hits**. +- `ROADMAP.md` collapsed to match; `STATUS.md`'s power-cut line moved from "What's broken" to "What + works right now" with the repeat count, plus one dated bullet. + +## 10. Teardown and CI + +**Provisioned nothing** — no VM, no guest, no scratch app, no storage. The apps used for validation +already existed and were **all restored**: `bookstack`, `immich`, `calibre-web`, `docmost` all +`desired=running` and running, 15 containers, controller `0.190.0` healthy. The drive is remounted +(`/dev/sdb` at both `/mnt/hdd_1` and `/mnt/felhom-drives/hdd_1`), and the temporary unmount-holder +loops are gone (`ps aux | grep -c '[u]mount …'` → **0**). The credential-bearing helper was deleted +from the guest (`/root/act.sh`, `/tmp/act.sh`) and from DooPlex. + +CI run number/task id for the final push and its conclusion: recorded in §11 below. + +## 11. Observations — noticed, documented, NOT acted on + +1. **`Manager.StartStack` has no drive gate of its own**, and now has fourteen callers. The API path + guards it (`startGatedByMissingDrive`), the boot sweep now guards it (R-171), and the remaining + machine callers each decide for themselves. Pushing the check *into* `StartStack` was NOT done — + it would break the legitimate callers that must start an app on a drive the gate is mid-way + through re-attaching (the drive gate's own `Return` branch is one). Recorded because the next + person to add a caller needs to know the gate is per-caller. +2. **The agent heals an unmounted drive within ~60 s** while its block device is present. This is + correct behaviour and it is why the Part 0 diagnosis needed the drive held off, but it also means + **"unmount to simulate an absent drive" does not work on this platform** — worth knowing before + the next drive-absence test. +3. **The accidental permission protection** (§2) is untested and unowned. Not pinned by a test here + because a test asserting it would enshrine an accident; the fix removes the dependency on it + instead. If a privileged guest is ever shipped, this is the note to re-read. +4. **`recreateBootStaleApps` runs before the boot sweep and can fix the same app first**, which is + why several runs logged `no boot-orphaned apps` after the gate had already acted. Not a defect — + the gates agree — but it means the sweep's log alone under-reports how much recovery happened.