felhom-controller/REPORT.md

REPORT — controller v0.221.1 (record) + v0.222.0 (R-384, R-383)

Session 2026-08-23, UNATTENDED. Live leg on demo-hp (Tier 0, disposable).

⚠ HALT DECLARED — §4's measurement found a real defect, filed as R-386, NOT fixed

The task's §4 asked for a measurement and named it a halt condition. It reproduced. A single-container app stopped out of band raises no alarm at all. Details in §10 below. Per §13 the fix was NOT attempted here. Live-walk step 4 (Scenario E) was dropped as a consequence — it is item (3) on the task's own drop list. Everything else completed.

---

1. Baselines used, and the hub's four numbers as read

Repo	main at start	Verified
felhom-controller	f7881787f434	matches the task
felhom.eu	1eb64bec5183	task said 4e488321bfd1+; it had moved on
felhom-agent	untouched	—

Hub's own numbers, read live from /configuration (ClusterIP + Basic auth) 2026-08-23:

Field	Value
golden_version	0.221.1
agent_version	0.130.0
min_agent	0.129.0
controller floor (min_controller_version)	0.221.1

The task expected golden/floor 0.220.2; the operator had already vouched 0.221.1 and raised the floor. Live controller on demo-hp at session start: 0.221.1 — so the running version, the golden and the floor all agreed, and only the RECORD disagreed. That is exactly R-385's shape.

2. Architecture documents read

• documentation/architecture/00-capability-map.md — its 2026-08-22 paragraph already NAMED R-384 as an open finding, from the held-app measurement.
• felhom-controller/internal/stacks/manager.go IsDownState + aggregateState + supervisedPolicy
• cmd/controller/main.go classifyRunStates and its three suppressions
• internal/quiesce/suppress.go, internal/bootrecon/bootrecon.go, internal/stacks/desiredstate.go
• felhom.eu/scripts/golden_currency_gate.py (all 133 lines)

§2's conditional applies and is answered: NO document owned the alarm ladder. That absence is reported as a finding, and documentation/architecture/08-alarm-ladder.md now owns it (Part N.4). It is why the ordering defect was legible only by reading one function top to bottom.

3. Part 0 — the record, pushed ALONE

Exact heading written:

## v0.221.1 — the undo-copy prune stopped running because another fix made its guard reachable (2026-08-23, R-361 follow-on)

Commit da75603, pushed alone before anything else. The reasoning was moved verbatim from the v0.221.0 entry (which no longer claims it), not rewritten and not duplicated.

4. Files changed, commits, CI runs

Commit	Contents
da75603	Part 0 — the v0.221.1 heading, alone
5da11c4	v0.222.0 — R-384 + R-383, tests, CHANGELOG

Modified: CHANGELOG.md, internal/stacks/manager.go, internal/stacks/degraded_test.go, internal/backup/offbox_reconstitute.go, cmd/controller/r361_classifier_control_test.go. Added: cmd/controller/r384_dead_db_alarm_test.go, internal/backup/r383_undo_phrase_test.go.

CI runs confirmed BY ID (id and run_number diverge, both printed):

Commit	CI id	run_number	Result
da75603	404	85	success
5da11c4	405	86	success

5. Red-proofs — four planted, FOUR SEEN FAILING

#	Mutation	Layer the guard sits at	Observed failure
1	hoisted block moved back below unhealthy > 0	aggregateState — the ORDERING	aggregateState = "unhealthy", want "degraded" and bookstack state = "unhealthy", want "degraded" (production-path wiring)
2	up narrowed back to running alone	aggregateState — the GUARD	same subtest, plus "starting" and "restarting" — all three survivor shapes convict
3	classifier drops StateDegraded from down	classifyRunStates — the CONSEQUENCE	dead-app banner = [], want exactly one entry for bookstack
4	undoCopyPhrase reverted to the unconditional claim	the phrase builder — where the CLAIM is made	phrase "…mentése megvan: …mariadb.sql" contains "mentése megvan" — it asserts a file that is not on disk; the empty set printed megvan: ., naming a file that never existed

Every mutation was asserted to have applied (the scripts assert the pre-fix text is present before rewriting and print MUTATION APPLIED). None passed first time. Mutations 1 and 2 convict independently, which is what proves the fix genuinely has two halves.

6. Test count

1494 → 1504 top-level test functions (measured by go test ./... -list '.*' on the stashed and unstashed tree, not estimated). Full green gate go build && go vet && go test ./... → exit 0, zero failures, run after Part 2 and again after Part 3.

7. Deployed version, and the golden

gitea.dooplex.hu/admin/felhom-controller:0.222.0 Up 20 seconds (healthy)

Golden BAKED and PUBLISHED: YES — version 0.222.0. GOLDEN_SHA256 = 19f5904f53792684f046ec0bc25426645cb87ad73d5cfc6c03639d9f82706037, upload OK (HTTP 201), round-trip HTTP 206 from the package URL. All acceptance markers counted and recorded. VOUCHING IS THE OPERATOR'S ACT AND WAS NOT DONE HERE.

8. The five IsDownState consumers, walked and named

Consumer	What changes
cmd/controller/main.go:2173 classifyRunStates	THE INTENDED CHANGE. A stack that read unhealthy now reads degraded → down=true → banner + app_start_failed. userStopped tests StateStopped specifically, so the whitelist cannot swallow degraded.
internal/bootrecon/bootrecon.go:213 (DesiredStateRunning)	CHANGES, and toward repair. A half-started stack at boot now reads degraded → an orphan → compose up -d. Previously it read unhealthy → not an orphan → left half-dead. Aligned with the file's own stated intent.
internal/bootrecon/bootrecon.go:216 (legacy DesiredStateUnknown)	Same shape, same direction.
internal/bootrecon/bootrecon.go:288 (recovery check)	CHANGES, and toward truth. A stack that came back with a dead supervised member is no longer counted Recovered; it stays pending and is retried, bounded by r.attempts. It used to be declared recovered while half-dead.
internal/stacks/desiredstate.go:150 isObservedUp	UNAFFECTED — verified, not assumed. It is an allow-list of {running, starting}; neither unhealthy nor degraded was ever in it, so a stack moving between them does not cross the boundary.
internal/quiesce/suppress.go	UNAFFECTED. It does not call IsDownState at all — the suppression is cycle-keyed and state-blind, which is precisely why R-97b's guarantee cannot be weakened by a state change. Pinned by TestR384_QuiesceSuppressionStillHoldsForDegraded.
dashboard state badge	Already handled. R-51 wired degraded through handlers.go:158/169 (counts with stopped) and funcmap.go:255 (filters with stopped). Verified live — the badge rendered (degraded).

9. The live walk

Method: endpoint-level. No browser exists on DooPlex; every read below is either the exact endpoint the UI calls (POST /api/stacks/<name>/<action>, GET /api/stacks, GET /dashboard) or the controller's own log. Guest clock is UTC.

Step 1 — Scenario A: a database dies behind a healthy-looking app ✅

Observable	Result
bookstack-db stopped out of band	05:30:07Z
front end went unhealthy	05:31:21Z — the state that used to swallow the alarm
aggregate state read	degraded while the front end was unhealthy (05:32:03Z)
app_start_failed	fired at 05:30:14Z, 7 s after the stop
new code path visible	manager.go:703: restart-policy of down member "bookstack-db" = "unless-stopped"
banner	„Telepített alkalmazás nem fut: BookStack (degraded)" on both /launcher and /dashboard
edge-triggered, not per-scan	1 event across 22 scans

The heartbeat, old beside new — same 8 apps evaluated:

2026/08/22 21:13:48  [deadapp] check alive: 580 scans since boot, 8 deployed app(s) evaluated, 0 currently down   ← v0.220.2/0.221.1
2026/08/23 05:37:44  [deadapp] check alive:  20 scans since boot, 8 deployed app(s) evaluated, 1 currently down   ← v0.222.0

Step 2 — Scenario B: unhealthy with nothing dead ✅

The database was restarted; the front end stayed unhealthy with nothing down. Aggregate read unhealthy, not degraded. 0 new app_start_failed, and — the positive observable — no restart-policy of down member line at all, meaning the supervised path was not entered. That absence is trustworthy because the same line HAD appeared on this box 10 minutes earlier. Banner cleared; bookstack state=running.

Honest limit: the live window in which docker reported the front end unhealthy with the database up was ~11 s wide, and the controller's cached read was taken at its edge. The three-shape unit test TestR384_UnhealthyWithNothingDeadDoesNotAlarm carries the rest of this case.

Step 3 — Scenario D: a full deploy cycle ✅ 0 alarms

POST /api/stacks/docmost/stop then /start — the exact calls the launcher's buttons make — on a 3-container stack, watched for 5 minutes to settled healthy.

Observable	Result
app_start_failed across the cycle	0
dead-app scans in the window	9
supervised-down path entered	1, at 05:42:34Z (docmost itself momentarily down beside two live members)

Alarms that v0.221.1 would NOT have produced: ZERO. The single degraded reading at 05:42:34Z has running > 0, so v0.221.1's mixed-case branch reaches the identical verdict. No moment in the cycle had a down supervised member with only non-running survivors, which is the only shape where the two versions differ.

Step 4 — Scenario E: the quiesce cycle — DROPPED, and why

Dropped as a direct consequence of the §4 halt (see the banner at the top), and it is item (3) on the task's own drop list. Running it would have meant triggering a real backup cycle on the box after a halt condition had already fired. Covered at unit level instead by TestR384_QuiesceSuppressionStillHoldsForDegraded, which asserts a degraded stack inside the quiesce set produces no banner and Down=false. Not proven live in this session — stated plainly rather than implied.

Step 5 — §4's measurement ✅ (it reproduced — see §10)

Step 6 — Part 1's gate, both directions ✅

Run	Gate	CHANGELOG	Golden	Exit
gate-01	old	v0.221.0	0.221.1	0 — the blindness, on the real history
gate-02	new	v0.221.0	0.221.1	1 — convicted, naming the missing heading and the route
gate-03	new	v0.221.1	0.221.1	0
gate-04	new	absent clone	—	2 — INCONCLUSIVE preserved
gate-05	new	v0.222.0	0.222.0	0 — post-bake

10. §4's answer, in plain words

A single-container app that is stopped out of band raises no alarm at all, and a comment in the code says the opposite.

aggregateState folds StateExited into the stopped counter, so when every member is down it returns StateStopped — StateExited never survives aggregation, which is the path the task suspected and could not find. classifyRunStates then whitelists StateStopped as a deliberate user stop. So the comment at cmd/controller/main.go — "An out-of-band docker compose stop leaves the containers present → StateExited → still alerts, which is correct: out-of-band tampering IS reportable" — is false, and so is the neighbouring I2 claim that a crashing app never comes to rest at stopped.

Measured: privatebin (1 container, unless-stopped) stopped 05:47:35Z. At 05:51:53Z: state=stopped, 9 dead-app scans had run, 0 events, 0 banner lines. Positive control first, per standing rule 3: app_start_failed fired for BookStack at 05:30:14Z on the same box 17 minutes earlier, so the detector was demonstrably alive.

Scoped honestly: a genuine crash under unless-stopped is restarted by Docker and surfaces as restarting → the 5-minute crash-loop path, which does alarm. The silent case is an explicit out-of-band stop of a stack with no surviving member.

Filed as R-386 (OPEN — MEDIUM). Not fixed here, per §12 and §13.

11. Evidence

felhom.eu/documentation/audits/DRILL-r384-dead-db-alarm-2026-08-23/evidence/ — 5 gate runs, 4 red-proof transcripts, 20 live-walk files including two full controller-log windows (1808 and 2139 lines) pulled off the guest. Both log windows were copied off before the app was restarted, per standing rule 5.

12. Teardown, three layers, and the box's end state

1. Guest 9201 / apps — nothing provisioned. bookstack-db restarted and bookstack confirmed healthy; privatebin restarted and healthy; docmost (all 3) healthy. Planted data untouched throughout — no app was rebuilt, redeployed or restored.
2. Bake VM — the drill VM on DooPlex ran the bake and its build guest 9100 is stopped inside it; the qcow2 reverts to the virgin snapshot. No storage was added anywhere, so pvesm status has nothing to compare.
3. Hub-side record — stated explicitly even though there is none. No appliance was registered, no customer created, no config written, no artifact manifest changed. The hub was READ ONLY (GET /configuration, GET /events). Nothing to discard.

End state: demo-hp guest 9201 runs controller 0.222.0, all 8 deployed apps healthy, golden 0.222.0 baked and published but NOT vouched — floor still 0.221.1.

13. Register size

File	Before	After
OPEN-ITEMS.md	327,266 B	328,325 B
CLOSED-ITEMS.md	68,464 B	71,441 B

R-383 and R-384 moved to CLOSED compressed; R-385 (closed) and R-386 (open) filed. OPEN grew by ~1 KB despite two closures because R-386 is a substantial new finding — recorded rather than smoothed over.

14. Observations — noticed, documented, NOT acted on

1. R-329 is live and now matters much more. app_start_failed is pushed with severity warn, which is not in the hub's vocabulary ({info, warning, error, critical}) and coerces silently to info, e-mailing nobody, while the POST still returns 200. Observed again today: PushEvent: type=app_start_failed severity=warn. R-384 makes this event actually fire, so a known-broken severity moved from unreachable to load-bearing. Not in scope; not touched.
2. Two files carry pre-existing gofmt drift — internal/backup/offbox.go and internal/backup/offbox_recovery_cli.go. Confirmed pre-existing by stashing this session's work and re-running gofmt -l. Not touched (§12 forbids nearby refactors).
3. The runbook's golden-bake step is missing pveam update. On the virgin snapshot the template index is stale, so pveam available offers 13.1-2 and downloading it fails with 400 Parameter verification failed. template: no such template. Recorded in the bake evidence README; the runbook itself was not edited.
4. The register's own suggested fix for R-384 was wrong — it proposed a sustained-unhealthy threshold on the crashLoopAfter model. The defect needed no threshold at all, only an ordering. Recorded in the CLOSED entry so the next reader sees that a register remedy is a hypothesis.
5. Deliberately left open, untouched: R-102, R-359, R-361's sibling surfaces.