chaos-night fixes: rulings recorded, guide reordered, three rows closed, two filed

Architecture: 08 records ruling A (45 m, the round-9 arithmetic, the cost) and the two new event types with their audiences; 03 records the slow counter as built; 07 records the restore-record persistence as a REVERSED design for the restore record only; CONTEXT.md carries the day's rulings. Guide: the recovery code moves after the first apps and waits for the yellow bar. Register: R-549 and R-550 closed PROVEN-LIVE; R-546 closed on red-proofed tests with its live walk owed by R-551 (no Tier-0 box is paused AND agent-connected). R-552 filed: an interrupted-restore notice for a removed app never clears - found in my own v0.246.0 after the release was built. Evidence: Part A (hub prints 45m/1h30m), Part C delivery on HP and N100, B.4(a) live proof and its teardown. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-17 11:00:29 +02:00
parent 06334e164c
commit 3c1882a0a4
11 changed files with 163 additions and 13 deletions
@@ -0,0 +1,19 @@
+# Part A - R-549, operator ruling A: the quiet-box alarm waits three report cycles
+# Deployed 2026-09-17 ~08:22Z. Manifest commit 06334e164c51; hub image 0.117.0.
+
+## ArgoCD, verified by revision and image - not by the rollout message
+  before sync: OutOfSync rev=06334e164c519a9926e868c23e6bda2f58f915cc
+  after:       Synced, health Healthy, rev == manifest HEAD: YES
+  image:       gitea.dooplex.hu/admin/felhom-hub:0.117.0
+  NOTE: `rollout status` printed "successfully rolled out" while the new pod was still 0/1 ready.
+        Readiness was waited for separately (ready=true at 08:22:51Z).
+
+## The live ConfigMap
+  stale_threshold: "45m"
+
+## THE PROOF - the running hub's own startup lines (these fields did not exist before v0.117.0)
+  2026/09/17 10:22:34 [INFO] Staleness checker initialized: 2 ok, 0 stale, 2 down (node_stale after 45m0s, node_down after 1h30m0s)
+  2026/09/17 10:22:34 [INFO] Host staleness checker initialized: 2 ok, 0 stale, 0 down (stale/down left unseeded → first Check emits; host_stale after 45m0s, host_down after 1h30m0s)
+  (log times are the pod's local CEST; 10:22 CEST = 08:22Z)
+
+## Hub serving: https://hub.felhom.eu/healthz -> 200
@@ -0,0 +1,25 @@
+2026-09-17T08:49:35Z === PROOF A: a restore interrupted 2 s in, on 9201, throwaway app homebox ===
+2026-09-17T08:49:35Z login: session=yes csrf_len=64
+tr: write error: Broken pipe
+2026-09-17T08:49:35Z deploy homebox -> 202 {"ok":true,"message":"Telepítés elindítva – az állapot a kártyán követhető"}
+2026-09-17T08:49:46Z homebox container: running
+2026-09-17T08:50:06Z backup run -> 200
+2026-09-17T08:51:51Z backup finished: {"ok":true,"data":{"db_dump":{"count":6,"duration":"1m43.068591229s","last_run":"2026-09-17T08:51:49.156859606Z","success":true},"enabled":true,"running":false}}
+2026-09-17T08:51:51Z login: session=yes csrf_len=64
+2026-09-17T08:51:51Z restore-status BEFORE: {"ok":true,"data":{"running":false,"started_at":"0001-01-01T00:00:00Z"}}
+2026-09-17T08:51:51Z POST /backup/restore homebox -> HTTP/1.1 302 Found
+2026-09-17T08:51:53Z restore-status 2 s in: {"ok":true,"data":{"running":true,"op":"restore","stack":"homebox","started_at":"2026-09-17T08:51:51.445442422Z"}}
+2026-09-17T08:51:53Z KILL: docker kill felhom-controller -> felhom-controller
+2026-09-17T08:51:53Z record on disk right after the kill: {"running":true,"op":"restore","stack":"homebox","started_at":"2026-09-17T08:51:51.445442422Z"}
+2026-09-17T08:52:33Z controller back: state=running after 42 s (NOT restarted by hand)
+2026-09-17T08:52:59Z login: session=yes csrf_len=64
+2026-09-17T08:52:59Z restore-status AFTER: {"ok":true,"data":{"running":false,"started_at":"0001-01-01T00:00:00Z","last":{"op":"restore","stack":"homebox","ok":false,"message":"A visszaállítás megszakadt (a doboz újraindult) — indítsd el újra.","finished_at":"2026-09-17T08:52:33.050939642Z","interrupted":true},"last_recent":true}}
+2026-09-17T08:53:14Z /backups/restore: bytes=68722 card('Megszakadt vissza')=1 msg('megszakadt')=1 NEGATIVE('zzzz-not-present')=0
+    card: <strong>homebox</strong>: A visszaállítás megszakadt (a doboz újraindult) — indítsd el újra.
+2026-09-17T08:53:14Z controller log lines:
+    2026/09/17 08:52:33 restore_record.go:88: [WARN] [backup] restore of homebox (restore, started 2026-09-17T08:51:51Z) was INTERRUPTED by a controller stop — recorded as failed; the household is told to run it again
+    2026/09/17 08:52:33 restore_record_wiring.go:35: [INFO] [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: true)
+    2026/09/17 08:52:33 notifier.go:208: [DEBUG] PushEvent: type=restore_interrupted severity=warning url=https://hub.felhom.eu/api/v1/event
+    2026/09/17 08:52:33 notifier.go:234: [DEBUG] PushEvent: restore_interrupted pushed OK (HTTP 200)
+    2026/09/17 08:52:33 notifier.go:236: [INFO] Event pushed: restore_interrupted (warning) — A(z) homebox visszaállítása megszakadt, mert a doboz újraindult — indítsd el újra.
+2026-09-17T08:53:14Z homebox after: running
@@ -0,0 +1,17 @@
+2026-09-17T08:54:29Z === TEARDOWN A (and the live half of 'the notice lasts until that app's next restore') ===
+2026-09-17T08:54:29Z second restore of homebox -> HTTP/1.1 302 Found
+2026-09-17T08:54:39Z restore-status: {"ok":true,"data":{"running":false,"op":"restore","stack":"homebox","started_at":"2026-09-17T08:54:29.123541604Z","last":{"op":"restore","stack":"homebox","ok":true,"message":"A(z) homebox: 1 adatkötet visszaállítva — az alkalmazás újraindult.","finished_at":"2026-09-17T08:54:38.543283958Z"},"last_recent":true}}
+2026-09-17T08:54:50Z /backups/restore card('Megszakadt vissza')=0  (must be 0 now)
+2026-09-17T08:54:50Z record on disk: {"running":false,"op":"restore","stack":"homebox","started_at":"2026-09-17T08:54:29.123541604Z","last":{"op":"restore","stack":"homebox","ok":true,"message":"A(z) homebox: 1 adatkötet visszaállítva — az alkalmazás újraindult.","finished_at":"2026-09-17T08:54:38.543283958Z"}}
+2026-09-17T08:54:50Z remove homebox (data + backups) -> 409 {"ok":false,"error":"stack \"homebox\" is still running — stop it first before removing"}
+2026-09-17T08:56:51Z homebox containers left: 1  volumes left: 1
+2026-09-17T08:56:51Z standing apps still up: 10 of 10 checked; controller: running
+2026-09-17T08:56:51Z guest secrets/scripts left: 1 (teardownA.sh removes itself next)
+left: 0
+2026-09-17T08:57:13Z === TEARDOWN A, second pass: stop first, then remove (my first pass skipped the stop and was refused 409) ===
+2026-09-17T08:57:13Z stop homebox -> 200 {"ok":true,"message":"Stack homebox stop completed"}
+2026-09-17T08:57:13Z homebox state: 
+2026-09-17T08:57:14Z remove homebox (data + backups) -> 200 {"ok":true,"data":{"removed":"homebox","volumes_removed":[],"hdd_paths_removed":[],"hdd_paths_preserved":[],"hdd_note":"Az alkalmazás nem tárolt saját adatot külső meghajtón, így ott nem volt m
+2026-09-17T08:57:14Z homebox containers left: 0  volumes left: 0  stack dir app.yaml deployed: 0
+2026-09-17T08:57:14Z standing apps: 10 of 10 up; controller running
+guest leftovers: 0
@@ -0,0 +1,20 @@
+# Part C - agent v0.132.0 delivered by operator-signed agent_update jobs (ruling 3 of 2026-09-16)
+# released sha256 4afe815749a41b327ebe4a98a4557ad2acffb1fa71740a3a8b8003473b835321
+signed+uploaded 2026-09-17T08:27:36Z: demo-hp-bb76ea nonce=dfc91f1c9bb7eb789fc99469adba4392; demo-felhom-8363b5 nonce=8e183e6634041822d6b161a6de7d7768; expires 09:12:36Z
+## 2026-09-17T08:30:07Z hp now runs 0.132.0 - waiting 75 s for the dwell/commit
+  [hp] Sep 17 10:29:25 demo-hp felhom-agent[1526161]: time=2026-09-17T10:29:25.418+02:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=dbc1b52384ade807 op=agent_update key_id=felhom-op-1 nonce=dfc91f1c9bb7eb789fc99469adba4392
+  [hp] Sep 17 10:29:25 demo-hp felhom-agent[1526161]: time=2026-09-17T10:29:25.886+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=dbc1b52384ade807 op=agent_update
+  [hp] Sep 17 10:29:29 demo-hp felhom-agent[1195783]: time=2026-09-17T10:29:29.858+02:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s guests_dir=/var/lib/felhom-agent/guests
+  [hp] Sep 17 10:29:29 demo-hp felhom-agent[1195783]: time=2026-09-17T10:29:29.859+02:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.132.0 prev=0.131.0 dwell=1m0s
+  [hp] Sep 17 10:30:29 demo-hp felhom-agent[1195783]: time=2026-09-17T10:30:29.884+02:00 level=WARN msg="selfupdate: update committed" version=0.132.0 wrapper=""
+  [hp] active: active
+  [hp] felhom-agent 0.132.0
+## 2026-09-17T08:35:11Z felhom-pve now runs 0.132.0 - waiting 75 s for the dwell/commit
+  [felhom-pve] Sep 17 10:34:48 demo-felhom felhom-agent[2416121]: time=2026-09-17T10:34:48.018+02:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=640bd623316627c5 op=agent_update key_id=felhom-op-1 nonce=8e183e6634041822d6b161a6de7d7768
+  [felhom-pve] Sep 17 10:34:48 demo-felhom felhom-agent[2416121]: time=2026-09-17T10:34:48.396+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=640bd623316627c5 op=agent_update
+  [felhom-pve] Sep 17 10:34:52 demo-felhom felhom-agent[3550256]: time=2026-09-17T10:34:52.247+02:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.132.0 prev=0.131.0 dwell=1m0s
+  [felhom-pve] Sep 17 10:34:52 demo-felhom felhom-agent[3550256]: time=2026-09-17T10:34:52.247+02:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s guests_dir=/var/lib/felhom-agent/guests
+  [felhom-pve] Sep 17 10:35:52 demo-felhom felhom-agent[3550256]: time=2026-09-17T10:35:52.283+02:00 level=WARN msg="selfupdate: update committed" version=0.132.0 wrapper=""
+  [felhom-pve] active: active
+  [felhom-pve] felhom-agent 0.132.0
+## 2026-09-17T08:36:26Z watcher end: hp=1 n100=1