diff --git a/CONTEXT.md b/CONTEXT.md index a39546c8..03ebbae2 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -16,6 +16,22 @@ > and holds nothing of its own; this file does hold its own content, namely the standing rulings below. +> **2026-10-29 (evening, attended) — RELEASED: hub v0.177.0, agent v0.170.0 (VOUCHED 18:57:48 CET), controller v0.349.0 +> (global floor 19:24 CET, MinAgent 0.158.1). Operator ruling D65 = A (15:50; `09` §3 decision 284, `04` §6.1):** +> `reboot_guest ` joins the AGENT's closed list, no signature (a reboot destroys nothing, `03`). **Built:** (1) the +> agent's `opreboot.go` — `pct reboot` of that guest only; refused for a stopped/locked/parked/unprovisioned guest, a held +> heavy-operation gate (held until the guest is up), `jobBlocker`, a fresh controller busy file (one new read-only sudo +> vector, fail closed but „no file"), a second reboot within the hour (persisted), the restart brake (a reboot counts); +> the watchdogs leave the guest alone meanwhile; the result names both times. (2) the controller's `internal/busymark` +> (`/ctl/busy`, 15 s) and the dashboard line „A Felhom ügyfélszolgálata HH:MM-kor újraindította a dobozt." for a +> day (`GET /guest/operator-reboot`). (3) the hub's button, the 30-minute expiry of an unpicked reboot, the slow-request +> line (> 5 s, path only, `/api/v1/wait` exempt) and the vouch save's per-leg timing. (4) R-1350: the hold clear records +> the stop as wanted. **Proven live on Tester 1:** refused during an off-site run; 67 s / 79 s; refused inside the hour; +> R-1350 no alarm in 6.5 min; on the release builds 66 s / 78 s with every app back and the household line. **R-1346 +> measured:** the vouch save's time is the hub's own „fleet check + save" (38.1 of 38.3 s; registry 135 ms; page 6 ms in +> the cluster); ingress-nginx's default 60-s read timeout makes the 504. **R-1344 is still alarm only** — read the health +> log after 2026-10-30 07:00 UTC (planned events noted in the row). New row R-1351 (FileBrowser boot race, P4). + > **2026-10-29 (afternoon, attended) — RELEASED: hub v0.176.1, agent v0.169.0 (VOUCHED), controller v0.348.0 (global > floor). Operator ruling D64 = B (`09` §3 decision 283, `04` §6.1):** „the hub's „Clear hold" works for every hold reason > with no operator signature; the operator accepted that a compromised hub could clear a data-protecting hold. R-379 diff --git a/REPORT-release-2026-10-29c.md b/REPORT-release-2026-10-29c.md new file mode 100644 index 00000000..5f6b417a --- /dev/null +++ b/REPORT-release-2026-10-29c.md @@ -0,0 +1,66 @@ +# REPORT — release run 2026-10-29 (evening, attended): „Reboot guest" from the hub (D65 = A), R-1350, R-1346 + +The brief was dated Friday 2026-10-30; it started Thursday 15:54 CET. Part 2 (the stuck-controller decision, R-1344) +needs the health log to 2026-10-30 07:00 UTC, so it is NOT in this run — it is the next session's first step. + +## Five lines + +1. **The reboot button works, and its guards refused when they should: yes.** On Tester 1, one press rebooted the guest + (up after 67 s, the controller answered after 79 s); a press during an off-site run was refused („the controller is + busy"); a second press inside the hour was refused. On the release builds: 66 s / 78 s, every app back, and the + household's dashboard said „A Felhom ügyfélszolgálata 20:05-kor újraindította a dobozt." +2. **The stuck-controller decision: not taken yet (alarm only).** The night log ends 2026-10-30 07:00 UTC. The logger + runs (1 730 lines at 18:26 UTC, not read tonight); today's planned restarts and reboots are listed in R-1344 so + tomorrow's reading leaves them out. +3. **R-1350 fixed: yes.** After the operator's hold clear the app stayed stopped and no „app is not running" alarm came + in 6.5 minutes (before: 30 s); the dead-app scan read „0 currently down". +4. **Where the vouch page's time goes: the hub's own database step of the save** — 38.1 s of 38.3 s in „fleet check + + save"; the registry took 135 ms, the page loads in 6 ms (in the cluster) / 65–255 ms (public name). ingress-nginx's + default 60-s read timeout turns a save over 60 s into the 504. Not fixed tonight (it needs one more hub release). +5. **Released:** hub 0.177.0, agent 0.170.0 (vouched), controller 0.349.0 (the global floor since 19:24 CET). + +## Versions, digests, CI by commit + +| Repo | Commit | CI job | What | +|---|---|---|---| +| agent | `ca15a31` | 2520, 2524 green | `reboot_guest`, its guards, the busy-file read, `GET /guest/operator-reboot` — **0.170.0 built from here** (`da8d12a0…`, bundle `03f2ff64…`, tag `v0.170.0`) | +| agent | `6578a8b` | 2525 green | v0.170.0 heading; `pinned.day0_agent` = 0.170.0 | +| controller | `b69e90b` | 2521 green | busymark, the household line, R-1350 — rc.1 `sha256:3bd08409…` | +| controller | `94049a7` | 2526 green | v0.349.0 heading — **0.349.0 built from here** (`sha256:c5341883cd58410ea631fc0c76262d0902a731a24c1096df551e7d35df1ea877`) | +| felhom.eu (hub) | `636022dc` | 2522 green | the button, the 30-min reboot expiry, the slow-request line, the vouch timing — **hub 0.177.0** (`sha256:727eeec76bf728c00f9a7e645ad26fddff31addc5bca9f8465098e264eb90ef9`) | +| felhom.eu | `8cc555be` | 2523 green | manifest → 0.177.0 (Deployment/hub only) | + +The close-out commits' CI is read at the end of the session. + +## Rows + +- **Before:** 229. **After:** 228. **Opened (1):** R-1351 (FileBrowser boot race, P4). **Closed (2):** R-1349 (guest + reboot from the hub), R-1350 (alarm after a hold clear). **Updated:** R-1346 (the numbers), R-1344 (planned events). + +## Observations + +- **FILED: R-1351** — at the release-build reboot two FileBrowser start paths ran `compose up` in the same second; one + logged two `[ERROR]` lines („container name already in use"); FileBrowser then ran healthy. Not seen on the first + reboot an hour earlier. +- **FILED: R-1346** (numbers added) — beyond the vouch: `/apps` 25–30 s, `/customers/tester-1` 7–9 s, `/system` 5.8 s in + the access log of the last days. +- **NOT-A-FINDING: the off-site run's „not deployed — offsite push is unit-only" warnings on Tester 1** — existing, + documented behaviour for apps with no recovery unit; unrelated to this run. +- **NOT-A-FINDING: the agent mutation „no swap guard" survived** — the reboot checked the swap twice (its own check and + `jobBlocker`'s). The duplicate was removed and `jobBlocker`'s check was proven by its own mutant. +- **NOT-A-FINDING: two controller mutants and one hub-side check first „failed" by not compiling** — re-made so they + compile; each then failed its test for the right reason. + +## Teardown + +Machine: Tester 1 — the throwaway Gokapi removed through the product (no container, no app.yaml, no volume); the +controller on 0.349.0 by the floor. 9202 runs rc.1 (not linked to the hub; stays until the next test). Host: nothing +provisioned. Hub: no test customer or host; the per-box floors cleared after the 30-minute watch (clean: three boxes +healthy on 0.349.0, 0 error/held/refused lines). + +## Docs + +`04` §6.1 [DESIGN] D65; `09` §3 decision 284; `03` [DESIGN] the reboot path; `05` [DESIGN] the button, the expiry, the +slow-request line; `08` [DESIGN] R-1350. The system poster: no fact changed (no machine, path, backup tier or time). + +Evidence: `documentation/audits/release-2026-10-29c/`. diff --git a/STATUS.md b/STATUS.md index d1372f82..b4220a54 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-10-29 (afternoon release).** On your yes: hub 0.176.1, agent 0.169.0 (your vouch) and controller 0.348.0 as the floor for every box. From the hub's host page you can now restart a box's controller (it works when the controller is stopped; 3 presses in 15 minutes, then 30 minutes refused) and clear any held app (D64 = B) — both proven on Tester 1 on the release builds. Register 226 → 229 (7 opened, 4 closed); a stuck-but-running controller only alarms you until CC reads tonight's measurement. +**Updated 2026-10-29 (evening release).** On your yes: hub 0.177.0, agent 0.170.0 (your vouch) and controller 0.349.0 for every box; „Reboot guest" on the hub works and refuses during backups, updates and twice in an hour, and no alarm follows your hold clear any more — proven on Tester 1. The slow vouch is the hub's own database step (38 of 38.3 s); the stuck-controller decision waits for tomorrow's night log. ## What works - **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241). @@ -32,7 +32,6 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked | |---|---|---|---|---|---|---| -| D65 | R-1349 | Add „Reboot guest" to the hub's host page, beside „Restart controller"? | **Not now** (restart controller covers a stuck controller; a reboot stops every app of the household for a minute and needs its own guards) | yes: a brief with its guards, about one session to build and prove | nothing changes; a stuck Docker or network still needs a person at the host | — | | D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — | | D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — | | D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — | diff --git a/documentation/architecture/03-host-agent.md b/documentation/architecture/03-host-agent.md index 2dadaad8..25d7783d 100644 --- a/documentation/architecture/03-host-agent.md +++ b/documentation/architecture/03-host-agent.md @@ -214,6 +214,19 @@ by verb**: `unresponsive_since` in the stanza, the hub's operator-only `controller_unresponsive`. Not counted during a swap, a parked or locked guest, or a vzdump. **It does not restart for it** until the R-1344 measurement (every probe, both test boxes, through the 2026-10-29/30 night chain) shows a busy night never reads unhealthy that long. +- **`[DESIGN]` 2026-10-29 — the operator's „Reboot guest" (D65 = A, agent `ca15a31`, released v0.170.0; `04` §6.1, + `09` §3 decision 284).** The agent's list gains `reboot_guest `: the existing guest-reboot path (`pct reboot` + of THAT guest — never the host), for a stuck Docker or a stuck network inside the box. Guards, each refusing with the + reason: provisioned, running, not locked, not parked; the host-wide heavy-operation gate free (held from the checks + until the guest is up); `jobBlocker` (agent backup job, controller swap, drive/NAS job, escrow ceremony, vzdump); the + controller's busy file not fresh (the controller rewrites `data/ctl/busy` every 15 s during a backup, restore, + off-site run, night catch-up, app update or app export/import; older than 90 s = a controller that stopped writing, + not a busy one; any read failure but „no file" refuses); at most one per hour per guest (`// + operator-reboot.json`, persisted); the restart brake (a reboot counts as a restart). The guest-power watchdog and the + supervisor leave the guest alone while it reboots. The result names both times from the press („rebooted; the guest was + up after N s, the controller answered after M s"). The record is written BEFORE the reboot; `GET /guest/operator-reboot` + (self-scoped) hands it to the controller that starts after it, and the household's dashboard says „A Felhom + ügyfélszolgálata HH:MM-kor újraindította a dobozt." for a day. **[RULING 2026-09-16, operator] The brake catches a FAST crash loop and is blind to a SLOW one — a second, slower counter is owed (R-539).** MEASURED the same day on the drill box: the controller was diff --git a/documentation/architecture/04-control-plane-authorization.md b/documentation/architecture/04-control-plane-authorization.md index 79455cee..be702b3f 100644 --- a/documentation/architecture/04-control-plane-authorization.md +++ b/documentation/architecture/04-control-plane-authorization.md @@ -193,7 +193,21 @@ report held now. **The AGENT has its own closed list on the host report's reply (`store.ValidateHostOperatorAction` / agent `localapi.OperatorActionNames`; never a shared verb, `TestHostOperatorActions_ClosedList`): the supervisor's restart with its guards, the 3-in-15-minutes brake counting operator restarts too (then 30 minutes refused), the result in the next report (requested at once), a vmid that is not a guest of that host refused at the hub. -Neither verb deletes data, starts a countdown or shortens one. A guest reboot from the hub is NOT on either list. +Neither verb deletes data, starts a countdown or shortens one. ~~A guest reboot from the hub is NOT on either list.~~ +(Superseded the same day by D65 below.) + +**`[DESIGN]` 2026-10-29 — D65 = A: `reboot_guest ` joins the AGENT's list (operator ruling 15:50; `09` §3 +decision 284).** No signature: a reboot destroys nothing (`03`). The agent's list is now `restart_controller` and +`reboot_guest`; the controller's list does not change and the two never share a verb (`TestHostOperatorActions_ClosedList`, +agent `TestOpActions_AgentClosedList`). The act is `pct reboot` of that guest over the existing exact sudo vector — never +the host. The agent refuses (reason in the result, nothing called) a guest it did not provision, a stopped, locked or +parked guest, a held heavy-operation gate, any agent job (`jobBlocker`: backup, controller swap, drive/NAS job, escrow +ceremony, vzdump), a fresh controller busy file (backup, restore, off-site run, night catch-up, app update, app +export/import — read over ONE new read-only grant, `pct exec -- cat /data/ctl/busy`; +any read failure but „no file" refuses), a second reboot within an hour (persisted), and the restart brake (a reboot +counts as a restart). It holds the heavy-operation gate until the guest is up. The hub expires an unpicked reboot after +30 minutes (a restart keeps the day). The closed list's rule stands: nothing on either list deletes data, starts a +countdown or shortens one. ### 6.2 Who may press the operator's buttons: session, Basic, or a scoped token [hub v0.152.0, `09` §3 decision 251] diff --git a/documentation/architecture/05-hub-architecture.md b/documentation/architecture/05-hub-architecture.md index e8cf4464..36fec2b9 100644 --- a/documentation/architecture/05-hub-architecture.md +++ b/documentation/architecture/05-hub-architecture.md @@ -824,4 +824,12 @@ agent does not report — found live, fixed in 0.176.1); it stores an AGENT row and shows the result under Operator Actions. „Clear hold" sits on each Held apps row; its confirm step names the app, the reason and since when, and for a restore or put-back hold warns that the app may start on half-restored data; the hub refuses an app the box does not report held now. Both are `hosts:write`, CSRF like every form (a token press needs none -and is recorded as `access token ""`), and neither is reachable without the confirm fold-out in the browser. \ No newline at end of file +and is recorded as `access token ""`), and neither is reachable without the confirm fold-out in the browser. + +**`[DESIGN]` 2026-10-29 — „Reboot guest" beside it (hub v0.177.0; D65 = A, `04` §6.1).** One per guest of a customer +host, behind its own confirm step (the whole guest reboots, every app stops for about a minute, when it is refused); it +stores an AGENT row `reboot_guest ` and pokes the box like the restart. **A reboot row the agent did not pick up in +30 minutes expires** (`HostOperatorActionRebootTTL`; a restart keeps the 24 h) — a box that was off must not reboot by +itself when it comes back. The agent's result closes the row with one `operator_action` event. **R-1346:** the access-log +middleware also writes `[WARN] slow request: ` for a request over 5 s (path only; +`/api/v1/wait` exempt), and the artifact vouch save logs each leg's time. diff --git a/documentation/architecture/08-alarm-ladder.md b/documentation/architecture/08-alarm-ladder.md index 59c3d4db..38de6bc3 100644 --- a/documentation/architecture/08-alarm-ladder.md +++ b/documentation/architecture/08-alarm-ladder.md @@ -127,6 +127,11 @@ live 2026-08-23 — one event across 22 scans. - **Operator/hub event + dashboard banner.** `app_start_failed` is **not** in `settings.DefaultEnabledEvents`, so by default it does **not** e-mail the customer. +- **`[DESIGN]` 2026-10-29 — no alarm after the operator's hold clear (R-1350).** The clear starts nothing, so a held app + stays stopped; the clear now records that stop as WANTED (desired state „stopped") for an app that is not up — once the + hold is found, before it is lifted — so the scan's tri-state reads „the household asked" and stays quiet. The + household's Start records „running" and ends it. Measured before the fix on Tester 1: `app_start_failed` 30 s after a + clear from the hub (controller, R-1350). - **F-OBS heartbeat**, every 20 scans (~10 min), at `[INFO]`: `[deadapp] check alive: N scans since boot, M deployed app(s) evaluated, K currently down`. This line exists because an absent alarm and a stopped detector look identical in a log. diff --git a/documentation/architecture/09-update-architecture.md b/documentation/architecture/09-update-architecture.md index ca5cf5a8..a4102460 100644 --- a/documentation/architecture/09-update-architecture.md +++ b/documentation/architecture/09-update-architecture.md @@ -1206,6 +1206,17 @@ Wanderer's Meilisearch went 256 → 384 MB and was measured again (it sizes itse `-tags faultinject` test build — never in the release image (`TestD63_TheReleaseBuildHasNoFaultSwitch`). *Operator, 2026-10-29 08:17.* `07` §6.3. +### 2026-10-29 (15:50, operator) — D65 = A: the hub may reboot a guest + +284. **D65 = A — `reboot_guest` joins the AGENT's closed operator-action list, with no signature.** The operator reboots + one guest from the hub's host page (`pct reboot` of THAT guest, the existing guest-reboot path — never the host), + for a stuck Docker or a stuck network inside the box that „Restart controller" cannot reach. A reboot destroys + nothing (`03`, „healing … non-destructive by construction"). Guards on the agent: no backup of any tier, restore, + update, controller swap or other heavy job (the controller's busy file included); not a locked, stopped or parked + guest; at most one per hour per guest; the restart brake counts it. A press the agent did not pick up in 30 minutes + expires. The household's dashboard says for a day that Felhom support restarted the box. `04` §6.1, `03`. + *Operator, 2026-10-29 15:50.* + ### 2026-10-29 (12:10, operator) — D64 = B: the hub may clear any hold 283. **D64 = B — the hub's „Clear hold" works for every hold reason, with no operator signature.** The operator accepted diff --git a/documentation/audits/release-2026-10-29c/final/gates-felhom-eu.txt b/documentation/audits/release-2026-10-29c/final/gates-felhom-eu.txt new file mode 100644 index 00000000..da22f8ce --- /dev/null +++ b/documentation/audits/release-2026-10-29c/final/gates-felhom-eu.txt @@ -0,0 +1,385 @@ +repo_gates (felhom.eu) — 24 gate(s) [--fast] + --fast SKIPPED (deliberate periodic runs, never in a hook): iso-bootstrap + +============================================================================== +== gate: site (site_gates.py) +============================================================================== +/mnt/5_hdd/felhom.eu/git/felhom.eu/scripts/site_gates.py:769: DeprecationWarning: 'maxsplit' is passed as positional argument + _first = re.split(r"[\s>+~]", _sel, 1)[0] + shell sidebar: 11 en page(s), all identical to en/apps.html + shell sidebar: 15 hu page(s), all identical to 404.html + download pages agree: felhom-installer-1.30.0-pve9.2-1.iso, sha 590d7b9b3ccb… + twins: 11 pairs checked both ways (hreflang, globe, canonical), 0 of them unlisted + apps pages agree: 0 cards, 64 logos + no-HU matcher controls OK (positive 'Jelentkezz be'/'Kapcsolat' convict, negative 'Angola…' does not) + gyik.html: JSON-LD equals the 42 visible questions and answers + en/faq.html: JSON-LD equals the 42 visible questions and answers + dashboard pictures: 24 references, each in its page's language + picture viewer: 390 openers, every one a link to an existing file + app logos: 65 catalog apps, each with its -logo file + shell leak: 759 site rules vs 119 shell rules, 9 shared selector(s), 0 leaking declarations (media-aware) + anchors: 416 in-site #fragment(s), every one resolves + sitemap: 24 URL(s), every public page once, every twin pair cross-named + unlisted: 0 page(s) — noindex, robots Disallow, not in the sitemap, linked from no other page +site gates OK — BOM, emoji=0, nav/footer per language, analytics, no CDN, no legacy tokens, no