diff --git a/documentation/architecture/08-alarm-ladder.md b/documentation/architecture/08-alarm-ladder.md index 2b4186e5..ef3e446c 100644 --- a/documentation/architecture/08-alarm-ladder.md +++ b/documentation/architecture/08-alarm-ladder.md @@ -610,7 +610,7 @@ normal use; work it does not explain is the sign of a hacked app. Numbers and me (Samba) has no counter; the push floor is a lower bound, the weekly test is close to the line. 7. **Release:** released 2026-10-24 on the operator's yes (controller v0.342.0, hub v0.169.0; R-1244) — corrected 2026-10-24. -## 10. A held app the operator can see [DESIGN — proposed, R-1291, 2026-10-26; no code] +## 10. A held app the operator can see [DESIGN — built 2026-10-26 (commit only): controller `5bd03ad`, hub `7e57e824`; D53 = operator-only, `09` §3 decision 265; R-1291] **Why.** A held app is stopped on purpose and stays stopped until a person acts — and today nobody is told after the first minute. demo-hp's Kimai sat held for 11 days (2026-10-14 → 10-26) and was found by accident (R-1286). The @@ -625,7 +625,7 @@ anything, and the hub has no held-app state at all. On a household box the same | `restore_mixed` | an off-site restore whose database replay failed | a person (the same) | | `update_failed` | a failed update | the household, by restoring the app from its backup | | `unhealthy_stop` | the crash-loop guard | the household's Start (one more try, `09` §3 decision 28) | -| `put_back_pending` | a restore waiting for its pre-restore copy | nobody — it clears itself when the drive comes back | +| `putback_pending` | a restore waiting for its pre-restore copy | nobody — it clears itself when the drive comes back | **What the controller reports.** The legacy report is frozen (hub rule), so the event channel carries it: `app_held` when a hold is written (stack, reason, since — the hold's own `At`) and `app_hold_cleared` when it goes. @@ -636,7 +636,7 @@ treats it as idempotent. Both types enter `allowedEventTypes` AND `customerMessa **Where the hub shows it.** The host page (and the customer page) get a „Held apps" list: app, reason in one line (the `holdWhy` sentence), since, and who may clear it. Built from the open `app_held` events per customer and stack. -**When it mails.** One operator mail per hold after a grace that follows the reason: `put_back_pending` 6 hours (it +**When it mails.** One operator mail per hold after a grace that follows the reason: `putback_pending` 6 hours (it should have cleared itself — if not, something is stuck); `restore_failed` / `restore_mixed` 1 day (only a person can act); `update_failed` and `unhealthy_stop` 3 days (the household can act first). Then one reminder a week while it lasts, through the dispatcher's existing per-type cooldown (§6.2). A cleared hold sends nothing. @@ -647,5 +647,16 @@ never clears a hold** — it only shows and reminds. **Cost.** One controller release (the two events + the daily re-send, ~½ day with tests) and one hub release (allow-list, the i18n keys, the list on two pages, a reminder checker; ~1 day). No agent change. -**Open pick (STATUS D53):** v1 operator-only (CC's pick), or also mail the household for the two holds it can clear -itself („press Start" / „restore from the backup")? +**Ruled (D53, operator 2026-10-26, `09` §3 decision 265):** v1 is operator-only. Household mails for the two holds it can +clear itself may follow once the operator list has run. + +**As built (2026-10-26, commit only — releases ride an attended run).** The wire reason is the controller's own constant: +`putback_pending` (one word), and the empty legacy reason is sent as `restore_failed`. The controller's reporter is ONE +observer of the hold store, ticked every minute (`internal/holdreport`), never a send inside a hold writer; the set it +last reported is kept in settings, so a hold cleared while the controller was down is still reported cleared. The hub +mirrors the events in `held_apps`, sends `app_held_reminder` (warning, operator-only, per-app cooldown, the subject +names the app) once after the grace and then weekly, records the reminder before it sends it, and does not remind about +a row the box stopped repeating for 50 h (the list still shows it, marked). `alerting.held_app_grace_override` replaces +every grace — a test configuration only. Live proof: `audits/d53-held-apps-2026-10-26/` (a crash-looped throwaway app on +9202 against a test hub with a 3-minute grace: listed on both pages, one operator mail, Start cleared it, the row went, +every household leg `skipped operator_only`). diff --git a/documentation/audits/d53-held-apps-2026-10-26/README.md b/documentation/audits/d53-held-apps-2026-10-26/README.md new file mode 100644 index 00000000..4e2df3c2 --- /dev/null +++ b/documentation/audits/d53-held-apps-2026-10-26/README.md @@ -0,0 +1,62 @@ +# D53 — a held app reaches the operator: build and live proof (2026-10-26 evening) + +**Design:** `architecture/08-alarm-ladder.md` §10. **Ruling:** D53 = operator-only reminders first (`09` §3 decision 265). +**Code (commit only, not released):** controller `5bd03ad` (CI 2377 ✓), hub `7e57e824` (CI 2378 ✓). + +## Red proofs + +- Controller `internal/holdreport`: with an inert `Tick`, all six behaviour tests failed; with the scheduler line + commented out, the wiring test failed. +- Hub: six mutations, each turned its test red — `redproof-hub.txt` (one first attempt was only a compile error; it was + redone with a compiling mutation, the last line). + +## Live proof — method + +No hub deploy is allowed in this run, so the hub side ran as a **test hub process** on DooPlex (`192.168.0.180:18090`), +built from a clean clone at `7e57e824`, scratch database, a throwaway login, its own random global key, operator mail to +`d53-test@felhom.eu` (the catch-all), and `alerting.held_app_grace_override: 3m` (the test configuration). Scratch guest +9202 ran a **local-only** controller image built from `5bd03ad` (`0.346.0-d53test`, `docker save`/`load`, never pushed to +the registry), pointed at the test hub for the test. Pages were read through the test hub's own login (the endpoints +the operator's browser uses); the mail was read in the mailbox (a different channel from the hub log). + +## Live proof — what happened (UTC) + +| Time | What | Evidence | +|---|---|---| +| 17:28 | Both lists empty: „No app is held on this box." / „… on any box." | `live/0-before_*.html` | +| 17:29:52 | Test controller up; „Notifier enabled (hub: …:18090)", job `hold-report` every 1m | `live/9202-controller-log.txt` | +| 17:30:52 | The first tick reported **four old holds 9202 has carried since 2026-09-24** (`chaoscrash`, `chaosoom`, `chaosoomb`, `gokapi`) | `live/test-hub.log` | +| 17:33:35 | Four operator reminders (one per app, per-app cooldown) | mailbox, `live/notification_log.txt` | +| 17:35:41 | Throwaway Radicale installed through the product | — | +| 17:36:18–17:37:48 | Seven `kill -9` of its main process, 15 s apart | — | +| 17:37:52 | The guard stopped it: „crash_loop — 7 in 10m0s; STOPPING it"; hold `unhealthy_stop` | `live/9202-controller-log.txt` | +| 17:38:52 | `app_held` radicale reached the hub (next tick) | `live/test-hub.log` | +| 17:41:34 | One reminder after the 3-minute grace; mail „[Felhom] ⚠️ demo-hp: the app radicale is still held — a person must act" in the inbox at 17:41:35 | mailbox | +| 17:42 | Both pages list 5 held apps, Radicale with its reason, sentence, and who may clear it | `live/1-held_*.html` | +| 17:45:37 | The household's Start (`POST /api/stacks/radicale/start`) lifted the hold | controller log | +| 17:45:52 | `app_hold_cleared` → the Radicale row went from both pages; no reminder after it | `live/2-cleared_*.html`, `live/held_apps_final.txt` | + +**No household mail:** every reminder's customer leg is `skipped operator_only` (`live/notification_log.txt`). + +## Teardown + +- **Machine (9202):** Radicale removed through the product (no container, volume, `app.yaml`, data or subdomain left); + config and image file put back byte-identical to the saved copies; the test's two settings traces (`reported_holds`, the + Radicale stop time) removed while the controller was stopped; the test image removed; controller 0.343.0 healthy, hub off, + exactly as found. The four old holds and an old PrivateBin record (below) were there before and stay. +- **Host (demo-hp):** nothing provisioned; the transfer file removed. +- **Hub:** the real hub was not touched. The test hub process stopped; its database, login, key and binary deleted. The + local test image on DooPlex removed. + +## Observations + +- FILED: R-1298 — 9202 carries leftovers made outside the product: four crash-loop holds since 2026-09-24 for apps that + are not installed, and a PrivateBin `app.yaml` (2026-10-25) with no container, so the product reads it as installed. + Once D53 is released, such orphan holds would be reminded weekly about apps that do not exist. +- NOT-A-FINDING: the test hub's log looked cut mid-line at 18:45:52 local — the `sec` wrapper holds back an unfinished + part of a long-running child's output; the full line was there when the process ended. +- NOT-A-FINDING: my first switch of 9202 used `systemctl start` on the bootstrap oneshot, which was still active, so the + controller was briefly not running (about one minute); `restart` is the right verb. Fixed in the run. + +Secrets: the evidence was scanned for the Resend key, the hub token, the report key and 9202's password — 0 hits; the +test hub's CSRF tokens are redacted. diff --git a/documentation/audits/d53-held-apps-2026-10-26/live/0-before_hosts.html b/documentation/audits/d53-held-apps-2026-10-26/live/0-before_hosts.html new file mode 100644 index 00000000..6c9d74c3 --- /dev/null +++ b/documentation/audits/d53-held-apps-2026-10-26/live/0-before_hosts.html @@ -0,0 +1,126 @@ + + + + + + Hosts — Felhom Hub + + + + + + + +
+
+

Felhom Hub

+ +
+ + +

Hosts

+ + + + +
+

Held apps

+ +
+

No app is held on any box.

+
+ +
+ + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
HostCustomerAgentProxmox / kernelStatusGuestsCPUMemDiskCloudflaredWorst Storage
d53-test-9202demo-hp—unknown
unknown
NO REPORT——————
+
+ + + +
+ + diff --git a/documentation/audits/d53-held-apps-2026-10-26/live/0-before_hosts_d53-test-9202.html b/documentation/audits/d53-held-apps-2026-10-26/live/0-before_hosts_d53-test-9202.html new file mode 100644 index 00000000..ea29de96 --- /dev/null +++ b/documentation/audits/d53-held-apps-2026-10-26/live/0-before_hosts_d53-test-9202.html @@ -0,0 +1,347 @@ + + + + + + d53-test-9202 — Felhom Hub + + + + +
+
+

Felhom Hub

+ +
+ + + ← Hosts + + + +
+
+

d53-test-9202

+ NO REPORT +
+
+
+ Host ID + d53-test-9202 +
+
+ Customer + demo-hp +
+
+ Agent Version + — +
+ + +
+ Enrolled + just now +
+
+ Last Report + waiting for first report +
+ + +
+ Box connection + unknown (the hub restarted recently, or the host has no customer) +
+
+ Desired Generation + 0 +
+ +
+
+ + +
+
+

Waiting for first report.

+

Vitals, guests and storage appear once this host's agent sends a host-report.

+
+
+ + + +
+

Guests

+ +
+

No guests reported on this host.

+
+ +
+ + +
+

Held apps

+ +
+

No app is held on this box.

+
+ +
+ + +
+

Storage Targets

+ +
+

No storage targets reported.

+
+ +
+ + +
+

Capabilities

+ + +
+

No capability report yet (waiting for the first host report).

+
+ +
+ + +
+

Diagnostics — Log Bundles

+

+ Pull-based: the box ships its debug ring on its own next cycle — controller ≤ one report interval (~15 min), + agent ≈ one heartbeat. The pull is recorded in the box's own log (customer-visible). Bundles expire after 72 h. +

+
+
+ + + +
+
+ + + +
+
+ +
+

No log bundles. Use the request buttons above — the box delivers on its next cycle.

+
+ +
+ + +
+

Operator Actions

+ +

+ The box acts on its next report reply (usually seconds, at most one report interval) and answers on the + report after that. Each press runs once. Recorded in the box's own log and in Events. Unanswered after a day: expired. +

+
+
+ + + +
+
+ + + + +
+
+ + + +
+
+ + + + +
+
+ +
+

No operator actions for this box yet.

+
+ + +
+ + +
+

Network

+
+
+ WireGuard + + no peer allocated + +
+
+ +

+ Waiting for the first host report. +

+ +
+ + +
+

Guest network + unknown + +

+ + +

+ This host reported no guest-network state, so it is unknown — not healthy. + The watchdog is default-on; a silent capable agent means it was switched off + (guest_net.disable), or this report predates it on this box. +

+ +
+ + +
+

DR / Backup

+
+
+ DR Recipe + none +
+
+ Key Escrow + none +
+
+ Last full check (off-site) + not reported (no off-site status in the box's report) +
+
+
+ + +
+

Console access

+ +

not vaulted

+

+ No console credential is vaulted for this host. Expected for a byo host — the owner manages root@pam. Otherwise the installer's step 4b did not run; re-run felhom-host-install.sh, or set and vault one per the break-glass runbook §5. +

+ +
+ + + + +
+

Danger zone

+

+ Removing this host deletes its reports, guests, log bundles and WireGuard peer, and + permanently invalidates its API key — a still-running agent would receive 401s. + Re-enrollment requires the Day-0 passphrase flow. +

+ + +
+ + + + + +
+ + diff --git a/documentation/audits/d53-held-apps-2026-10-26/live/1-held_hosts.html b/documentation/audits/d53-held-apps-2026-10-26/live/1-held_hosts.html new file mode 100644 index 00000000..0d61863a --- /dev/null +++ b/documentation/audits/d53-held-apps-2026-10-26/live/1-held_hosts.html @@ -0,0 +1,177 @@ + + + + + + Hosts — Felhom Hub + + + + + + + +
+
+

Felhom Hub

+ +
+ + +

Hosts

+ + + + +
+

Held apps

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
CustomerAppReasonSinceWho may clear it
demo-hpgokapiunhealthy_stop32d agothe household's Start (one more try)
demo-hpchaosoomunhealthy_stop32d agothe household's Start (one more try)
demo-hpchaoscrashunhealthy_stop32d agothe household's Start (one more try)
demo-hpchaosoombunhealthy_stop32d agothe household's Start (one more try)
demo-hpradicaleunhealthy_stop7 min agothe household's Start (one more try)
+ +
+ + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
HostCustomerAgentProxmox / kernelStatusGuestsCPUMemDiskCloudflaredWorst Storage
d53-test-9202Demo HP (scratch guest 9202)—unknown
unknown
NO REPORT——————
+
+ + + +
+ + diff --git a/documentation/audits/d53-held-apps-2026-10-26/live/1-held_hosts_d53-test-9202.html b/documentation/audits/d53-held-apps-2026-10-26/live/1-held_hosts_d53-test-9202.html new file mode 100644 index 00000000..8e34b80c --- /dev/null +++ b/documentation/audits/d53-held-apps-2026-10-26/live/1-held_hosts_d53-test-9202.html @@ -0,0 +1,399 @@ + + + + + + d53-test-9202 — Felhom Hub + + + + +
+
+

Felhom Hub

+ +
+ + + ← Hosts + + + +
+
+

d53-test-9202

+ NO REPORT +
+
+
+ Host ID + d53-test-9202 +
+ +
+ Agent Version + — +
+ + +
+ Enrolled + 16 min ago +
+
+ Last Report + waiting for first report +
+ + +
+ Box connection + not connected since the hub started (2026-10-26 17:28 UTC, 17 min ago) + +
+
+ Desired Generation + 0 +
+ +
+
+ + +
+
+

Waiting for first report.

+

Vitals, guests and storage appear once this host's agent sends a host-report.

+
+
+ + + +
+

Guests

+ +
+

No guests reported on this host.

+
+ +
+ + +
+

Held apps

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
AppReasonSinceWhyWho may clear it
gokapiunhealthy_stop32d agostopped by the box for a crash loop (2026-09-24T10:37:48Z) — the household's Start gives it one more trythe household's Start (one more try) (operator reminded)
chaosoomunhealthy_stop32d agostopped by the box for a oom storm (2026-09-24T11:48:01Z) — the household's Start gives it one more trythe household's Start (one more try) (operator reminded)
chaoscrashunhealthy_stop32d agostopped by the box for a crash loop (2026-09-24T12:16:24Z) — the household's Start gives it one more trythe household's Start (one more try) (operator reminded)
chaosoombunhealthy_stop32d agostopped by the box for a oom storm (2026-09-24T12:39:51Z) — the household's Start gives it one more trythe household's Start (one more try) (operator reminded)
radicaleunhealthy_stop7 min agostopped by the box for a crash loop (2026-10-26T17:37:52Z) — the household's Start gives it one more trythe household's Start (one more try) (operator reminded)
+ +
+ + +
+

Storage Targets

+ +
+

No storage targets reported.

+
+ +
+ + +
+

Capabilities

+ + +
+

No capability report yet (waiting for the first host report).

+
+ +
+ + +
+

Diagnostics — Log Bundles

+

+ Pull-based: the box ships its debug ring on its own next cycle — controller ≤ one report interval (~15 min), + agent ≈ one heartbeat. The pull is recorded in the box's own log (customer-visible). Bundles expire after 72 h. +

+
+
+ + + +
+
+ + + +
+
+ +
+

No log bundles. Use the request buttons above — the box delivers on its next cycle.

+
+ +
+ + +
+

Operator Actions

+ +

+ The box acts on its next report reply (usually seconds, at most one report interval) and answers on the + report after that. Each press runs once. Recorded in the box's own log and in Events. Unanswered after a day: expired. +

+
+
+ + + +
+
+ + + + +
+
+ + + +
+
+ + + + +
+
+ +
+

No operator actions for this box yet.

+
+ + +
+ + +
+

Network

+
+
+ WireGuard + + no peer allocated + +
+
+ +

+ Waiting for the first host report. +

+ +
+ + +
+

Guest network + unknown + +

+ + +

+ This host reported no guest-network state, so it is unknown — not healthy. + The watchdog is default-on; a silent capable agent means it was switched off + (guest_net.disable), or this report predates it on this box. +

+ +
+ + +
+

DR / Backup

+
+
+ DR Recipe + none +
+
+ Key Escrow + none +
+
+ Last full check (off-site) + not reported (no off-site status in the box's report) +
+
+
+ + +
+

Console access

+ +

not vaulted

+

+ No console credential is vaulted for this host. Expected for a byo host — the owner manages root@pam. Otherwise the installer's step 4b did not run; re-run felhom-host-install.sh, or set and vault one per the break-glass runbook §5. +

+ +
+ + + + +
+

Danger zone

+

+ Removing this host deletes its reports, guests, log bundles and WireGuard peer, and + permanently invalidates its API key — a still-running agent would receive 401s. + Re-enrollment requires the Day-0 passphrase flow. +

+ + +
+ + + + + +
+ + diff --git a/documentation/audits/d53-held-apps-2026-10-26/live/2-cleared_hosts.html b/documentation/audits/d53-held-apps-2026-10-26/live/2-cleared_hosts.html new file mode 100644 index 00000000..5f377976 --- /dev/null +++ b/documentation/audits/d53-held-apps-2026-10-26/live/2-cleared_hosts.html @@ -0,0 +1,169 @@ + + + + + + Hosts — Felhom Hub + + + + + + + +
+
+

Felhom Hub

+ +
+ + +

Hosts

+ + + + +
+

Held apps

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
CustomerAppReasonSinceWho may clear it
demo-hpgokapiunhealthy_stop32d agothe household's Start (one more try)
demo-hpchaosoomunhealthy_stop32d agothe household's Start (one more try)
demo-hpchaoscrashunhealthy_stop32d agothe household's Start (one more try)
demo-hpchaosoombunhealthy_stop32d agothe household's Start (one more try)
+ +
+ + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
HostCustomerAgentProxmox / kernelStatusGuestsCPUMemDiskCloudflaredWorst Storage
d53-test-9202Demo HP (scratch guest 9202)—unknown
unknown
NO REPORT——————
+
+ + + +
+ + diff --git a/documentation/audits/d53-held-apps-2026-10-26/live/2-cleared_hosts_d53-test-9202.html b/documentation/audits/d53-held-apps-2026-10-26/live/2-cleared_hosts_d53-test-9202.html new file mode 100644 index 00000000..31334f94 --- /dev/null +++ b/documentation/audits/d53-held-apps-2026-10-26/live/2-cleared_hosts_d53-test-9202.html @@ -0,0 +1,391 @@ + + + + + + d53-test-9202 — Felhom Hub + + + + +
+
+

Felhom Hub

+ +
+ + + ← Hosts + + + +
+
+

d53-test-9202

+ NO REPORT +
+
+
+ Host ID + d53-test-9202 +
+ +
+ Agent Version + — +
+ + +
+ Enrolled + 22 min ago +
+
+ Last Report + waiting for first report +
+ + +
+ Box connection + not connected since the hub started (2026-10-26 17:28 UTC, 23 min ago) + +
+
+ Desired Generation + 0 +
+ +
+
+ + +
+
+

Waiting for first report.

+

Vitals, guests and storage appear once this host's agent sends a host-report.

+
+
+ + + +
+

Guests

+ +
+

No guests reported on this host.

+
+ +
+ + +
+

Held apps

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
AppReasonSinceWhyWho may clear it
gokapiunhealthy_stop32d agostopped by the box for a crash loop (2026-09-24T10:37:48Z) — the household's Start gives it one more trythe household's Start (one more try) (operator reminded)
chaosoomunhealthy_stop32d agostopped by the box for a oom storm (2026-09-24T11:48:01Z) — the household's Start gives it one more trythe household's Start (one more try) (operator reminded)
chaoscrashunhealthy_stop32d agostopped by the box for a crash loop (2026-09-24T12:16:24Z) — the household's Start gives it one more trythe household's Start (one more try) (operator reminded)
chaosoombunhealthy_stop32d agostopped by the box for a oom storm (2026-09-24T12:39:51Z) — the household's Start gives it one more trythe household's Start (one more try) (operator reminded)
+ +
+ + +
+

Storage Targets

+ +
+

No storage targets reported.

+
+ +
+ + +
+

Capabilities

+ + +
+

No capability report yet (waiting for the first host report).

+
+ +
+ + +
+

Diagnostics — Log Bundles

+

+ Pull-based: the box ships its debug ring on its own next cycle — controller ≤ one report interval (~15 min), + agent ≈ one heartbeat. The pull is recorded in the box's own log (customer-visible). Bundles expire after 72 h. +

+
+
+ + + +
+
+ + + +
+
+ +
+

No log bundles. Use the request buttons above — the box delivers on its next cycle.

+
+ +
+ + +
+

Operator Actions

+ +

+ The box acts on its next report reply (usually seconds, at most one report interval) and answers on the + report after that. Each press runs once. Recorded in the box's own log and in Events. Unanswered after a day: expired. +

+
+
+ + + +
+
+ + + + +
+
+ + + +
+
+ + + + +
+
+ +
+

No operator actions for this box yet.

+
+ + +
+ + +
+

Network

+
+
+ WireGuard + + no peer allocated + +
+
+ +

+ Waiting for the first host report. +

+ +
+ + +
+

Guest network + unknown + +

+ + +

+ This host reported no guest-network state, so it is unknown — not healthy. + The watchdog is default-on; a silent capable agent means it was switched off + (guest_net.disable), or this report predates it on this box. +

+ +
+ + +
+

DR / Backup

+
+
+ DR Recipe + none +
+
+ Key Escrow + none +
+
+ Last full check (off-site) + not reported (no off-site status in the box's report) +
+
+
+ + +
+

Console access

+ +

not vaulted

+

+ No console credential is vaulted for this host. Expected for a byo host — the owner manages root@pam. Otherwise the installer's step 4b did not run; re-run felhom-host-install.sh, or set and vault one per the break-glass runbook §5. +

+ +
+ + + + +
+

Danger zone

+

+ Removing this host deletes its reports, guests, log bundles and WireGuard peer, and + permanently invalidates its API key — a still-running agent would receive 401s. + Re-enrollment requires the Day-0 passphrase flow. +

+ + +
+ + + + + +
+ + diff --git a/documentation/audits/d53-held-apps-2026-10-26/live/9202-controller-log.txt b/documentation/audits/d53-held-apps-2026-10-26/live/9202-controller-log.txt new file mode 100644 index 00000000..f1f56b72 --- /dev/null +++ b/documentation/audits/d53-held-apps-2026-10-26/live/9202-controller-log.txt @@ -0,0 +1,253 @@ +2026/10/26 17:29:52 notifier.go:81: [INFO] Notifier enabled (hub: http://192.168.0.180:18090) +2026/10/26 17:29:52 scheduler.go:120: [INFO] [scheduler] Registered periodic job: hold-report (every 1m0s) +2026/10/26 17:29:52 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="hold-report" interval=1m0s totalJobs=8 +2026/10/26 17:29:52 notifier.go:1145: [DEBUG] SyncPreferences: url=http://192.168.0.180:18090/api/v1/preferences source=default email= events=[backup_failed db_dump_failed disk_warning disk_critical storage_disconnected node_down health_critical expected_backup_missed expected_dbdump_missed offbox_enlarge_blocked app_update_undone app_update_held app_stopped_unhealthy storage_reconnected backup_integrity_failed crossdrive_failed health_recovered] cooldown=6h +2026/10/26 17:29:52 notifier.go:1176: [INFO] Notification preferences synced to hub: source=default status=ok email=, events=[backup_failed db_dump_failed disk_warning disk_critical storage_disconnected node_down health_critical expected_backup_missed expected_dbdump_missed offbox_enlarge_blocked app_update_undone app_update_held app_stopped_unhealthy storage_reconnected backup_integrity_failed crossdrive_failed health_recovered], cooldown=6h, email_cleared=false +2026/10/26 17:29:52 sync.go:559: [DEBUG] [sync] radicale/docker-compose.yml: hash match, skipped +2026/10/26 17:29:52 sync.go:559: [DEBUG] [sync] radicale/.felhom.yml: hash match, skipped +2026/10/26 17:30:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:30:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:30:52 holdreport.go:137: [INFO] [holdreport] chaoscrash held (unhealthy_stop since 2026-09-24T12:16:24Z) — reported to the hub +2026/10/26 17:30:52 holdreport.go:137: [INFO] [holdreport] chaosoom held (unhealthy_stop since 2026-09-24T11:48:01Z) — reported to the hub +2026/10/26 17:30:52 holdreport.go:137: [INFO] [holdreport] chaosoomb held (unhealthy_stop since 2026-09-24T12:39:51Z) — reported to the hub +2026/10/26 17:30:52 holdreport.go:137: [INFO] [holdreport] gokapi held (unhealthy_stop since 2026-09-24T10:37:48Z) — reported to the hub +2026/10/26 17:30:52 holdreport.go:137: [DEBUG] [holdreport] full re-send: 4 hold(s) sent, 0 cleared +2026/10/26 17:30:52 notifier.go:317: [DEBUG] PushEvent: type=app_held severity=info url=http://192.168.0.180:18090/api/v1/event +2026/10/26 17:30:52 notifier.go:317: [DEBUG] PushEvent: type=app_held severity=info url=http://192.168.0.180:18090/api/v1/event +2026/10/26 17:30:52 notifier.go:317: [DEBUG] PushEvent: type=app_held severity=info url=http://192.168.0.180:18090/api/v1/event +2026/10/26 17:30:52 notifier.go:317: [DEBUG] PushEvent: type=app_held severity=info url=http://192.168.0.180:18090/api/v1/event +2026/10/26 17:30:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 1ms) +2026/10/26 17:30:52 notifier.go:343: [DEBUG] PushEvent: app_held pushed OK (HTTP 200) +2026/10/26 17:30:52 notifier.go:355: [INFO] Event pushed: app_held (info) [hu-only] — App chaosoom is held: stopped by the box for a oom storm (2026-09-24T11:48:01Z) — the household's Start gives it one more try +2026/10/26 17:30:52 notifier.go:343: [DEBUG] PushEvent: app_held pushed OK (HTTP 200) +2026/10/26 17:30:52 notifier.go:355: [INFO] Event pushed: app_held (info) [hu-only] — App gokapi is held: stopped by the box for a crash loop (2026-09-24T10:37:48Z) — the household's Start gives it one more try +2026/10/26 17:30:52 notifier.go:343: [DEBUG] PushEvent: app_held pushed OK (HTTP 200) +2026/10/26 17:30:52 notifier.go:355: [INFO] Event pushed: app_held (info) [hu-only] — App chaoscrash is held: stopped by the box for a crash loop (2026-09-24T12:16:24Z) — the household's Start gives it one more try +2026/10/26 17:30:52 notifier.go:343: [DEBUG] PushEvent: app_held pushed OK (HTTP 200) +2026/10/26 17:30:52 notifier.go:355: [INFO] Event pushed: app_held (info) [hu-only] — App chaosoomb is held: stopped by the box for a oom storm (2026-09-24T12:39:51Z) — the household's Start gives it one more try +2026/10/26 17:31:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:31:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:31:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/26 17:32:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:32:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:32:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/26 17:33:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:33:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:33:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/26 17:34:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:34:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:34:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/26 17:35:41 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/stacks/radicale +2026/10/26 17:35:41 router.go:78: [DEBUG] [api] GET /api/stacks/radicale (path=/stacks/radicale) +2026/10/26 17:35:41 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/stacks/radicale/deploy-fields +2026/10/26 17:35:41 router.go:78: [DEBUG] [api] GET /api/stacks/radicale/deploy-fields (path=/stacks/radicale/deploy-fields) +2026/10/26 17:35:41 auth.go:144: [DEBUG] [web] auth: valid session for POST /api/stacks/radicale/deploy +2026/10/26 17:35:41 router.go:78: [DEBUG] [api] POST /api/stacks/radicale/deploy (path=/stacks/radicale/deploy) +2026/10/26 17:35:41 router.go:442: [INFO] [api] Deploy requested for stack: radicale +2026/10/26 17:35:41 router.go:78: [DEBUG] [api] deployStack: name=radicale contentLength=145 +2026/10/26 17:35:41 deploy.go:338: [DEBUG] Deploy radicale: received 4 user values +2026/10/26 17:35:41 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 1 encrypted, 1 sensitive fields +2026/10/26 17:35:41 [INFO] [stacks] SaveAppConfig: saved config for radicale +2026/10/26 17:35:41 deploy.go:574: [INFO] [stacks] Deploying stack radicale with 4 env vars: [SUBDOMAIN, RADICALE_USER, RADICALE_PASSWORD, DOMAIN] +2026/10/26 17:35:41 manager.go:1856: [INFO] [stacks] Deploying stack radicale — checking 1 images... +2026/10/26 17:35:41 manager.go:1860: [DEBUG] ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c — not found locally, will pull +2026/10/26 17:35:41 lifecycle.go:70: [DEBUG] [integrations] OnStackStart: stack=radicale, waiting 5s for state refresh +2026/10/26 17:35:41 manager.go:1701: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/radicale) +2026/10/26 17:35:46 lifecycle.go:86: [DEBUG] [integrations] OnStackStart: stack=radicale integrationsFound=0 +2026/10/26 17:35:46 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/stacks/radicale +2026/10/26 17:35:46 router.go:78: [DEBUG] [api] GET /api/stacks/radicale (path=/stacks/radicale) +2026/10/26 17:35:50 deploy.go:664: [INFO] [stacks] Stack radicale deployed successfully (took 9.0s) +2026/10/26 17:35:50 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 1 encrypted, 1 sensitive fields +2026/10/26 17:35:50 [INFO] [stacks] SaveAppConfig: saved config for radicale +2026/10/26 17:35:50 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 0 encrypted, 1 sensitive fields +2026/10/26 17:35:50 [INFO] [stacks] SaveAppConfig: saved config for radicale +2026/10/26 17:35:50 installed.go:420: [INFO] [stacks] installed-images radicale: recorded 1 service(s) (radicale=ghcr.io/kozea/radicale:3.8.1 (sha256:54d9406cd30f…)) +2026/10/26 17:35:50 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 0 encrypted, 1 sensitive fields +2026/10/26 17:35:50 [INFO] [stacks] SaveAppConfig: saved config for radicale +2026/10/26 17:35:50 pin.go:93: [INFO] [stacks] pin radicale: radicale=ghcr.io/kozea/radicale:3.8.1 +2026/10/26 17:35:50 [DEBUG] [backup] radicale: off-site default not applied: no off-site target +2026/10/26 17:35:51 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/stacks/radicale +2026/10/26 17:35:51 router.go:78: [DEBUG] [api] GET /api/stacks/radicale (path=/stacks/radicale) +2026/10/26 17:35:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:35:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:35:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/26 17:35:53 manager.go:1701: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/radicale) +2026/10/26 17:35:53 manager.go:1817: [INFO] [stacks] Stack radicale post-start status: +2026/10/26 17:35:53 manager.go:1820: [INFO] [stacks] radicale ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c running Up 3 seconds (health: starting) +2026/10/26 17:35:56 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/stacks/radicale +2026/10/26 17:35:56 router.go:78: [DEBUG] [api] GET /api/stacks/radicale (path=/stacks/radicale) +2026/10/26 17:36:01 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/stacks/radicale +2026/10/26 17:36:01 router.go:78: [DEBUG] [api] GET /api/stacks/radicale (path=/stacks/radicale) +2026/10/26 17:36:06 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/stacks/radicale +2026/10/26 17:36:06 router.go:78: [DEBUG] [api] GET /api/stacks/radicale (path=/stacks/radicale) +2026/10/26 17:36:12 healthprobe.go:187: [DEBUG] Health probe radicale: API GET :5232/.web/ → 200 (4ms) +2026/10/26 17:36:22 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 10s ago, effective interval 5m0s, healthy=true +2026/10/26 17:36:42 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 30s ago, effective interval 5m0s, healthy=true +2026/10/26 17:36:52 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 40s ago, effective interval 5m0s, healthy=true +2026/10/26 17:36:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:36:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:36:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/26 17:37:02 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 50s ago, effective interval 5m0s, healthy=true +2026/10/26 17:37:12 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/10/26 17:37:32 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/10/26 17:37:42 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/10/26 17:37:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:37:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:37:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/26 17:37:52 main.go:4311: [WARN] [deadapp] radicale: crash_loop — 7 in 10m0s; STOPPING it (decision 28) +2026/10/26 17:37:52 manager.go:1443: [DEBUG] [stacks] StopStack radicale: current state=starting deployed=true containers=1 +2026/10/26 17:37:52 manager.go:1446: [INFO] [stacks] Stopping stack: radicale +2026/10/26 17:37:52 manager.go:1701: [DEBUG] Running: docker compose down (in /opt/docker/stacks/radicale) +2026/10/26 17:37:52 manager.go:1455: [INFO] [stacks] Stack radicale stopped successfully (took 0.3s) +2026/10/26 17:37:52 settings.go:2043: [WARN] [settings] restore hold SET for radicale — the app stays stopped until it is cleared +2026/10/26 17:37:52 update_guard.go:873: [WARN] [backup] radicale is STOPPED by the box: crash_loop (trip 1 within 24h0m0s) — Start gives it one more try (decision 28) +2026/10/26 17:37:52 notifier.go:317: [DEBUG] PushEvent: type=app_stopped_unhealthy severity=warning url=http://192.168.0.180:18090/api/v1/event +2026/10/26 17:37:52 notifier.go:343: [DEBUG] PushEvent: app_stopped_unhealthy pushed OK (HTTP 200) +2026/10/26 17:37:52 notifier.go:355: [INFO] Event pushed: app_stopped_unhealthy (warning) [hu-only] — A(z) radicale alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod. +2026/10/26 17:38:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:38:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:38:52 holdreport.go:137: [INFO] [holdreport] radicale held (unhealthy_stop since 2026-10-26T17:37:52Z) — reported to the hub +2026/10/26 17:38:52 notifier.go:317: [DEBUG] PushEvent: type=app_held severity=info url=http://192.168.0.180:18090/api/v1/event +2026/10/26 17:38:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 1ms) +2026/10/26 17:38:52 notifier.go:343: [DEBUG] PushEvent: app_held pushed OK (HTTP 200) +2026/10/26 17:38:52 notifier.go:355: [INFO] Event pushed: app_held (info) [hu-only] — App radicale is held: stopped by the box for a crash loop (2026-10-26T17:37:52Z) — the household's Start gives it one more try +2026/10/26 17:39:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:39:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:39:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/26 17:39:52 backup.go:540: [DEBUG] groupStacksByDrive: /mnt/sys_drive → [privatebin, radicale] +2026/10/26 17:40:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:40:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:40:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/26 17:41:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:41:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:41:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/26 17:42:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:42:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:42:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/26 17:43:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:43:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:43:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/26 17:44:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:44:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:44:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/26 17:44:52 backup.go:540: [DEBUG] groupStacksByDrive: /mnt/sys_drive → [privatebin, radicale] +2026/10/26 17:44:52 sync.go:559: [DEBUG] [sync] radicale/docker-compose.yml: hash match, skipped +2026/10/26 17:44:52 sync.go:559: [DEBUG] [sync] radicale/.felhom.yml: hash match, skipped +2026/10/26 17:45:37 auth.go:144: [DEBUG] [web] auth: valid session for POST /api/stacks/radicale/start +2026/10/26 17:45:37 router.go:78: [DEBUG] [api] POST /api/stacks/radicale/start (path=/stacks/radicale/start) +2026/10/26 17:45:37 router.go:663: [INFO] [api] start requested for stack: radicale +2026/10/26 17:45:37 router.go:78: [DEBUG] [api] actionStack: action=start name=radicale +2026/10/26 17:45:37 settings.go:2960: [INFO] [settings] unhealthy-stop hold LIFTED for radicale (Start) +2026/10/26 17:45:37 router.go:687: [INFO] [api] radicale: the unhealthy stop is LIFTED by Start — one more try (decision 28) +2026/10/26 17:45:37 manager.go:1354: [DEBUG] [stacks] StartStack radicale: current state=stopped deployed=true +2026/10/26 17:45:37 manager.go:1357: [INFO] [stacks] Starting stack: radicale +2026/10/26 17:45:37 manager.go:1364: [DEBUG] [stacks] StartStack radicale: prepared 10 env vars for compose +2026/10/26 17:45:37 manager.go:1701: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/radicale) +2026/10/26 17:45:37 manager.go:1372: [INFO] [stacks] Stack radicale started successfully (took 0.4s) +2026/10/26 17:45:38 installed.go:409: [DEBUG] [stacks] installed-images radicale: unchanged (1 service(s)) — app.yaml not rewritten +2026/10/26 17:45:38 lifecycle.go:70: [DEBUG] [integrations] OnStackStart: stack=radicale, waiting 5s for state refresh +2026/10/26 17:45:41 manager.go:1701: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/radicale) +2026/10/26 17:45:41 manager.go:1817: [INFO] [stacks] Stack radicale post-start status: +2026/10/26 17:45:41 manager.go:1820: [INFO] [stacks] radicale ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c running Up 3 seconds (health: starting) +2026/10/26 17:45:43 lifecycle.go:86: [DEBUG] [integrations] OnStackStart: stack=radicale integrationsFound=0 +2026/10/26 17:45:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:45:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:45:52 holdreport.go:137: [INFO] [holdreport] radicale hold cleared — reported to the hub +2026/10/26 17:45:52 notifier.go:317: [DEBUG] PushEvent: type=app_hold_cleared severity=info url=http://192.168.0.180:18090/api/v1/event +2026/10/26 17:45:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 1ms) +2026/10/26 17:45:52 notifier.go:343: [DEBUG] PushEvent: app_hold_cleared pushed OK (HTTP 200) +2026/10/26 17:45:52 notifier.go:355: [INFO] Event pushed: app_hold_cleared (info) [hu-only] — App radicale is no longer held +2026/10/26 17:45:52 healthprobe.go:187: [DEBUG] Health probe radicale: API GET :5232/.web/ → 200 (3ms) +2026/10/26 17:46:02 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 10s ago, effective interval 5m0s, healthy=true +2026/10/26 17:46:12 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 20s ago, effective interval 5m0s, healthy=true +2026/10/26 17:46:22 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 30s ago, effective interval 5m0s, healthy=true +2026/10/26 17:46:32 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 40s ago, effective interval 5m0s, healthy=true +2026/10/26 17:46:42 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 50s ago, effective interval 5m0s, healthy=true +2026/10/26 17:46:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:46:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:46:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/26 17:46:52 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/10/26 17:47:02 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/10/26 17:47:12 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/10/26 17:47:22 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/10/26 17:47:32 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/10/26 17:47:42 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/10/26 17:47:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:47:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:47:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/26 17:47:52 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/10/26 17:48:02 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/10/26 17:48:12 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/10/26 17:48:22 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/10/26 17:48:32 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/10/26 17:48:42 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/10/26 17:48:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:48:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:48:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/26 17:48:52 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/10/26 17:49:02 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/10/26 17:49:12 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/10/26 17:49:22 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/10/26 17:49:32 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/10/26 17:49:42 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/10/26 17:49:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:49:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:49:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/26 17:49:52 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/10/26 17:49:52 backup.go:540: [DEBUG] groupStacksByDrive: /mnt/sys_drive → [privatebin, radicale] +2026/10/26 17:49:52 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 9fa4e86f3613 radicale radicale ghcr.io/kozea/radicale:3.8.1 +2026/10/26 17:49:52 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container radicale (image=ghcr.io/kozea/radicale:3.8.1, not a database) +2026/10/26 17:49:52 recovery_unit.go:297: [INFO] [backup] Recovery unit captured for radicale → /mnt/sys_drive/felhom-data/backups/primary/radicale (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1) +2026/10/26 17:50:02 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/10/26 17:50:12 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/10/26 17:50:22 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/10/26 17:50:32 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/10/26 17:50:42 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/10/26 17:50:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:50:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:50:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/26 17:50:52 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 5m0s ago, effective interval 5m0s, healthy=true +2026/10/26 17:51:02 healthprobe.go:187: [DEBUG] Health probe radicale: API GET :5232/.web/ → 200 (3ms) +2026/10/26 17:51:12 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 10s ago, effective interval 5m0s, healthy=true +2026/10/26 17:51:22 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 20s ago, effective interval 5m0s, healthy=true +2026/10/26 17:51:32 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping radicale — last check 30s ago, effective interval 5m0s, healthy=true +2026/10/26 17:51:34 auth.go:144: [DEBUG] [web] auth: valid session for POST /api/stacks/radicale/stop +2026/10/26 17:51:34 router.go:78: [DEBUG] [api] POST /api/stacks/radicale/stop (path=/stacks/radicale/stop) +2026/10/26 17:51:34 router.go:663: [INFO] [api] stop requested for stack: radicale +2026/10/26 17:51:34 router.go:78: [DEBUG] [api] actionStack: action=stop name=radicale +2026/10/26 17:51:34 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 0 encrypted, 1 sensitive fields +2026/10/26 17:51:34 [INFO] [stacks] SaveAppConfig: saved config for radicale +2026/10/26 17:51:34 desiredstate.go:81: [INFO] [stacks] desired state for radicale recorded as "stopped" (was "running") +2026/10/26 17:51:34 manager.go:1443: [DEBUG] [stacks] StopStack radicale: current state=running deployed=true containers=1 +2026/10/26 17:51:34 manager.go:1446: [INFO] [stacks] Stopping stack: radicale +2026/10/26 17:51:34 manager.go:1701: [DEBUG] Running: docker compose down (in /opt/docker/stacks/radicale) +2026/10/26 17:51:35 manager.go:1455: [INFO] [stacks] Stack radicale stopped successfully (took 0.4s) +2026/10/26 17:51:35 lifecycle.go:24: [DEBUG] [integrations] OnStackStop: stack=radicale integrationsFound=0 +2026/10/26 17:51:40 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/stacks/radicale +2026/10/26 17:51:40 router.go:78: [DEBUG] [api] GET /api/stacks/radicale (path=/stacks/radicale) +2026/10/26 17:51:40 auth.go:144: [DEBUG] [web] auth: valid session for POST /api/stacks/radicale/remove +2026/10/26 17:51:40 router.go:78: [DEBUG] [api] POST /api/stacks/radicale/remove (path=/stacks/radicale/remove) +2026/10/26 17:51:40 router.go:1025: [INFO] [api] Remove requested for stack: radicale +2026/10/26 17:51:40 router.go:78: [DEBUG] [api] removeStack: name=radicale +2026/10/26 17:51:40 router.go:78: [DEBUG] [api] removeStack: name=radicale removeHDDData=true removeBackups=true +2026/10/26 17:51:40 delete.go:626: [DEBUG] [stacks] RemoveStack called: name="radicale", removeHDDData=true, backupPathsToRemove=3 +2026/10/26 17:51:40 delete.go:640: [DEBUG] [stacks] RemoveStack radicale: state=stopped, deployed=true, orphaned=false, deploying=false +2026/10/26 17:51:40 delete.go:697: [INFO] Removing deployed stack: radicale (removeHDDData=true, hddDeclared=false, backupPaths=3) +2026/10/26 17:51:40 delete.go:710: [DEBUG] [stacks] RemoveStack radicale: found 0 HDD mounts from compose file +2026/10/26 17:51:40 manager.go:1701: [DEBUG] Running: docker compose down --volumes (in /opt/docker/stacks/radicale) +2026/10/26 17:51:40 delete.go:736: [DEBUG] [stacks] RemoveStack radicale: compose down output: +2026/10/26 17:51:52 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/26 17:51:52 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/26 17:51:52 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/26 17:52:06 delete.go:774: [INFO] [stacks] RemoveStack radicale: removed volume(s) [radicale_radicale_data] +2026/10/26 17:52:06 delete.go:837: [DEBUG] [stacks] RemoveStack radicale: processing 3 backup paths for removal (base=/mnt/sys_drive/felhom-data/backups) +2026/10/26 17:52:06 delete.go:854: [INFO] Removed backup data: /mnt/sys_drive/felhom-data/backups/primary/radicale (36K) +2026/10/26 17:52:06 delete.go:862: [DEBUG] [stacks] RemoveStack radicale: removing app.yaml at /opt/docker/stacks/radicale/app.yaml +2026/10/26 17:52:06 delete.go:879: [INFO] Stack radicale removed successfully (took 26.8s) +2026/10/26 17:52:07 lifecycle.go:176: [DEBUG] [integrations] OnStackRemove: stack=radicale integrationsFound=0 +2026/10/26 17:52:07 notifier.go:355: [INFO] Event pushed: app_removed (info) [hu-only] — Alkalmazás eltávolítva: radicale +2026/10/26 17:52:07 image_retention.go:265: [INFO] [stacks] image retention (remove of radicale): deleted [ghcr.io/kozea/radicale:] (0139a3fcd1ed, 96.6MB) — no container, installed app or undo names it (decision 53) +2026/10/26 17:52:07 image_retention.go:238: [INFO] [stacks] image retention (remove of radicale): pass over 40 image(s) of [ghcr.io/kozea/radicale] — 1 candidate(s), 1 deleted, the rest kept +2026/10/26 17:52:12 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/stacks/radicale +2026/10/26 17:52:12 router.go:78: [DEBUG] [api] GET /api/stacks/radicale (path=/stacks/radicale) +privatebin diff --git a/documentation/audits/d53-held-apps-2026-10-26/live/held_apps_final.txt b/documentation/audits/d53-held-apps-2026-10-26/live/held_apps_final.txt new file mode 100644 index 00000000..3a4ebe65 --- /dev/null +++ b/documentation/audits/d53-held-apps-2026-10-26/live/held_apps_final.txt @@ -0,0 +1 @@ +held_apps after: [('chaosoom', 'unhealthy_stop'), ('gokapi', 'unhealthy_stop'), ('chaoscrash', 'unhealthy_stop'), ('chaosoomb', 'unhealthy_stop')] diff --git a/documentation/audits/d53-held-apps-2026-10-26/live/notification_log.txt b/documentation/audits/d53-held-apps-2026-10-26/live/notification_log.txt new file mode 100644 index 00000000..5845f811 --- /dev/null +++ b/documentation/audits/d53-held-apps-2026-10-26/live/notification_log.txt @@ -0,0 +1,14 @@ +['id', 'customer_id', 'event_type', 'severity', 'message', 'status', 'error_message', 'created_at', 'channel'] +['1', 'demo-hp', 'app_definition_refused', 'warning', 'immich: the box REFUSED a catalog definition — it asks for c', 'sent', '', '2026-10-26 17:29:52', 'operator'] +['2', 'demo-hp', 'app_definition_refused', 'warning', 'immich: the box REFUSED a catalog definition — it asks for c', 'skipped', 'operator_only', '2026-10-26 17:29:52', 'customer'] +['3', 'demo-hp', 'app_held_reminder', 'warning', 'Customer demo-hp: the app gokapi has been held for 32 days (', 'sent', '', '2026-10-26 17:33:35', 'operator'] +['4', 'demo-hp', 'app_held_reminder', 'warning', 'Customer demo-hp: the app gokapi has been held for 32 days (', 'skipped', 'operator_only', '2026-10-26 17:33:35', 'customer'] +['5', 'demo-hp', 'app_held_reminder', 'warning', 'Customer demo-hp: the app chaosoom has been held for 32 days', 'sent', '', '2026-10-26 17:33:35', 'operator'] +['6', 'demo-hp', 'app_held_reminder', 'warning', 'Customer demo-hp: the app chaosoom has been held for 32 days', 'skipped', 'operator_only', '2026-10-26 17:33:35', 'customer'] +['7', 'demo-hp', 'app_held_reminder', 'warning', 'Customer demo-hp: the app chaoscrash has been held for 32 da', 'sent', '', '2026-10-26 17:33:35', 'operator'] +['8', 'demo-hp', 'app_held_reminder', 'warning', 'Customer demo-hp: the app chaoscrash has been held for 32 da', 'skipped', 'operator_only', '2026-10-26 17:33:35', 'customer'] +['9', 'demo-hp', 'app_held_reminder', 'warning', 'Customer demo-hp: the app chaosoomb has been held for 32 day', 'sent', '', '2026-10-26 17:33:35', 'operator'] +['10', 'demo-hp', 'app_held_reminder', 'warning', 'Customer demo-hp: the app chaosoomb has been held for 32 day', 'skipped', 'operator_only', '2026-10-26 17:33:35', 'customer'] +['11', 'demo-hp', 'app_stopped_unhealthy', 'warning', 'A(z) radicale alkalmazást leállítottuk, mert újra és újra ös', 'sent', '', '2026-10-26 17:37:53', 'operator'] +['12', 'demo-hp', 'app_held_reminder', 'warning', 'Customer demo-hp: the app radicale has been held for 4m0s (u', 'sent', '', '2026-10-26 17:41:35', 'operator'] +['13', 'demo-hp', 'app_held_reminder', 'warning', 'Customer demo-hp: the app radicale has been held for 4m0s (u', 'skipped', 'operator_only', '2026-10-26 17:41:35', 'customer'] diff --git a/documentation/audits/d53-held-apps-2026-10-26/live/test-hub.log b/documentation/audits/d53-held-apps-2026-10-26/live/test-hub.log new file mode 100644 index 00000000..7f654916 --- /dev/null +++ b/documentation/audits/d53-held-apps-2026-10-26/live/test-hub.log @@ -0,0 +1,82 @@ +2026/10/26 18:28:34 [INFO] felhom-hub 0.173.0-d53test starting +2026/10/26 18:28:34 [INFO] [store] app-update event types added to 0 household(s)' notification prefs (one-time, add-only): [] +2026/10/26 18:28:34 [INFO] [store] app_stopped_unhealthy added to 0 household(s)' notification prefs (one-time, add-only): [] +2026/10/26 18:28:34 [ERROR] OFFSITE_SECRET_KEY unset — off-site passwords cannot be sealed; provisioning and the key registrar are DISABLED +2026/10/26 18:28:34 [INFO] Database opened at /tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0b76c4d7-9b39-4e4e-869d-d0d2c6006331/scratchpad/testhub/data/hub.db +2026/10/26 18:28:34 [WARN] Cannot read seed dir /usr/share/felhom/assets-seed: open /usr/share/felhom/assets-seed: no such file or directory (no assets will be available until uploaded) +2026/10/26 18:28:34 [INFO] Asset manifest built: 0 files +2026/10/26 18:28:34 [INFO] App-email relay enabled (limit 30/min/customer, From domains [felhom.eu]) +2026/10/26 18:28:34 [INFO] osupdates: approval rule = every ring-0 box healthy for 24h0m0s and 1 night run(s) +2026/10/26 18:28:34 [INFO] osupdates: the Docker engine set is approved only by the operator, after 2 healthy ring-0 night(s) +2026/10/26 18:28:34 [INFO] osupdates: alarms = no OS leg 168h0m0s, reboot needed 336h0m0s, ring 0 stalled 168h0m0s, not covered 336h0m0s +2026/10/26 18:28:34 [INFO] access log: /tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0b76c4d7-9b39-4e4e-869d-d0d2c6006331/scratchpad/testhub/data/access-log, one file per UTC day, kept 90 days +2026/10/26 18:28:34 [INFO] Registry version checker disabled (no credentials configured) +2026/10/26 18:28:34 [INFO] WG peer-sync disabled (endpoint not configured) +2026/10/26 18:28:34 [INFO] PBS DR tenantsync disabled (key or endpoint not configured) +2026/10/26 18:28:34 [INFO] agent-plane poke disabled (key or endpoint not configured) +2026/10/26 18:28:34 [INFO] PBS-DR self-heal reconciler started (interval 5m) +2026/10/26 18:28:34 [INFO] offsite credential self-heal reconciler started (interval 5m, debounce 2 reports) +2026/10/26 18:28:34 [INFO] Staleness checker initialized: 0 ok, 0 stale, 0 down (node_stale after 45m0s, node_down after 1h30m0s) +2026/10/26 18:28:34 [INFO] prune: next run at 2026-10-27 04:30 CET (in 10h1m25s) +2026/10/26 18:28:34 [INFO] Host staleness checker initialized: 0 ok, 0 already alarmed (seeded silently, R-1001), 0 stale, 0 down (left unseeded → first Check emits; host_stale after 45m0s, host_down after 1h30m0s) +2026/10/26 18:28:34 [INFO] Host capability checker initialized: 0 ok, 0 degraded, 0 with a refused whole-guest backup (degraded/refused left unseeded → first Check emits) +2026/10/26 18:28:34 [INFO] Host leaf checker initialized: 0 host fingerprint(s) seeded +2026/10/26 18:28:34 [INFO] Host disk checker initialized: warn=90% crit=95%, 0 ok seeded, 0 already-breached left unseeded (first Check emits) +2026/10/26 18:28:34 [INFO] Storage fill checker initialized: warn=90% crit=95%, 0 ok seeded, 0 already-breached left unseeded, 0 root-backed excluded +2026/10/26 18:28:34 [INFO] Host mgmt-plane checker initialized: 0 host heal-state(s) seeded +2026/10/26 18:28:34 [INFO] Host OOB checker initialized: 0 host(s) seeded degraded +2026/10/26 18:28:34 [INFO] Offsite checker initialized: fill warn=90% crit=95%, stale after 48h0m0s, 0 ok-seeded +2026/10/26 18:28:34 [WARN] Held apps: the reminder grace is OVERRIDDEN to 3m0s for every reason — a test configuration +2026/10/26 18:28:34 [WARN] Golden vouch check OFF: no registry credentials, so the hub cannot list published goldens +2026/10/26 18:28:34 [INFO] db-snapshot 08:00: next run at 2026-10-27 08:00 CET (in 13h31m25s) +2026/10/26 18:28:34 [INFO] db-snapshot 20:00: next run at 2026-10-26 20:00 CET (in 1h31m25s) +2026/10/26 18:28:34 [INFO] deadline-check: next run at 2026-10-27 05:00 CET (in 10h31m25s) +2026/10/26 18:28:34 [INFO] Listening on 192.168.0.180:18090 +2026/10/26 18:28:34 [INFO] db-snapshot 14:00: next run at 2026-10-27 14:00 CET (in 19h31m25s) +2026/10/26 18:28:34 [INFO] db-snapshot 02:00: next run at 2026-10-27 02:00 CET (in 7h31m25s) +2026/10/26 18:28:34 [INFO] db snapshot written: hub-20261026T172834Z.db (409600 bytes, 13ms); pruned 0, keeping 2 +2026/10/26 18:29:34 [INFO] osupdates: guest candidate (0 packages, first seen 0001-01-01T00:00:00Z): waiting — no ring-0 box +2026/10/26 18:29:34 [INFO] osupdates: host candidate (0 packages, first seen 0001-01-01T00:00:00Z): waiting — no ring-0 box +2026/10/26 18:29:34 [INFO] osupdates: alarm check done in 4ms, 0 alarm(s) sent [] +2026/10/26 18:29:52 [DEBUG] Notification prefs read for demo-hp: exists=false household_saved=false +2026/10/26 18:29:52 [INFO] Notification preferences updated for demo-hp: address set=false, household=false, events=[backup_failed db_dump_failed disk_warning disk_critical storage_disconnected node_down health_critical expected_backup_missed expected_dbdump_missed offbox_enlarge_blocked app_update_undone app_update_held app_stopped_unhealthy storage_reconnected backup_integrity_failed crossdrive_failed health_recovered] +2026/10/26 18:29:52 [INFO] Event from demo-hp: app_definition_refused (warning) — immich: the box REFUSED a catalog definition — it asks for container rights not on the box's allow-list. Nothing was written; the app keeps its last good compose. Check who pushed to the catalog. the catalog definition of immich asks for container rights the box does not allow: immich-server: bind /mnt/sizing-bench/appdata/immich -> /usr/src/app/upload rw; immich-server: bind /mnt/sizing-bench/userdata/media/photos -> /external/photos ro +2026/10/26 18:29:52 [INFO] Operator email sent for demo-hp/app_definition_refused +2026/10/26 18:29:57 [INFO] Event from demo-hp: controller_started (info) — Controller elindult (0.346.0-d53test) +2026/10/26 18:29:58 [INFO] DR-recipe app-half stored for customer demo-hp (v1) +2026/10/26 18:29:58 [INFO] Received report from demo-hp (8171 bytes) +2026/10/26 18:30:52 [INFO] Event from demo-hp: app_held (info) — App chaosoom is held: stopped by the box for a oom storm (2026-09-24T11:48:01Z) — the household's Start gives it one more try +2026/10/26 18:30:52 [DEBUG] [held-apps] demo-hp/chaosoom held (unhealthy_stop since 2026-09-24T11:48:01Z) +2026/10/26 18:30:52 [INFO] Event from demo-hp: app_held (info) — App gokapi is held: stopped by the box for a crash loop (2026-09-24T10:37:48Z) — the household's Start gives it one more try +2026/10/26 18:30:52 [DEBUG] [held-apps] demo-hp/gokapi held (unhealthy_stop since 2026-09-24T10:37:48Z) +2026/10/26 18:30:52 [INFO] Event from demo-hp: app_held (info) — App chaoscrash is held: stopped by the box for a crash loop (2026-09-24T12:16:24Z) — the household's Start gives it one more try +2026/10/26 18:30:52 [DEBUG] [held-apps] demo-hp/chaoscrash held (unhealthy_stop since 2026-09-24T12:16:24Z) +2026/10/26 18:30:52 [INFO] Event from demo-hp: app_held (info) — App chaosoomb is held: stopped by the box for a oom storm (2026-09-24T12:39:51Z) — the household's Start gives it one more try +2026/10/26 18:30:52 [DEBUG] [held-apps] demo-hp/chaosoomb held (unhealthy_stop since 2026-09-24T12:39:51Z) +2026/10/26 18:33:34 [INFO] Held apps: reminder for demo-hp/gokapi (unhealthy_stop, held 32 days, repeat=false) +2026/10/26 18:33:35 [INFO] Operator email sent for demo-hp/app_held_reminder +2026/10/26 18:33:35 [INFO] Held apps: reminder for demo-hp/chaosoom (unhealthy_stop, held 32 days, repeat=false) +2026/10/26 18:33:35 [INFO] Operator email sent for demo-hp/app_held_reminder +2026/10/26 18:33:35 [INFO] Held apps: reminder for demo-hp/chaoscrash (unhealthy_stop, held 32 days, repeat=false) +2026/10/26 18:33:35 [INFO] Operator email sent for demo-hp/app_held_reminder +2026/10/26 18:33:35 [INFO] Held apps: reminder for demo-hp/chaosoomb (unhealthy_stop, held 32 days, repeat=false) +2026/10/26 18:33:35 [INFO] Operator email sent for demo-hp/app_held_reminder +2026/10/26 18:35:41 [INFO] Event from demo-hp: app_deploy_started (info) — Alkalmazás telepítése elindult: Radicale +2026/10/26 18:35:44 [INFO] DR-recipe app-half stored for customer demo-hp (v1) +2026/10/26 18:35:44 [INFO] Received report from demo-hp (8070 bytes) +2026/10/26 18:35:50 [INFO] Event from demo-hp: app_deployed (info) — Alkalmazás telepítve: Radicale +2026/10/26 18:37:52 [INFO] Event from demo-hp: app_stopped_unhealthy (warning) — A(z) radicale alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod. +2026/10/26 18:37:53 [INFO] Operator email sent for demo-hp/app_stopped_unhealthy +2026/10/26 18:38:52 [INFO] Event from demo-hp: app_held (info) — App radicale is held: stopped by the box for a crash loop (2026-10-26T17:37:52Z) — the household's Start gives it one more try +2026/10/26 18:38:52 [DEBUG] [held-apps] demo-hp/radicale held (unhealthy_stop since 2026-10-26T17:37:52Z) +2026/10/26 18:41:34 [INFO] Held apps: reminder for demo-hp/radicale (unhealthy_stop, held 4m0s, repeat=false) +2026/10/26 18:41:35 [INFO] Operator email sent for demo-hp/app_held_reminder +2026/10/26 18:44:52 [INFO] DR-recipe app-half stored for customer demo-hp (v1) +2026/10/26 18:44:52 [INFO] Received report from demo-hp (7771 bytes) +2026/10/26 18:45:52 [INFO] Event from demo-hp: app_hold_cleared (info) — App radicale is no longer held +2026/10/26 18:45:52 [INFO] [held-apps] demo-hp/radicale no longer held (a row existed: true) +2026/10/26 18:52:07 [INFO] Event from demo-hp: app_removed (info) — Alkalmazás eltávolítva: radicale +2026/10/26 18:52:09 [INFO] DR-recipe app-half stored for customer demo-hp (v1) +2026/10/26 18:52:09 [INFO] Received report from demo-hp (7122 bytes) +2026/10/26 18:52:49 [INFO] Received signal terminated, shutting down... +2026/10/26 18:52:49 [INFO] felhom-hub stopped diff --git a/documentation/audits/r1297-ci-trim-2026-10-26/README.md b/documentation/audits/r1297-ci-trim-2026-10-26/README.md new file mode 100644 index 00000000..db1a61e5 --- /dev/null +++ b/documentation/audits/r1297-ci-trim-2026-10-26/README.md @@ -0,0 +1,55 @@ +# R-1297 — faster CI: two runners, the trim, the numbers, the decoys (2026-10-26 evening) + +## 1. Both runners work (R-1297 B, done by the operator) + +Two harmless docs commits pushed to two repos, read from the Gitea jobs API (`runner_name` is the job page's runner): + +| Job | Repo, commit | Runner | Started | Ended | +|---|---|---|---|---| +| 2374 | felhom.eu `8143a2c8` | `felhom-gates-runner` (id 2) | 17:11:15 | 17:17:06 | +| 2375 | felhom-agent `ad80ee7` | `felhom-gates-runner-2` (id 3) | 17:11:41 | 17:12:56 | + +**At 17:11:41 UTC both jobs ran at the same time, on two different runners.** The agent job queued 3 s. + +## 2. The trim (R-1297 A, `ecd82aba`) + +- `scripts/script_tests_gate.py` `PATH_FILTERED`: on a CI **branch push with a known file list**, three suites run only + when one of their paths changed — `test_gate_decoys.py` (`scripts/`, `.gitea/`, `.githooks/`), + `test_due_checks_gate.py` (its gate, itself, `repo_gates.py`, `script_tests_gate.py`), + `dooplex-offsite/test_dooplex_offsite.py` (`scripts/dooplex-offsite/`). No list, an empty list or an unreadable list + runs every suite. The pre-push hook sets no list, so it still runs every suite. +- `.gitea/workflows/gates.yml`: a nightly schedule (03:20 UTC); the list is set only for `push:refs/heads/*`. A tag, the + nightly run and a manual dispatch run everything. A trimmed commits array (`total_commits` > listed) is an unknown list. +- `test_repo_gates.py` **tests the wiring** instead of running the whole gate set a second time: the real runner, + `GATES` narrowed by label to the five fingerprinted gates (130 s → 2.7 s). The same for `test_due_checks_gate.py` + group G (one more full run inside it, ~130 s → under 1 s). +- Red-proved: the filter cases and the workflow pins (`test_script_tests_gate.py`, red on the old `gates.yml`), an + inert `run_gate` (`test_repo_gates.py` red), an unregistered due-checks gate (group G red). + +**Not done: a Go build cache for `hub-gofmt`.** The R-1297 row was wrong about it: CI has no Go toolchain, so `hub-gofmt` +says NOT CHECKED there in 0.1 s. The ~110 s after it in job 2366 was `security-scan` (gitleaks over the depth-1 tree); +on the 2-CPU runners it takes 27 s. Nothing to cache. + +## 3. Times + +| Commit | Before | After | +|---|---|---| +| (a) docs only | job 2370 (1 runner, 1 CPU): **810 s**; job 2374 (2 CPUs, before the trim): **351 s** | see the R-1297 row (the next docs-only push) | +| (b) touches a gate script | 351 s (every suite ran on every push) | job 2376 `ecd82aba`: **203 s** (decoys ran; off-site skipped) | +| (c) a hub change | 351 s | job 2378 `7e57e824`: **118 s** (all three skipped) | + +Per-suite lines: `ci-job-*-suites.txt`. + +## 4. Decoys — a broken gate still goes red with the filter ON + +A red commit cannot be pushed (the pre-push hook refuses it; `--no-verify` is forbidden), so each decoy was run through +**CI's own entry point** (`repo_gates.py --fast`, `FELHOM_CHANGED_FILES` = the commit's file list, exactly as +`gates.yml` sets it) in a scratch clone of all four repos. + +- **Decoy 1 (docs only):** R-1297's row loses its state and owner cells. All three heavy suites SKIPPED; + `register-shape` CONVICTED; **rc=1**. `decoy1.txt` +- **Decoy 2 (a gate made inert):** `register_shape_gate.py` main() convicts nothing. The decoy suite RAN (its path + changed) — and stayed **green, rc=0**: its register-shape cases called `check()` directly, never the script the runner + calls. A real hole, older than the trim. `decoy2.txt` +- **Fixed in the same session** (small; `test_gate_decoys.py` now asserts the script's exit code on every + register-shape case). **Decoy 2 again: 9 decoys exposed the hole, `script-tests` FAILED, rc=1.** `decoy2b.txt` diff --git a/documentation/audits/r1297-ci-trim-2026-10-26/ci-job-2370-suites.txt b/documentation/audits/r1297-ci-trim-2026-10-26/ci-job-2370-suites.txt new file mode 100644 index 00000000..a3769516 --- /dev/null +++ b/documentation/audits/r1297-ci-trim-2026-10-26/ci-job-2370-suites.txt @@ -0,0 +1,40 @@ +2026-10-26T13:38:54.0405384Z == gate: script-tests (script_tests_gate.py /root/.cache/act/cdd17f98aeae57bd/hostexecutor) +2026-10-26T13:49:17.0478186Z OK scripts/test_build_sizing_web.py 0.3s rc=0 +2026-10-26T13:49:17.0753209Z OK scripts/test_dockerfile_arg_order.py 0.1s rc=0 +2026-10-26T13:49:17.0753966Z OK scripts/test_due_checks_gate.py 135.3s rc=0 +2026-10-26T13:49:17.0754283Z OK scripts/test_felhom_restore_beside.py 1.2s rc=0 +2026-10-26T13:49:17.0754632Z OK scripts/test_gate_decoys.py 229.5s rc=0 +2026-10-26T13:49:17.0754899Z OK scripts/test_golden_currency_gate.py 3.0s rc=0 +2026-10-26T13:49:17.0755567Z OK scripts/test_guide_quote_gate.py 1.1s rc=0 +2026-10-26T13:49:17.0755890Z OK scripts/test_host_swap_rule_gate.py 0.5s rc=0 +2026-10-26T13:49:17.0756131Z OK scripts/test_hostinstall.py 1.7s rc=0 +2026-10-26T13:49:17.0756359Z OK scripts/test_hu_grep.py 0.2s rc=0 +2026-10-26T13:49:17.0756622Z OK scripts/test_hub_copy_drift.py 0.1s rc=0 +2026-10-26T13:49:17.0756954Z OK scripts/test_instructions_gate.py 4.5s rc=0 +2026-10-26T13:49:17.0757265Z OK scripts/test_iso_bootstrap_gate.py 2.2s rc=0 +2026-10-26T13:49:17.0757563Z OK scripts/test_no_latest_push.py 0.1s rc=0 +2026-10-26T13:49:17.0757833Z OK scripts/test_push_scope.py 0.1s rc=0 +2026-10-26T13:49:17.0758121Z OK scripts/test_read_credential.py 0.3s rc=0 +2026-10-26T13:49:17.0758362Z OK scripts/test_repo_gates.py 130.2s rc=0 +2026-10-26T13:49:17.0758605Z OK scripts/test_repo_gates_docstring.py 0.1s rc=0 +2026-10-26T13:49:17.0758848Z OK scripts/test_repo_gates_scope.py 0.8s rc=0 +2026-10-26T13:49:17.0759179Z OK scripts/test_reuse_refs_check.py 0.5s rc=0 +2026-10-26T13:49:17.0759423Z OK scripts/test_script_tests_gate.py 0.9s rc=0 +2026-10-26T13:49:17.0759648Z OK scripts/test_security_scan_gate.py 16.7s rc=0 +2026-10-26T13:49:17.0759885Z OK scripts/test_status_view_gate.py 5.7s rc=0 +2026-10-26T13:49:17.0760140Z OK scripts/test_tenantsync_prune.py 0.1s rc=0 +2026-10-26T13:49:17.0760413Z OK scripts/test_tenantsync_selfcheck.py 0.2s rc=0 +2026-10-26T13:49:17.0760694Z OK scripts/bench/test_plugcut.py 0.2s rc=0 +2026-10-26T13:49:17.0760943Z OK scripts/bench/test_plugswitch.py 0.2s rc=0 +2026-10-26T13:49:17.0761189Z OK scripts/dooplex-offsite/test_dooplex_offsite.py 68.5s rc=0 +2026-10-26T13:49:17.0761482Z OK scripts/ep0-copy-gc/test_ep0_copy_gc.py 3.5s rc=0 +2026-10-26T13:49:17.0761739Z OK scripts/facebook/test_fb_probe.py 0.4s rc=0 +2026-10-26T13:49:17.0762015Z OK scripts/hub-db-backup/test_hub_db_backup.py 14.0s rc=0 +2026-10-26T13:49:17.0762307Z OK scripts/iso/test/test_bound_banner_clears.py 0.1s rc=0 +2026-10-26T13:49:17.0762538Z OK scripts/iso/test/test_console_glyphs.py 0.1s rc=0 +2026-10-26T13:49:17.0762801Z OK scripts/iso/test/test_rootpw_guard.py 0.2s rc=0 +2026-10-26T13:49:17.0763104Z OK scripts/release-copy/test_release_copy.py 0.5s rc=0 +2026-10-26T13:49:17.0763407Z script-tests: OK — 35 suite(s), every one exited 0 +2026-10-26T13:49:17.0781930Z == gate: decoy-coverage (decoy_coverage_gate.py /root/.cache/act/cdd17f98aeae57bd/hostexecutor /root/.cache/act/cdd17f98aeae57bd/felhom-controller /root/.cache/act/cdd17 +2026-10-26T13:49:17.6516884Z == gate: security-scan (security_scan_gate.py /root/.cache/act/cdd17f98aeae57bd/hostexecutor) +2026-10-26T13:51:06.6220333Z security-scan: 10 new hit(s), 42 accepted — warn-first, the push goes on diff --git a/documentation/audits/r1297-ci-trim-2026-10-26/ci-job-2374-suites.txt b/documentation/audits/r1297-ci-trim-2026-10-26/ci-job-2374-suites.txt new file mode 100644 index 00000000..172b00f6 --- /dev/null +++ b/documentation/audits/r1297-ci-trim-2026-10-26/ci-job-2374-suites.txt @@ -0,0 +1,40 @@ +2026-10-26T17:12:36.5594913Z == gate: script-tests (script_tests_gate.py /root/.cache/act/e82d471840c0952e/hostexecutor) +2026-10-26T17:16:39.1744210Z OK scripts/test_build_sizing_web.py 0.2s rc=0 +2026-10-26T17:16:39.1745588Z OK scripts/test_dockerfile_arg_order.py 0.0s rc=0 +2026-10-26T17:16:39.1745890Z OK scripts/test_due_checks_gate.py 38.3s rc=0 +2026-10-26T17:16:39.1746163Z OK scripts/test_felhom_restore_beside.py 0.6s rc=0 +2026-10-26T17:16:39.1746490Z OK scripts/test_gate_decoys.py 89.9s rc=0 +2026-10-26T17:16:39.1746772Z OK scripts/test_golden_currency_gate.py 1.4s rc=0 +2026-10-26T17:16:39.1747079Z OK scripts/test_guide_quote_gate.py 0.5s rc=0 +2026-10-26T17:16:39.1747586Z OK scripts/test_host_swap_rule_gate.py 0.2s rc=0 +2026-10-26T17:16:39.1747831Z OK scripts/test_hostinstall.py 0.8s rc=0 +2026-10-26T17:16:39.1748125Z OK scripts/test_hu_grep.py 0.1s rc=0 +2026-10-26T17:16:39.1748354Z OK scripts/test_hub_copy_drift.py 0.1s rc=0 +2026-10-26T17:16:39.1748579Z OK scripts/test_instructions_gate.py 2.1s rc=0 +2026-10-26T17:16:39.1748923Z OK scripts/test_iso_bootstrap_gate.py 1.1s rc=0 +2026-10-26T17:16:39.1749157Z OK scripts/test_no_latest_push.py 0.1s rc=0 +2026-10-26T17:16:39.1749396Z OK scripts/test_push_scope.py 0.1s rc=0 +2026-10-26T17:16:39.1749660Z OK scripts/test_read_credential.py 0.1s rc=0 +2026-10-26T17:16:39.1749895Z OK scripts/test_repo_gates.py 37.6s rc=0 +2026-10-26T17:16:39.1750116Z OK scripts/test_repo_gates_docstring.py 0.1s rc=0 +2026-10-26T17:16:39.1750381Z OK scripts/test_repo_gates_scope.py 0.4s rc=0 +2026-10-26T17:16:39.1750758Z OK scripts/test_reuse_refs_check.py 0.3s rc=0 +2026-10-26T17:16:39.1750986Z OK scripts/test_script_tests_gate.py 0.5s rc=0 +2026-10-26T17:16:39.1751248Z OK scripts/test_security_scan_gate.py 6.8s rc=0 +2026-10-26T17:16:39.1751480Z OK scripts/test_status_view_gate.py 2.3s rc=0 +2026-10-26T17:16:39.1751694Z OK scripts/test_tenantsync_prune.py 0.1s rc=0 +2026-10-26T17:16:39.1751957Z OK scripts/test_tenantsync_selfcheck.py 0.1s rc=0 +2026-10-26T17:16:39.1752243Z OK scripts/bench/test_plugcut.py 0.1s rc=0 +2026-10-26T17:16:39.1752541Z OK scripts/bench/test_plugswitch.py 0.1s rc=0 +2026-10-26T17:16:39.1752820Z OK scripts/dooplex-offsite/test_dooplex_offsite.py 48.6s rc=0 +2026-10-26T17:16:39.1753036Z OK scripts/ep0-copy-gc/test_ep0_copy_gc.py 1.5s rc=0 +2026-10-26T17:16:39.1753269Z OK scripts/facebook/test_fb_probe.py 0.2s rc=0 +2026-10-26T17:16:39.1753526Z OK scripts/hub-db-backup/test_hub_db_backup.py 8.1s rc=0 +2026-10-26T17:16:39.1753762Z OK scripts/iso/test/test_bound_banner_clears.py 0.0s rc=0 +2026-10-26T17:16:39.1753979Z OK scripts/iso/test/test_console_glyphs.py 0.0s rc=0 +2026-10-26T17:16:39.1754267Z OK scripts/iso/test/test_rootpw_guard.py 0.1s rc=0 +2026-10-26T17:16:39.1754491Z OK scripts/release-copy/test_release_copy.py 0.2s rc=0 +2026-10-26T17:16:39.1754729Z script-tests: OK — 35 suite(s), every one exited 0 +2026-10-26T17:16:39.1885982Z == gate: decoy-coverage (decoy_coverage_gate.py /root/.cache/act/e82d471840c0952e/hostexecutor /root/.cache/act/e82d471840c0952e/felhom-controller /root/.cache/act/e82d4 +2026-10-26T17:16:39.4762840Z == gate: security-scan (security_scan_gate.py /root/.cache/act/e82d471840c0952e/hostexecutor) +2026-10-26T17:17:06.0127522Z security-scan: 10 new hit(s), 42 accepted — warn-first, the push goes on diff --git a/documentation/audits/r1297-ci-trim-2026-10-26/ci-job-2376-suites.txt b/documentation/audits/r1297-ci-trim-2026-10-26/ci-job-2376-suites.txt new file mode 100644 index 00000000..c02dd191 --- /dev/null +++ b/documentation/audits/r1297-ci-trim-2026-10-26/ci-job-2376-suites.txt @@ -0,0 +1,41 @@ +2026-10-26T17:18:09.7889931Z suite filter: ON (a branch push with a known file list) +2026-10-26T17:18:18.4276734Z == gate: script-tests (script_tests_gate.py /root/.cache/act/f015d6564ba3139c/hostexecutor) +2026-10-26T17:20:21.8188080Z OK scripts/test_build_sizing_web.py 0.2s rc=0 +2026-10-26T17:20:21.8189127Z OK scripts/test_dockerfile_arg_order.py 0.0s rc=0 +2026-10-26T17:20:21.8189544Z OK scripts/test_due_checks_gate.py 0.9s rc=0 +2026-10-26T17:20:21.8189909Z OK scripts/test_felhom_restore_beside.py 0.5s rc=0 +2026-10-26T17:20:21.8190332Z OK scripts/test_gate_decoys.py 90.1s rc=0 +2026-10-26T17:20:21.8190886Z OK scripts/test_golden_currency_gate.py 1.4s rc=0 +2026-10-26T17:20:21.8191212Z OK scripts/test_guide_quote_gate.py 0.5s rc=0 +2026-10-26T17:20:21.8191644Z OK scripts/test_host_swap_rule_gate.py 0.2s rc=0 +2026-10-26T17:20:21.8191975Z OK scripts/test_hostinstall.py 0.8s rc=0 +2026-10-26T17:20:21.8192218Z OK scripts/test_hu_grep.py 0.1s rc=0 +2026-10-26T17:20:21.8192480Z OK scripts/test_hub_copy_drift.py 0.1s rc=0 +2026-10-26T17:20:21.8192723Z OK scripts/test_instructions_gate.py 2.1s rc=0 +2026-10-26T17:20:21.8192961Z OK scripts/test_iso_bootstrap_gate.py 1.1s rc=0 +2026-10-26T17:20:21.8193205Z OK scripts/test_no_latest_push.py 0.1s rc=0 +2026-10-26T17:20:21.8193467Z OK scripts/test_push_scope.py 0.1s rc=0 +2026-10-26T17:20:21.8193700Z OK scripts/test_read_credential.py 0.1s rc=0 +2026-10-26T17:20:21.8193927Z OK scripts/test_repo_gates.py 3.5s rc=0 +2026-10-26T17:20:21.8194139Z OK scripts/test_repo_gates_docstring.py 0.0s rc=0 +2026-10-26T17:20:21.8194382Z OK scripts/test_repo_gates_scope.py 0.4s rc=0 +2026-10-26T17:20:21.8194624Z OK scripts/test_reuse_refs_check.py 0.2s rc=0 +2026-10-26T17:20:21.8194894Z OK scripts/test_script_tests_gate.py 1.4s rc=0 +2026-10-26T17:20:21.8195299Z OK scripts/test_security_scan_gate.py 6.7s rc=0 +2026-10-26T17:20:21.8195534Z OK scripts/test_status_view_gate.py 2.3s rc=0 +2026-10-26T17:20:21.8195829Z OK scripts/test_tenantsync_prune.py 0.1s rc=0 +2026-10-26T17:20:21.8196204Z OK scripts/test_tenantsync_selfcheck.py 0.1s rc=0 +2026-10-26T17:20:21.8196478Z OK scripts/bench/test_plugcut.py 0.1s rc=0 +2026-10-26T17:20:21.8196727Z OK scripts/bench/test_plugswitch.py 0.1s rc=0 +2026-10-26T17:20:21.8196990Z SKIP scripts/dooplex-offsite/test_dooplex_offsite.py none of its paths (scripts/dooplex-offsite/) changed in this push — it runs on a release tag and nightly +2026-10-26T17:20:21.8197523Z OK scripts/ep0-copy-gc/test_ep0_copy_gc.py 1.5s rc=0 +2026-10-26T17:20:21.8197782Z OK scripts/facebook/test_fb_probe.py 0.2s rc=0 +2026-10-26T17:20:21.8198009Z OK scripts/hub-db-backup/test_hub_db_backup.py 8.0s rc=0 +2026-10-26T17:20:21.8198222Z OK scripts/iso/test/test_bound_banner_clears.py 0.0s rc=0 +2026-10-26T17:20:21.8198521Z OK scripts/iso/test/test_console_glyphs.py 0.0s rc=0 +2026-10-26T17:20:21.8198736Z OK scripts/iso/test/test_rootpw_guard.py 0.1s rc=0 +2026-10-26T17:20:21.8198950Z OK scripts/release-copy/test_release_copy.py 0.2s rc=0 +2026-10-26T17:20:21.8199183Z script-tests: OK — 34 suite(s) ran, every one exited 0, 1 skipped by path (R-1297) +2026-10-26T17:20:21.8329076Z == gate: decoy-coverage (decoy_coverage_gate.py /root/.cache/act/f015d6564ba3139c/hostexecutor /root/.cache/act/f015d6564ba3139c/felhom-controller /root/.cache/act/f015d +2026-10-26T17:20:22.1007937Z == gate: security-scan (security_scan_gate.py /root/.cache/act/f015d6564ba3139c/hostexecutor) +2026-10-26T17:20:50.4881408Z security-scan: 10 new hit(s), 42 accepted — warn-first, the push goes on diff --git a/documentation/audits/r1297-ci-trim-2026-10-26/ci-job-2378-suites.txt b/documentation/audits/r1297-ci-trim-2026-10-26/ci-job-2378-suites.txt new file mode 100644 index 00000000..7a74dfa7 --- /dev/null +++ b/documentation/audits/r1297-ci-trim-2026-10-26/ci-job-2378-suites.txt @@ -0,0 +1,41 @@ +2026-10-26T17:28:47.7729918Z suite filter: ON (a branch push with a known file list) +2026-10-26T17:28:56.1037450Z == gate: script-tests (script_tests_gate.py /root/.cache/act/cc3bec12ce9ea276/hostexecutor) +2026-10-26T17:29:28.5067637Z OK scripts/test_build_sizing_web.py 0.2s rc=0 +2026-10-26T17:29:28.5068146Z OK scripts/test_dockerfile_arg_order.py 0.0s rc=0 +2026-10-26T17:29:28.5068305Z SKIP scripts/test_due_checks_gate.py none of its paths (scripts/due_checks_gate.py, scripts/test_due_checks_gate.py, scripts/repo_gates.py, scrip +2026-10-26T17:29:28.5068488Z OK scripts/test_felhom_restore_beside.py 0.5s rc=0 +2026-10-26T17:29:28.5069032Z SKIP scripts/test_gate_decoys.py none of its paths (scripts/, .gitea/, .githooks/) changed in this push — it runs on a release tag and nigh +2026-10-26T17:29:28.5069271Z OK scripts/test_golden_currency_gate.py 1.4s rc=0 +2026-10-26T17:29:28.5069398Z OK scripts/test_guide_quote_gate.py 0.5s rc=0 +2026-10-26T17:29:28.5069518Z OK scripts/test_host_swap_rule_gate.py 0.2s rc=0 +2026-10-26T17:29:28.5069622Z OK scripts/test_hostinstall.py 0.8s rc=0 +2026-10-26T17:29:28.5069725Z OK scripts/test_hu_grep.py 0.1s rc=0 +2026-10-26T17:29:28.5069822Z OK scripts/test_hub_copy_drift.py 0.1s rc=0 +2026-10-26T17:29:28.5069925Z OK scripts/test_instructions_gate.py 2.1s rc=0 +2026-10-26T17:29:28.5070020Z OK scripts/test_iso_bootstrap_gate.py 1.1s rc=0 +2026-10-26T17:29:28.5070167Z OK scripts/test_no_latest_push.py 0.1s rc=0 +2026-10-26T17:29:28.5070276Z OK scripts/test_push_scope.py 0.1s rc=0 +2026-10-26T17:29:28.5070379Z OK scripts/test_read_credential.py 0.1s rc=0 +2026-10-26T17:29:28.5070499Z OK scripts/test_repo_gates.py 3.4s rc=0 +2026-10-26T17:29:28.5070600Z OK scripts/test_repo_gates_docstring.py 0.1s rc=0 +2026-10-26T17:29:28.5070702Z OK scripts/test_repo_gates_scope.py 0.4s rc=0 +2026-10-26T17:29:28.5070806Z OK scripts/test_reuse_refs_check.py 0.3s rc=0 +2026-10-26T17:29:28.5070977Z OK scripts/test_script_tests_gate.py 1.4s rc=0 +2026-10-26T17:29:28.5072435Z OK scripts/test_security_scan_gate.py 6.8s rc=0 +2026-10-26T17:29:28.5072830Z OK scripts/test_status_view_gate.py 2.3s rc=0 +2026-10-26T17:29:28.5073074Z OK scripts/test_tenantsync_prune.py 0.1s rc=0 +2026-10-26T17:29:28.5073299Z OK scripts/test_tenantsync_selfcheck.py 0.1s rc=0 +2026-10-26T17:29:28.5073586Z OK scripts/bench/test_plugcut.py 0.1s rc=0 +2026-10-26T17:29:28.5073872Z OK scripts/bench/test_plugswitch.py 0.1s rc=0 +2026-10-26T17:29:28.5074106Z SKIP scripts/dooplex-offsite/test_dooplex_offsite.py none of its paths (scripts/dooplex-offsite/) changed in this push — it runs on a release tag and nightly +2026-10-26T17:29:28.5074484Z OK scripts/ep0-copy-gc/test_ep0_copy_gc.py 1.6s rc=0 +2026-10-26T17:29:28.5074721Z OK scripts/facebook/test_fb_probe.py 0.2s rc=0 +2026-10-26T17:29:28.5074931Z OK scripts/hub-db-backup/test_hub_db_backup.py 8.0s rc=0 +2026-10-26T17:29:28.5075217Z OK scripts/iso/test/test_bound_banner_clears.py 0.0s rc=0 +2026-10-26T17:29:28.5075436Z OK scripts/iso/test/test_console_glyphs.py 0.0s rc=0 +2026-10-26T17:29:28.5075680Z OK scripts/iso/test/test_rootpw_guard.py 0.1s rc=0 +2026-10-26T17:29:28.5075908Z OK scripts/release-copy/test_release_copy.py 0.2s rc=0 +2026-10-26T17:29:28.5076184Z script-tests: OK — 32 suite(s) ran, every one exited 0, 3 skipped by path (R-1297) +2026-10-26T17:29:28.5150602Z == gate: decoy-coverage (decoy_coverage_gate.py /root/.cache/act/cc3bec12ce9ea276/hostexecutor /root/.cache/act/cc3bec12ce9ea276/felhom-controller /root/.cache/act/cc3be +2026-10-26T17:29:28.7683218Z == gate: security-scan (security_scan_gate.py /root/.cache/act/cc3bec12ce9ea276/hostexecutor) +2026-10-26T17:29:59.0587029Z security-scan: 10 new hit(s), 42 accepted — warn-first, the push goes on diff --git a/documentation/audits/r1297-ci-trim-2026-10-26/decoy1.txt b/documentation/audits/r1297-ci-trim-2026-10-26/decoy1.txt new file mode 100644 index 00000000..3fc89352 --- /dev/null +++ b/documentation/audits/r1297-ci-trim-2026-10-26/decoy1.txt @@ -0,0 +1,63 @@ +register-shape: CONVICTED — 1 problem(s) + line 402 R-1297: RULE 5 — 6 cells where the header declares 8. A literal `|` outside backticks makes a cell; put it inside `code`. + OK scripts/test_build_sizing_web.py 0.1s rc=0 + OK scripts/test_dockerfile_arg_order.py 0.0s rc=0 + SKIP scripts/test_due_checks_gate.py none of its paths (scripts/due_checks_gate.py, scripts/test_due_checks_gate.py, scripts/repo_gates.py, scripts/script_tests_gate.py) changed in this push — + OK scripts/test_felhom_restore_beside.py 0.5s rc=0 + SKIP scripts/test_gate_decoys.py none of its paths (scripts/, .gitea/, .githooks/) changed in this push — it runs on a release tag and nightly + OK scripts/test_golden_currency_gate.py 1.0s rc=0 + OK scripts/test_guide_quote_gate.py 0.3s rc=0 + OK scripts/test_host_swap_rule_gate.py 0.2s rc=0 + OK scripts/test_hostinstall.py 0.7s rc=0 + OK scripts/test_hu_grep.py 0.0s rc=0 + OK scripts/test_hub_copy_drift.py 0.0s rc=0 + OK scripts/test_instructions_gate.py 1.6s rc=0 + OK scripts/test_iso_bootstrap_gate.py 0.8s rc=0 + OK scripts/test_no_latest_push.py 0.1s rc=0 + OK scripts/test_push_scope.py 0.0s rc=0 + OK scripts/test_read_credential.py 0.1s rc=0 + OK scripts/test_repo_gates.py 2.8s rc=0 + OK scripts/test_repo_gates_docstring.py 0.0s rc=0 + OK scripts/test_repo_gates_scope.py 0.3s rc=0 + OK scripts/test_reuse_refs_check.py 0.1s rc=0 + OK scripts/test_script_tests_gate.py 1.1s rc=0 + OK scripts/test_security_scan_gate.py 6.1s rc=0 + OK scripts/test_status_view_gate.py 1.7s rc=0 + OK scripts/test_tenantsync_prune.py 0.7s rc=0 + OK scripts/test_tenantsync_selfcheck.py 1.1s rc=0 + OK scripts/bench/test_plugcut.py 0.1s rc=0 + OK scripts/bench/test_plugswitch.py 0.1s rc=0 + SKIP scripts/dooplex-offsite/test_dooplex_offsite.py none of its paths (scripts/dooplex-offsite/) changed in this push — it runs on a release tag and nightly + OK scripts/ep0-copy-gc/test_ep0_copy_gc.py 1.2s rc=0 + OK scripts/facebook/test_fb_probe.py 0.1s rc=0 + OK scripts/hub-db-backup/test_hub_db_backup.py 5.7s rc=0 + OK scripts/iso/test/test_bound_banner_clears.py 0.0s rc=0 + OK scripts/iso/test/test_console_glyphs.py 0.0s rc=0 + OK scripts/iso/test/test_rootpw_guard.py 0.1s rc=0 + OK scripts/release-copy/test_release_copy.py 0.2s rc=0 + site OK (exit 0) + hostinstall OK (exit 0) + hostinstall-harness OK (exit 0) + hub-confirm OK (exit 0) + manifest-bearer OK (exit 0) + reuse-refs OK (exit 0) + instructions OK (exit 0) + golden-currency OK (exit 0) + wire-contract OK (exit 0) + hub-copy OK (exit 0) + guide-quote OK (exit 0) + host-swap-rule OK (exit 0) + due-checks OK (exit 0) + one-register OK (exit 0) + closed-register OK (exit 0) + observations OK (exit 0) + register-shape FAILED (exit 1) + stands OK (exit 0) + script-tests OK (exit 0) + decoy-coverage OK (exit 0) + poster-facts OK (exit 0) + hub-gofmt OK (exit 0) + security-scan OK (exit 0) + status-view OK (exit 0) +CONVICTED: register-shape +decoy1 rc=1 diff --git a/documentation/audits/r1297-ci-trim-2026-10-26/decoy2.txt b/documentation/audits/r1297-ci-trim-2026-10-26/decoy2.txt new file mode 100644 index 00000000..18eaee88 --- /dev/null +++ b/documentation/audits/r1297-ci-trim-2026-10-26/decoy2.txt @@ -0,0 +1,60 @@ + OK scripts/test_build_sizing_web.py 0.1s rc=0 + OK scripts/test_dockerfile_arg_order.py 0.0s rc=0 + SKIP scripts/test_due_checks_gate.py none of its paths (scripts/due_checks_gate.py, scripts/test_due_checks_gate.py, scripts/repo_gates.py, scripts/script_tests_gate.py) changed in this push — + OK scripts/test_felhom_restore_beside.py 0.5s rc=0 + OK scripts/test_gate_decoys.py 82.5s rc=0 + OK scripts/test_golden_currency_gate.py 1.1s rc=0 + OK scripts/test_guide_quote_gate.py 0.4s rc=0 + OK scripts/test_host_swap_rule_gate.py 0.2s rc=0 + OK scripts/test_hostinstall.py 0.7s rc=0 + OK scripts/test_hu_grep.py 0.0s rc=0 + OK scripts/test_hub_copy_drift.py 0.0s rc=0 + OK scripts/test_instructions_gate.py 1.6s rc=0 + OK scripts/test_iso_bootstrap_gate.py 0.8s rc=0 + OK scripts/test_no_latest_push.py 0.1s rc=0 + OK scripts/test_push_scope.py 0.0s rc=0 + OK scripts/test_read_credential.py 0.1s rc=0 + OK scripts/test_repo_gates.py 2.7s rc=0 + OK scripts/test_repo_gates_docstring.py 0.0s rc=0 + OK scripts/test_repo_gates_scope.py 0.3s rc=0 + OK scripts/test_reuse_refs_check.py 0.1s rc=0 + OK scripts/test_script_tests_gate.py 1.2s rc=0 + OK scripts/test_security_scan_gate.py 6.3s rc=0 + OK scripts/test_status_view_gate.py 1.7s rc=0 + OK scripts/test_tenantsync_prune.py 0.7s rc=0 + OK scripts/test_tenantsync_selfcheck.py 1.0s rc=0 + OK scripts/bench/test_plugcut.py 0.1s rc=0 + OK scripts/bench/test_plugswitch.py 0.1s rc=0 + SKIP scripts/dooplex-offsite/test_dooplex_offsite.py none of its paths (scripts/dooplex-offsite/) changed in this push — it runs on a release tag and nightly + OK scripts/ep0-copy-gc/test_ep0_copy_gc.py 1.2s rc=0 + OK scripts/facebook/test_fb_probe.py 0.1s rc=0 + OK scripts/hub-db-backup/test_hub_db_backup.py 5.6s rc=0 + OK scripts/iso/test/test_bound_banner_clears.py 0.0s rc=0 + OK scripts/iso/test/test_console_glyphs.py 0.0s rc=0 + OK scripts/iso/test/test_rootpw_guard.py 0.1s rc=0 + OK scripts/release-copy/test_release_copy.py 0.2s rc=0 + site OK (exit 0) + hostinstall OK (exit 0) + hostinstall-harness OK (exit 0) + hub-confirm OK (exit 0) + manifest-bearer OK (exit 0) + reuse-refs OK (exit 0) + instructions OK (exit 0) + golden-currency OK (exit 0) + wire-contract OK (exit 0) + hub-copy OK (exit 0) + guide-quote OK (exit 0) + host-swap-rule OK (exit 0) + due-checks OK (exit 0) + one-register OK (exit 0) + closed-register OK (exit 0) + observations OK (exit 0) + register-shape OK (exit 0) + stands OK (exit 0) + script-tests OK (exit 0) + decoy-coverage OK (exit 0) + poster-facts OK (exit 0) + hub-gofmt OK (exit 0) + security-scan OK (exit 0) + status-view OK (exit 0) +decoy2 rc=0 diff --git a/documentation/audits/r1297-ci-trim-2026-10-26/decoy2b.txt b/documentation/audits/r1297-ci-trim-2026-10-26/decoy2b.txt new file mode 100644 index 00000000..9ee522e8 --- /dev/null +++ b/documentation/audits/r1297-ci-trim-2026-10-26/decoy2b.txt @@ -0,0 +1,63 @@ + OK scripts/test_build_sizing_web.py 0.1s rc=0 + OK scripts/test_dockerfile_arg_order.py 0.0s rc=0 + SKIP scripts/test_due_checks_gate.py none of its paths (scripts/due_checks_gate.py, scripts/test_due_checks_gate.py, scripts/repo_gates.py, scripts/script_tests_gate.py) changed in this push — + OK scripts/test_felhom_restore_beside.py 0.5s rc=0 + FAIL scripts/test_gate_decoys.py 83.2s rc=1 + | FAIL: register-shape [pipe-outside-backticks]: the SCRIPT exited 0, want 1 — check() and the entry point disagree + | 9 decoy(s) of 95 exposed a hole + OK scripts/test_golden_currency_gate.py 1.1s rc=0 + OK scripts/test_guide_quote_gate.py 0.3s rc=0 + OK scripts/test_host_swap_rule_gate.py 0.2s rc=0 + OK scripts/test_hostinstall.py 0.6s rc=0 + OK scripts/test_hu_grep.py 0.0s rc=0 + OK scripts/test_hub_copy_drift.py 0.0s rc=0 + OK scripts/test_instructions_gate.py 1.5s rc=0 + OK scripts/test_iso_bootstrap_gate.py 0.8s rc=0 + OK scripts/test_no_latest_push.py 0.1s rc=0 + OK scripts/test_push_scope.py 0.0s rc=0 + OK scripts/test_read_credential.py 0.1s rc=0 + OK scripts/test_repo_gates.py 2.9s rc=0 + OK scripts/test_repo_gates_docstring.py 0.0s rc=0 + OK scripts/test_repo_gates_scope.py 0.3s rc=0 + OK scripts/test_reuse_refs_check.py 0.1s rc=0 + OK scripts/test_script_tests_gate.py 1.2s rc=0 + OK scripts/test_security_scan_gate.py 6.1s rc=0 + OK scripts/test_status_view_gate.py 1.7s rc=0 + OK scripts/test_tenantsync_prune.py 0.7s rc=0 + OK scripts/test_tenantsync_selfcheck.py 1.0s rc=0 + OK scripts/bench/test_plugcut.py 0.1s rc=0 + OK scripts/bench/test_plugswitch.py 0.1s rc=0 + SKIP scripts/dooplex-offsite/test_dooplex_offsite.py none of its paths (scripts/dooplex-offsite/) changed in this push — it runs on a release tag and nightly + OK scripts/ep0-copy-gc/test_ep0_copy_gc.py 1.1s rc=0 + OK scripts/facebook/test_fb_probe.py 0.1s rc=0 + OK scripts/hub-db-backup/test_hub_db_backup.py 5.7s rc=0 + OK scripts/iso/test/test_bound_banner_clears.py 0.0s rc=0 + OK scripts/iso/test/test_console_glyphs.py 0.0s rc=0 + OK scripts/iso/test/test_rootpw_guard.py 0.1s rc=0 + OK scripts/release-copy/test_release_copy.py 0.2s rc=0 + site OK (exit 0) + hostinstall OK (exit 0) + hostinstall-harness OK (exit 0) + hub-confirm OK (exit 0) + manifest-bearer OK (exit 0) + reuse-refs OK (exit 0) + instructions OK (exit 0) + golden-currency OK (exit 0) + wire-contract OK (exit 0) + hub-copy OK (exit 0) + guide-quote OK (exit 0) + host-swap-rule OK (exit 0) + due-checks OK (exit 0) + one-register OK (exit 0) + closed-register OK (exit 0) + observations OK (exit 0) + register-shape OK (exit 0) + stands OK (exit 0) + script-tests FAILED (exit 1) + decoy-coverage OK (exit 0) + poster-facts OK (exit 0) + hub-gofmt OK (exit 0) + security-scan OK (exit 0) + status-view OK (exit 0) +CONVICTED: script-tests +decoy2b rc=1 diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index c355638c..6bcfc8da 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -305,7 +305,7 @@ stopping line that lies. | **R-1223** | Monitoring & notifications | P3 | **After a reboot, a box whose tunnel ep0 refuses is invisible: the future stamp is gone, so neither the D30 re-key nor the hub's alarm sees it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, A8 + H1): the kernel forgets `latest-handshake` when the interface restarts (0 = never); the agent then sends neither `last_handshake_age_s` nor `handshake_future_s`, and `checkWGTunnel` (`hub/internal/api/host_tunnel.go`) does not judge a registered, active tunnel with no handshake at all. Shape: the clock ran ahead, was corrected, the box rebooted (some households switch off at night), ep0 still holds the future stamp and starts nothing. | OPEN | — | Agent: report `tunnel_up_s` (how long the unit has been active) with no handshake; hub: a registered, active tunnel with no handshake for > 30 min while reporting → `wg_tunnel_stale`; the agent re-keys the same way after 30 min with no handshake. Tests both sides; rides an agent release + a hub deploy | CC | | **R-1224** | Monitoring & notifications | P4 | **The operator's 1-hour mail cooldown for `wg_tunnel_stale` is per customer, not per host, and `wg_tunnel_ok` does not end it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, H3): `notify/dispatcher.go:620` keys the cooldown `customerID:eventType`. Two hosts of one household going stale within an hour, or stale → ok → stale, store a second event row but mail once (the suppression is in `notification_log`). | OPEN | — | Key the cooldown by `host_id` from the details for host-scoped operator events, and let the matching ok event end it; a dispatcher test with two hosts; rides a hub deploy | CC | | **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC | -| **R-1291** | Monitoring & notifications | P3 | **A held app gives the operator no lasting signal: one event at the stop, then nothing — demo-hp's Kimai sat held 11 days unseen.** Found 2026-10-25 night (R-1286). No held-app state reaches the hub (the report carries none; the hub shows none); the crash-loop stop sends `app_stopped_unhealthy` once. On a household box an app held after a crash loop, a failed update or a failed restore stays stopped until someone opens its page. | **OPEN** — ruled 2026-10-26 evening: **D53 = operator-only reminders first** (`09` §3 decision 265); CC builds `08` §10, commit only. **Earlier:** WAITING-ON-OPERATOR — 2026-10-26: the design is written (`08` §10 [DESIGN — proposed]: `app_held` / `app_hold_cleared` events + a daily re-send, a „Held apps" list on the host page, an operator reminder after a grace per reason, the hub never clears a hold; ~½ day controller + ~1 day hub). The pick is STATUS D53. **Earlier:** OPEN | — | a two-repo design: the controller reports its holds (stack, reason, since), the hub shows them on the host and mails a reminder after N days (the D-level of the reason); one row's worth of design first | CC | +| **R-1291** | Monitoring & notifications | P3 | **A held app gives the operator no lasting signal: one event at the stop, then nothing — demo-hp's Kimai sat held 11 days unseen.** Found 2026-10-25 night (R-1286). No held-app state reaches the hub (the report carries none; the hub shows none); the crash-loop stop sends `app_stopped_unhealthy` once. On a household box an app held after a crash loop, a failed update or a failed restore stays stopped until someone opens its page. | **OPEN — BUILT, commit only, NOT released** — 2026-10-26 evening: controller `5bd03ad` (`internal/holdreport`) + hub `7e57e824` (`held_apps`, `HeldAppsChecker`, the „Held apps" list on the host page and the Hosts page); red-proved; live on 9202 against a test hub with a 3-min grace (listed, one operator mail, Start cleared it, the row went, household leg `skipped operator_only`) — `audits/d53-held-apps-2026-10-26/`. Closes when both release. Ruled D53 = operator-only reminders first (`09` §3 decision 265). **Earlier:** WAITING-ON-OPERATOR — 2026-10-26: the design is written (`08` §10 [DESIGN — proposed]: `app_held` / `app_hold_cleared` events + a daily re-send, a „Held apps" list on the host page, an operator reminder after a grace per reason, the hub never clears a hold; ~½ day controller + ~1 day hub). The pick is STATUS D53. **Earlier:** OPEN | — | a two-repo design: the controller reports its holds (stack, reason, since), the hub shows them on the host and mails a reminder after N days (the D-level of the reason); one row's worth of design first | CC | | **R-1296** | Monitoring & notifications | P4 | **When ep0's tenancy script refuses, the hub log shows only its one-line error — PBS's own reason (the 403 `missing Datastore.Audit|Datastore.Backup on …`) stays in the script's stderr, which the hub drops on a refusal.** Found 2026-10-26 diagnosing R-1295: the cause was visible only by reading ep0's proxy journal over root SSH. `hub/internal/tenantsync/client.go` `parseResponse` returns the JSON error and ignores stderr; the script keeps secrets off stderr by design (its header), so the tail is safe to log. | OPEN | — | append the last ~3 stderr lines to the refusal error (bounded, never stdout); a test that a secret on stdout never appears | CC | ## Hub & operator — 12 rows (P3 3, P4 9) @@ -400,3 +400,4 @@ stopping line that lies. | R-1035 | 2026-12-09 | the same three numbers 8 weeks after publishing, compared with the 4-week read; a null at 4 weeks is not a verdict, both reads are needed; detail in the R-1035 row | | **R-1297** | Process & tooling | P3 | **CI waits dominate a working day: felhom.eu's CI run takes ~13 min and the ONE runner makes every repo's job wait behind it (up to 10–17 min queued).** Measured 2026-10-26 (operator: „these CI waiting times are getting out of hand"): since 10-25 felhom.eu runs median 793 s, the controller/agent/catalog 57–102 s; queue median 5 s, max 548–1023 s. In run 2366, ~10 of 13 minutes are four suites — `test_gate_decoys.py` 232 s, `test_due_checks_gate.py` 133 s, `test_repo_gates.py` 132 s (it runs the whole gate set a SECOND time), `dooplex-offsite/test_dooplex_offsite.py` 69 s — plus `hub-gofmt` ~110 s (a cold Go cache per run). **Ruled 2026-10-26 by the operator: BOTH** — (A) trim felhom.eu CI: the heavy suites only when their own files change, no duplicate full run, a Go cache for hub-gofmt; (B) a second CI runner on DooPlex. | OPEN — (B) DONE by the operator 2026-10-26: a second runner, both runners 2 CPUs (`09` §3 decision 267); (A) CC, 2026-10-26 evening | — | (A) CC, a run of its own (a CI scope change, ruled); (B) a DooPlex change — a brief or the homelab session; and CC waits on CI only before a release or at the end of a run | CC | +| **R-1298** | Monitoring & notifications | P3 | **Scratch guest 9202 carries leftovers made outside the product, and D53 would remind about them every week once released: four crash-loop holds since 2026-09-24 for apps that are not installed (`chaoscrash`, `chaosoom`, `chaosoomb`, `gokapi`), and a PrivateBin `app.yaml` (2026-10-25) with no container that the product reads as „installed".** Found 2026-10-26 evening in the D53 live proof: the new reporter sent the four holds at its first tick and the test hub mailed four reminders („held 32 days"). The product's removal clears an install's hold (R-704); these apps were removed by hand in a chaos drill. On a household box the same shape would mail the operator weekly about an app that does not exist. | OPEN | — | decide whether the reporter (or the hub) skips an `unhealthy_stop` / `update_failed` / `putback_pending` hold whose app is not installed (a restore hold stays reported — operator-cleared by design, R-379); then clear 9202's four holds with `-clear-restore-hold` and remove PrivateBin through the product | CC | diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index 845ba5f9..44f89cc4 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,3 +1,16 @@ +## 2026-10-26 (evening) — CI: heavy suites by path, no second full gate run, a nightly run (R-1297 A); a decoy hole closed + +- **`script_tests_gate.py` `PATH_FILTERED`:** on a CI branch push with a known file list, `test_gate_decoys.py`, + `test_due_checks_gate.py` and `dooplex-offsite/test_dooplex_offsite.py` run only when one of their paths changed; no + list (a tag, the nightly run, a dispatch, the pre-push hook), an empty or unreadable list = every suite (`ecd82aba`). +- **`.gitea/workflows/gates.yml`:** a nightly schedule 03:20 UTC; `FELHOM_CHANGED_FILES` only for `push:refs/heads/*`; a + trimmed commits array is an unknown list. Pinned by `test_script_tests_gate.py` (filter cases + workflow pins). +- **`test_repo_gates.py` and `test_due_checks_gate.py` group G** drive the real runner with `GATES` narrowed by label — + the wiring, not the whole gate set a second time (~130 s each in CI). +- **`test_gate_decoys.py` register-shape cases assert the SCRIPT's exit code too.** Found by R-1297's decoy: an inert + `main()` passed every case because the cases called `check()` only; red-proved with that inert `main()` (9 cases red). +- Numbers and decoys: `documentation/audits/r1297-ci-trim-2026-10-26/`. + ## 2026-10-26 — felhom-tenantsync v1.4.0: the self-check waits out the PBS permission cache (R-1295) — INSTALLED on ep0 2026-10-26 13:09Z on the operator's yes (sha256 `751b3c26…` = the repository's; v1.3.0 kept as `.v1.3.0`; `fingerprint` and `usage` ok) - **`felhom-tenantsync.sh` v1.4.0:** the new token is re-checked every 0.5 s for up to 15 s (a log line per try) before diff --git a/scripts/test_gate_decoys.py b/scripts/test_gate_decoys.py index e22b0fba..b28b1449 100644 --- a/scripts/test_gate_decoys.py +++ b/scripts/test_gate_decoys.py @@ -882,7 +882,15 @@ def _rsg_case(name, text, must_convict, expect_rule=None): fh.write(text) rows, ids, bad = _rsg.check(fp) convicted = bool(bad) - if convicted != must_convict: + # 2026-10-26 (R-1297 A's decoy): the RUNNER calls the script, not check(). An inert main() (convicting + # nothing) passed every case here while check() still worked — so the script's own exit code is + # asserted too, on every case. + _p = subprocess.run([sys.executable, os.path.join(ROOT, "scripts", "register_shape_gate.py"), fp], + stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + if _p.returncode != (1 if must_convict else 0): + fails.append("register-shape [%s]: the SCRIPT exited %d, want %d — check() and the entry point disagree" + % (name, _p.returncode, 1 if must_convict else 0)) + elif convicted != must_convict: fails.append("register-shape [%s]: expected %s, got %s (%s)" % (name, "CONVICT" if must_convict else "PASS", "CONVICT" if convicted else "PASS",