From ef8e23f42a4bacf8e27e0c377db50324440ed9e5 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sat, 14 Nov 2026 09:11:16 +0100 Subject: [PATCH] Health check 2026-11-14: R-528 dated check read (one failed oom_check, date moved to 11-28), R-1391 + R-1392 filed - R-528: NOT closed. demo-felhom's night oom_check failed once (2026-11-11 04:40 UTC, Docker 29.9.3, containerd 2.4.2-3); 56 others passed; all four app_oom events since 10-12 explained. Due date 2026-11-28. This clears the due-checks gate that kept nightly CI red (runs 2628, 2630, 2632). - R-1391: Go 1.27.3 (hub, controller, agent) has GO-2026-6859; fixed in 1.27.4. - R-1392: the hub's "root files behind" compares the whole bundle sha, which moves with every agent release even when configs/ did not change; the os_config_bundle_behind alarm is due ~11-17 on all three boxes. - STATUS: D81 + the 11-14 do line; old Updated paragraph moved to STATUS-HISTORY. REPORT overwritten. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS --- REPORT.md | 53 +++++++++++++++++-------- STATUS.md | 5 ++- documentation/backlog/OPEN-ITEMS.md | 10 +++-- documentation/backlog/STATUS-HISTORY.md | 4 ++ 4 files changed, 50 insertions(+), 22 deletions(-) diff --git a/REPORT.md b/REPORT.md index cd3349d8..c8a0cab1 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,21 +1,42 @@ -# REPORT — the three Space fixes are released (2026-11-10, afternoon, attended) +# REPORT — health check after a few unattended days (2026-11-14, morning, attended) -**Plain words first.** Three fixes are released. The agent cleans its own update folder; the operator vouched it. The -controller sends one warning for one full disk; the global floor is set. The hub sends the host-disk alarm to the -operator only; it was deployed on the operator's yes. The HP box booted its kept older kernel once, was healthy, and -went back. The five rulings D71–D75 are written down and none is built. +**Plain words first.** The three test boxes are healthy. Backups ran last night on all three, off-site too, and each +restore proof passed. The night CI mail was red for three nights for one reason: a dated check (R-528) was due. The +check is now read and written down, and CI is green for it. Two new rows are filed. Nothing was changed on any box. -**The full report is `REPORT-release-space-fixes-2026-11-10.md`.** It names first the claims that turned out wrong, -and one deviation from the rules (the hub deploy ran before CI for the manifest-bump commit was read). +**Method.** Read only. The hub's own pages with CC's token (Dashboard, Hosts, System, Offsite); one copy of the hub +database **with its WAL** (newest host report 7 minutes old, `quick_check` ok; the copy was shredded after the read); +the hub log's deadline lines; the operator mailbox (label Monitoring, last 5 days); the CI run log of run id 2632 +(job 2636) through the Gitea API. | | | |---|---| -| Agent | 0.173.0, vouched by the operator 15:29 UTC. The self-update folder on the three test boxes: 49 → 1, 48 → 1, 33 → 1 files. A rollback worked after the clean-up | -| Controller | 0.352.0, MinAgent 0.158.1. Global floor 15:31 UTC. The three boxes that report are on it. Each watches 2 filesystems, one target each | -| Hub | 0.180.0, live since 15:12 UTC, deployed by manifest bump. Synced/Healthy, no alarm sent in the 30 minutes after | -| The older kernel | demo-hp booted it to healthy in 313 seconds and went back in 324 seconds. Nothing was left behind | -| Rulings | D71–D75 in `documentation/architecture/09-update-architecture.md` §3, decisions 291–296 | -| Register | 236 open at the start, 236 at the end. Opened 0, closed 0. Dated lines on R-1368, R-1371, R-1301, R-1353 | -| Not proven | That a household which saved its own alarm list no longer gets the host-disk alarm, on the running hub. A real fill on a household's box. The kernel clean-up as a program | -| Evidence | `documentation/audits/release-space-fixes-2026-11-10/` and `documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/` | -| Not done, by the brief | No golden. No catalog push. No change on ep0. No test alarm. No mail to a household | +| Boxes | demo-felhom, demo-hp, Tester 1: controller 0.352.0, agent 0.173.0, ONLINE, all containers running, crash guard armed, 0 crash restarts | +| Backups last night | DB dumps 03:15–03:17 UTC; off-site `ok` 03:15–03:18 UTC; off-site restore proof `pass` 04:30 UTC (bentopdf, opengist, bookstack); whole-guest backups `success`; host restore tests `pass` (demo-hp from PBS 2026-11-14 02:43 UTC) | +| CI red ×3 nights (runs 2628, 2630, 2632) | only gate `due-checks`: R-528 was due 2026-11-12. Read and recorded (below), date moved to 2026-11-28 with the reason in the row | +| R-528 reading | NOT closed. One failed `oom_check` — demo-felhom 2026-11-11 04:40:30 UTC, Docker 29.9.3 / containerd 2.4.2-3: exit 137, `OOMKilled=false`, no `oom` event. 56 others passed. The four `app_oom` events since 2026-10-12 are all explained (a freeze, a squeeze test, a release proof) | +| Tester4 | Host deleted by the operator 07:54:31 UTC. The pairing-link mail at the same second is by design (a host delete that keeps the customer). The 05:00 `expected_dbdump_missed` came before the delete; the host's reports are gone with it, so the deadline check reads „not judged yet" from tomorrow (as Tester 2 does) | +| demo-hp blips | `agent_channel_unknown` 2026-11-11 13:52 → recovered 14:00 UTC (the certificate-pin spike on demo-hp that afternoon); `health_degraded` 2026-11-12 03:17 → recovered 03:22 UTC (a controller restart in the night step) | +| Security watch | New: 4 FileBrowser notices (one high, arbitrary file write) fixed only on 2.1.0 — belongs to R-1305; Go 1.27.3 GO-2026-6859 had no row → R-1391; cloudflared is R-1358 | +| Register | 248 rows at the start, 250 at the end. Opened R-1391 (Go 1.27.4), R-1392 (root files amber after every agent release). Closed 0. Dated line on R-528 | + +## Findings + +1. **R-1392 — the yellow „root files behind the vouched agent's" on all three boxes is a false amber, and it becomes + three operator alarms around 2026-11-17.** The files on the boxes are the vouched ones (`configs/` did not change + between 0.172.1 and 0.173.0), but the hub compares the whole bundle's sha256, which carries the version. +2. **R-1391 — Go 1.27.3 in hub, controller and agent; fixed in 1.27.4.** Reach not measured. +3. **CI turns red again after 2026-11-15:** R-1050 (HSTS one year) is due that day and needs a controller release. + +## Waiting on the operator + +- Rotate `GITEA_GIT_TOKEN` (the secret sweep mail of 2026-11-12 found it in 2 session records), then `sec put`. +- Approve the Proxmox set 9.2.26 and kernel 7.0.14-28 on the System page when you want Tester 1 to follow. +- R-1392: A or B (in the row). Sending the 0.173.0 bundle to the three boxes clears today's amber either way. + +**Fixed without a row:** the local `instructions` gate (and its decoy test) failed because the memory index was 201 +lines / 25 725 bytes (ceilings 200 / 25 600). Two index lines were merged; now 199 / 25 571. Not a repo file. + +**Not done:** no box touched, no signed job sent, no hub change, no release. No area CHANGELOG entry: the change is +register-only (`hub/`, `scripts/`, `website/`, `marketing/` untouched). Provisioned nothing; teardown: the hub DB copy +in the scratchpad was shredded. diff --git a/STATUS.md b/STATUS.md index 90273885..47c704d2 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for Tester 2? Yes — tell the friend go (`documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-11-11 (afternoon).** Golden 0.352.0 is vouched, so code pushes to `felhom.eu` work again. The N100 keeps no time: one hand step repairs it (below). A new Proxmox certificate blinds the agent; the hand repair is proven on demo-hp and one decision waits (D80). +**Updated 2026-11-14 (morning).** Health check: the three test boxes are healthy; last night's backups and restore proofs passed. CI was red three nights for one due check; it is read now. The yellow „root files behind" is a false alarm that mails on 11-17 unless you answer D81. ## What works - **The Plex attack did not reach Felhom's secrets** (R-1241). @@ -32,6 +32,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked | |---|---|---|---|---|---|---| +| D81 | R-1392 | When an agent release changes no root file, should the hub stop calling the boxes' root files „behind"? | **Yes — the hub compares the files, not the bundle's label** (fixes every future release once; one hub change and deploy) | keep the hub: every release also sends a root-file job to each box (about 15 minutes per release, forever) | on 11-17 three alarm mails; the same after every agent release | — | | D80 | R-1380 | How shall the agent trust Proxmox after its certificate changes: a copy of the box's own root CA, or a helper that follows the new fingerprint (your ruling of 11-11 13:17)? | **The root CA copy** (proven on demo-hp: the agent went through a new certificate with no hand step; no new privileged code) | follow the fingerprint: a root helper, a sudo rule and re-pin logic to write and test; trust stays on one exact certificate | nothing is built; the first box goes blind about 2028-07-02 | the fix of R-1380 | | D66 | R-1365 | Should a security-watch source that stays down mail again every 7 days, as an open notice does? Today it mails once, then only the Monday summary lists it. | **Yes** (a real outage then cannot be forgotten; the seven pins that can never be read no longer count, so it stays quiet when all is well) | no: one line in a long Monday mail stays the only reminder | a source that stays down mails once; the Monday list still shows it | — | | D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — | @@ -40,7 +41,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | D7 | R-990, R-989 | Rule on the forum ideas and the selfh.st ideas now, or park them? | **Park them** (P4; nothing waits on them) | Rule now: about one hour of reading | nothing changes; the research gets old | — | ## Waiting on you — do -**2026-11-11.** 1. N100, as root, one time: `apparmor_parser -r -T -W /etc/apparmor.d/usr.sbin.chronyd && systemctl restart chrony && chronyc tracking` (R-1301; not tested). If not: about mid-January 2027 it refuses every agent update. 2. Hub Hosts page: delete host `Tester4-d35551`, Discard appliance `3fb5fed3…` (today's test box). If not: a dead entry alarms. 3. Rotate login `kisfenyo`'s Gitea token into the store as `GITEA_GIT_TOKEN` (R-925); the push after `sec put` is in R-1390. 4. Home DNS: remove the `felhom.eu` override (R-1389). 5. Read the dates only you can read (R-1381). 6. Search Console read (R-1035, 11-18). +**2026-11-14.** Before 11-17: say „send the bundle" to clear the yellow (D81). If not: 3 alarm mails. **2026-11-11.** 1. N100, as root, one time: `apparmor_parser -r -T -W /etc/apparmor.d/usr.sbin.chronyd && systemctl restart chrony && chronyc tracking` (R-1301; not tested). If not: about mid-January 2027 it refuses every agent update. 2. Hub Hosts page: Discard appliance `3fb5fed3…` (11-11 test box). 3. Rotate login `kisfenyo`'s Gitea token into the store as `GITEA_GIT_TOKEN` (R-925); the push after `sec put` is in R-1390. 4. Home DNS: remove the `felhom.eu` override (R-1389). 5. Read the dates only you can read (R-1381). 6. Search Console read (R-1035, 11-18). **Still needed? (yes/no)** — answer per line. 1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). (since 10-16) diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index c54a60ee..7d3cd66a 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -242,7 +242,7 @@ stopping line that lies. | **R-1373** | Storage & devices | P3 | **Three endings have a second door that does less than the first.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC13, EA08; read in source, not measured on a box). (1) **Orphan delete** of an app (`internal/stacks/delete.go:305-460`, `internal/api/router.go:1150-1178`) removes the containers, the volumes and the stack record; it does not remove the undo copy, the update record or the backup folders that the normal remove takes away. (2) **A move of ONE app to another drive** (`internal/stacks/migrate.go:616-654`) removes the app's data and its backup unit from the old drive; the app's restore copy, a set-aside backup and kept data stay there. Only the move of all apps clears the old drive. (3) **A drive retired by a signed operator job** (agent `internal/signedjobs/decommission.go:86-103`) sets the record and does not take the drive out of the running guest, which the household's own door does (`internal/localapi/disks.go:559-567`). **Members (answers marked N):** EC03-4, EC13-4, EA08-4. | OPEN — filed 2026-11-10 (endings list) | — | Controller: make orphan delete call the same clean-up steps as the normal remove; make a one-app move take the app's other folders with it, or list them. Agent: say in `03-host-agent.md` whether the signed retire is meant to leave the drive in the guest. | CC | | **R-1378** | Storage & devices | P3 | **Three endings can stop half-way with no finish, or have no way back that nobody chose.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC15, EC03, EC27; read in source, not measured). (1) **Wipe of a drive** (`internal/web/storage_handlers.go:911-1000`): the handler removes the drive from the registered places BEFORE the format. If the format fails, or the controller stops between the two steps, the drive is not registered and still holds the old data; there is no record of a wipe in progress, and a failed eject is only a warning. The handler does not look for apps on the drive. (2) **Orphan delete** writes no 'remove pending' record, so a controller that stops in the middle does not finish it at the next start (the normal remove does). (3) **Import over an installed app** (`internal/web/handler_export.go:312-356`, `internal/appexport/restore.go:288`) replaces the app's volumes and makes no undo copy first. **Members (answers marked N):** EC15-4, EC15-7, EC03-7, EC27-5. | OPEN — filed 2026-11-10 (endings list) | — | Controller: format first and unregister after, or write a wipe-in-progress record that the next start reads; give orphan delete the pending record; decide if an import makes an undo copy (a restore does). | CC | -## Security & access — 28 rows (P1 2, P2 5, P3 14, P4 7) +## Security & access — 29 rows (P1 2, P2 5, P3 15, P4 7) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -273,6 +273,7 @@ stopping line that lies. | **R-1194** | Security & access | P3 | **On a fresh box the operator's break-glass SSH door was broken from the first minute: `oob_degraded` „felhom-sshd config invalid (sshd -t fails)".** Rehearsal 2026-10-20, Tester4-816e21, mail at 12:03 UTC (one minute after the host record was created): details `active:false, config_invalid:true, felhom_sshd_port:0, operator_key_configured:false, operator_key_reported:true, reachable:false`. Possibly the shape of a customer with no operator key set (Tester4 is a test customer) — then the alarm is a false one on every such box; or a real config defect. Not diagnosed (no shell). Checked the same day: the only `oob_degraded` in the operator mailbox in 30 days, and no recovery mail followed; the report says port 0. Two candidate mechanisms in the agent (`internal/felhomsshd/health.go:63-66`): `sshd -t -f ` run by the NON-root agent fails on unreadable host keys, or a config rendered before the desired state arrived (port 0). **CAUSE FOUND AND FIXED 2026-10-21 night** (`audits/night-2026-10-21/P5-checks/`): reproduced on a fresh box from the public ISO — `oob_degraded` mailed 20:34 CEST while felhom-sshd was applied 0.4 s after the check and is healthy (port 8822 listening, `sshd -t` passes as the agent's user through its sudo grant — so the non-root theory is FALSE; without sudo it fails on the host key, which the agent never does). The first host report runs before the felhom-sshd loop's first pass and reads a config never rendered. Fix: agent `0da792e` — no OOB stanza until the first pass has run (the hub does not evaluate a missing one; a failed pass counts). `TestR1194_NoOOBStanzaBeforeTheFirstPass` red-proved. | WATCHING — released in agent v0.162.0 (2026-10-22, operator yes; on the 3 test boxes); the false alarm comes only on a FRESH box's first report | the next fresh install with agent ≥ 0.162.0 | At the next fresh install read the hub's events for `oob_degraded` in the first hour (none expected); then close | CC | | **R-1237** | Security & access | P3 | **The nightly CC-secrets sweep found stored one-time codes in a session record dated 2026-10-23: 14 names in one file (bind tokens and setup codes of test boxes — NIGHT_2026_10_23_*, FIXES_2026_10_21_*, NIGHT_2026_10_21_*, REHEARSAL_2026_10_20_* — and R_DEMO_FELHOM in a file of 2026-10-20).** Mail „[DooPlex CC secrets] 2 secret name(s) found in old session records", 2026-10-23 01:57 UTC, read in the day session (names only). One file holding all of them suggests one command printed many stored values at once (a listing or a dump) in the night session. All but R_DEMO_FELHOM are one-time install codes of test boxes; NIGHT_2026_10_23_* die when Tester4-16fdef is deleted (the operator's do list). | **OPEN** — 2026-10-27 (operator yes): the dead names were REMOVED from the store with `sec rm` (46, one commit each; `audits/r1299-window-button-2026-10-27/sec-rm.txt`, names only): every one-time code this row names is gone; R_DEMO_FELHOM stays (unsure). Consequence: the nightly sweep no longer watches for the removed values — they open nothing (their boxes are destroyed). The session transcripts themselves are unchanged — that question stays here. **Earlier:** OPEN | — | CC: find the file and the command by the sweep's own report (names only), add a hook shape for that command if it is not one; R_DEMO_FELHOM (demo-felhom's recovery code) — the operator decides whether to rotate it | CC | | **R-1248** | Security & access | P3 | **An app's own numbers can lie: an attacker with code inside an app's container can answer the app's API with whatever matches its flood, and the upload watch (R-1244) would then call it explained.** Measured remedy, not built: a per-container upload ceiling with `tc` from a helper container in the app's network namespace — on scratch 9202 (unprivileged) a flood was held at 19.9–20.1 Mbit/s and the app kept answering (`audits/SPIKE-upload-watch-2026-10-23.md` §5). Costs: a helper image with `tc` (a new external image, or `tc` in the controller image), `NET_ADMIN` for that helper, re-applied after each app restart. | OPEN | an operator decision (the new image / capability) | Decide; if yes, CC builds the ceiling for the download and media apps first | operator | +| **R-1391** | Security & access | P3 | **The hub, the controller and the agent are built on Go 1.27.3, which has GO-2026-6859 (an HTTP/1 server can crash with a runtime panic when a `MaxBytesReader` limit is hit while the handler reads the body and writes a header at the same time); fixed in Go 1.27.4.** Found 2026-11-14 (health-check session) from the security watch's 2026-11-11 mail, which names only the hub image (`golang 1.27.3-alpine3.24`); `go.mod` in all three repos reads `go 1.27.3`. Reach not measured: the crash needs a handler that reads the request body concurrently with writing the response header — whether any Felhom handler does is not read yet. A crash restarts the process (the controller's supervisor, the agent's unit, the hub's pod), so the effect is an outage of seconds, not data loss. Source: https://vuln.go.dev/ID/GO-2026-6859.json | OPEN — filed 2026-11-14 | — | build each component's next release on Go 1.27.4 (`go.mod` + the build image); until then, grep each repo for `MaxBytesReader`/`MaxBytesHandler` and a handler that writes headers while a goroutine reads the body | CC | | **R-1309** | Security & access | P4 | **What no automated source shows: a fix announced only by mail, forum or a blog we do not read; a security fix nobody calls one (cloudflared 2026.10.2's „bump distroless image"); closed-source apps (Plex, Emby).** The watch (R-1303) shows these only as BEHIND (weekly). The vendor mails the operator receives stay a source. `audits/security-watch-2026-10-26/SPIKE.md` | OPEN — filed 2026-10-26 (a stated limit) | — | none until a vendor offers a feed; add a vendor's feed to `security_watch.FEEDS` when one exists | CC | ## Box system & updates — 21 rows (P3 16, P4 5) @@ -303,7 +304,7 @@ stopping line that lies. | **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — 2026-10-27: agent v0.167.0 RELEASED; in the NORMAL signed delivery the pass now runs right after the commit on all three boxes („running the bundle policy pass after the self-update commit (R-1285)” → `BUNDLE POLICY DONE install_mode=appliance removed=0`; `audits/release-2026-10-27/agent-0167-tester1-demofelhom.txt`). Not live: a BYO box coming FROM 0.164.0 (none in the fleet). **Earlier:** **VERIFY** — 2026-10-26: agent `b1bc63d` (unreleased) — the new agent runs the bundle-policy pass once right after its self-update commits (`selfupdate.SetOnCommitted`); test in the real order (commit, then one pass), red-proved. **Not live:** only a box coming FROM 0.164.0 discriminates (0.165.0's wrapper already applies D43), which needs a signed downgrade chain on a BYO box — the live proof rides the next agent release. **Earlier:** **OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:** **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC | -## Monitoring & notifications — 28 rows (P2 3, P3 17, P4 8) +## Monitoring & notifications — 29 rows (P2 3, P3 18, P4 8) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -327,13 +328,14 @@ stopping line that lies. | **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC | | **R-1296** | Monitoring & notifications | P4 | **When ep0's tenancy script refuses, the hub log shows only its one-line error — PBS's own reason (the 403 `missing Datastore.Audit|Datastore.Backup on …`) stays in the script's stderr, which the hub drops on a refusal.** Found 2026-10-26 diagnosing R-1295: the cause was visible only by reading ep0's proxy journal over root SSH. `hub/internal/tenantsync/client.go` `parseResponse` returns the JSON error and ignores stderr; the script keeps secrets off stderr by design (its header), so the tail is safe to log. | OPEN | — | append the last ~3 stderr lines to the refusal error (bounded, never stdout); a test that a secret on stdout never appears | CC | | **R-1328** | Monitoring & notifications | P2 | **DooPlex's own workloads were not in the security watch at all** — about ten public apps there carry a bad Brotli/Node (HTTP2Shell), on the host that runs the hub, Gitea, the registry and the backups. | **VERIFY** — BUILT 2026-10-27 night (commit only): `scripts/dooplex_inventory.py` lists every `image:` of homelab-manifests (89 images, `documentation/security-watch/dooplex-workloads.txt`; NOT Helm-chart images — said in the file); the watch reads them as the group „dooplex" (operator mails only, own block, „Next: the homelab session"); an image in the private registry is NOT CHECKED with the reason. Tests W12, red-proved. The homelab session's list: `documentation/audits/http2shell-2026-10-27/HOMELAB-LIST.md`. Felhom changes nothing on DooPlex. | — | the first CI night shows the DooPlex block; the homelab session works its list. 2026-10-28: `minio/minio:latest` answers HTTP 401 on Docker Hub (the image is no longer published there) — for the homelab session: the pin cannot be checked anonymously any more | CC | -| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | WATCHING — **released 2026-10-29:** agent 0.168.0 (vouched 10:21:56 CET) + controller 0.347.0 (the global floor). Proven on the release on Tester 1: a hog in a 128 MB Gokapi — host counter 1, one `app_oom` „seen by docker,cgroup", one operator mail (`audits/d63-2026-10-29/run-p4-oom.log`; the repeated WARN is R-1341). Closes at the dated check 2026-11-12 if it holds. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC | +| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | WATCHING — **2026-11-14 dated check read (health-check session, read only; hub DB copy with its WAL, newest host report 7 min old): NOT closed — one failed `oom_check`.** demo-felhom-8363b5, 2026-11-11 04:40:30 UTC, night Docker step (Docker 29.9.3, containerd 2.4.2-3, outcome „nothing”): exit 137 but `OOMKilled=false` and no `oom` event — the engine missed the test kill. Every other `oom_check` since 2026-10-12 passed (demo-felhom 33, demo-hp 23; 2026-11-12 onward on containerd 2.4.2-4 all pass). The four `app_oom` events since 2026-10-12 are all explained: Tester4 2026-10-20 (the R-1192/R-1193 freeze), Tester 1 2026-10-28 radicale (the R-528 squeeze test, `audits/night-2026-10-28/r528-oom/`) and 2026-10-29 gokapi ×2 (the release proof). So the household alarm path (host cgroup counter) holds, and Docker’s own signal still drops a kill now and then — the original finding. **Date moved to 2026-11-28:** two more weeks of night `oom_check`s; close if no further fail, else decide whether the sentinel should judge on the cgroup counter. WATCHING — **released 2026-10-29:** agent 0.168.0 (vouched 10:21:56 CET) + controller 0.347.0 (the global floor). Proven on the release on Tester 1: a hog in a 128 MB Gokapi — host counter 1, one `app_oom` „seen by docker,cgroup", one operator mail (`audits/d63-2026-10-29/run-p4-oom.log`; the repeated WARN is R-1341). Closes at the dated check 2026-11-12 if it holds. | — | 2026-11-28: read the hub (`os_reports.report_json` → `oom_check`, `events` → `app_oom`) for any failed `oom_check` after 2026-11-11 or an `app_oom` with no restart reason; none → close | CC | | **R-1361** | Monitoring & notifications | P2 | **An event the box sends while the hub cannot be reached is lost: 3 tries over 9 s, then dropped.** Walked 2026-11-03 on Tester 1 (hub blackholed 13 min): the box ran normally, an app install worked, a crash loop stopped the app and held it, and `app_stopped_unhealthy` and `app_held` failed after 3 attempts (`notifier.go`). Afterwards NOTHING was re-sent: the hub's log shows no event from the outage, its „Held apps" had no row (`a row existed: false`), and the hold reporter had already recorded the app as reported (`holdreport.go`, the push is asynchronous) so only its periodic re-send can repair it. Any DooPlex restart (2026-11-01 was one) can swallow a household's and the operator's mails this way. `audits/security-2026-11-02/part5/outage-hub.log` **2026-11-11:** the Clock list's class „nobody is told” (R-1383) is beside this row; it does not repeat it. | OPEN — filed 2026-11-03 **2026-11-03, a second cost seen:** the walk's cut (01:17–01:39 UTC) covered Tester 1's 01:30 dump, its `db_dump_completed` was lost, and at 04:00 UTC the hub mailed `expected_dbdump_missed` „No DB dump completed or failed since midnight", while the box's own report read `last_db_dump` 03:15 UTC. A lost event becomes a false alarm. | — | a small on-disk outbox for events, sent in order when the hub answers (a design: which types, how long kept); the hold reporter marks „reported" only after a 2xx. Test red first | CC | | **R-1363** | Monitoring & notifications | P3 | **A household's mail ends with raw JSON: `Megjegyzés: {"avail_gb":5.192043304443359,"band":"warning",...}`.** Seen in the walks of 2026-11-02 (`audits/security-2026-11-02/part5/mail-fill-household.txt`, `mail-crash-household.txt`). `api/handler.go` adds the details as the „Megjegyzés" line; a golden pins it (`mail_golden_test.go`), so it was once wanted. The sentence itself is right and Hungarian. | OPEN — filed 2026-11-03 | — | drop the line from a household's mail, or render known fields in words; the operator's mail keeps it | CC | | **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) | | **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC | | **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC | | **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC | +| **R-1392** | Monitoring & notifications | P3 | **Every agent release whose `configs/` did not change turns every box's „Root files” amber, and after 7 days the hub mails `os_config_bundle_behind` for each box, although the files on the box are the vouched ones.** Found 2026-11-14 (health-check session): the System page shows „root files behind the vouched agent's” on demo-felhom, demo-hp and Tester 1 (bundle 0.172.1, sha `3e7acaee…`); the vouched agent 0.173.0 carries bundle sha `51ebc1f4…`; `git diff v0.172.1 v0.173.0 -- configs/` is EMPTY, and the 0.173.0 release delivered no bundle job on purpose („`configs/` did not change”, felhom-agent CHANGELOG). The hub compares the WHOLE bundle's sha256 (`hub/internal/web/system.go` `bundleCell`, `hub/internal/osupdates/service.go` ~line 1182), and the bundle carries its version, so the sha moves with every release. Vouched 2026-11-10 16:29 CET → the alarm is due about 2026-11-17 on all three boxes. **Two ways out:** (A) the release sends a signed `agent_config_update` every time, even with no file change (cheap, one job per box, keeps the hub as is); (B) the hub compares the files (the box reports `config_bundle.live` per-file sha256s) with the vouched bundle's files, not the bundle's own sha. | OPEN — filed 2026-11-14 | operator: A or B | today: send the 0.173.0 bundle to the three boxes (clears the amber and the 11-17 alarm); then A in `felhom-build-deploy` or B in the hub | CC | | **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC | ## Hub & operator — 17 rows (P2 1, P3 5, P4 11) @@ -424,7 +426,7 @@ stopping line that lies. | R-1293 | 2027-03-29 | the spring clock change: the hub's 02:00 snapshot and the 02:30 push on 2027-03-28 (02:00–03:00 does not exist) — which file each push took; detail in the R-1293 row | | R-1050 | 2026-11-15 | HSTS one year if no https trouble since 2026-10-15; detail in the R-1050 row | | R-1035 | 2026-11-18 | „felhő tárhely” + „saját felhő tárhely” impressions and average position, and the new page's own impressions/clicks/position, 4 weeks after publishing; operator supplies the read; detail in the R-1035 row | -| R-528 | 2026-11-12 | no failed oom_check / unexplained app_oom since 2026-10-12; detail in the R-528 row | +| R-528 | 2026-11-28 | no further failed oom_check after the one on 2026-11-11 (demo-felhom), no unexplained app_oom; detail in the R-528 row | | R-1035 | 2026-12-09 | the same three numbers 8 weeks after publishing, compared with the 4-week read; a null at 4 weeks is not a verdict, both reads are needed; detail in the R-1035 row | | **R-1297** | Process & tooling | P3 | **CI waits dominate a working day: felhom.eu's CI run takes ~13 min and the ONE runner makes every repo's job wait behind it (up to 10–17 min queued).** Measured 2026-10-26 (operator: „these CI waiting times are getting out of hand"): since 10-25 felhom.eu runs median 793 s, the controller/agent/catalog 57–102 s; queue median 5 s, max 548–1023 s. In run 2366, ~10 of 13 minutes are four suites — `test_gate_decoys.py` 232 s, `test_due_checks_gate.py` 133 s, `test_repo_gates.py` 132 s (it runs the whole gate set a SECOND time), `dooplex-offsite/test_dooplex_offsite.py` 69 s — plus `hub-gofmt` ~110 s (a cold Go cache per run). **Ruled 2026-10-26 by the operator: BOTH** — (A) trim felhom.eu CI: the heavy suites only when their own files change, no duplicate full run, a Go cache for hub-gofmt; (B) a second CI runner on DooPlex. | **OPEN — (A) BUILT 2026-10-26 evening, closes after the first nightly run (2026-10-27 03:20 UTC) is read.** Both runners proven at once (jobs 2374 + 2375 on `felhom-gates-runner` / `-2`, 17:11:41Z). felhom.eu CI, one push each: **(a) docs only** 810 s (old runner) → 351 s (2 CPUs) → see the close-out; **(b) a gate script** 351 → 203 s (`ecd82aba`, job 2376); **(c) a hub change** 351 → 118 s (`7e57e824`, job 2378). Heavy suites by path + tag + nightly; `test_repo_gates.py` and due-checks group G test the wiring (130 s each → under 3 s). Decoys through CI's entry point with the filter ON: a docs-only register break → red; an inert gate → GREEN at first (a decoy hole, fixed in `e33ec821`) → red. No Go cache: CI has no Go (the 110 s was security-scan, now 27 s). `audits/r1297-ci-trim-2026-10-26/`. (B) DONE by the operator (`09` §3 decision 267). | — | (A) CC, a run of its own (a CI scope change, ruled); (B) a DooPlex change — a brief or the homelab session; and CC waits on CI only before a release or at the end of a run | CC | diff --git a/documentation/backlog/STATUS-HISTORY.md b/documentation/backlog/STATUS-HISTORY.md index 80f75efe..1d997a81 100644 --- a/documentation/backlog/STATUS-HISTORY.md +++ b/documentation/backlog/STATUS-HISTORY.md @@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S **Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.** +## Moved from STATUS on 2026-11-14 (morning) + +**Updated 2026-11-11 (afternoon).** Golden 0.352.0 is vouched, so code pushes to `felhom.eu` work again. The N100 keeps no time: one hand step repairs it (below). A new Proxmox certificate blinds the agent; the hand repair is proven on demo-hp and one decision waits (D80). + ## Moved from STATUS on 2026-11-11 (midday) **Updated 2026-11-10 (evening).** Three releases are live: the agent that cleans its own update folder (you vouched it), the controller that sends one warning for one full disk (every box takes it by itself), and the hub that sends the host-disk alarm to you only. You ruled D71–D75 at 15:19 and none is built; the smallest system disk is 200 GB, and what the installer does below 200 GB (refuse, or warn) is open (R-1370). The report of the run is `REPORT-release-space-fixes-2026-11-10.md`.