Release 2a: agent 0.174.0 vouched — old kernels removed by the box (11 §5.11.1 built), security profiles refreshed after a kernel change (11 §5.12, spike + build); R-1388, R-1301, R-1391 closed; R-1394, R-1395 filed; D82
gates / gates (push) Successful in 2m1s

This commit is contained in:
admin committed 2026-11-14 15:30:44 +01:00
1 parent a2173830c1
commit 8955b4e55f
51 files changed
+1385 -19

No files matched your search

+10
View File
@@ -26,6 +26,16 @@
---
## 2026-11-14 — release 2a: R-1388, R-1301, R-1391 closed (full text of the rows: `git show a2173830:documentation/backlog/OPEN-ITEMS.md`)
| Row | What | Closed | Evidence |
|---|---|---|---|
| **R-1388** | **After a kernel went back and forth, a box could load an old security profile for an updated package — on the N100 this stopped the time service for good.** (P3) | CLOSED 2026-11-14 — agent 0.174.0 (vouched 15:21:45 CET): the root wrapper's mode `profile-refresh` loads every AppArmor profile from a fresh compile once per kernel the box starts on, and restarts only a failed service whose own program's cached profile was stale. Spiked first on demo-hp with both guests running (the fault reproduced with one planted stale profile; 0.5 s; no container touched). Ran by the product's own path on Tester 1, demo-hp and the N100 | `audits/profile-refresh-spike-2026-11-14/`; `audits/release-2a-2026-11-14/` `B/`, `D/06`, `D/13`; `11-os-updates.md` §5.12; felhom-agent `cc63dd5` |
| **R-1301** | **The N100's Proxmox host kept no time: `chrony` failed at every boot since 2026-10-16.** (P3) | CLOSED 2026-11-14 — repaired by the product, no hand step (`09` §3 decision 305): when the N100 took agent 0.174.0's bundle, the profile refresh rewrote the stale chrony profile and restarted `chrony.service`; the clock stepped +32.717 s. Offset to DooPlex −32.55 s before, +0.18 s after and 2.5 hours later; chrony active, 0.0002 s off | `audits/release-2a-2026-11-14/D/09-n100-before.txt`, `D/13-n100-after.txt` |
| **R-1391** | **The hub, the controller and the agent were built on Go 1.27.3, which has GO-2026-6859.** (P3) | CLOSED 2026-11-14 — all three on Go 1.27.4: controller 0.352.1 and hub 0.181.0 (release 1), agent 0.174.0 (the running binary reads `go1.27.4` on Tester 1, demo-hp and the N100). No handler of the three matched the advisory's crash pattern | `audits/release-1-dates-2026-11-14/`; `audits/release-2a-2026-11-14/D/17-go-version-on-boxes.txt` |
**Reasoning kept (R-1388).** A stale copy is told by the CACHED FILE changing when it is rewritten — never by the kernel's loaded hash, which differs between a cache load and a fresh load of the same policy for a file with several profiles. The refresh runs once per kernel, not at every agent start: each run floods the kernel log and hides other audit lines for a few seconds.
## 2026-11-14 — R-1392 closed: root files are „behind” only when the files differ (full text of the row: `git show 561a4d98:documentation/backlog/OPEN-ITEMS.md`)
| Row | What | Closed | Evidence |
+8 -9
View File
@@ -242,7 +242,7 @@ stopping line that lies.
| **R-1373** | Storage & devices | P3 | **Three endings have a second door that does less than the first.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC13, EA08; read in source, not measured on a box). (1) **Orphan delete** of an app (`internal/stacks/delete.go:305-460`, `internal/api/router.go:1150-1178`) removes the containers, the volumes and the stack record; it does not remove the undo copy, the update record or the backup folders that the normal remove takes away. (2) **A move of ONE app to another drive** (`internal/stacks/migrate.go:616-654`) removes the app's data and its backup unit from the old drive; the app's restore copy, a set-aside backup and kept data stay there. Only the move of all apps clears the old drive. (3) **A drive retired by a signed operator job** (agent `internal/signedjobs/decommission.go:86-103`) sets the record and does not take the drive out of the running guest, which the household's own door does (`internal/localapi/disks.go:559-567`). **Members (answers marked N):** EC03-4, EC13-4, EA08-4. | OPEN — filed 2026-11-10 (endings list) | — | Controller: make orphan delete call the same clean-up steps as the normal remove; make a one-app move take the app's other folders with it, or list them. Agent: say in `03-host-agent.md` whether the signed retire is meant to leave the drive in the guest. | CC |
| **R-1378** | Storage & devices | P3 | **Three endings can stop half-way with no finish, or have no way back that nobody chose.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC15, EC03, EC27; read in source, not measured). (1) **Wipe of a drive** (`internal/web/storage_handlers.go:911-1000`): the handler removes the drive from the registered places BEFORE the format. If the format fails, or the controller stops between the two steps, the drive is not registered and still holds the old data; there is no record of a wipe in progress, and a failed eject is only a warning. The handler does not look for apps on the drive. (2) **Orphan delete** writes no 'remove pending' record, so a controller that stops in the middle does not finish it at the next start (the normal remove does). (3) **Import over an installed app** (`internal/web/handler_export.go:312-356`, `internal/appexport/restore.go:288`) replaces the app's volumes and makes no undo copy first. **Members (answers marked N):** EC15-4, EC15-7, EC03-7, EC27-5. | OPEN — filed 2026-11-10 (endings list) | — | Controller: format first and unregister after, or write a wipe-in-progress record that the next start reads; give orphan delete the pending record; decide if an import makes an undo copy (a restore does). | CC |
## Security & access — 29 rows (P1 2, P2 5, P3 15, P4 7)
## Security & access — 28 rows (P1 2, P2 5, P3 14, P4 7)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -273,17 +273,15 @@ stopping line that lies.
| **R-1194** | Security & access | P3 | **On a fresh box the operator's break-glass SSH door was broken from the first minute: `oob_degraded` „felhom-sshd config invalid (sshd -t fails)".** Rehearsal 2026-10-20, Tester4-816e21, mail at 12:03 UTC (one minute after the host record was created): details `active:false, config_invalid:true, felhom_sshd_port:0, operator_key_configured:false, operator_key_reported:true, reachable:false`. Possibly the shape of a customer with no operator key set (Tester4 is a test customer) — then the alarm is a false one on every such box; or a real config defect. Not diagnosed (no shell). Checked the same day: the only `oob_degraded` in the operator mailbox in 30 days, and no recovery mail followed; the report says port 0. Two candidate mechanisms in the agent (`internal/felhomsshd/health.go:63-66`): `sshd -t -f <conf>` run by the NON-root agent fails on unreadable host keys, or a config rendered before the desired state arrived (port 0). **CAUSE FOUND AND FIXED 2026-10-21 night** (`audits/night-2026-10-21/P5-checks/`): reproduced on a fresh box from the public ISO — `oob_degraded` mailed 20:34 CEST while felhom-sshd was applied 0.4 s after the check and is healthy (port 8822 listening, `sshd -t` passes as the agent's user through its sudo grant — so the non-root theory is FALSE; without sudo it fails on the host key, which the agent never does). The first host report runs before the felhom-sshd loop's first pass and reads a config never rendered. Fix: agent `0da792e` — no OOB stanza until the first pass has run (the hub does not evaluate a missing one; a failed pass counts). `TestR1194_NoOOBStanzaBeforeTheFirstPass` red-proved. | WATCHING — released in agent v0.162.0 (2026-10-22, operator yes; on the 3 test boxes); the false alarm comes only on a FRESH box's first report | the next fresh install with agent ≥ 0.162.0 | At the next fresh install read the hub's events for `oob_degraded` in the first hour (none expected); then close | CC |
| **R-1237** | Security & access | P3 | **The nightly CC-secrets sweep found stored one-time codes in a session record dated 2026-10-23: 14 names in one file (bind tokens and setup codes of test boxes — NIGHT_2026_10_23_*, FIXES_2026_10_21_*, NIGHT_2026_10_21_*, REHEARSAL_2026_10_20_* — and R_DEMO_FELHOM in a file of 2026-10-20).** Mail „[DooPlex CC secrets] 2 secret name(s) found in old session records", 2026-10-23 01:57 UTC, read in the day session (names only). One file holding all of them suggests one command printed many stored values at once (a listing or a dump) in the night session. All but R_DEMO_FELHOM are one-time install codes of test boxes; NIGHT_2026_10_23_* die when Tester4-16fdef is deleted (the operator's do list). | **OPEN** — 2026-10-27 (operator yes): the dead names were REMOVED from the store with `sec rm` (46, one commit each; `audits/r1299-window-button-2026-10-27/sec-rm.txt`, names only): every one-time code this row names is gone; R_DEMO_FELHOM stays (unsure). Consequence: the nightly sweep no longer watches for the removed values — they open nothing (their boxes are destroyed). The session transcripts themselves are unchanged — that question stays here. **Earlier:** OPEN | — | CC: find the file and the command by the sweep's own report (names only), add a hook shape for that command if it is not one; R_DEMO_FELHOM (demo-felhom's recovery code) — the operator decides whether to rotate it | CC |
| **R-1248** | Security & access | P3 | **An app's own numbers can lie: an attacker with code inside an app's container can answer the app's API with whatever matches its flood, and the upload watch (R-1244) would then call it explained.** Measured remedy, not built: a per-container upload ceiling with `tc` from a helper container in the app's network namespace — on scratch 9202 (unprivileged) a flood was held at 19.9–20.1 Mbit/s and the app kept answering (`audits/SPIKE-upload-watch-2026-10-23.md` §5). Costs: a helper image with `tc` (a new external image, or `tc` in the controller image), `NET_ADMIN` for that helper, re-applied after each app restart. | OPEN | an operator decision (the new image / capability) | Decide; if yes, CC builds the ceiling for the download and media apps first | operator |
| **R-1391** | Security & access | P3 | **The hub, the controller and the agent are built on Go 1.27.3, which has GO-2026-6859 (an HTTP/1 server can crash with a runtime panic when a `MaxBytesReader` limit is hit while the handler reads the body and writes a header at the same time); fixed in Go 1.27.4.** Found 2026-11-14 (health-check session) from the security watch's 2026-11-11 mail, which names only the hub image (`golang 1.27.3-alpine3.24`); `go.mod` in all three repos reads `go 1.27.3`. Reach not measured: the crash needs a handler that reads the request body concurrently with writing the response header — whether any Felhom handler does is not read yet. A crash restarts the process (the controller's supervisor, the agent's unit, the hub's pod), so the effect is an outage of seconds, not data loss. Source: https://vuln.go.dev/ID/GO-2026-6859.json | NARROWED — **2026-11-14: the controller (0.352.1, global floor 09:21 UTC, on every box that reports) and the hub (0.181.0, live 09:58 UTC) are built on Go 1.27.4. LEFT: the agent (still 1.27.3), release 2.** Reach read the same day: no controller handler matches the crash pattern (seven `MaxBytesReader` places, each reads the body to its end before it answers; no `TimeoutHandler`); the hub has no `MaxBytesReader` at all; the agent has ONE (`internal/localapi/server.go`, a JSON decode in the handler's own goroutine — not read further). So the agent's turn is hygiene, not urgent. Evidence `audits/release-1-dates-2026-11-14/`. Earlier: OPEN — filed 2026-11-14 | — | build each component's next release on Go 1.27.4 (`go.mod` + the build image); until then, grep each repo for `MaxBytesReader`/`MaxBytesHandler` and a handler that writes headers while a goroutine reads the body | CC |
| **R-1309** | Security & access | P4 | **What no automated source shows: a fix announced only by mail, forum or a blog we do not read; a security fix nobody calls one (cloudflared 2026.10.2's „bump distroless image"); closed-source apps (Plex, Emby).** The watch (R-1303) shows these only as BEHIND (weekly). The vendor mails the operator receives stay a source. `audits/security-watch-2026-10-26/SPIKE.md` | OPEN — filed 2026-10-26 (a stated limit) | — | none until a vendor offers a feed; add a vendor's feed to `security_watch.FEEDS` when one exists | CC |
## Box system & updates — 21 rows (P3 16, P4 5)
## Box system & updates — 20 rows (P3 14, P4 6)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` **2026-11-11:** the Clock list measured it again (32.6 s behind, time service not running) and leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CA03-6; class R-1384). **2026-11-11 (afternoon, read only): the cause is found.** The kernel loaded an OLD cached AppArmor profile for chrony (the one of chrony 4.6.1); chrony 4.8 writes to `/run/systemd/notify`, the old profile refuses that write, chrony treats it as fatal. Proof: loaded profile sha256 `dbf67f16…` against a fresh compile `83417e83…` on the N100, equal on demo-hp; the kernel's `DENIED` line of 2026-10-16. The order that made it: a new kernel started once (2026-10-07 12:11), the box went back to the old kernel, chrony was updated there (14:15), the new kernel started again and used its stale cache folder (`documentation/audits/n100-clock-2026-11-11/README.md`). The product side is R-1388. The hand step is in `STATUS.md` „do”; it was not run and not tested. | READY — **2026-11-14 09:55 — RULED (`09` §3 decision 305): no hand repair now; release 2 repairs it through the product. If release 2 is not live by 2026-12-15, the operator runs the hand step.** OPEN — filed 2026-10-26 **2026-11-10:** still so. The host clock is 32 s behind DooPlex (it was 14 s on 2026-10-26); `chrony` failed again at the 2026-11-08 05:35 boot with the same line; `NTPSynchronized=no`. Seen in a release run, where the box's log times read half a minute early (`documentation/audits/release-space-fixes-2026-11-10/agent/13-n100-clock.txt`). Not fixed by hand: the N100 takes no hand step in a release run. | — | the operator runs the one hand step on the N100 (`STATUS.md`); then read `chronyc tracking` and close. The product fix is R-1388 | CC |
| **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CC02, CC06-6, CC07-6, CC22-6). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC |
| **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H7: the N100 (4.28 GB, 74 files) and Tester 1 (4.29 GB, 81 files) sit at the journal's own cap; Tester 1 writes about 150 MB a day. Lines written by `sudo` are 92 / 92 / 90 % of all lines in one hour — this counts three lines for each call (the command and the session pair), while this row's 61 % counts the session pair only: the same fact, counted two ways. No rank change is proposed: the journal stops at its own cap (about 4.3 GB), so it costs space and log history, not safety. | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC |
| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. **2026-11-10 (release run) — part (2) RELEASED, VOUCHED and PROVEN on three test boxes** (agent 0.173.0, vouched by the operator 15:29 UTC; `documentation/audits/release-space-fixes-2026-11-10/agent/`): the folder went 49 → 1, 48 → 1 and 33 → 1 files (about 700 / 690 / 485 MB back) at the commit of THIS update, not at the next one (the earlier sentence here and in the agent's CHANGELOG was wrong, and is corrected there). A rollback after the clean-up worked on Tester 1: a signed update to the older version was crashed once, 9 seconds into the 60-second dwell, and the wrapper put the released version back in the same second; the failed version's file stays until the next good update (by design). The kept older kernel on demo-hp booted to healthy in 313 s and the box went back (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). **Still open:** (1) the kernel clean-up as a program (`11-os-updates.md` §5.11.1, not built) and a second box; (3) template archives; (4) the `.bak-*` copies — the release does not reach them; (5) the failed-run logs. A household box gets part (2) at its first good agent update after the vouch. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC |
| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | NARROWED — **2026-11-14 (release 2a): the old-kernel part is BUILT — agent 0.174.0, vouched 15:21 CET (`11` §5.11.1 `[FACT — built]`).** A ring-0 box removes its old kernels at night by the wrapper's own rule. On boxes: a real removal on Tester 1 (3 → 2 kernels, 1.12 GB, 8.5 s; the default, the pinned file, the generators and the state file unchanged; a second run removed nothing; one clean restart) — started by hand with a ring-0 mark set for the test on the operator's yes, because a night cannot be started by day (R-1394); dry runs on demo-hp (nothing to remove; refusals P1, P2, P3) and the N100 (6 of 8 will go). **LEFT for this part: the step by the agent at night — read the N100's and demo-hp's night of 2026-11-15; then a ruling for ring-1 boxes (D82).** Evidence `audits/release-2a-2026-11-14/`. The other four places of this row are unchanged. Earlier: OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. **2026-11-10 (release run) — part (2) RELEASED, VOUCHED and PROVEN on three test boxes** (agent 0.173.0, vouched by the operator 15:29 UTC; `documentation/audits/release-space-fixes-2026-11-10/agent/`): the folder went 49 → 1, 48 → 1 and 33 → 1 files (about 700 / 690 / 485 MB back) at the commit of THIS update, not at the next one (the earlier sentence here and in the agent's CHANGELOG was wrong, and is corrected there). A rollback after the clean-up worked on Tester 1: a signed update to the older version was crashed once, 9 seconds into the 60-second dwell, and the wrapper put the released version back in the same second; the failed version's file stays until the next good update (by design). The kept older kernel on demo-hp booted to healthy in 313 s and the box went back (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallbLine truncated
| **R-1369** | Box system & updates | P3 | **When a disk is full there is no written way out: no runbook and no self-repair, for the host system disk, the thin pool, the guest disks or a data drive.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 7). One missing runbook, seen from 21 places; the search of the runbooks folder for a disk-full procedure gave 0 hits (`evidence/searches.txt` S6). The alarms exist (hub: 90 / 95 % host disk, 85 / 90 % thin pool; controller: 85 / 95 % on the data volume and each drive) but they end in a mail, not in a step. For three small places nobody wrote what fails when they are full (EFI partition, host and guest temp folders). **Members (answers marked N):** H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7. | OPEN — filed 2026-11-10 (space list) **2026-11-10 (afternoon) — the runbook is written:** `documentation/runbooks/disk-full.md`, five parts, each hand step marked PROVEN (date and evidence; all on demo-hp) or NOT PROVEN. Still open: one rehearsal of the whole runbook on demo-hp; the NOT PROVEN steps (copies on a former target, template archives, the `.bak-*` binaries, the guest root disk); what fails when the EFI partition or a temp folder is full. | — | Docs: one runbook „a disk is full” with one section per disk (host system disk, thin pool, guest root, data volume, data drive): what to look at first, what is safe to delete (the places of R-1368 and R-1367), what is never deleted by hand (household data, the kept-data rulings). Rehearse it once on demo-hp. | CC |
| **R-1370** | Box system & updates | P3 | **The host's system disk has no space budget: no document says how much the host's own files may take, and the installer's sizing rule covers the guest only.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` place H1, answer H1-3; search S5 — 0 hits). Today: demo-hp 58 % of 41 GB, N100 38 % of 101 GB, Tester 1 34 % of 63 GB — and the parts that grow are the journal (R-1367), old kernels and agent binaries (R-1368), old backup copies (R-1087) and the local whole-guest copy (R-548). Second half, known but with no row until now: on a small disk the guest's own guards read the guest's 70 GiB while the thin pool behind it holds 54 GiB (demo-hp: 102 GiB promised on a 53.9 GiB pool, 70.67 % used), so only the hub's thin-pool alarm stands before a full pool. **Member:** H1-3. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D69 = yes; `09` §3 decision 288):** the section goes inside `16-storage-physical-identity.md`, no new document. **2026-11-10 (afternoon) — the section is written:** `16-storage-physical-identity.md` §14 (45 places, the list „no limit, by ruling”, the budget, the open design points); `PROMPT-TEMPLATE.md` §N.5 keeps it true. The budget from measured numbers: the root needs about 14 GB when the whole-box copies go to a backup drive, and 28–54 GB at keep 3 when they do not. **D75 (for the operator):** the smallest system disk Felhom accepts. CC's pick: 128 GB only with a backup drive, 200 GB without one. The other option, 128 GB always, lets a box with many apps and no backup drive stop its local copies. With no ruling the installer accepts any disk. Open after the ruling: the installer's refusal. **2026-11-10 15:19 — RULED (D75 = one number, 200 GB, with or without a backup drive; `09` §3 decision 295).** This replaces CC's two-step pick. **Open, not ruled:** what the installer does below 200 GB — refuse, or warn. demo-hp (128 GB) stays as a test box. `16` §14.3 carries the ruling. | — | Design first: what the installer does below 200 GB (refuse, or warn) — a later session. Then the installer. | CC |
| **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC |
@@ -296,12 +294,12 @@ stopping line that lies.
| **R-1358** | Box system & updates | P3 | **cloudflared 2026.10.4 misses two dependency fixes (VULN-183456 chi, 2026.10.5; VULN-184686 go-jose, 2026.10.6; the newest is 2026.10.7).** No public text on reach. Not pinned tonight: the tunnel is a box's only door, and 9202 has none to prove it on. | OPEN — filed 2026-11-03 | — | raise `CloudflaredImage` on the next controller release with a tunnel proof on Tester 1 | CC |
| **R-1380** | Box system & updates | P3 | **Proxmox renews its own certificate and the agent's pin does not follow: from that day the agent cannot control or back up a guest (5 answers).** Found 2026-11-11 (the End-dates list, class T2, `documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB02-3 to DB02-7). MEASURED: the agent trusts one exact Proxmox certificate (a pin), and the certificate on the three test boxes ends about two years after install — the N100's about 2028-07-16, then demo-hp and Tester 1 within three months. Proxmox makes a new certificate by itself about 14 days before the old one ends (so about 2028-07-02 on the N100). No code path gives the agent the new fingerprint, no alarm names the cause, no runbook holds the hand step, and nobody has tried it. Every box installed in the same season fails in the same fortnight. P3 only because the first date is in 2028; it is certain, not a risk. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 297). Not built.** **2026-11-11 (afternoon): tried on demo-hp (`documentation/audits/certpin-spike-2026-11-11/README.md`).** A forced new certificate (`pvecm updatecerts --force`, then a reload of the web service; Proxmox's own renewal does that reload by itself) blinds the agent at once: `TLS pin mismatch`, no reconcile, no host report to the hub, 502 to the controller; guests and apps stay up. One operator mail names a symptom (`agent_channel_unknown`), none names the cause; the hub's Hosts page still read ONLINE eleven minutes after the last report (when it turns offline was not measured). The hand repair works and is written down (`documentation/runbooks/RUNBOOK-agent-proxmox-pin-repair.md`, 2.1 s from restart to a working call). **A CA file in place of the fingerprint works with no agent code:** with `proxmox.tls.ca_file` on a readable copy of the box's root CA and the fingerprint empty, the agent went through a second new certificate with no hand step; a wrong CA file was refused. Two facts for the fix: the agent's user cannot read Proxmox's certificate files, and a set fingerprint wins over the CA file. The design with both ways is `03-host-agent.md` §12a; CC's pick differs from the ruled shape, so it is decision D80. | READY — **2026-11-14 09:55 — RULED (D80, `09` §3 decision 303): the root CA copy.** Built in release 2 (the host release). OPEN — filed 2026-11-11 (end-dates list) | — | the operator rules D80 (CA file, or follow the fingerprint); then the design's open parts in `03-host-agent.md` §12a (the root step that refreshes the copy; an alarm that names the cause), then build, then prove on a scratch host with a forced renewal | CC |
| **R-1382** | Box system & updates | P3 | **A box that was off past an end date: nobody knows if it comes back by itself (5 answers).** Found 2026-11-11 (the End-dates list, class T4, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; read in source and documents, never tried). After months without power: the web certificate has ended (DB01-6: does Traefik renew at the first start, and what does the household see until then), Debian's package keys may be too old to verify an update (DB16-6), the tunnel client may be below what Cloudflare accepts (DB19-6), a 30-day restore copy is past its date (DB24-6), and the 14-day countdown after a stopped off-site copy ran out while the box was off (DB04-6). Tester 2 has been off since 2026-10-04 and is the natural test. | OPEN — filed 2026-11-11 (end-dates list) | — | When Tester 2 comes back (or with a scratch host and a moved clock on Tier 0, as a task): record what renews by itself and what needs a hand; put the hand steps in a runbook. | CC |
| **R-1384** | Box system & updates | P3 | **A verdict or an „already ran” check trusts a clock that can be wrong (30 answers).** Found 2026-11-11 (the Clock list, class W2, `documentation/audits/clock-list-2026-11-11/TABLE.md`). MEASURED: the N100's host is 32.6 s behind and its time service does not run (R-1301); the other two boxes are in step. Nothing checks a box's clock except one hub alarm for a tunnel handshake stamped in the future. The hub takes the box's own time for backup age, off-site age and proof age (CH02-6, CH07-6, CH07-8, CH08-6); a box with a wrong clock refuses every signed job (CA09-6); a stamp from the future blocks the OS-update leg, the trim and the line test until the clock passes it (CA02-6, CA02-8, CA11-6, CA11-8, CA12-6); the crash guard reads the clock at early boot (CA04-6, CA04-8); a hub clock jump moves its own daily jobs (CH03-6, CH04-6, CH06-6, CH09-6, CH16-6, CH18-6, CH20-6, CH24-6). The others: CA05-6, CA05-8, CA06-6, CA07-6, CA10-6, CA20-6, CC04-6, CC05-6, CC14-6, CC18-6. Known parts stay in their rows: R-1189, R-1206, R-1260. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 300). Not built.** **2026-11-11 (afternoon): why the N100's clock is wrong is known (R-1301, `documentation/audits/n100-clock-2026-11-11/README.md`).** Numbers from source: a host clock more than 120 s behind refuses every signed job (`felhom-agent/internal/authz/verifier.go:21`, `:153-162`); at about 1.25 s a day the N100 gets there in about 70 days (mid-January 2027, an estimate from one measurement). | OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): the hub compares the time in each report with its own, alarms the operator above one minute, and does not take a time from the future as „fresh”. | CC |
| **R-1388** | Box system & updates | P3 | **After a kernel went back and forth, the box can load an old security profile for an updated package — on the N100 this stopped the time service for good.** Found 2026-11-11 (`documentation/audits/n100-clock-2026-11-11/README.md`; read only, on three boxes). AppArmor keeps one cache folder for each kernel. A package update writes the new profile only into the folder of the kernel that runs. If the box then starts another kernel that it started once before, that kernel's folder still holds the old profile, and AppArmor uses it (the cached copy looks newer than the package's file). MEASURED on the N100: chrony 4.8 under the profile of chrony 4.6.1; the kernel refuses its write to `/run/systemd/notify`; the unit fails at every start since 2026-10-16. On a household box the product's own kernel rollback (the crash guard) makes the same order, and it is true for every profile, not only chrony's. Not measured: a box where the rollback really ran. Also seen: Tester 1's host still has chrony 4.6.1 while the other two have 4.8 from the OS-update leg; why was not looked at. | READY — **2026-11-14 09:55 — RULED (`09` §3 decision 305): the N100 is this fix's real case in release 2; no hand step before.** OPEN — filed 2026-11-11 (N100 clock diagnosis) | — | Design first, in `11-os-updates.md`: after an OS-update leg and after a kernel rollback, reload the profiles with the cache ignored (`apparmor_parser -r -T -W`), or clear the cache folders of the kernels that do not run. Red-proof on a scratch host: make the order, see the stale profile, see the fix remove it. Then say why Tester 1 did not take chrony 4.8. | CC |
| **R-1384** | Box system & updates | P3 | **A verdict or an „already ran” check trusts a clock that can be wrong (30 answers).** Found 2026-11-11 (the Clock list, class W2, `documentation/audits/clock-list-2026-11-11/TABLE.md`). MEASURED: the N100's host is 32.6 s behind and its time service does not run (R-1301); the other two boxes are in step. Nothing checks a box's clock except one hub alarm for a tunnel handshake stamped in the future. The hub takes the box's own time for backup age, off-site age and proof age (CH02-6, CH07-6, CH07-8, CH08-6); a box with a wrong clock refuses every signed job (CA09-6); a stamp from the future blocks the OS-update leg, the trim and the line test until the clock passes it (CA02-6, CA02-8, CA11-6, CA11-8, CA12-6); the crash guard reads the clock at early boot (CA04-6, CA04-8); a hub clock jump moves its own daily jobs (CH03-6, CH04-6, CH06-6, CH09-6, CH16-6, CH18-6, CH20-6, CH24-6). The others: CA05-6, CA05-8, CA06-6, CA07-6, CA10-6, CA20-6, CC04-6, CC05-6, CC14-6, CC18-6. Known parts stay in their rows: R-1189, R-1206, R-1260. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 300). Not built.** **2026-11-11 (afternoon): why the N100's clock is wrong is known (R-1301, `documentation/audits/n100-clock-2026-11-11/README.md`).** Numbers from source: a host clock more than 120 s behind refuses every signed job (`felhom-agent/internal/authz/verifier.go:21`, `:153-162`); at about 1.25 s a day the N100 gets there in about 70 days (mid-January 2027, an estimate from one measurement). | OPEN — **2026-11-14: the one box whose clock was wrong (the N100, 32.6 s) is right again (R-1301 closed); this row's check is still not built.** Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): the hub compares the time in each report with its own, alarms the operator above one minute, and does not take a time from the future as „fresh”. | CC |
| **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC |
| **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC |
| **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC |
| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — 2026-10-27: agent v0.167.0 RELEASED; in the NORMAL signed delivery the pass now runs right after the commit on all three boxes („running the bundle policy pass after the self-update commit (R-1285)” → `BUNDLE POLICY DONE install_mode=appliance removed=0`; `audits/release-2026-10-27/agent-0167-tester1-demofelhom.txt`). Not live: a BYO box coming FROM 0.164.0 (none in the fleet). **Earlier:** **VERIFY** — 2026-10-26: agent `b1bc63d` (unreleased) — the new agent runs the bundle-policy pass once right after its self-update commits (`selfupdate.SetOnCommitted`); test in the real order (commit, then one pass), red-proved. **Not live:** only a box coming FROM 0.164.0 discriminates (0.165.0's wrapper already applies D43), which needs a signed downgrade chain on a BYO box — the live proof rides the next agent release. **Earlier:** **OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:** **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
| **R-1395** | Box system & updates | P4 | **A step of the host's clock makes the controller read a new boot: it then recreates every drive-backed app, once.** Found 2026-11-14 on the N100 (`audits/release-2a-2026-11-14/D/13b-n100-container-near-the-step.txt`): the time service came back and stepped the clock by +32.7 s; 9 seconds later the controller logged „[gate] boot 1794112550-…: recreating drive-backed app” for radarr, sonarr and qbittorrent (each down for 1 to 5 s; all healthy after). Cause, read on the box: the controller's boot identity carries `/proc/stat` `btime`, and the kernel computes `btime` from the wall clock, so it moved by the same 32 s (the crash guard's `last_boot_at` reads 04:35:18Z, `btime` now 04:35:50Z). The crash guard was NOT fooled: it reads the kernel's `boot_id`. A step this large needs a clock that was wrong for weeks, so a healthy box does not meet it; a fresh box with a wrong hardware clock can, at its first time sync. | OPEN — filed 2026-11-14 | — | make the controller's boot identity the kernel's `boot_id` (as the crash guard's is), with a test that a moved `btime` is not a new boot | CC |
## Monitoring & notifications — 28 rows (P2 3, P3 17, P4 8)
@@ -334,7 +332,7 @@ stopping line that lies.
| **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) |
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC |
| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC |
| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | OPEN — **2026-11-14 (release 2a), two more things the hub does not tell:** (1) a kernel clean-up that removed kernels shows only as „cleaned <time ago>” in the System page's „Kernel step” cell — what was removed, kept and gained is stored and shown nowhere; (2) the security-profile refresh sends nothing to the hub at all, also when it restarted a failed service (it did on the N100). Both for release 3. Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC |
| **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC |
## Hub & operator — 18 rows (P2 1, P3 5, P4 12)
@@ -379,7 +377,7 @@ stopping line that lies.
| **R-990** | Business & legal | P4 | **Operator: read `documentation/audits/forum-research-2026-10-12/FELHOM.md` §5 and rule on each recommendation.** READ-ONLY research 2026-10-12: three years (#7943–#11092, 2,908 posts, 195 people, 82 beginner threads) of the public PROHARDVER! topic „Home server / házi szerver építése”, tagged and counted by person (`FINDINGS.md`, method and spot-check in `METHOD.md`; no username or post text in git). Ten ranked recommendations, none acted on; the top three: a „which machine works” page (8 of 33 beginners who name their hardware own only a Synology, a router disk or a Raspberry Pi — the installer cannot run there; the same gap is R-989's first recommendation from a different source, and the shipped-page half is R-991), „you do not choose an operating system — we do” (the biggest beginner question, 20 of 82 threads), and „bring your old office PC” as the cheap entry. Catalog gap (≥ 3 people): a DLNA server (7), an ad/DNS blocker (5), a camera recorder (5). | **WAITING-ON-OPERATOR** — filed 2026-10-12 | the operator's reading | Operator: rule on each of the ten (yes / no / later). If nothing is done: nothing changes; the research ages | operator |
| **R-1035** | Business & legal | P4 | **Did the „felhő tárhely” page earn anything? Two dated checks.** **Baseline (reported by the operator from Search Console, 2026-10-14, last 3 months):** 1 click, 48 impressions, CTR 2.1 %, average position **73.6**; „felhő tárhely” 19 impressions, „felhőtárhely” 5, „felhő alapú tárhely” 3. CC measured separately that the phrase „felhő tárhely” appeared **0 times** on the home, FAQ and backup pages — Google was showing the home page for those queries by accident. `/sajat-felho-tarhely` + `/en/own-cloud-storage` published **2026-10-14**; full baseline, price sources and the power provenance in `audits/seo-2026-10-14/BASELINE.md`. **What to measure, both times:** impressions and average position for „felhő tárhely” and „saját felhő tárhely”, plus the new page's own impressions, clicks and position. **A null result at +4 weeks means nothing on its own** — a new page on a site at position 73.6 takes weeks to months to move, and „felhő tárhely” alone is held by Microsoft, Google and the big list sites; the page aims at the two longer phrases. That is why there are two checks and not one, and why neither should be read as a verdict on the copy. **There is no Search Console connector in this project, so CC cannot read these numbers** — the operator reads them, CC records and compares. **Blocked until the operator asks for indexing** (Search Console → URL-ellenőrzés → the URL → „Indexelés kérése”): until that happens, or Google finds the page itself, both checks read zero for a reason that is not the page. **This is a slow channel and does not replace talking to people.** **2026-11-11:** the 4-week check came due today in a run with no operator present; only the operator can read Search Console, so the date moved to 2026-11-18 (the gate's rule allows a move with a reason). The 8-week check keeps its date. | **WATCHING** | operator: request indexing, then supply the Search Console read | 2026-11-11: first read (+4 weeks); 2026-12-09: second (+8 weeks), compared with the first | operator + CC |
## Process & tooling — 34 rows (P3 8, P4 26)
## Process & tooling — 35 rows (P3 8, P4 27)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -434,3 +432,4 @@ stopping line that lies.
| **R-1387** | Process & tooling | P4 | **A document and the source disagree on a time, in four places (4 answers).** Found 2026-11-11 (the Clock list, class W5, `documentation/audits/clock-list-2026-11-11/TABLE.md`). (1) `documentation/architecture/03-host-agent.md` names `Restart=always` and a watchdog for the agent's unit; the unit has `Restart=on-failure` and no watchdog (CA18-6, CA18-8). Which one is the design is not stated, so this session did not edit either. (2) The hub-database copy alarm: 13 hours in one place, 26 in another (CH23-7; a DooPlex procedure, listed for the operator, not edited). (3) Old „24 h / 02:30” text for the hub's daily job (CH17-6). **Seen beside the lists, same kind, not in the count:** (4) the comment at `felhom-controller/controller/cmd/controller/main.go:1412` says 14 days where the code keeps restore copies 30 days (D62); (5) `documentation/architecture/07-backup-architecture.md:455-457` cites `main.go` lines that have moved; (6) `documentation/operations/tailscale.md` says „both nodes” where three test hosts exist, and Tailscale was not running on demo-hp and Tester 1 on 2026-11-11 (not known if that is meant); (7) the threat model and the recovery inventory do not say the same about how often the hub bearer is rotated. | OPEN — filed 2026-11-11 (clock list) | — | Operator or a task: say which side is right for (1) and (2); then one docs commit. (3) is a text fix in the same commit. | CC |
| **R-1389** | Process & tooling | P4 | **From the operator's home network the installer address gives 404.** Found 2026-11-11 (`documentation/tests/golden-0.352.0-2026-11-11/fresh-install/installer-address-404-on-lan.txt`). The home router's DNS (192.168.0.250) answers `felhom.eu` with 192.168.0.192, the site copy on DooPlex that was retired on 2026-11-07 (R-1228). So `https://felhom.eu/scripts/…` gives 404 for every machine that uses that DNS; from DooPlex (public DNS) it gives 200. A household is not touched. A test box on the home network, and Tester 2 if it sits there, cannot get the installer by its public address. | OPEN — filed 2026-11-11 (golden fresh install) | — | The operator removes the `felhom.eu` override from the home DNS (it is his router), then one `curl -sI https://felhom.eu/scripts/` from a machine on that network reads 200. | operator |
| **R-1390** | Process & tooling | P4 | **`sec put` cannot push the store any more, so a rotated token is saved on DooPlex only.** Found 2026-11-11. `sec put` ends with a plain `git push` of the `cc-secrets` clone (`cc-secrets/ccsec/cli.py:51-56`). Since this run that clone's address holds no login, and its helper answers only inside `sec run GITEA_GIT_TOKEN -- …`. A plain push now fails for lack of a login; the new entry is committed in the clone and not pushed. Until it is fixed, the push is a second command: `sec run GITEA_GIT_TOKEN -- git -C /mnt/5_hdd/felhom.eu/git/cc-secrets push origin HEAD:main`. Not tried in this run (no store change was allowed). | OPEN — filed 2026-11-11 (token fix) | — | In `cc-secrets`: `sec put` runs its push through the store's own token (or says plainly „committed, not pushed: run …”); a test that a `put` on a clone with no login in the address ends pushed or says so. | CC |
| **R-1394** | Process & tooling | P4 | **The night's OS-update leg cannot be started by day on a test box, so a step that lives in it is proven only by waiting for a night.** Found 2026-11-14 (release 2a). The agent starts the leg only after the night's OWN whole-guest backup; a manual backup does not start it, by design (R-899: a household press must not start OS updates), and the debug pass (`--selftest=os-update`) runs with trigger `debug`, which the kernel step and the kernel clean-up refuse, also by design. So the first real kernel clean-up „by the program” on Tester 1 was started by hand with the agent's own command line (`audits/release-2a-2026-11-14/D/08-tester1-cleanup.txt`), and „the step at night by the agent” waits for a real night. The same held for the kernel lane's own night tests. | OPEN — filed 2026-11-14 | — | decide if a test box needs a signed „run the night leg now” (operator-signed, never a household's press), or if waiting one night stays the method | CC |
+4
View File
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
## Moved from STATUS on 2026-11-14 (afternoon)
**Updated 2026-11-14 (midday).** Release 1 („dates”) is out: controller 0.352.1 and hub 0.181.0 with the Go security fix; golden 0.352.1 is vouched. The false „root files behind” warning is fixed; no alarm mail is due on 11-17. The one-year HTTPS setting is NOT in (next controller release, from 11-15).
## Moved from STATUS on 2026-11-14 (midday)
**Updated 2026-11-14 (morning).** Health check: the three test boxes are healthy; last night's backups and restore proofs passed. CI was red three nights for one due check; it is read now. The yellow „root files behind" is a false alarm that mails on 11-17 unless you answer D81.