Release 2a: agent 0.174.0 vouched — old kernels removed by the box (11 §5.11.1 built), security profiles refreshed after a kernel change (11 §5.12, spike + build); R-1388, R-1301, R-1391 closed; R-1394, R-1395 filed; D82
gates / gates (push) Successful in 2m1s

This commit is contained in:
admin committed 2026-11-14 15:30:44 +01:00
1 parent a2173830c1
commit 8955b4e55f
51 files changed
+1385 -19

No files matched your search

@@ -249,7 +249,7 @@
| <a id="g15"></a>[G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a consistent copy every 6 hours since D34 (nightly before 2026-10-23; corrected 2026-10-24), encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — be7e97dc + 52af38df (D34 6-hourly) | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
| <a id="g16"></a>[G16](capability-map-detail/G.md#g16) | Gitea (all code), the operator's password manager (Vaultwarden, since 2026-10-09 afternoon, R-923) and DooPlex's secrets survive the loss of DooPlex: a nightly encrypted copy on ep0; restore-tested weekly; an alarm when either stops; a failure mail | `scripts/dooplex-offsite/` (R-232), homelab-manifests rules | **PROVEN-LIVE** | — | `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
| <a id="g17"></a>[G17](capability-map-detail/G.md#g17) | The operator surface refuses a cross-site form post on BOTH login paths (session token; Basic auth + `X-Felhom-Operator`) | hub v0.135.0 (R-135) | **PROVEN-LIVE** | — | `audits/hub-safety-2026-10-05/partA/live.txt` (2026-10-05) |
| <a id="g18"></a>[G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo; **re-check (2026-10-24):** mechanism changed after its evidence — agent a0c88e5 (os-apply sshd check, D4) | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) |
| <a id="g18"></a>[G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo; **added 2026-11-14 (agent v0.174.0):** a box reloads its security profiles after a kernel change — PROVEN-LIVE on three boxes, it repaired the N100's time service (`audits/release-2a-2026-11-14/D/13-n100-after.txt`); a ring-0 box removes its old kernels — the removal PROVEN on Tester 1, started by hand (`D/08-tester1-cleanup.txt`), the step at night NOT yet; **re-check (2026-10-24):** mechanism changed after its evidence — agent a0c88e5 (os-apply sshd check, D4) | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) |
| <a id="g19"></a>[G19](capability-map-detail/G.md#g19) | An ENGLISH-SPEAKING household's first hour: download, install, pair, bind, claim, two apps | controller v0.259.0 + hub v0.119.0 + ISO 1.29.0 + the whole catalog | **PROVEN-LIVE** | walked on 0.258.0 with one blocker; fixed, the walk not repeated | `audits/DRILL-first-hour-en-0258-2026-09-20.md` (2026-09-20); `audits/i18n-closing-2026-09-21/live/` (2026-09-21) |
| <a id="g20"></a>[G20](capability-map-detail/G.md#g20) | A deletion of a customer's off-site history is NOTICED within a day | hub v0.111.0 (R-431) | **IMPLEMENTED** | not yet proven live | R-431 |
+97 -4
View File
@@ -609,7 +609,7 @@ another kernel), and the kernel step (the newest result, the kernel due, whether
7.0.14-20), so a soft lockup turning into a panic is recorded as unmeasured (no tool was built). A true dead freeze
needs a person (accepted, decision 172). Evidence and proofs: `audits/kernel-lane-2026-10-07/`.
### 5.11.1 Removing old kernels `[DESIGN — proven by hand on demo-hp 2026-11-10; not built]`
### 5.11.1 Removing old kernels — BUILT 2026-11-14 (agent v0.174.0; R-1368, `09` §3 decision 286) `[FACT — built]`
**The ruling** (`09` §3 decision 286): the box may remove old kernels by itself, as its own step with its own narrow
rule. **Rule R4 does not change**: an update plan that removes a package is still refused. Why a step is needed: the
@@ -642,14 +642,107 @@ self-revert into `from`, must be over first. In the night's order it belongs whe
whole-guest backup. It needs no reboot and stops no guest. A box that fell back or reverted keeps all its kernels
until a later step ends `good`.
**Proven** on demo-hp by hand: 8 kernels → 2, 6.4 GB gained on the system disk, the default, the pinned file and both
**As built** (`felhom-agent` `configs/felhom-os-apply`, layer `kernel`, mode `kernel-cleanup`; the agent's
`internal/osupdate/kernel.go`):
- **The plan names nothing.** The wrapper computes the set from the box; a plan that carries packages, a select, a
signed job or an expected kernel is refused (C1). `dry_run: true` is the same code path: it reports the keep set, the
removal set and what it would do, and changes nothing (`felhom-agent --selftest=kernel-cleanup-dry-run`).
- **Who may:** an appliance (R12, the root-owned install record) with the **root-owned ring-0 mark**
(`/etc/felhom/os-trust.json`), read by the wrapper itself — never the agent's config, never the hub's ring. A box
without the mark answers „not run". Ring 1 needs an operator ruling first.
- **„Not run"** (exit 0, nothing changed, a log line): no ring-0 mark; no kernel step yet (no state file); the newest
step ended in another phase than `good` (`fell_back`, `self_reverted`, `revert_failed`, `cancelled`); the box does
not run its newest installed kernel; a step was staged, booted or ended within the last 20 hours.
- **Refused** (exit 2, nothing changed): the table above, each with its own code; the wrapper repairs dpkg first
(§5.4.1), so a purge that a power cut interrupted is made whole and the rule is computed again from the box.
- **K1 reads the box a second time.** The removal set is built from the keep set; K1 then reads the running kernel, the
step's `from`/`to`/`default_before`, both defaults, the flag and the boot pin again and refuses a name that any of
them still needs — also before apt's simulation.
- **After the purge** the wrapper proves that the default in `grub.cfg`, the pinned file, both generators and the
step's state file did not change, and that every kept kernel is still installed. Anything else is `failed`, said
loudly; nothing is repaired in code. A left-over `/lib/modules/<version>` or `/boot` file is reported, never deleted.
- **When (the agent):** the night leg's own step, after a healthy host step (and Proxmox step, when one ran), the last
call of the night, never in a night with a kernel step (due, staged or reported), never in a debug pass, never on a
BYO host. Not retried the same night.
- **The sudo rule did not change:** the wrapper is reached by the one `--plan` line.
- **What the hub shows (hub v0.181.0, no hub change in this release):** a clean-up that removed kernels arrives as a
kernel-layer report with outcome `cleaned`; the System page's „Kernel step" cell then reads „cleaned <time ago>", and
the report's text (what was removed, kept and gained) is stored but shown nowhere. A refusal or a failure raises the
existing operator event „Kernel step on <box> refused/failed: …". „Nothing to remove" and „not run" are log lines
on the box only. The report carries no kernel version on purpose: the hub reads that as a kernel step. What is missing
is in R-1383 (release 3).
**Proven on boxes with the program** (2026-11-14, `audits/release-2a-2026-11-14/D/`): Tester 1 — 3 kernels → 2
(`proxmox-kernel-7.0.14-20-pve-signed`), 1.12 GB gained, 8.5 s, nothing else changed, no left-over, the second call
removed nothing, one orderly restart back onto the same kernel with a clean stop for the crash guard. **That run was
started by hand** with the agent's own command line, and with the ring-0 mark set for the test on the operator's yes
(Tester 1 is ring 1; a night cannot be started by day, R-1394). demo-hp — „nothing to remove"; P1, P2 and P3 refused by
dry run against states set and put back. Tester 1 without the mark — „not run". The N100 — the dry run names 6 of 8.
**Not proven:** the step by the agent at night (the night of 2026-11-15 on the N100 and demo-hp); K1 on a box (it
guards a wrong computation — tests only); a box whose running kernel is not its newest, and a self-revert into the
kept kernel chosen by the product (tests only); a box that boots with `proxmox-boot-tool`.
**Proven by hand** on demo-hp (2026-11-10): 8 kernels → 2, 6.4 GB gained on the system disk, the default, the pinned file and both
generators unchanged, five refusals by simulation, one orderly reboot back onto the same kernel with a clean stop for
the crash guard (§5.9), a second run that removed nothing. The kept old kernel then booted the box to healthy through
the one-shot flag (313 s), and the box went back to its newest kernel (324 s); the pinned file, `grub.cfg` and the
step's state file did not change. **Not proven:** a box whose running kernel is not its newest; a self-revert into
the kept kernel chosen by the product; the step as a program at night; a box that boots with `proxmox-boot-tool`. Evidence, the rule as a read-only script, and the full "not measured" list:
step's state file did not change. Evidence, the rule as a read-only script, and the full "not measured" list:
`audits/kernel-cleanup-spike-2026-11-10/`.
### 5.12 Security profiles after a kernel change — BUILT 2026-11-14 (agent v0.174.0; R-1388, R-1301) `[FACT — built]` `[FACT — measured on demo-hp]`
**The fault** (`audits/n100-clock-2026-11-11/`). AppArmor keeps one cache folder for each kernel feature set. When a
package changes a profile while one kernel runs, only that kernel's folder gets the new compiled copy. A box that then
starts on another kernel loads the OLD copy from that kernel's folder — the copy's time looks newer than its source.
The order „start a new kernel once → go back → a profile is updated → start the new kernel again" is made by a kernel
test, and by the kernel lane's own fall-back or self-revert (§5.11). On the N100 it stopped the time service.
**The mode** (`felhom-os-apply`, layer `host`, mode `profile-refresh`; an appliance only; it changes no package and
takes nothing from the plan but `dry_run`, F1):
1. It runs **once per kernel the box starts on**: the wrapper keeps the kernel of the last refresh in a root-owned
record (`/var/lib/felhom-os-apply/profile-refresh.json`) and answers „not needed" for the same kernel. A box with no
record counts as changed.
2. `apparmor_parser --replace --skip-read-cache --write-cache -- /etc/apparmor.d` — every profile from a fresh
compile, the cache ignored and rewritten.
3. **The narrow restart rule:** a service in state `failed`, whose `ExecStart` program is the path-name of a profile
whose cached copy the rewrite CHANGED, gets one `systemctl restart`. At most 3 a run. Never „whatever failed", never
a unit a plan names. A running program needs nothing: the kernel swaps its profile in place.
4. A parser failure writes no record; the next start tries again.
**The agent** asks at every start and after a config bundle was installed (a new wrapper may know the mode the old one
refused), under the heavy-op gate (up to 12 tries, 5 minutes apart) and never beside an OS-update pass. Nothing goes to
the hub (no field exists for it; R-1383).
**Measured on demo-hp, both guests running** (`audits/profile-refresh-spike-2026-11-14/`): the fault was reproduced
with one planted stale profile (chrony failed with the N100's own error); the refresh took 0.51 s; the same 119
profiles stayed loaded in the same modes; no container restarted; no new kernel refusal in 10 minutes (with a positive
control — see below); the restart rule brought chrony back.
**Three facts the spike found, each built into the mode:**
- A stale copy is told by the **cached file changing at the rewrite**. The kernel's loaded hash is NOT a signal: for a
file with several profiles it differs between a cache load and a fresh load of the same policy (17 of 119 profiles).
- `lxc-containers` compiles to a different binary every time, so it is „rewritten" at every refresh. It names no
program path, so the restart rule never matches it.
- A refresh writes about 119 audit lines, and the kernel then **suppresses other audit lines for a few seconds**
(a real refusal in those seconds is not in the kernel log). This is the cost that makes „once per kernel" the right
trigger and „at every agent start" the wrong one.
**Proven on boxes by the product's own path** (2026-11-14, `audits/release-2a-2026-11-14/D/`): on Tester 1, demo-hp
and the N100 the old wrapper refused the mode at the agent's start (logged), and the refresh ran right after the
bundle install — 0.5 s each. Tester 1 and demo-hp: only `lxc-containers` rewritten, nothing restarted. **The N100:**
the stale chrony profile was rewritten, `chrony.service` (failed since 2026-10-16) was restarted, the clock stepped
+32.717 s; offset to DooPlex −32.55 s → +0.18 s, the same 2.5 hours later. After a restart of Tester 1 the agent
answered „not needed" for the same kernel.
**Seen on the N100, a consequence of the clock step, not of the refresh:** the controller read a new boot and
recreated its three drive-backed apps once (R-1395).
**Not proven:** a restart by this rule of any program but chrony; the refresh after a real fall-back or self-revert of
the kernel lane (the trigger is the same: another kernel than at the last refresh); the spike's „Not measured" list.
## 6. Risks and edge cases
| # | What can go wrong | What the design does |
@@ -275,7 +275,7 @@ The alarms, by short name:
| H6 | backup logs and half-written copies | each run | a failed run's log: none (R-1368) | Proxmox; the agent | host disk |
| H7 | the host journal | the agent's `sudo` calls (R-1367) | the journald default: 10 % of the disk, at most 4 GB | journald | host disk |
| H8 | other host logs, task logs, graphs | each task | rotation | `logrotate`, Proxmox | host disk |
| H9 | old kernels | each approved kernel | none today. Ruled: keep the running one, the one before it and the newest (decision 286, `11-os-updates.md` §5.11.1). Proven by hand, not built | nobody today | host disk; later `os_update_stale` |
| H9 | old kernels | each approved kernel | the running one, the one before it and the newest: 3 at most, about 3.4 GB (decision 286, `11-os-updates.md` §5.11.1) | the box itself at night, on a ring-0 box (agent v0.174.0); a ring-1 box: nobody yet (needs a ruling) | host disk; a refused or failed clean-up is an operator event |
| H10 | the EFI partition | a boot-loader update | 1.07 GB, 1 % used | none needed | none (R-1371) |
| H11 | the host package cache | each OS update | emptied in each run | the update wrapper | host disk |
| H12 | the agent's self-update folder | each agent update | the running version's binary, and the file of a failed update until the next good one (about 16 MB each; measured 2026-11-10 on three boxes, R-1368) | the agent, at the end of each good update | host disk |
@@ -410,7 +410,7 @@ below 200 GB — refuse, or warn (R-1370). demo-hp (128 GB) stays as a test box.
| A drive in the backup role reports as `usb` or `local-dir`. So the household gets the `storage_fill` mail for a drive it must not clean by hand | R-1371 |
| The hub's first list for a household still holds `storage_fill_critical`. It is replaced when the box sends its own list. **Not measured on the running hub:** the release run of 2026-11-10 deployed the hub that blocks the host-disk alarm at send time, but the pages that show a household's list refuse a token („session only"), and no test alarm was made. The proof is the tests and the deployed commit (`audits/release-space-fixes-2026-11-10/hub/`) | R-1371 |
| Helper images carry no version; the guest's journal has no stated cap; off-site restores have no limit | R-1372 |
| Old kernels: the keep rule is ruled and proven by hand, not built | `11-os-updates.md` §5.11.1 |
| Old kernels: the clean-up is built for ring-0 boxes (agent v0.174.0); a ring-1 box still keeps every kernel | `11-os-updates.md` §5.11.1 |
| The parts of the host disk with no limit and no cleaner (H4, H5, H6, H13) | R-1087, R-1368 |
| The smallest disk is ruled (200 GB, `09` §3 decision 295). Nothing enforces it: what the installer does below 200 GB (refuse, or warn) is open | R-1370 |
| What to do by hand when a disk is full | `runbooks/disk-full.md` (R-1369) |