Release 2a: agent 0.174.0 vouched — old kernels removed by the box (11 §5.11.1 built), security profiles refreshed after a kernel change (11 §5.12, spike + build); R-1388, R-1301, R-1391 closed; R-1394, R-1395 filed; D82
gates / gates (push) Successful in 2m1s
gates / gates (push) Successful in 2m1s
This commit is contained in:
1 parent
a2173830c1
commit
8955b4e55f
51 files changed
+1385
-19
No files matched your search
@@ -249,7 +249,7 @@
|
||||
| <a id="g15"></a>[G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a consistent copy every 6 hours since D34 (nightly before 2026-10-23; corrected 2026-10-24), encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — be7e97dc + 52af38df (D34 6-hourly) | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
|
||||
| <a id="g16"></a>[G16](capability-map-detail/G.md#g16) | Gitea (all code), the operator's password manager (Vaultwarden, since 2026-10-09 afternoon, R-923) and DooPlex's secrets survive the loss of DooPlex: a nightly encrypted copy on ep0; restore-tested weekly; an alarm when either stops; a failure mail | `scripts/dooplex-offsite/` (R-232), homelab-manifests rules | **PROVEN-LIVE** | — | `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
|
||||
| <a id="g17"></a>[G17](capability-map-detail/G.md#g17) | The operator surface refuses a cross-site form post on BOTH login paths (session token; Basic auth + `X-Felhom-Operator`) | hub v0.135.0 (R-135) | **PROVEN-LIVE** | — | `audits/hub-safety-2026-10-05/partA/live.txt` (2026-10-05) |
|
||||
| <a id="g18"></a>[G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo; **re-check (2026-10-24):** mechanism changed after its evidence — agent a0c88e5 (os-apply sshd check, D4) | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) |
|
||||
| <a id="g18"></a>[G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo; **added 2026-11-14 (agent v0.174.0):** a box reloads its security profiles after a kernel change — PROVEN-LIVE on three boxes, it repaired the N100's time service (`audits/release-2a-2026-11-14/D/13-n100-after.txt`); a ring-0 box removes its old kernels — the removal PROVEN on Tester 1, started by hand (`D/08-tester1-cleanup.txt`), the step at night NOT yet; **re-check (2026-10-24):** mechanism changed after its evidence — agent a0c88e5 (os-apply sshd check, D4) | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) |
|
||||
| <a id="g19"></a>[G19](capability-map-detail/G.md#g19) | An ENGLISH-SPEAKING household's first hour: download, install, pair, bind, claim, two apps | controller v0.259.0 + hub v0.119.0 + ISO 1.29.0 + the whole catalog | **PROVEN-LIVE** | walked on 0.258.0 with one blocker; fixed, the walk not repeated | `audits/DRILL-first-hour-en-0258-2026-09-20.md` (2026-09-20); `audits/i18n-closing-2026-09-21/live/` (2026-09-21) |
|
||||
| <a id="g20"></a>[G20](capability-map-detail/G.md#g20) | A deletion of a customer's off-site history is NOTICED within a day | hub v0.111.0 (R-431) | **IMPLEMENTED** | not yet proven live | R-431 |
|
||||
|
||||
|
||||
@@ -609,7 +609,7 @@ another kernel), and the kernel step (the newest result, the kernel due, whether
|
||||
7.0.14-20), so a soft lockup turning into a panic is recorded as unmeasured (no tool was built). A true dead freeze
|
||||
needs a person (accepted, decision 172). Evidence and proofs: `audits/kernel-lane-2026-10-07/`.
|
||||
|
||||
### 5.11.1 Removing old kernels `[DESIGN — proven by hand on demo-hp 2026-11-10; not built]`
|
||||
### 5.11.1 Removing old kernels — BUILT 2026-11-14 (agent v0.174.0; R-1368, `09` §3 decision 286) `[FACT — built]`
|
||||
|
||||
**The ruling** (`09` §3 decision 286): the box may remove old kernels by itself, as its own step with its own narrow
|
||||
rule. **Rule R4 does not change**: an update plan that removes a package is still refused. Why a step is needed: the
|
||||
@@ -642,14 +642,107 @@ self-revert into `from`, must be over first. In the night's order it belongs whe
|
||||
whole-guest backup. It needs no reboot and stops no guest. A box that fell back or reverted keeps all its kernels
|
||||
until a later step ends `good`.
|
||||
|
||||
**Proven** on demo-hp by hand: 8 kernels → 2, 6.4 GB gained on the system disk, the default, the pinned file and both
|
||||
**As built** (`felhom-agent` `configs/felhom-os-apply`, layer `kernel`, mode `kernel-cleanup`; the agent's
|
||||
`internal/osupdate/kernel.go`):
|
||||
|
||||
- **The plan names nothing.** The wrapper computes the set from the box; a plan that carries packages, a select, a
|
||||
signed job or an expected kernel is refused (C1). `dry_run: true` is the same code path: it reports the keep set, the
|
||||
removal set and what it would do, and changes nothing (`felhom-agent --selftest=kernel-cleanup-dry-run`).
|
||||
- **Who may:** an appliance (R12, the root-owned install record) with the **root-owned ring-0 mark**
|
||||
(`/etc/felhom/os-trust.json`), read by the wrapper itself — never the agent's config, never the hub's ring. A box
|
||||
without the mark answers „not run". Ring 1 needs an operator ruling first.
|
||||
- **„Not run"** (exit 0, nothing changed, a log line): no ring-0 mark; no kernel step yet (no state file); the newest
|
||||
step ended in another phase than `good` (`fell_back`, `self_reverted`, `revert_failed`, `cancelled`); the box does
|
||||
not run its newest installed kernel; a step was staged, booted or ended within the last 20 hours.
|
||||
- **Refused** (exit 2, nothing changed): the table above, each with its own code; the wrapper repairs dpkg first
|
||||
(§5.4.1), so a purge that a power cut interrupted is made whole and the rule is computed again from the box.
|
||||
- **K1 reads the box a second time.** The removal set is built from the keep set; K1 then reads the running kernel, the
|
||||
step's `from`/`to`/`default_before`, both defaults, the flag and the boot pin again and refuses a name that any of
|
||||
them still needs — also before apt's simulation.
|
||||
- **After the purge** the wrapper proves that the default in `grub.cfg`, the pinned file, both generators and the
|
||||
step's state file did not change, and that every kept kernel is still installed. Anything else is `failed`, said
|
||||
loudly; nothing is repaired in code. A left-over `/lib/modules/<version>` or `/boot` file is reported, never deleted.
|
||||
- **When (the agent):** the night leg's own step, after a healthy host step (and Proxmox step, when one ran), the last
|
||||
call of the night, never in a night with a kernel step (due, staged or reported), never in a debug pass, never on a
|
||||
BYO host. Not retried the same night.
|
||||
- **The sudo rule did not change:** the wrapper is reached by the one `--plan` line.
|
||||
- **What the hub shows (hub v0.181.0, no hub change in this release):** a clean-up that removed kernels arrives as a
|
||||
kernel-layer report with outcome `cleaned`; the System page's „Kernel step" cell then reads „cleaned <time ago>", and
|
||||
the report's text (what was removed, kept and gained) is stored but shown nowhere. A refusal or a failure raises the
|
||||
existing operator event „Kernel step on <box> refused/failed: …". „Nothing to remove" and „not run" are log lines
|
||||
on the box only. The report carries no kernel version on purpose: the hub reads that as a kernel step. What is missing
|
||||
is in R-1383 (release 3).
|
||||
|
||||
**Proven on boxes with the program** (2026-11-14, `audits/release-2a-2026-11-14/D/`): Tester 1 — 3 kernels → 2
|
||||
(`proxmox-kernel-7.0.14-20-pve-signed`), 1.12 GB gained, 8.5 s, nothing else changed, no left-over, the second call
|
||||
removed nothing, one orderly restart back onto the same kernel with a clean stop for the crash guard. **That run was
|
||||
started by hand** with the agent's own command line, and with the ring-0 mark set for the test on the operator's yes
|
||||
(Tester 1 is ring 1; a night cannot be started by day, R-1394). demo-hp — „nothing to remove"; P1, P2 and P3 refused by
|
||||
dry run against states set and put back. Tester 1 without the mark — „not run". The N100 — the dry run names 6 of 8.
|
||||
**Not proven:** the step by the agent at night (the night of 2026-11-15 on the N100 and demo-hp); K1 on a box (it
|
||||
guards a wrong computation — tests only); a box whose running kernel is not its newest, and a self-revert into the
|
||||
kept kernel chosen by the product (tests only); a box that boots with `proxmox-boot-tool`.
|
||||
|
||||
**Proven by hand** on demo-hp (2026-11-10): 8 kernels → 2, 6.4 GB gained on the system disk, the default, the pinned file and both
|
||||
generators unchanged, five refusals by simulation, one orderly reboot back onto the same kernel with a clean stop for
|
||||
the crash guard (§5.9), a second run that removed nothing. The kept old kernel then booted the box to healthy through
|
||||
the one-shot flag (313 s), and the box went back to its newest kernel (324 s); the pinned file, `grub.cfg` and the
|
||||
step's state file did not change. **Not proven:** a box whose running kernel is not its newest; a self-revert into
|
||||
the kept kernel chosen by the product; the step as a program at night; a box that boots with `proxmox-boot-tool`. Evidence, the rule as a read-only script, and the full "not measured" list:
|
||||
step's state file did not change. Evidence, the rule as a read-only script, and the full "not measured" list:
|
||||
`audits/kernel-cleanup-spike-2026-11-10/`.
|
||||
|
||||
### 5.12 Security profiles after a kernel change — BUILT 2026-11-14 (agent v0.174.0; R-1388, R-1301) `[FACT — built]` `[FACT — measured on demo-hp]`
|
||||
|
||||
**The fault** (`audits/n100-clock-2026-11-11/`). AppArmor keeps one cache folder for each kernel feature set. When a
|
||||
package changes a profile while one kernel runs, only that kernel's folder gets the new compiled copy. A box that then
|
||||
starts on another kernel loads the OLD copy from that kernel's folder — the copy's time looks newer than its source.
|
||||
The order „start a new kernel once → go back → a profile is updated → start the new kernel again" is made by a kernel
|
||||
test, and by the kernel lane's own fall-back or self-revert (§5.11). On the N100 it stopped the time service.
|
||||
|
||||
**The mode** (`felhom-os-apply`, layer `host`, mode `profile-refresh`; an appliance only; it changes no package and
|
||||
takes nothing from the plan but `dry_run`, F1):
|
||||
|
||||
1. It runs **once per kernel the box starts on**: the wrapper keeps the kernel of the last refresh in a root-owned
|
||||
record (`/var/lib/felhom-os-apply/profile-refresh.json`) and answers „not needed" for the same kernel. A box with no
|
||||
record counts as changed.
|
||||
2. `apparmor_parser --replace --skip-read-cache --write-cache -- /etc/apparmor.d` — every profile from a fresh
|
||||
compile, the cache ignored and rewritten.
|
||||
3. **The narrow restart rule:** a service in state `failed`, whose `ExecStart` program is the path-name of a profile
|
||||
whose cached copy the rewrite CHANGED, gets one `systemctl restart`. At most 3 a run. Never „whatever failed", never
|
||||
a unit a plan names. A running program needs nothing: the kernel swaps its profile in place.
|
||||
4. A parser failure writes no record; the next start tries again.
|
||||
|
||||
**The agent** asks at every start and after a config bundle was installed (a new wrapper may know the mode the old one
|
||||
refused), under the heavy-op gate (up to 12 tries, 5 minutes apart) and never beside an OS-update pass. Nothing goes to
|
||||
the hub (no field exists for it; R-1383).
|
||||
|
||||
**Measured on demo-hp, both guests running** (`audits/profile-refresh-spike-2026-11-14/`): the fault was reproduced
|
||||
with one planted stale profile (chrony failed with the N100's own error); the refresh took 0.51 s; the same 119
|
||||
profiles stayed loaded in the same modes; no container restarted; no new kernel refusal in 10 minutes (with a positive
|
||||
control — see below); the restart rule brought chrony back.
|
||||
|
||||
**Three facts the spike found, each built into the mode:**
|
||||
|
||||
- A stale copy is told by the **cached file changing at the rewrite**. The kernel's loaded hash is NOT a signal: for a
|
||||
file with several profiles it differs between a cache load and a fresh load of the same policy (17 of 119 profiles).
|
||||
- `lxc-containers` compiles to a different binary every time, so it is „rewritten" at every refresh. It names no
|
||||
program path, so the restart rule never matches it.
|
||||
- A refresh writes about 119 audit lines, and the kernel then **suppresses other audit lines for a few seconds**
|
||||
(a real refusal in those seconds is not in the kernel log). This is the cost that makes „once per kernel" the right
|
||||
trigger and „at every agent start" the wrong one.
|
||||
|
||||
**Proven on boxes by the product's own path** (2026-11-14, `audits/release-2a-2026-11-14/D/`): on Tester 1, demo-hp
|
||||
and the N100 the old wrapper refused the mode at the agent's start (logged), and the refresh ran right after the
|
||||
bundle install — 0.5 s each. Tester 1 and demo-hp: only `lxc-containers` rewritten, nothing restarted. **The N100:**
|
||||
the stale chrony profile was rewritten, `chrony.service` (failed since 2026-10-16) was restarted, the clock stepped
|
||||
+32.717 s; offset to DooPlex −32.55 s → +0.18 s, the same 2.5 hours later. After a restart of Tester 1 the agent
|
||||
answered „not needed" for the same kernel.
|
||||
|
||||
**Seen on the N100, a consequence of the clock step, not of the refresh:** the controller read a new boot and
|
||||
recreated its three drive-backed apps once (R-1395).
|
||||
|
||||
**Not proven:** a restart by this rule of any program but chrony; the refresh after a real fall-back or self-revert of
|
||||
the kernel lane (the trigger is the same: another kernel than at the last refresh); the spike's „Not measured" list.
|
||||
|
||||
## 6. Risks and edge cases
|
||||
|
||||
| # | What can go wrong | What the design does |
|
||||
|
||||
@@ -275,7 +275,7 @@ The alarms, by short name:
|
||||
| H6 | backup logs and half-written copies | each run | a failed run's log: none (R-1368) | Proxmox; the agent | host disk |
|
||||
| H7 | the host journal | the agent's `sudo` calls (R-1367) | the journald default: 10 % of the disk, at most 4 GB | journald | host disk |
|
||||
| H8 | other host logs, task logs, graphs | each task | rotation | `logrotate`, Proxmox | host disk |
|
||||
| H9 | old kernels | each approved kernel | none today. Ruled: keep the running one, the one before it and the newest (decision 286, `11-os-updates.md` §5.11.1). Proven by hand, not built | nobody today | host disk; later `os_update_stale` |
|
||||
| H9 | old kernels | each approved kernel | the running one, the one before it and the newest: 3 at most, about 3.4 GB (decision 286, `11-os-updates.md` §5.11.1) | the box itself at night, on a ring-0 box (agent v0.174.0); a ring-1 box: nobody yet (needs a ruling) | host disk; a refused or failed clean-up is an operator event |
|
||||
| H10 | the EFI partition | a boot-loader update | 1.07 GB, 1 % used | none needed | none (R-1371) |
|
||||
| H11 | the host package cache | each OS update | emptied in each run | the update wrapper | host disk |
|
||||
| H12 | the agent's self-update folder | each agent update | the running version's binary, and the file of a failed update until the next good one (about 16 MB each; measured 2026-11-10 on three boxes, R-1368) | the agent, at the end of each good update | host disk |
|
||||
@@ -410,7 +410,7 @@ below 200 GB — refuse, or warn (R-1370). demo-hp (128 GB) stays as a test box.
|
||||
| A drive in the backup role reports as `usb` or `local-dir`. So the household gets the `storage_fill` mail for a drive it must not clean by hand | R-1371 |
|
||||
| The hub's first list for a household still holds `storage_fill_critical`. It is replaced when the box sends its own list. **Not measured on the running hub:** the release run of 2026-11-10 deployed the hub that blocks the host-disk alarm at send time, but the pages that show a household's list refuse a token („session only"), and no test alarm was made. The proof is the tests and the deployed commit (`audits/release-space-fixes-2026-11-10/hub/`) | R-1371 |
|
||||
| Helper images carry no version; the guest's journal has no stated cap; off-site restores have no limit | R-1372 |
|
||||
| Old kernels: the keep rule is ruled and proven by hand, not built | `11-os-updates.md` §5.11.1 |
|
||||
| Old kernels: the clean-up is built for ring-0 boxes (agent v0.174.0); a ring-1 box still keeps every kernel | `11-os-updates.md` §5.11.1 |
|
||||
| The parts of the host disk with no limit and no cleaner (H4, H5, H6, H13) | R-1087, R-1368 |
|
||||
| The smallest disk is ruled (200 GB, `09` §3 decision 295). Nothing enforces it: what the installer does below 200 GB (refuse, or warn) is open | R-1370 |
|
||||
| What to do by hand when a disk is full | `runbooks/disk-full.md` (R-1369) |
|
||||
|
||||
Reference in new issue
Block a user