The close-out of the attended release run of 2026-11-10 (afternoon). - Evidence: documentation/audits/release-space-fixes-2026-11-10/ (agent, controller, hub) and the hub row after the fallback-kernel test. - Rows: dated lines on R-1368, R-1371, R-1301, R-1353. 236 open before and after; opened 0, closed 0. - Capability map: dated notes under C13 and G5; no status changed. - disk-full runbook and 16 §14: the self-update folder is PROVEN as a product path on three test boxes. - STATUS: new "Updated" paragraph; the old one moved to STATUS-HISTORY. - REPORT-release-space-fixes-2026-11-10.md: wrong claims first, one deviation (the hub deploy ran before CI for the manifest-bump commit was read), what stays not proven. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
14 KiB
REPORT — today's rulings, the fallback kernel, the release of the three Space fixes (2026-11-10, afternoon, attended)
Plain words first. CC wrote down the five rulings of 15:19 and built none. The HP box booted its kept older kernel once, was healthy, and went back. Three fixes are released: the agent cleans its own update folder (the operator vouched it), the controller sends one warning for one full disk (the global floor is set), and the hub sends the host-disk alarm to the operator only (deployed on the operator's yes). No golden, no catalog push, no change on ep0. No household got a mail from this run.
1. Claims that turned out wrong — in the brief, and in what the brief told CC to read first
| Claim | Verdict |
|---|---|
| The brief, Part 2 step 3: "If the folder is cleaned only at the next good update after this one, say so plainly" | It is not so. The folder is cleaned at the commit of the update that brings the fix. The commit runs in the new binary. Measured on three boxes (removed=49, 48, 33). No second version was needed |
| The morning's report (§3, limits) and the agent's CHANGELOG entry: "a box cleans its folder at the first good update AFTER the release" | Wrong, for the same reason. The CHANGELOG entry is corrected. The morning's report is a dated record and is not edited |
| The morning's report (§1): the double fill warning was "not seen as two real mails" | Now seen on a box. Scratch guest 9202 on the version before, one fill: two warnings, two names. 9202 has no hub, so they are two warnings in the box's own record, not two mails in a mailbox |
The brief, Part 4 step 4: "Read what the deployed hub lets you read about the open point in 16 §14.4" |
It lets CC read nothing about it. The pages that show a household's alarm list answer a token with 403, "session only". That is by design (05 §17). The point stays not measured on the running hub |
| The brief, for the operator: "The HP box restarts twice, about 5 minutes each" | True: 313 s and 324 s to healthy |
2. Five lines
- The older kernel booted to healthy: yes, in 313 seconds. The box went back to its newest kernel in 324 seconds.
Nothing was left behind: the pinned default,
grub.cfgand the kernel-step state have the same checksum before and after; the hub got no kernel-step report from the test. - Agent 0.173.0 is vouched (the operator, 15:29:16 UTC, read back from the hub's log). The self-update folder: demo-hp 49 files → 1, demo-felhom 48 → 1, Tester 1 33 → 1 (722,596 / 706,968 / 497,244 KB → 16,028 KB each). A rollback after the clean-up worked on Tester 1.
- Every box that reports is on controller 0.352.0: Tester 1, demo-hp and demo-felhom (the hub's host list holds these three; Tester 2 is off). The fill watch's target list on each: 2 filesystems, one target each (it was 3 targets on 2 filesystems). The global floor is 0.352.0 with MinAgent 0.158.1, set 15:31:34 UTC in its own save. In the 30 minutes after: every box on the release and healthy, 4 reports from each box, no event, no alarm and no mail in the hub's log. A second channel agrees: the operator's mailbox got no mail in that time (the same search found the night's mail, so the search works).
- Hub 0.180.0 is live since 15:12:48 UTC: Synced/Healthy, no restart, no error line and no alarm sent in the 30 minutes after; 6 reports came from each box in that time.
- What stays not proven: see §6.
3. The parts
| Part | State | Note |
|---|---|---|
| 0 — the rulings D71–D75 written down | done | e83a78ce. 09 §3, decisions 291–296. None is built |
| 1 — the fallback kernel on demo-hp | done | 5f408cac. documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/ |
| 2 — the agent release | done, vouched | felhom-agent 776025c8 (the release), 5cebb766, 9edaa8c0 (notes, the D50 pin) |
| 3 — the controller release | done | felhom-controller 3dbb68c1 (the release), a9189c9d (notes) |
| 4 — the hub release | done | 97b7e471 (the image's commit), d0a54c2f (the manifest bump) |
Nothing was dropped. No STOP ran out: the operator answered both asks (the hub deploy: "Yes, approved"; the vouch: "I vouched it").
Part 1, the method. The product's own boot-side switch, set by hand: one flag in the GRUB environment file, then an orderly restart. GRUB cleared the flag and booted the older kernel once. The second restart had no flag. No wrapper verb ran; no kernel-step result was written; the pinned default did not change. Not covered: a step where the product itself chooses the old kernel.
Part 2, the rollback proof. The path the repo already has: a signed update on Tester 1 to the older version, then one crash 9 seconds into the 60-second dwell. The wrapper put 0.173.0 back in the same second. After it: the unit active, no pending marker, 2 files in the folder (the failed version's file stays until the next good update, by design). This is one crash, not a crash loop.
Part 3, red and green on a box. Scratch guest 9202, one test file of a known name inside the free space, never
into the reserve, removed by its exact name. On the version before: two warnings. On 0.352.0-rc.1: one warning; ten
minutes after the file was gone the watch said "cleared silently, re-armed". Then the release build, per-box floors on
the three test boxes (each moved in under 15 seconds), the box walk green (containers 10 / 26 / 9, none unhealthy, no
error line), the https-only check green, and the global floor last.
Part 4. Tests green on the image's commit (TestD68_*). Only the hub's Deployment was out of sync before the
sync, and only it was synced. No test alarm was made.
4. One deviation from the rules, said plainly
The hub deploy ran before CI for the manifest-bump commit was read. Rule 2 says CI green by commit hash before
each release. The deploy was 15:12:48 UTC; CI for d0a54c2f was read green at 15:15:35 UTC (run 2617). CC had
chained the deploy to the push check and not to the CI read. What limits the harm: the image was built from
97b7e471, and CI for that commit was green before the build (run 2616); the bump commit changes one image tag. It
went well, but it was not the order the rule asks for.
5. Versions, digests, CI by commit
| What | Version | Digest / checksum | Built from | CI (read by commit on the public Actions page) |
|---|---|---|---|---|
| Agent | 0.173.0 | binary sha256 752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea; config bundle sha256 51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1 |
776025c8 |
run 2608, gates #311, success |
| Controller | 0.352.0 (MinAgent 0.158.1) | image sha256:3c785dd396b091f19d4c777db9d7a206e75ab018b6238b5d887e1a4c6cfd3b29 |
3dbb68c1 |
run 2615, gates #558, success |
| Hub | 0.180.0 | image sha256:768e73ab1ab502a8c3fe23a9d42cc038443a6456a1f40dcec853a2c588d04a8f |
97b7e471 |
run 2616, gates #1418, success |
| Repo | Other commits of this run | CI |
|---|---|---|
| felhom.eu | e83a78ce Part 0 |
run 2613, gates #1417, success |
| felhom.eu | 5f408cac Part 1 |
pushed in one push with 97b7e471; CI ran on the head, run 2616 |
| felhom.eu | d0a54c2f the manifest bump |
run 2617, gates #1419, success (read after the deploy, §4) |
| felhom-agent | 5cebb766 release notes |
run 2618, gates #313, success |
| felhom-agent | 9edaa8c0 the vouch, pinned.day0_agent |
run 2619, gates #314, success |
| felhom-controller | a9189c9d release notes |
run 2620, gates #559, success |
| felhom.eu | the commit that holds this report | read after the push; the result is in the session's closing note |
Register: 236 open at the start, 236 at the end. Opened 0. Closed 0. Dated lines on R-1368, R-1371, R-1301 and R-1353. R-1368 and R-1371 stay open: each has parts no box proves yet.
Other close-out facts. Capability map: no status changed; two dated notes (under C13 and G5) for what a box now
proves. STATUS.md: a new "Updated" paragraph; the old one moved to documentation/backlog/STATUS-HISTORY.md; D71–D75
were taken out of the decide table in Part 0; the open point of D75 (what the installer does below 200 GB, R-1370) is
named. scripts/unproven.py --summary at the end: walked 32, partial 15, built 4, missing 4, not walked 23 of 55; no
status changed in this run, so no number moved. System poster: none of the three actions; no listed fact changed. Instruction files: none edited. CHANGELOG
and REPORT: updated in felhom-agent, felhom-controller and here (hub/CHANGELOG.md).
6. What stays not proven
- The hub, on the running system: that a household which saved its own alarm list before no longer gets the
host-disk alarm (
16§14.4). The proof is the tests and the deployed commit. The operator can read it: a household's page on the hub, the list "Events". - The controller: a real fill on a household's box; a fill at the critical line (95 %) on any box. The guest's own root disk still has no watcher (R-1371 part 1).
- The agent: the
.bak-*copies are not reached (R-1368 part 4). A household box cleans its folder at its first good agent update after the vouch; none was watched. - The kernel clean-up: it is not built as a program, and no second box did it (R-1368 part 1).
7. Machines changed, and how each was left
| Machine | Change | State now |
|---|---|---|
| demo-hp host | Two orderly restarts (14:30 and 14:40 UTC). Agent updated by a signed job | Newest kernel runs; agent 0.173.0; online |
| demo-hp guest 9201, N100 guest, Tester 1 guest | Controller updated by the per-box floor, the normal path | 0.352.0, healthy |
| N100 host, Tester 1 host | Agent updated by a signed job. On Tester 1 also the rollback proof | Agent 0.173.0 |
| Scratch guest 9202 | A release candidate, then the release, set by hand. One test file written and removed | On the release; the test file is gone; the watch's state is empty |
| DooPlex | Three builds, the hub deploy, the manifest bump | Hub 0.180.0, Synced/Healthy |
ep0, Cloudflare, Hetzner, the sec store |
Not touched. No login to ep0. Nothing removed from the store | — |
Teardown, three layers. Machine: the test file on 9202 removed by its exact name. Host: nothing was provisioned.
Hub: the three per-box floors were cleared at 16:03:11 UTC (three 303; the hub's log names each one), and the global
floor holds the boxes; no appliance was registered. On DooPlex the clean clone used for the agent release was deleted
(its git settings held a credential).
8. Architecture documents read
11-os-updates.md (§5.11, §5.11.1, §8.2) · 09-update-architecture.md (§0, §3: decisions 250, 251, 262, 286–296) ·
16-storage-physical-identity.md (§14, §14.4) · 08-alarm-ladder.md (the operator-only register, the fill alarms) ·
05-hub-architecture.md (§17, what a token may read) · 03-host-agent.md (the self-update part).
9. Observations outside the scope
| Observation | Verdict |
|---|---|
CC ran git remote -v in felhom-agent. It printed the origin address with a login and a token into the session |
CC's mistake. Nothing was committed and no evidence file holds it (scanned). The token is one of the logins R-925 already lists for rotation. FILED: no new row; a note in CC's memory so that the next session does not run it |
| The first gate run at the close was red: CC's new memory line put the memory index 43 bytes over its limit | Fixed in the session (small): CC shortened one other index line; the detail is in its topic file. The second run was green |
The N100's host clock is 32 seconds behind; chrony failed again at the 11-08 boot |
FILED: R-1301 (dated line; documentation/audits/release-space-fixes-2026-11-10/agent/13-n100-clock.txt) |
| The hub's customer, events and notifications pages answer CC's token with 403; so does the OS fleet page | NOT-A-FINDING: "session only — never by a token" is the design (05 §17) |
| CC took no copy of the hub's database to read the households' lists | NOT-A-FINDING: a choice. The copy holds mail addresses and old token bytes, and the fence is by design |
| 11 slow database lines in the hub's first 2 minutes after the deploy (the longest 19.1 s); one snapshot took 127.8 s | FILED: R-1353 (dated line). The same statements as the 11-02 read |
| The hub logs "household cleared its mail address — deleted" for Tester 1 at each controller start | NOT-A-FINDING: the same line at every start since at least 11-09 13:09 CET, before this run; Tester 1's box holds no address of its own |
| The hub's row for demo-hp said "unknown (no running customer guest)" while the box was on the older kernel | NOT-A-FINDING: it was the agent's first report after the boot, and no newer report came in the six minutes on that kernel |
| 9202's watch line went from "checked 11 filesystem(s)" to 3 | NOT-A-FINDING: the drill's test drives lie on the one data volume; the fix watches a filesystem once. The side effect is in the controller's CHANGELOG |
| The first crossing try on 9202 read "ok" | NOT-A-FINDING: the box freed 1.6 GB by itself between the size read and the check. The second try, with a new size read, crossed the line |
| 9202 logs "DROPPED event" for each of its events | NOT-A-FINDING: 9202 has no hub configured; its log says so ("notifier disabled (no hub configured)") |
After a rollback /usr/local/bin/felhom-agent.prev is the same version as the running binary |
NOT-A-FINDING: how the wrapper works; it is not new. Written in the agent's report |
| demo-hp's controller prints three warnings at each start: an installed app (calibre-web, plex, bookstack) got a new field, and a person must fill it | NOT-A-FINDING: the controller's designed note. It never fills a text or a password field by itself. This release did not change that code. The lines hold field names only |
| The first box walk hid two of these three lines | NOT-A-FINDING: the walk's own filter drops each line that holds the word "password". The lines were read again without the filter and are in controller/12-watch.txt |