Files
felhom.eu/REPORT-release-space-fixes-2026-11-10.md
T
admin bcc73b428d
gates / gates (push) Successful in 2m1s
Space fixes released: agent 0.173.0 vouched, controller 0.352.0 global floor, hub 0.180.0 live; evidence, rows, status, report
The close-out of the attended release run of 2026-11-10 (afternoon).
- Evidence: documentation/audits/release-space-fixes-2026-11-10/ (agent, controller, hub) and the hub row after the
  fallback-kernel test.
- Rows: dated lines on R-1368, R-1371, R-1301, R-1353. 236 open before and after; opened 0, closed 0.
- Capability map: dated notes under C13 and G5; no status changed.
- disk-full runbook and 16 §14: the self-update folder is PROVEN as a product path on three test boxes.
- STATUS: new "Updated" paragraph; the old one moved to STATUS-HISTORY.
- REPORT-release-space-fixes-2026-11-10.md: wrong claims first, one deviation (the hub deploy ran before CI for the
  manifest-bump commit was read), what stays not proven.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-10 17:17:04 +01:00

14 KiB
Raw Blame History

REPORT — today's rulings, the fallback kernel, the release of the three Space fixes (2026-11-10, afternoon, attended)

Plain words first. CC wrote down the five rulings of 15:19 and built none. The HP box booted its kept older kernel once, was healthy, and went back. Three fixes are released: the agent cleans its own update folder (the operator vouched it), the controller sends one warning for one full disk (the global floor is set), and the hub sends the host-disk alarm to the operator only (deployed on the operator's yes). No golden, no catalog push, no change on ep0. No household got a mail from this run.

1. Claims that turned out wrong — in the brief, and in what the brief told CC to read first

Claim Verdict
The brief, Part 2 step 3: "If the folder is cleaned only at the next good update after this one, say so plainly" It is not so. The folder is cleaned at the commit of the update that brings the fix. The commit runs in the new binary. Measured on three boxes (removed=49, 48, 33). No second version was needed
The morning's report (§3, limits) and the agent's CHANGELOG entry: "a box cleans its folder at the first good update AFTER the release" Wrong, for the same reason. The CHANGELOG entry is corrected. The morning's report is a dated record and is not edited
The morning's report (§1): the double fill warning was "not seen as two real mails" Now seen on a box. Scratch guest 9202 on the version before, one fill: two warnings, two names. 9202 has no hub, so they are two warnings in the box's own record, not two mails in a mailbox
The brief, Part 4 step 4: "Read what the deployed hub lets you read about the open point in 16 §14.4" It lets CC read nothing about it. The pages that show a household's alarm list answer a token with 403, "session only". That is by design (05 §17). The point stays not measured on the running hub
The brief, for the operator: "The HP box restarts twice, about 5 minutes each" True: 313 s and 324 s to healthy

2. Five lines

  1. The older kernel booted to healthy: yes, in 313 seconds. The box went back to its newest kernel in 324 seconds. Nothing was left behind: the pinned default, grub.cfg and the kernel-step state have the same checksum before and after; the hub got no kernel-step report from the test.
  2. Agent 0.173.0 is vouched (the operator, 15:29:16 UTC, read back from the hub's log). The self-update folder: demo-hp 49 files → 1, demo-felhom 48 → 1, Tester 1 33 → 1 (722,596 / 706,968 / 497,244 KB → 16,028 KB each). A rollback after the clean-up worked on Tester 1.
  3. Every box that reports is on controller 0.352.0: Tester 1, demo-hp and demo-felhom (the hub's host list holds these three; Tester 2 is off). The fill watch's target list on each: 2 filesystems, one target each (it was 3 targets on 2 filesystems). The global floor is 0.352.0 with MinAgent 0.158.1, set 15:31:34 UTC in its own save. In the 30 minutes after: every box on the release and healthy, 4 reports from each box, no event, no alarm and no mail in the hub's log. A second channel agrees: the operator's mailbox got no mail in that time (the same search found the night's mail, so the search works).
  4. Hub 0.180.0 is live since 15:12:48 UTC: Synced/Healthy, no restart, no error line and no alarm sent in the 30 minutes after; 6 reports came from each box in that time.
  5. What stays not proven: see §6.

3. The parts

Part State Note
0 — the rulings D71–D75 written down done e83a78ce. 09 §3, decisions 291–296. None is built
1 — the fallback kernel on demo-hp done 5f408cac. documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/
2 — the agent release done, vouched felhom-agent 776025c8 (the release), 5cebb766, 9edaa8c0 (notes, the D50 pin)
3 — the controller release done felhom-controller 3dbb68c1 (the release), a9189c9d (notes)
4 — the hub release done 97b7e471 (the image's commit), d0a54c2f (the manifest bump)

Nothing was dropped. No STOP ran out: the operator answered both asks (the hub deploy: "Yes, approved"; the vouch: "I vouched it").

Part 1, the method. The product's own boot-side switch, set by hand: one flag in the GRUB environment file, then an orderly restart. GRUB cleared the flag and booted the older kernel once. The second restart had no flag. No wrapper verb ran; no kernel-step result was written; the pinned default did not change. Not covered: a step where the product itself chooses the old kernel.

Part 2, the rollback proof. The path the repo already has: a signed update on Tester 1 to the older version, then one crash 9 seconds into the 60-second dwell. The wrapper put 0.173.0 back in the same second. After it: the unit active, no pending marker, 2 files in the folder (the failed version's file stays until the next good update, by design). This is one crash, not a crash loop.

Part 3, red and green on a box. Scratch guest 9202, one test file of a known name inside the free space, never into the reserve, removed by its exact name. On the version before: two warnings. On 0.352.0-rc.1: one warning; ten minutes after the file was gone the watch said "cleared silently, re-armed". Then the release build, per-box floors on the three test boxes (each moved in under 15 seconds), the box walk green (containers 10 / 26 / 9, none unhealthy, no error line), the https-only check green, and the global floor last.

Part 4. Tests green on the image's commit (TestD68_*). Only the hub's Deployment was out of sync before the sync, and only it was synced. No test alarm was made.

4. One deviation from the rules, said plainly

The hub deploy ran before CI for the manifest-bump commit was read. Rule 2 says CI green by commit hash before each release. The deploy was 15:12:48 UTC; CI for d0a54c2f was read green at 15:15:35 UTC (run 2617). CC had chained the deploy to the push check and not to the CI read. What limits the harm: the image was built from 97b7e471, and CI for that commit was green before the build (run 2616); the bump commit changes one image tag. It went well, but it was not the order the rule asks for.

5. Versions, digests, CI by commit

What Version Digest / checksum Built from CI (read by commit on the public Actions page)
Agent 0.173.0 binary sha256 752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea; config bundle sha256 51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1 776025c8 run 2608, gates #311, success
Controller 0.352.0 (MinAgent 0.158.1) image sha256:3c785dd396b091f19d4c777db9d7a206e75ab018b6238b5d887e1a4c6cfd3b29 3dbb68c1 run 2615, gates #558, success
Hub 0.180.0 image sha256:768e73ab1ab502a8c3fe23a9d42cc038443a6456a1f40dcec853a2c588d04a8f 97b7e471 run 2616, gates #1418, success
Repo Other commits of this run CI
felhom.eu e83a78ce Part 0 run 2613, gates #1417, success
felhom.eu 5f408cac Part 1 pushed in one push with 97b7e471; CI ran on the head, run 2616
felhom.eu d0a54c2f the manifest bump run 2617, gates #1419, success (read after the deploy, §4)
felhom-agent 5cebb766 release notes run 2618, gates #313, success
felhom-agent 9edaa8c0 the vouch, pinned.day0_agent run 2619, gates #314, success
felhom-controller a9189c9d release notes run 2620, gates #559, success
felhom.eu the commit that holds this report read after the push; the result is in the session's closing note

Register: 236 open at the start, 236 at the end. Opened 0. Closed 0. Dated lines on R-1368, R-1371, R-1301 and R-1353. R-1368 and R-1371 stay open: each has parts no box proves yet.

Other close-out facts. Capability map: no status changed; two dated notes (under C13 and G5) for what a box now proves. STATUS.md: a new "Updated" paragraph; the old one moved to documentation/backlog/STATUS-HISTORY.md; D71–D75 were taken out of the decide table in Part 0; the open point of D75 (what the installer does below 200 GB, R-1370) is named. scripts/unproven.py --summary at the end: walked 32, partial 15, built 4, missing 4, not walked 23 of 55; no status changed in this run, so no number moved. System poster: none of the three actions; no listed fact changed. Instruction files: none edited. CHANGELOG and REPORT: updated in felhom-agent, felhom-controller and here (hub/CHANGELOG.md).

6. What stays not proven

  • The hub, on the running system: that a household which saved its own alarm list before no longer gets the host-disk alarm (16 §14.4). The proof is the tests and the deployed commit. The operator can read it: a household's page on the hub, the list "Events".
  • The controller: a real fill on a household's box; a fill at the critical line (95 %) on any box. The guest's own root disk still has no watcher (R-1371 part 1).
  • The agent: the .bak-* copies are not reached (R-1368 part 4). A household box cleans its folder at its first good agent update after the vouch; none was watched.
  • The kernel clean-up: it is not built as a program, and no second box did it (R-1368 part 1).

7. Machines changed, and how each was left

Machine Change State now
demo-hp host Two orderly restarts (14:30 and 14:40 UTC). Agent updated by a signed job Newest kernel runs; agent 0.173.0; online
demo-hp guest 9201, N100 guest, Tester 1 guest Controller updated by the per-box floor, the normal path 0.352.0, healthy
N100 host, Tester 1 host Agent updated by a signed job. On Tester 1 also the rollback proof Agent 0.173.0
Scratch guest 9202 A release candidate, then the release, set by hand. One test file written and removed On the release; the test file is gone; the watch's state is empty
DooPlex Three builds, the hub deploy, the manifest bump Hub 0.180.0, Synced/Healthy
ep0, Cloudflare, Hetzner, the sec store Not touched. No login to ep0. Nothing removed from the store —

Teardown, three layers. Machine: the test file on 9202 removed by its exact name. Host: nothing was provisioned. Hub: the three per-box floors were cleared at 16:03:11 UTC (three 303; the hub's log names each one), and the global floor holds the boxes; no appliance was registered. On DooPlex the clean clone used for the agent release was deleted (its git settings held a credential).

8. Architecture documents read

11-os-updates.md (§5.11, §5.11.1, §8.2) · 09-update-architecture.md (§0, §3: decisions 250, 251, 262, 286–296) · 16-storage-physical-identity.md (§14, §14.4) · 08-alarm-ladder.md (the operator-only register, the fill alarms) · 05-hub-architecture.md (§17, what a token may read) · 03-host-agent.md (the self-update part).

9. Observations outside the scope

Observation Verdict
CC ran git remote -v in felhom-agent. It printed the origin address with a login and a token into the session CC's mistake. Nothing was committed and no evidence file holds it (scanned). The token is one of the logins R-925 already lists for rotation. FILED: no new row; a note in CC's memory so that the next session does not run it
The first gate run at the close was red: CC's new memory line put the memory index 43 bytes over its limit Fixed in the session (small): CC shortened one other index line; the detail is in its topic file. The second run was green
The N100's host clock is 32 seconds behind; chrony failed again at the 11-08 boot FILED: R-1301 (dated line; documentation/audits/release-space-fixes-2026-11-10/agent/13-n100-clock.txt)
The hub's customer, events and notifications pages answer CC's token with 403; so does the OS fleet page NOT-A-FINDING: "session only — never by a token" is the design (05 §17)
CC took no copy of the hub's database to read the households' lists NOT-A-FINDING: a choice. The copy holds mail addresses and old token bytes, and the fence is by design
11 slow database lines in the hub's first 2 minutes after the deploy (the longest 19.1 s); one snapshot took 127.8 s FILED: R-1353 (dated line). The same statements as the 11-02 read
The hub logs "household cleared its mail address — deleted" for Tester 1 at each controller start NOT-A-FINDING: the same line at every start since at least 11-09 13:09 CET, before this run; Tester 1's box holds no address of its own
The hub's row for demo-hp said "unknown (no running customer guest)" while the box was on the older kernel NOT-A-FINDING: it was the agent's first report after the boot, and no newer report came in the six minutes on that kernel
9202's watch line went from "checked 11 filesystem(s)" to 3 NOT-A-FINDING: the drill's test drives lie on the one data volume; the fix watches a filesystem once. The side effect is in the controller's CHANGELOG
The first crossing try on 9202 read "ok" NOT-A-FINDING: the box freed 1.6 GB by itself between the size read and the check. The second try, with a new size read, crossed the line
9202 logs "DROPPED event" for each of its events NOT-A-FINDING: 9202 has no hub configured; its log says so ("notifier disabled (no hub configured)")
After a rollback /usr/local/bin/felhom-agent.prev is the same version as the running binary NOT-A-FINDING: how the wrapper works; it is not new. Written in the agent's report
demo-hp's controller prints three warnings at each start: an installed app (calibre-web, plex, bookstack) got a new field, and a person must fill it NOT-A-FINDING: the controller's designed note. It never fills a text or a password field by itself. This release did not change that code. The lines hold field names only
The first box walk hid two of these three lines NOT-A-FINDING: the walk's own filter drops each line that holds the word "password". The lines were read again without the filter and are in controller/12-watch.txt