Files
felhom.eu/REPORT-endings-kernels-fixes-2026-11-10.md
T
admin 025a1bcd40
gates / gates (push) Successful in 2m7s
docs: the disk-space section (16 §14), the runbook "a disk is full", the run's report
C4: documentation/runbooks/disk-full.md (R-1369). Five disks; each hand step
is PROVEN with its date and evidence, or NOT PROVEN.
C5: 16-storage-physical-identity.md §14 (D69, decision 288). 45 places, each
with limit, cleaner and alarm; the rulings behind "no limit"; the host disk
budget; the open design points. PROMPT-TEMPLATE §N.5 gets the keep-it-true line.
Close-out: STATUS (D75 new), CONTEXT, dated lines on R-1369, R-1370, R-1371,
three small stale-document fixes, the report of the run.

Documents only. No release, no build, no deploy.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-10 14:47:25 +01:00

12 KiB
Raw Blame History

REPORT — the Endings list, the kernel clean-up test, the first Space fixes (2026-11-10, operator brief, daytime)

Plain words first. CC listed every way a thing ends on a box or in the hub. CC removed six old kernels from the HP box by hand, with the keep rule the operator ruled, and restarted the box once. CC wrote three small fixes for disk space. They are in source only. Nothing was released, built or deployed. No household got a mail.

1. Claims in the brief that were wrong, or that needed a check

Claim Verdict
"07 §6.6 (retiring a drive)" Wrong. §6.6 is "which version a restore brings back". Retiring a drive is in 03-host-agent.md:390 and capability-map-detail/D.md:184
"The kernel step names its packages, and that may be why old kernels stay" True, and it is one of two causes. The other: nothing runs an autoremove, and rule R4 refuses a removal (audits/kernel-cleanup-spike-2026-11-10/evidence/01-why-still-there.txt)
"The operator-only register blocks a type a household already saved" True. Proven by a test seen failing: with the register lines out, the saved type reached the household (TestD68_HostDiskIsOperatorOnly_EvenWhenTheHouseholdSavedIt)
"The fill watch counts one filesystem twice on a one-volume box" True as measured on three guests (audits/space-list-2026-11-10/evidence/extra/fs-identity.txt). Not seen as two real mails: no disk was filled

2. The parts

Part State Note
0 — the rulings of 12:22 written down done 449b065f
A — the Endings list, with the ageing drill done 807175c2; documentation/audits/endings-list-2026-11-10/
B — kernel clean-up by hand on demo-hp done 10ea4c2c; documentation/audits/kernel-cleanup-spike-2026-11-10/
C1 — hub: a full host disk is the operator's alarm done, commit only b14eb0fa
C2 — agent: the self-update folder cleans itself done, commit only felhom-agent 776025c8
C3 — controller: one fill warning for one filesystem done, commit only felhom-controller 844f19f1
C4 — the runbook "a disk is full" done documentation/runbooks/disk-full.md
C5 — the "disk space" section done documentation/architecture/16-storage-physical-identity.md §14

Nothing was dropped. Nothing was changed in scope.

3. Results

Part A, five lines (documentation/audits/endings-list-2026-11-10/README.md):

  1. 74 endings: 67 with a code path, 7 with none. 20 more are tiny.
  2. 9 of the 67 are named in no architecture document; 24 are only mentioned.
  3. 536 answers: A 499, D 0, K 15, N 22.
  4. The worst N: the operator unticks a household's off-site tier, then resets or deletes the household. The paid account stays at the provider and the hub has no record of it.
  5. Four questions for the operator: D71–D74 in STATUS.md.

Method verdict. It worked. Weak point: a tool opened all 691 pointers, but a person-like read covered only 86 of 536 cells (every K, every N, the main A cells). The other A cells are a map, not proof. The ageing drill: ten cheap rounds found nothing; the drive round and the update round each found or proved something. Repeat it only with drive apps, on a box with an agent.

Part B, two lines. The ruled rule (keep the running kernel, the one before it and the newest) is safe by hand on demo-hp: 8 → 2 kernels, 6,411 MB back, 50 seconds, one restart, all containers healthy after 311 seconds. apt alone is not safe: it did not refuse the running kernel, and one meta-package pulled proxmox-ve — so the build must name exact packages and check the plan first (11-os-updates.md §5.11.1). Not proven: a second box; a boot of the kept older kernel.

Part C, by fix.

Fix Commit Tests Red seen
C1 hub b14eb0fa TestD68_HostDiskIsOperatorOnly_EvenWhenTheHouseholdSavedIt, TestD68_StorageFillReachesAHouseholdOnlyForADriveItCanFree, TestD68_RegisterAndGate, TestD68_SeedHoldsNoOperatorOnlyType 3 times: the register lines out; the gate reading the register only; the type back in the first list
C2 agent 776025c8 TestPrune_*, 6 tests; one runs the wrapper's real commit and rollback 3 times: no prune call; no refusal while an update is pending; no regular-file check
C3 controller 844f19f1 TestR1371_OneFillSendsOneWarning, TestR1371_TargetsByFilesystem, TestR1371_ProductionListUsesTheRealIdentity, TestFilesystemKey_AnOverlayRootIsTheDockerDataRoot, TestFilesystemKey_RealSyscalls 3 times: no filesystem check; the production list not wired; an overlay keeping its own device number
C4 runbook this commit — (document). 5 parts; each hand step PROVEN with date and evidence, or NOT PROVEN —
C5 section this commit — (document). 45 places; 8 "no limit, by ruling"; the budget; open points with rows —

Limits, said plainly. None of C1–C3 runs on any box. C1: a household that saved the type keeps the saved text; the register blocks it at send time, with no migration. C2: a box cleans its folder at the first good update after the release; the .bak-* files are not reached. C3: not proven on a running box; the guest's root disk still has no watcher. C4: every PROVEN step was proven on demo-hp only; the whole runbook was not rehearsed. C5: the budget is from three boxes; the size of the root between the three measured disks is not known.

The budget in one line (C5). The host's own files need about 14 GB when the whole-box copies go to a backup drive, and 28–54 GB when three copies stay on the system disk. The smallest disk is not decided: it is D75.

Decisions CC took in this run (the operator may reverse each). Part B: purge, not remove. C1: the warning moves with the critical; the kinds usb, local-dir, nfs, cifs are the household's, every other kind and a missing kind are the operator's. C2: the staged file of a failed version stays until the next good update. C3: the first target of a filesystem keeps its name; the identity of the container's / comes from a file Docker keeps. All four are in CONTEXT.md of felhom.eu, today's note, with the tag. The notes in the agent's and the controller's CONTEXT.md describe the change and do not carry the tag.

4. Machines changed, and how each was put back

Machine Change State now
Scratch guest 9202 The drill: 5 rounds of install, update, restore, remove As found (documentation/audits/endings-list-2026-11-10/evidence/drill/putback-1.txt, putback-2.txt)
demo-hp host Six kernel packages purged. One orderly restart, sent 12:54:26 UTC Kernels 8 → 2 (7.0.14-28 runs, 7.0.14-26 kept). / 62 % → 45 %. Hub: online, tunnel connected. Read again at 13:36 UTC: up, 45 %. The kernels are not put back; that was the test
demo-hp guest 9201, N100, Tester 1 Read only Unchanged
ep0, DooPlex workloads, Cloudflare, Hetzner, mail Not touched. No login to ep0 —

felhom-os-apply, the config bundle, the agent on the box and rule R4 did not change. Teardown: machine — 9202 put back; host — nothing provisioned; hub — nothing provisioned.

5. Commits, CI, register

Repo Commit CI (read by commit on the public Actions page)
felhom.eu 449b065f Part 0 run 2606, gates #1412, success
felhom.eu 807175c2 Part A run 2607, gates #1413, success
felhom.eu 10ea4c2c Part B run 2609, gates #1414, success
felhom.eu b14eb0fa C1 run 2611, gates #1415, success
felhom-agent 776025c8 C2 run 2608, gates #311, success
felhom-controller 844f19f1 C3 run 2610, gates #557, success
felhom.eu the commit that holds this report (C4, C5, close-out) read after the push; the result is in the session's closing note

The run's job detail could not be read: the jobs address answers 401 and no stored secret has Actions rights. The run status on the public page is what was read.

Register: 230 open at the start, 236 at the end. Opened 6 (R-1373…R-1378). Closed 0. Dated lines were added to existing rows; in this last commit to R-1369, R-1370 and R-1371.

Other close-out facts. Capability map: no status changed, because an unreleased commit proves nothing on a box; one dated note was added under A21 (see §7). System poster: none of the three actions; no listed fact changed. unproven.py --summary: 55 claims, 32 walked, 23 not walked — unchanged. CHANGELOG entries, marked unreleased: hub, agent, controller. STATUS.md: new "Updated" paragraph, the old one moved to documentation/backlog/STATUS-HISTORY.md; D71–D75 in the decide table.

Instruction file edited: documentation/PROMPT-TEMPLATE.md §N.5, one new paragraph after the poster rule (before: no line; after: "A place that holds data … updates the 'disk space' section (§14) …"). Why: the brief asked for it (D69, decision 288). No rule was loosened.

6. Architecture documents read

By the main session: 11-os-updates.md (§5.4.1, §5.5, §5.6, §5.9, §5.11) · 09-update-architecture.md §3 (decisions 40, 139, 286–290) · 08-alarm-ladder.md (the operator-only register, the fill alarms) · 16-storage-physical-identity.md (in full) · 03-host-agent.md (the restore-test space note, the self-update part) · 07-backup-architecture.md §6.

By the read-only helpers, for Part A (their pointers were opened by a tool; see the method verdict): 07-backup-architecture.md (§6.5, §8.2a–§8.2e) · 05-hub-architecture.md · 06-offsite-connectivity.md · 01-topology-and-trust.md · 00-capability-map.md with its detail files.

No architecture document covers endings as a whole. That is the finding the list tested.

7. Observations outside the scope

Observation Verdict
Two "SMART read failed … context canceled" lines in the agent's log at 12:57:35 UTC, just after the agent started NOT-A-FINDING: the first read at start was cancelled; 680 reads ran after it with no failure line (read 13:36 UTC)
host_swap_short warning after the restart NOT-A-FINDING: report only, by ruling (closed R-1207, D27)
apt's own keep list still names 7.0.14-23, a kernel that is gone NOT-A-FINDING: it had no effect on the plan; noted in the spike's README
CC's hub token reads the host list, not a host's detail page or the System page (403) NOT-A-FINDING: the token is limited on purpose
runbooks/day0-install.md said the hub has no host delete Small; FIXED in this commit (the hub has POST /hosts/{id}/delete)
runbooks/ep0-datastore-copy.md heading said "NOT INSTALLED"; closed R-901 says the job was installed on 2026-10-09 Small; FIXED in this commit, from the closed row. Not re-checked on DooPlex (fence)
Map row A21 says "tunnel + zone removed"; no code calls Cloudflare (closed R-688, accepted) Small; a dated note added under A21 in capability-map-detail/A.md. No status change
R-1268 (a restore's undo copy stays after "delete backups too") reads as fixed: the source removes it and the drill saw it go FILED: a dated line on R-1268; left open for a session that can red-prove it
Closed R-521 and a test comment call the container's / "the guest's root filesystem"; it is an overlay on the data volume FILED: R-1371 (the guest's real root disk has no watcher), in its dated line of this run
R-810 lives in ROADMAP.md only NOT-A-FINDING: the roadmap is its home; D74 points at it
The drill sent an install with no kept-data choice and expected a refusal (409). The product answered 202 and installed, because the kept folder was empty NOT-A-FINDING for the product; the tool's expectation was wrong. It was on scratch guest 9202, inside a round that installs and removes that same app. The app was removed through the product, and 9202 is back as found. Said here because .claude/rules/live-probes.md asks for it