C4: documentation/runbooks/disk-full.md (R-1369). Five disks; each hand step is PROVEN with its date and evidence, or NOT PROVEN. C5: 16-storage-physical-identity.md §14 (D69, decision 288). 45 places, each with limit, cleaner and alarm; the rulings behind "no limit"; the host disk budget; the open design points. PROMPT-TEMPLATE §N.5 gets the keep-it-true line. Close-out: STATUS (D75 new), CONTEXT, dated lines on R-1369, R-1370, R-1371, three small stale-document fixes, the report of the run. Documents only. No release, no build, no deploy. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
12 KiB
REPORT — the Endings list, the kernel clean-up test, the first Space fixes (2026-11-10, operator brief, daytime)
Plain words first. CC listed every way a thing ends on a box or in the hub. CC removed six old kernels from the HP box by hand, with the keep rule the operator ruled, and restarted the box once. CC wrote three small fixes for disk space. They are in source only. Nothing was released, built or deployed. No household got a mail.
1. Claims in the brief that were wrong, or that needed a check
| Claim | Verdict |
|---|---|
"07 §6.6 (retiring a drive)" |
Wrong. §6.6 is "which version a restore brings back". Retiring a drive is in 03-host-agent.md:390 and capability-map-detail/D.md:184 |
| "The kernel step names its packages, and that may be why old kernels stay" | True, and it is one of two causes. The other: nothing runs an autoremove, and rule R4 refuses a removal (audits/kernel-cleanup-spike-2026-11-10/evidence/01-why-still-there.txt) |
| "The operator-only register blocks a type a household already saved" | True. Proven by a test seen failing: with the register lines out, the saved type reached the household (TestD68_HostDiskIsOperatorOnly_EvenWhenTheHouseholdSavedIt) |
| "The fill watch counts one filesystem twice on a one-volume box" | True as measured on three guests (audits/space-list-2026-11-10/evidence/extra/fs-identity.txt). Not seen as two real mails: no disk was filled |
2. The parts
| Part | State | Note |
|---|---|---|
| 0 — the rulings of 12:22 written down | done | 449b065f |
| A — the Endings list, with the ageing drill | done | 807175c2; documentation/audits/endings-list-2026-11-10/ |
| B — kernel clean-up by hand on demo-hp | done | 10ea4c2c; documentation/audits/kernel-cleanup-spike-2026-11-10/ |
| C1 — hub: a full host disk is the operator's alarm | done, commit only | b14eb0fa |
| C2 — agent: the self-update folder cleans itself | done, commit only | felhom-agent 776025c8 |
| C3 — controller: one fill warning for one filesystem | done, commit only | felhom-controller 844f19f1 |
| C4 — the runbook "a disk is full" | done | documentation/runbooks/disk-full.md |
| C5 — the "disk space" section | done | documentation/architecture/16-storage-physical-identity.md §14 |
Nothing was dropped. Nothing was changed in scope.
3. Results
Part A, five lines (documentation/audits/endings-list-2026-11-10/README.md):
- 74 endings: 67 with a code path, 7 with none. 20 more are tiny.
- 9 of the 67 are named in no architecture document; 24 are only mentioned.
- 536 answers:
A499,D0,K15,N22. - The worst
N: the operator unticks a household's off-site tier, then resets or deletes the household. The paid account stays at the provider and the hub has no record of it. - Four questions for the operator: D71–D74 in
STATUS.md.
Method verdict. It worked. Weak point: a tool opened all 691 pointers, but a person-like read covered only 86 of
536 cells (every K, every N, the main A cells). The other A cells are a map, not proof. The ageing drill: ten
cheap rounds found nothing; the drive round and the update round each found or proved something. Repeat it only with
drive apps, on a box with an agent.
Part B, two lines. The ruled rule (keep the running kernel, the one before it and the newest) is safe by hand on
demo-hp: 8 → 2 kernels, 6,411 MB back, 50 seconds, one restart, all containers healthy after 311 seconds. apt alone
is not safe: it did not refuse the running kernel, and one meta-package pulled proxmox-ve — so the build must name
exact packages and check the plan first (11-os-updates.md §5.11.1). Not proven: a second box; a boot of the kept
older kernel.
Part C, by fix.
| Fix | Commit | Tests | Red seen |
|---|---|---|---|
| C1 hub | b14eb0fa |
TestD68_HostDiskIsOperatorOnly_EvenWhenTheHouseholdSavedIt, TestD68_StorageFillReachesAHouseholdOnlyForADriveItCanFree, TestD68_RegisterAndGate, TestD68_SeedHoldsNoOperatorOnlyType |
3 times: the register lines out; the gate reading the register only; the type back in the first list |
| C2 agent | 776025c8 |
TestPrune_*, 6 tests; one runs the wrapper's real commit and rollback |
3 times: no prune call; no refusal while an update is pending; no regular-file check |
| C3 controller | 844f19f1 |
TestR1371_OneFillSendsOneWarning, TestR1371_TargetsByFilesystem, TestR1371_ProductionListUsesTheRealIdentity, TestFilesystemKey_AnOverlayRootIsTheDockerDataRoot, TestFilesystemKey_RealSyscalls |
3 times: no filesystem check; the production list not wired; an overlay keeping its own device number |
| C4 runbook | this commit | — (document). 5 parts; each hand step PROVEN with date and evidence, or NOT PROVEN |
— |
| C5 section | this commit | — (document). 45 places; 8 "no limit, by ruling"; the budget; open points with rows | — |
Limits, said plainly. None of C1–C3 runs on any box. C1: a household that saved the type keeps the saved text; the
register blocks it at send time, with no migration. C2: a box cleans its folder at the first good update after the
release; the .bak-* files are not reached. C3: not proven on a running box; the guest's root disk still has no
watcher. C4: every PROVEN step was proven on demo-hp only; the whole runbook was not rehearsed. C5: the budget is
from three boxes; the size of the root between the three measured disks is not known.
The budget in one line (C5). The host's own files need about 14 GB when the whole-box copies go to a backup drive, and 28–54 GB when three copies stay on the system disk. The smallest disk is not decided: it is D75.
Decisions CC took in this run (the operator may reverse each). Part B: purge, not remove. C1: the warning moves
with the critical; the kinds usb, local-dir, nfs, cifs are the household's, every other kind and a missing kind
are the operator's. C2: the staged file of a failed version stays until the next good update. C3: the first target of a
filesystem keeps its name; the identity of the container's / comes from a file Docker keeps. All four are in
CONTEXT.md of felhom.eu, today's note, with the tag. The notes in the agent's and the controller's CONTEXT.md
describe the change and do not carry the tag.
4. Machines changed, and how each was put back
| Machine | Change | State now |
|---|---|---|
| Scratch guest 9202 | The drill: 5 rounds of install, update, restore, remove | As found (documentation/audits/endings-list-2026-11-10/evidence/drill/putback-1.txt, putback-2.txt) |
| demo-hp host | Six kernel packages purged. One orderly restart, sent 12:54:26 UTC | Kernels 8 → 2 (7.0.14-28 runs, 7.0.14-26 kept). / 62 % → 45 %. Hub: online, tunnel connected. Read again at 13:36 UTC: up, 45 %. The kernels are not put back; that was the test |
| demo-hp guest 9201, N100, Tester 1 | Read only | Unchanged |
| ep0, DooPlex workloads, Cloudflare, Hetzner, mail | Not touched. No login to ep0 | — |
felhom-os-apply, the config bundle, the agent on the box and rule R4 did not change. Teardown: machine — 9202 put
back; host — nothing provisioned; hub — nothing provisioned.
5. Commits, CI, register
| Repo | Commit | CI (read by commit on the public Actions page) |
|---|---|---|
| felhom.eu | 449b065f Part 0 |
run 2606, gates #1412, success |
| felhom.eu | 807175c2 Part A |
run 2607, gates #1413, success |
| felhom.eu | 10ea4c2c Part B |
run 2609, gates #1414, success |
| felhom.eu | b14eb0fa C1 |
run 2611, gates #1415, success |
| felhom-agent | 776025c8 C2 |
run 2608, gates #311, success |
| felhom-controller | 844f19f1 C3 |
run 2610, gates #557, success |
| felhom.eu | the commit that holds this report (C4, C5, close-out) | read after the push; the result is in the session's closing note |
The run's job detail could not be read: the jobs address answers 401 and no stored secret has Actions rights. The
run status on the public page is what was read.
Register: 230 open at the start, 236 at the end. Opened 6 (R-1373…R-1378). Closed 0. Dated lines were added to existing rows; in this last commit to R-1369, R-1370 and R-1371.
Other close-out facts. Capability map: no status changed, because an unreleased commit proves nothing on a box; one
dated note was added under A21 (see §7). System poster: none of the three actions; no listed fact changed.
unproven.py --summary: 55 claims, 32 walked, 23 not walked — unchanged. CHANGELOG entries, marked unreleased: hub,
agent, controller. STATUS.md: new "Updated" paragraph, the old one moved to documentation/backlog/STATUS-HISTORY.md;
D71–D75 in the decide table.
Instruction file edited: documentation/PROMPT-TEMPLATE.md §N.5, one new paragraph after the poster rule (before:
no line; after: "A place that holds data … updates the 'disk space' section (§14) …"). Why: the brief asked for it
(D69, decision 288). No rule was loosened.
6. Architecture documents read
By the main session: 11-os-updates.md (§5.4.1, §5.5, §5.6, §5.9, §5.11) · 09-update-architecture.md §3
(decisions 40, 139, 286–290) · 08-alarm-ladder.md (the operator-only register, the fill alarms) ·
16-storage-physical-identity.md (in full) · 03-host-agent.md (the restore-test space note, the self-update part) ·
07-backup-architecture.md §6.
By the read-only helpers, for Part A (their pointers were opened by a tool; see the method verdict):
07-backup-architecture.md (§6.5, §8.2a–§8.2e) · 05-hub-architecture.md · 06-offsite-connectivity.md ·
01-topology-and-trust.md · 00-capability-map.md with its detail files.
No architecture document covers endings as a whole. That is the finding the list tested.
7. Observations outside the scope
| Observation | Verdict |
|---|---|
| Two "SMART read failed … context canceled" lines in the agent's log at 12:57:35 UTC, just after the agent started | NOT-A-FINDING: the first read at start was cancelled; 680 reads ran after it with no failure line (read 13:36 UTC) |
host_swap_short warning after the restart |
NOT-A-FINDING: report only, by ruling (closed R-1207, D27) |
apt's own keep list still names 7.0.14-23, a kernel that is gone |
NOT-A-FINDING: it had no effect on the plan; noted in the spike's README |
| CC's hub token reads the host list, not a host's detail page or the System page (403) | NOT-A-FINDING: the token is limited on purpose |
runbooks/day0-install.md said the hub has no host delete |
Small; FIXED in this commit (the hub has POST /hosts/{id}/delete) |
runbooks/ep0-datastore-copy.md heading said "NOT INSTALLED"; closed R-901 says the job was installed on 2026-10-09 |
Small; FIXED in this commit, from the closed row. Not re-checked on DooPlex (fence) |
| Map row A21 says "tunnel + zone removed"; no code calls Cloudflare (closed R-688, accepted) | Small; a dated note added under A21 in capability-map-detail/A.md. No status change |
| R-1268 (a restore's undo copy stays after "delete backups too") reads as fixed: the source removes it and the drill saw it go | FILED: a dated line on R-1268; left open for a session that can red-prove it |
Closed R-521 and a test comment call the container's / "the guest's root filesystem"; it is an overlay on the data volume |
FILED: R-1371 (the guest's real root disk has no watcher), in its dated line of this run |
R-810 lives in ROADMAP.md only |
NOT-A-FINDING: the roadmap is its home; D74 points at it |
| The drill sent an install with no kept-data choice and expected a refusal (409). The product answered 202 and installed, because the kept folder was empty | NOT-A-FINDING for the product; the tool's expectation was wrong. It was on scratch guest 9202, inside a round that installs and removes that same app. The app was removed through the product, and 9202 is back as found. Said here because .claude/rules/live-probes.md asks for it |