Space fixes released: agent 0.173.0 vouched, controller 0.352.0 global floor, hub 0.180.0 live; evidence, rows, status, report
gates / gates (push) Successful in 2m1s

The close-out of the attended release run of 2026-11-10 (afternoon).
- Evidence: documentation/audits/release-space-fixes-2026-11-10/ (agent, controller, hub) and the hub row after the
  fallback-kernel test.
- Rows: dated lines on R-1368, R-1371, R-1301, R-1353. 236 open before and after; opened 0, closed 0.
- Capability map: dated notes under C13 and G5; no status changed.
- disk-full runbook and 16 §14: the self-update folder is PROVEN as a product path on three test boxes.
- STATUS: new "Updated" paragraph; the old one moved to STATUS-HISTORY.
- REPORT-release-space-fixes-2026-11-10.md: wrong claims first, one deviation (the hub deploy ran before CI for the
  manifest-bump commit was read), what stays not proven.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-10 17:17:04 +01:00
1 parent d0a54c2f8e
commit bcc73b428d
40 files changed
+1037 -59

No files matched your search

+12
View File
@@ -16,6 +16,18 @@
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
> **2026-11-10 (afternoon, attended release run) — the three Space fixes are RELEASED; the note below is history.**
> (1) Agent 0.173.0, vouched by the operator 15:29 UTC: the self-update folder cleans itself at the commit of the
> update that brings the fix (49 / 48 / 33 files → 1 on the three test boxes); a rollback after the clean-up was proven
> on Tester 1. (2) Controller 0.352.0, global floor 15:31 UTC, MinAgent 0.158.1: one fill warning per filesystem; seen
> on scratch guest 9202 as two warnings before and one after. (3) Hub 0.180.0, deployed 15:12 UTC on the operator's
> yes: the host-disk alarm is the operator's only. **Not proven on the running hub:** a household that saved its own
> alarm list — the pages that show the list refuse a token and no test alarm was made (`16` §14.4). (4) The kept older
> kernel booted demo-hp to healthy in 313 s, and the box went back. (5) The rulings D71–D75 are written down
> (`09` §3, decisions 291–296); none is built. **One deviation:** the hub deploy ran 3 minutes before CI for the
> manifest-bump commit was read green; the image's own source commit had green CI before the build. Evidence:
> `documentation/audits/release-space-fixes-2026-11-10/`; report `REPORT-release-space-fixes-2026-11-10.md`.
> **2026-11-10 (day, operator brief) — COMMITTED, NOT RELEASED: D68 in the hub, and two sibling fixes.** (1) Hub source:
> `host_disk_warning`/`host_disk_critical` are operator-only and the critical is not seeded at claim; `storage_fill_*`
> is split by `storage_type` in `notify.operatorOnlyFor` (household: `usb`, `local-dir`, `nfs`, `cifs`; operator only:
+155
View File
@@ -0,0 +1,155 @@
# REPORT — today's rulings, the fallback kernel, the release of the three Space fixes (2026-11-10, afternoon, attended)
**Plain words first.** CC wrote down the five rulings of 15:19 and built none. The HP box booted its kept older
kernel once, was healthy, and went back. Three fixes are released: the agent cleans its own update folder (the
operator vouched it), the controller sends one warning for one full disk (the global floor is set), and the hub sends
the host-disk alarm to the operator only (deployed on the operator's yes). No golden, no catalog push, no change on
ep0. No household got a mail from this run.
## 1. Claims that turned out wrong — in the brief, and in what the brief told CC to read first
| Claim | Verdict |
|---|---|
| The brief, Part 2 step 3: "If the folder is cleaned only at the *next* good update after this one, say so plainly" | **It is not so.** The folder is cleaned at the commit of the update that brings the fix. The commit runs in the new binary. Measured on three boxes (`removed=49`, `48`, `33`). No second version was needed |
| The morning's report (§3, limits) and the agent's CHANGELOG entry: "a box cleans its folder at the first good update AFTER the release" | **Wrong, for the same reason.** The CHANGELOG entry is corrected. The morning's report is a dated record and is not edited |
| The morning's report (§1): the double fill warning was "not seen as two real mails" | **Now seen on a box.** Scratch guest 9202 on the version before, one fill: two warnings, two names. 9202 has no hub, so they are two warnings in the box's own record, not two mails in a mailbox |
| The brief, Part 4 step 4: "Read what the deployed hub lets you read about the open point in `16` §14.4" | **It lets CC read nothing about it.** The pages that show a household's alarm list answer a token with 403, "session only". That is by design (`05` §17). The point stays not measured on the running hub |
| The brief, for the operator: "The HP box restarts twice, about 5 minutes each" | True: 313 s and 324 s to healthy |
## 2. Five lines
1. **The older kernel booted to healthy: yes, in 313 seconds.** The box went back to its newest kernel in 324 seconds.
Nothing was left behind: the pinned default, `grub.cfg` and the kernel-step state have the same checksum before and
after; the hub got no kernel-step report from the test.
2. **Agent 0.173.0 is vouched** (the operator, 15:29:16 UTC, read back from the hub's log). The self-update folder:
demo-hp 49 files → 1, demo-felhom 48 → 1, Tester 1 33 → 1 (722,596 / 706,968 / 497,244 KB → 16,028 KB each). A
rollback after the clean-up worked on Tester 1.
3. **Every box that reports is on controller 0.352.0:** Tester 1, demo-hp and demo-felhom (the hub's host list holds
these three; Tester 2 is off). The fill watch's target list on each: 2 filesystems, one target each (it was 3
targets on 2 filesystems). The global floor is 0.352.0 with MinAgent 0.158.1, set 15:31:34 UTC in its own save.
In the 30 minutes after: every box on the release and healthy, 4 reports from each box, no event, no alarm and no
mail in the hub's log. A second channel agrees: the operator's mailbox got no mail in that time (the same search
found the night's mail, so the search works).
4. **Hub 0.180.0 is live** since 15:12:48 UTC: Synced/Healthy, no restart, no error line and no alarm sent in the 30
minutes after; 6 reports came from each box in that time.
5. **What stays not proven:** see §6.
## 3. The parts
| Part | State | Note |
|---|---|---|
| 0 — the rulings D71–D75 written down | done | `e83a78ce`. `09` §3, decisions 291–296. None is built |
| 1 — the fallback kernel on demo-hp | done | `5f408cac`. `documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/` |
| 2 — the agent release | done, vouched | felhom-agent `776025c8` (the release), `5cebb766`, `9edaa8c0` (notes, the D50 pin) |
| 3 — the controller release | done | felhom-controller `3dbb68c1` (the release), `a9189c9d` (notes) |
| 4 — the hub release | done | `97b7e471` (the image's commit), `d0a54c2f` (the manifest bump) |
Nothing was dropped. No STOP ran out: the operator answered both asks (the hub deploy: "Yes, approved"; the vouch: "I
vouched it").
**Part 1, the method.** The product's own boot-side switch, set by hand: one flag in the GRUB environment file, then
an orderly restart. GRUB cleared the flag and booted the older kernel once. The second restart had no flag. No wrapper
verb ran; no kernel-step result was written; the pinned default did not change. Not covered: a step where the product
itself chooses the old kernel.
**Part 2, the rollback proof.** The path the repo already has: a signed update on Tester 1 to the older version, then
one crash 9 seconds into the 60-second dwell. The wrapper put 0.173.0 back in the same second. After it: the unit
active, no pending marker, 2 files in the folder (the failed version's file stays until the next good update, by
design). This is one crash, not a crash loop.
**Part 3, red and green on a box.** Scratch guest 9202, one test file of a known name inside the free space, never
into the reserve, removed by its exact name. On the version before: two warnings. On `0.352.0-rc.1`: one warning; ten
minutes after the file was gone the watch said "cleared silently, re-armed". Then the release build, per-box floors on
the three test boxes (each moved in under 15 seconds), the box walk green (containers 10 / 26 / 9, none unhealthy, no
error line), the https-only check green, and the global floor last.
**Part 4.** Tests green on the image's commit (`TestD68_*`). Only the hub's Deployment was out of sync before the
sync, and only it was synced. No test alarm was made.
## 4. One deviation from the rules, said plainly
**The hub deploy ran before CI for the manifest-bump commit was read.** Rule 2 says CI green by commit hash before
each release. The deploy was 15:12:48 UTC; CI for `d0a54c2f` was read green at 15:15:35 UTC (run 2617). CC had
chained the deploy to the push check and not to the CI read. What limits the harm: the image was built from
`97b7e471`, and CI for that commit was green before the build (run 2616); the bump commit changes one image tag. It
went well, but it was not the order the rule asks for.
## 5. Versions, digests, CI by commit
| What | Version | Digest / checksum | Built from | CI (read by commit on the public Actions page) |
|---|---|---|---|---|
| Agent | 0.173.0 | binary sha256 `752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea`; config bundle sha256 `51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1` | `776025c8` | run 2608, `gates #311`, success |
| Controller | 0.352.0 (MinAgent 0.158.1) | image `sha256:3c785dd396b091f19d4c777db9d7a206e75ab018b6238b5d887e1a4c6cfd3b29` | `3dbb68c1` | run 2615, `gates #558`, success |
| Hub | 0.180.0 | image `sha256:768e73ab1ab502a8c3fe23a9d42cc038443a6456a1f40dcec853a2c588d04a8f` | `97b7e471` | run 2616, `gates #1418`, success |
| Repo | Other commits of this run | CI |
|---|---|---|
| felhom.eu | `e83a78ce` Part 0 | run 2613, `gates #1417`, success |
| felhom.eu | `5f408cac` Part 1 | pushed in one push with `97b7e471`; CI ran on the head, run 2616 |
| felhom.eu | `d0a54c2f` the manifest bump | run 2617, `gates #1419`, success (read after the deploy, §4) |
| felhom-agent | `5cebb766` release notes | run 2618, `gates #313`, success |
| felhom-agent | `9edaa8c0` the vouch, `pinned.day0_agent` | run 2619, `gates #314`, success |
| felhom-controller | `a9189c9d` release notes | run 2620, `gates #559`, success |
| felhom.eu | the commit that holds this report | read after the push; the result is in the session's closing note |
**Register:** 236 open at the start, 236 at the end. Opened 0. Closed 0. Dated lines on R-1368, R-1371, R-1301 and
R-1353. R-1368 and R-1371 stay open: each has parts no box proves yet.
**Other close-out facts.** Capability map: no status changed; two dated notes (under C13 and G5) for what a box now
proves. `STATUS.md`: a new "Updated" paragraph; the old one moved to `documentation/backlog/STATUS-HISTORY.md`; D71–D75
were taken out of the decide table in Part 0; the open point of D75 (what the installer does below 200 GB, R-1370) is
named. `scripts/unproven.py --summary` at the end: walked 32, partial 15, built 4, missing 4, not walked 23 of 55; no
status changed in this run, so no number moved. System poster: none of the three actions; no listed fact changed. Instruction files: none edited. CHANGELOG
and REPORT: updated in felhom-agent, felhom-controller and here (`hub/CHANGELOG.md`).
## 6. What stays not proven
- **The hub, on the running system:** that a household which saved its own alarm list before no longer gets the
host-disk alarm (`16` §14.4). The proof is the tests and the deployed commit. The operator can read it: a
household's page on the hub, the list "Events".
- **The controller:** a real fill on a household's box; a fill at the critical line (95 %) on any box. The guest's
own root disk still has no watcher (R-1371 part 1).
- **The agent:** the `.bak-*` copies are not reached (R-1368 part 4). A household box cleans its folder at its first
good agent update after the vouch; none was watched.
- **The kernel clean-up:** it is not built as a program, and no second box did it (R-1368 part 1).
## 7. Machines changed, and how each was left
| Machine | Change | State now |
|---|---|---|
| demo-hp host | Two orderly restarts (14:30 and 14:40 UTC). Agent updated by a signed job | Newest kernel runs; agent 0.173.0; online |
| demo-hp guest 9201, N100 guest, Tester 1 guest | Controller updated by the per-box floor, the normal path | 0.352.0, healthy |
| N100 host, Tester 1 host | Agent updated by a signed job. On Tester 1 also the rollback proof | Agent 0.173.0 |
| Scratch guest 9202 | A release candidate, then the release, set by hand. One test file written and removed | On the release; the test file is gone; the watch's state is empty |
| DooPlex | Three builds, the hub deploy, the manifest bump | Hub 0.180.0, Synced/Healthy |
| ep0, Cloudflare, Hetzner, the `sec` store | Not touched. No login to ep0. Nothing removed from the store | — |
**Teardown, three layers.** Machine: the test file on 9202 removed by its exact name. Host: nothing was provisioned.
Hub: the three per-box floors were cleared at 16:03:11 UTC (three `303`; the hub's log names each one), and the global
floor holds the boxes; no appliance was registered. On DooPlex the clean clone used for the agent release was deleted
(its git settings held a credential).
## 8. Architecture documents read
`11-os-updates.md` (§5.11, §5.11.1, §8.2) · `09-update-architecture.md` (§0, §3: decisions 250, 251, 262, 286–296) ·
`16-storage-physical-identity.md` (§14, §14.4) · `08-alarm-ladder.md` (the operator-only register, the fill alarms) ·
`05-hub-architecture.md` (§17, what a token may read) · `03-host-agent.md` (the self-update part).
## 9. Observations outside the scope
| Observation | Verdict |
|---|---|
| CC ran `git remote -v` in felhom-agent. It printed the origin address with a login and a token into the session | CC's mistake. Nothing was committed and no evidence file holds it (scanned). The token is one of the logins R-925 already lists for rotation. FILED: no new row; a note in CC's memory so that the next session does not run it |
| The first gate run at the close was red: CC's new memory line put the memory index 43 bytes over its limit | Fixed in the session (small): CC shortened one other index line; the detail is in its topic file. The second run was green |
| The N100's host clock is 32 seconds behind; `chrony` failed again at the 11-08 boot | FILED: R-1301 (dated line; `documentation/audits/release-space-fixes-2026-11-10/agent/13-n100-clock.txt`) |
| The hub's customer, events and notifications pages answer CC's token with 403; so does the OS fleet page | NOT-A-FINDING: "session only — never by a token" is the design (`05` §17) |
| CC took no copy of the hub's database to read the households' lists | NOT-A-FINDING: a choice. The copy holds mail addresses and old token bytes, and the fence is by design |
| 11 slow database lines in the hub's first 2 minutes after the deploy (the longest 19.1 s); one snapshot took 127.8 s | FILED: R-1353 (dated line). The same statements as the 11-02 read |
| The hub logs "household cleared its mail address — deleted" for Tester 1 at each controller start | NOT-A-FINDING: the same line at every start since at least 11-09 13:09 CET, before this run; Tester 1's box holds no address of its own |
| The hub's row for demo-hp said "unknown (no running customer guest)" while the box was on the older kernel | NOT-A-FINDING: it was the agent's first report after the boot, and no newer report came in the six minutes on that kernel |
| 9202's watch line went from "checked 11 filesystem(s)" to 3 | NOT-A-FINDING: the drill's test drives lie on the one data volume; the fix watches a filesystem once. The side effect is in the controller's CHANGELOG |
| The first crossing try on 9202 read "ok" | NOT-A-FINDING: the box freed 1.6 GB by itself between the size read and the check. The second try, with a new size read, crossed the line |
| 9202 logs "DROPPED event" for each of its events | NOT-A-FINDING: 9202 has no hub configured; its log says so ("notifier disabled (no hub configured)") |
| After a rollback `/usr/local/bin/felhom-agent.prev` is the same version as the running binary | NOT-A-FINDING: how the wrapper works; it is not new. Written in the agent's report |
| demo-hp's controller prints three warnings at each start: an installed app (calibre-web, plex, bookstack) got a new field, and a person must fill it | NOT-A-FINDING: the controller's designed note. It never fills a text or a password field by itself. This release did not change that code. The lines hold field names only |
| The first box walk hid two of these three lines | NOT-A-FINDING: the walk's own filter drops each line that holds the word "password". The lines were read again without the filter and are in `controller/12-watch.txt` |
+17 -50
View File
@@ -1,54 +1,21 @@
# REPORT — demo-hp's full local tier, and what the whole-box copy holds (2026-11-09, attended)
# REPORT — the three Space fixes are released (2026-11-10, afternoon, attended)
**Plain words first.** demo-hp did not restart on 11-08/09. Its local whole-box copy was refused for space again
(12.5 GiB free, 13.3 GiB needed), so the night's OS run and its kernel step waited. With the operator's yes, CC freed
space and filed the causes. The controller now deletes two kinds of old images by itself: v0.351.0, **released the
same day** (global floor 12:18 UTC, MinAgent 0.158.1).
**Plain words first.** Three fixes are released. The agent cleans its own update folder; the operator vouched it. The
controller sends one warning for one full disk; the global floor is set. The hub sends the host-disk alarm to the
operator only; it was deployed on the operator's yes. The HP box booted its kept older kernel once, was healthy, and
went back. The five rulings D71–D75 are written down and none is built.
## What the whole-box copy holds (demo-hp guest 9201, `docker system df -v`, measured)
**The full report is `REPORT-release-space-fixes-2026-11-10.md`.** It names first the claims that turned out wrong,
and one deviation from the rules (the hub deploy ran before CI for the manifest-bump commit was read).
| Part | Size |
| | |
|---|---|
| Images used by containers | 13.2 GB |
| The apps' undo images (decision 53, on purpose) | 9.8 GB |
| Old controller images (25 of them test builds) | 2.2 GB |
| Old infrastructure and helper images | 1.35 GB |
| Docker volumes (app data) | ~1.5 GB |
| `sys_drive` | 1.3 GB |
Packed: 9.8 GB, on a 39 GB system disk. The N100 writes its copy to its 1 TB backup drive (`felhom-backup`), 1.5 GB.
Tester 1 and demo-hp write to the system disk (`local`). The same old-image pattern was on all three boxes (43–48% of
image space unused; most of it is undo images).
## What was done on demo-hp (operator's yes)
- Inside guest 9201: 36 unused images deleted by name (25 old controller test builds; traefik v3.6.7/v3.7.13;
cloudflared 2026.6.0/2026.9.3; Samba 1.1.0; FileBrowser 1.3.3; alpine:3.20, curl, hello-world). The undo images,
the previous controller (0.349.0) and the previous pins were kept. Images 27.56 → 24.15 GB.
- On the host: installer ISO 1.30.0 removed from `local` (`pvesm free`); its sha256 `590d7b9b…` matched the copy on
DooPlex and the file on iso.felhom.eu (HTTP 200). The old golden archive 0.236.0 stays (no other copy found).
- `local`: 12.5 → 14.0 GiB free, above the 13.3 GiB the space check asks.
- Evidence off-git: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/` (before, cleanup, journal noise).
**Not yet observed:** tonight's local copy, OS run and kernel step on demo-hp. Positive observable: a new
`vzdump-lxc-9201-2026_11_10-*.tar.zst` in `/var/lib/vz/dump/`, then the host's boot time after the kernel step.
## Product changes
- Controller v0.351.0 (felhom-controller CHANGELOG): decision 56 amended (old `-rc` controller images go) and decision
285 (pinned infrastructure images keep the pin and the previous pin). Six mutants red-proved.
- **Released 2026-11-09** (`documentation/audits/release-2026-11-09/README.md`): rc.1 by hand on Tester 1 (37 → 24
images, 12 controller + 2 infrastructure images deleted, about 1 GB freed, because the test builds share layers);
per-box floors 12:14 UTC, all three test boxes on 0.351.0 by 12:15; each box's own pass deleted what the rule says
(demo-felhom: 1 controller + 5 infrastructure images); walk green on all three, `https_only_check.py` rc 0; global
floor 0.351.0 / MinAgent 0.158.1 at 12:18 UTC, read back from the hub log and the configuration page; per-box floors
cleared.
- `09-update-architecture.md` §3: decision 285 added, decision 56 amended.
## Register
Rows before 223, after 225; opened 2 (R-1366 the whole-box copy carries every image — operator decision; R-1367 the
host journal's sudo session noise), closed 0. R-548 extended with the 11-09 refusal and the clean-up.
Architecture read: `07-backup-architecture.md` (tiers), `09-update-architecture.md` §3 (decisions 53, 56).
System poster: no fact changed (no backup tier, time or retention moved).
| Agent | 0.173.0, vouched by the operator 15:29 UTC. The self-update folder on the three test boxes: 49 → 1, 48 → 1, 33 → 1 files. A rollback worked after the clean-up |
| Controller | 0.352.0, MinAgent 0.158.1. Global floor 15:31 UTC. The three boxes that report are on it. Each watches 2 filesystems, one target each |
| Hub | 0.180.0, live since 15:12 UTC, deployed by manifest bump. Synced/Healthy, no alarm sent in the 30 minutes after |
| The older kernel | demo-hp booted it to healthy in 313 seconds and went back in 324 seconds. Nothing was left behind |
| Rulings | D71–D75 in `documentation/architecture/09-update-architecture.md` §3, decisions 291–296 |
| Register | 236 open at the start, 236 at the end. Opened 0, closed 0. Dated lines on R-1368, R-1371, R-1301, R-1353 |
| Not proven | That a household which saved its own alarm list no longer gets the host-disk alarm, on the running hub. A real fill on a household's box. The kernel clean-up as a program |
| Evidence | `documentation/audits/release-space-fixes-2026-11-10/` and `documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/` |
| Not done, by the brief | No golden. No catalog push. No change on ep0. No test alarm. No mail to a household |
+1 -1
View File
@@ -2,7 +2,7 @@
**Ready for Tester 2? Yes — tell the friend go (`documentation/pilot/TESTER2-handover.md`).**
**Updated 2026-11-10 (late afternoon).** You ruled D71–D75 at 15:19 (`documentation/architecture/09-update-architecture.md` §3, decisions 291–296); none is built. The smallest system disk is 200 GB, and what the installer does below 200 GB (refuse, or warn) is open (R-1370). The report of this run is `REPORT-release-space-fixes-2026-11-10.md`.
**Updated 2026-11-10 (evening).** Three releases are live: the agent that cleans its own update folder (you vouched it), the controller that sends one warning for one full disk (every box takes it by itself), and the hub that sends the host-disk alarm to you only. You ruled D71–D75 at 15:19 and none is built; the smallest system disk is 200 GB, and what the installer does below 200 GB (refuse, or warn) is open (R-1370). The report of the run is `REPORT-release-space-fixes-2026-11-10.md`.
## What works
- **The Plex attack did not reach Felhom's secrets** (R-1241).
@@ -278,7 +278,7 @@ The alarms, by short name:
| H9 | old kernels | each approved kernel | none today. Ruled: keep the running one, the one before it and the newest (decision 286, `11-os-updates.md` §5.11.1). Proven by hand, not built | nobody today | host disk; later `os_update_stale` |
| H10 | the EFI partition | a boot-loader update | 1.07 GB, 1 % used | none needed | none (R-1371) |
| H11 | the host package cache | each OS update | emptied in each run | the update wrapper | host disk |
| H12 | the agent's self-update folder | each agent update | none on a released agent; one binary after the next agent release (in source, R-1368) | the agent, after that release | host disk |
| H12 | the agent's self-update folder | each agent update | the running version's binary, and the file of a failed update until the next good one (about 16 MB each; measured 2026-11-10 on three boxes, R-1368) | the agent, at the end of each good update | host disk |
| H13 | old agent binaries | `.prev`: one file. `.bak-*`: each re-run of the installer | none for `.bak-*` (R-1368) | nobody | host disk |
| H14 | wrapper state and previous config bundles | each bundle | the last 3 | the wrapper | host disk |
| H15 | host `/tmp` and `/var/tmp` | nothing that lasts | `/tmp` is in memory: half of memory | systemd, after 10 and 30 days | none for `/tmp` (R-1371) |
@@ -408,7 +408,7 @@ below 200 GB — refuse, or warn (R-1370). demo-hp (128 GB) stays as a test box.
| Nobody watches the guest's root disk (G1, G13, G16). It is 4–5 % used today; app definitions lie on it | R-1371 |
| No alarm for the EFI partition (H10) or for the places in memory (H15, G15, G17) | R-1371 |
| A drive in the backup role reports as `usb` or `local-dir`. So the household gets the `storage_fill` mail for a drive it must not clean by hand | R-1371 |
| The hub's first list for a household still holds `storage_fill_critical`. It is replaced when the box sends its own list. Not measured on the running hub in the 2026-11-10 run | R-1371 |
| The hub's first list for a household still holds `storage_fill_critical`. It is replaced when the box sends its own list. **Not measured on the running hub:** the release run of 2026-11-10 deployed the hub that blocks the host-disk alarm at send time, but the pages that show a household's list refuse a token („session only"), and no test alarm was made. The proof is the tests and the deployed commit (`audits/release-space-fixes-2026-11-10/hub/`) | R-1371 |
| Helper images carry no version; the guest's journal has no stated cap; off-site restores have no limit | R-1372 |
| Old kernels: the keep rule is ruled and proven by hand, not built | `11-os-updates.md` §5.11.1 |
| The parts of the host disk with no limit and no cleaner (H4, H5, H6, H13) | R-1087, R-1368 |
@@ -229,6 +229,8 @@
**Gap / roadmap (as written):** **Nothing warned before this.** The only prior signal was the healthcheck's generic `health_degraded` at 90%, for REGISTERED STORAGE PATHS ONLY — it never looked at the docker area or the system-data area, never gave a free-byte figure and never named a drive. **The two event types already existed with NO PRODUCER** (`disk_warning`/`disk_critical`: allowlisted, copy'd, in `DefaultEnabledEvents`, checkbox'd) — the **sixth** *built-but-never-wired* instance here; this ships their producer rather than a seventh near-duplicate type. **Two threshold terms, whichever trips first, and the live proof vindicated the design:** the critical crossing fired on the FREE-BYTE term (1.7 GB) at only **91%** used — a percentage-only rule would have missed it. The hub's generic `customerMessages` entries were REMOVED, because `FormatCustomerEmail` prefers the entry over the message and would discard the label and figures. **Known gap → R-177:** there is no operator-triggerable run-now path; the check is daily 03:30 + once at startup, so confirming a cleared warning on a support call needs a controller restart or a wait
**Noted 2026-11-10 (release run):** on a box with one data volume the watch counted that volume twice. Seen on scratch guest 9202 with one real fill: the old controller logged TWO warnings for it, under two names (`audits/release-space-fixes-2026-11-10/controller/03-crossing-on-0.350.0.txt`); the fixed controller logged ONE warning, and cleared it without a message when the file was removed (`04-rc1-on-9202.txt`). On the three test boxes the watch now checks 2 filesystems, was 3 (`09-fs-identity-after.txt`). 9202 has no hub, so no mail was sent; one mail for one fill is proven by the box's own record, not by a mailbox. Still open: the guest's root disk has no watcher (R-1371).
---
<a id="c14"></a>
@@ -91,6 +91,8 @@
**Gap / roadmap (as written):** Remote-customer proof pending → R-1
**Noted 2026-11-10 (release run):** the agent now removes old downloaded binaries after a good update: 49, 48 and 33 files became 1 on demo-hp, the N100 and Tester 1. A rollback after that clean-up was proven on Tester 1: a signed update to the older version, one crash 9 seconds into the 60-second dwell, and the wrapper put the previous binary back in the same second (`audits/release-space-fixes-2026-11-10/agent/07-rollback-proof.txt`). This is one crash, not a crash loop.
---
<a id="g6"></a>
@@ -0,0 +1 @@
MATCH 2026-11-10T14:58:14Z: demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 48% 42% running (connected) 71% local-lvm
@@ -0,0 +1,61 @@
# Evidence — the release of the three Space fixes (2026-11-10, attended release run)
**What this is.** The files that prove what the release run did on the boxes and on the hub. The report is
`REPORT-release-space-fixes-2026-11-10.md` at the repository root. The fallback-kernel test of the same run has its own
folder: `audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`.
**Method.** No browser. CC read the boxes over SSH, read the hub's log from its pod, and called the hub's own
addresses with CC's token (the same addresses the hub's pages call). Times are UTC, except lines copied from a host's
journal or the hub's log, which print CET (UTC+1). The N100's host clock is about 32 seconds behind (R-1301); its
lines read early by that much.
**Secrets.** The files were scanned before the commit. They hold no token, no password and no key. The signed job
envelopes are not here.
## `agent/` — the agent release (R-1368 part 2)
| File | What it shows |
|---|---|
| `01-folder-before.txt` | The self-update folder on the three test boxes before: 49 / 48 / 33 entries |
| `03-sign-upload.txt` | One signed `agent_update` per box, uploaded to the hub |
| `04-delivery.txt` | Each box took the job, committed, and removed the old files (`removed=49`, `48`, `33`) |
| `05-after-demo-hp.txt`, `05-after-demo-felhom.txt`, `05-after-tester1.txt` | The folder after: 1 entry each; no pending marker; the unit active |
| `06-rollback-sign.txt` | The signed job to the older version, for the rollback proof on Tester 1 |
| `07-rollback-proof.txt` | One crash 9 seconds into the dwell; the wrapper put the released version back in the same second |
| `08-after-rollback-tester1.txt` | After the rollback: the released version runs; 2 files in the folder (the failed one stays, by design) |
| `12-vouch-readback.txt` | The operator's vouch, read from the hub's log |
| `13-n100-clock.txt` | The N100's host clock against DooPlex and demo-hp; `chrony` failed at boot (R-1301) |
## `controller/` — the controller release (R-1371 part 2)
| File | What it shows |
|---|---|
| `00-fill-before.txt` | The fill watch on the boxes before the release: 3 filesystems checked, two of them the same one |
| `03-crossing-on-0.350.0.txt` | **Red on a box.** Scratch guest 9202, the version before, one test file: TWO warnings |
| `04-rc1-on-9202.txt` | **Green on a box.** 9202 on the release candidate, the same fill: ONE warning; cleared silently after the file was removed by its exact name |
| `05-boxwalk-before.txt` | The three test boxes before the per-box floors |
| `07-per-box-floors.txt` | The three per-box floors, each `303 … flash=floor_set` |
| `08-boxwalk-after.txt` | The three boxes on the release: healthy, containers 10 / 26 / 9, none unhealthy |
| `09-fs-identity-after.txt` | The watch's target list on each box after: 2 filesystems, one target each |
| `10-https-only.txt` | `scripts/https_only_check.py`, exit 0 |
| `11-global-floor.txt` | The global floor, in its own save; the hub's read-back |
| `12-watch.txt` | 30 minutes after the global floor: every box on the release, no new alarm; the mailbox as a second channel |
| `13-per-box-floors-cleared.txt` | The three per-box floors cleared; the boxes stay on the release by the global floor |
## `hub/` — the hub release (R-1371 part 3, D68)
| File | What it shows |
|---|---|
| `02-d68-tests.txt` | The `TestD68_*` tests, green, on the released commit |
| `03-ci-heading.txt` | CI for the commit the image was built from, read by commit hash |
| `07-deploy.txt` | Before and after the deploy: only the hub's Deployment was out of sync; Synced/Healthy; the image id |
| `08-ci-bump.txt` | CI for the manifest-bump commit. **Read 3 minutes after the deploy, not before** (a deviation; see the report) |
| `09-first-list-read.txt` | The pages that show a household's alarm list refuse a token (403, by design). So the open point of `16` §14.4 is not measured on the running hub |
| `10-watch.txt` | 30 minutes after the deploy: Synced/Healthy, no restart, no error line, 6 reports from each box, no alarm sent |
## Not proven by this folder
- That a household which saved its own alarm list before no longer gets the host-disk alarm. The proof is the tests
and the deployed commit. No test alarm was made, by the brief.
- A real fill on a household's box. A fill at the critical line (95 %) on any box.
- A removal by hand in the agent's self-update folder; the `.bak-*` copies of the agent (R-1368 part 4).
@@ -0,0 +1,27 @@
== demo-hp 2026-11-10T14:34:22Z
demo-hp
felhom-agent 0.172.1
entries=49 staged=49 size_kb=722596
felhom-agent-0.171.0
felhom-agent-0.172.0
felhom-agent-0.172.1
1
-rwxr-xr-x 1 root root 16393888 Nov 3 08:56 /usr/local/bin/felhom-agent.prev
== felhom-pve 2026-11-10T14:34:22Z
demo-felhom
felhom-agent 0.172.1
entries=48 staged=48 size_kb=706968
felhom-agent-0.171.0
felhom-agent-0.172.0
felhom-agent-0.172.1
11
-rwxr-xr-x 1 root root 16393888 Nov 3 08:49 /usr/local/bin/felhom-agent.prev
== root@192.168.0.154 2026-11-10T14:34:22Z
felhom
felhom-agent 0.172.1
entries=33 staged=33 size_kb=497244
felhom-agent-0.171.0
felhom-agent-0.172.0
felhom-agent-0.172.1
0
-rwxr-xr-x 1 root root 16393888 Nov 3 08:57 /usr/local/bin/felhom-agent.prev
@@ -0,0 +1,12 @@
== tester-1-d70be4 2026-11-10T14:46:49Z
signed: op=agent_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=fef383cc866be77372a2c7421c0f70ac expires=2026-11-10T15:16:49Z
uploaded signed op to the hub jobs queue
rc=0
== demo-hp-bb76ea 2026-11-10T14:46:49Z
signed: op=agent_update host=demo-hp-bb76ea guest="" key_id=felhom-op-1 nonce=c230909e8a25f4e33c0ac9c6e5b3b1fe expires=2026-11-10T15:16:49Z
uploaded signed op to the hub jobs queue
rc=0
== demo-felhom-8363b5 2026-11-10T14:46:49Z
signed: op=agent_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=481078428450ad9576174647ba6db78b expires=2026-11-10T15:16:50Z
uploaded signed op to the hub jobs queue
rc=0
@@ -0,0 +1,5 @@
signed ops uploaded 2026-11-10T14:46:49Z
+330s felhom-pve: version 0.173.0, committed
+753s demo-hp: version 0.173.0, committed
+815s root@192.168.0.154: version 0.173.0, committed
AGWATCH RESULT=ok all three committed at 2026-11-10T15:00:37Z
@@ -0,0 +1,31 @@
host=demo-felhom at=2026-11-10T14:53:03Z agent=felhom-agent 0.173.0
entries=1 staged=1 size_kb=16028
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-10T15:51:50 .
drwxr-x--- 9 felhom-agent felhom-agent 4096 2026-11-10T10:22:07 ..
-rwxr-xr-x 1 felhom-agent felhom-agent 16407361 2026-11-10T15:50:45 felhom-agent-0.173.0
pending.json: absent
-rwxr-xr-x 1 root root 16407361 2026-11-10T15:50:46 /usr/local/bin/felhom-agent
-rwxr-xr-x 1 root root 16393888 2026-11-07T23:04:48 /usr/local/bin/felhom-agent.prev
prev version: felhom-agent 0.172.1
bak files: 11
-- journal (host local time)
2026-11-10T15:50:46+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T15:50:46.007+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.173.0 version=0.173.0
2026-11-10T15:50:46+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T15:50:46.327+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.173.0 report="{\"agent_update\": {\"sha256\": \"752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea\", \"version\": \"
2026-11-10T15:50:46+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T15:50:46.327+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=77bd95f4bec54d8e op=agent_update
2026-11-10T15:50:48+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:48.455+01:00 level=INFO msg="felhom-agent daemon starting" version=0.173.0 host_id=demo-felhom-8363b5 hub_url=https://hub.felhom.eu interval_s=900
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.046+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.046+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.047+01:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.938+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.938+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.173.0 prev=0.172.1 dwell=1m0s
2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=20
2026-11-10T15:51:50+01:00 demo-felhom sudo[3986865]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-selfupdate-guarded commit
2026-11-10T15:51:50+01:00 demo-felhom felhom-selfupdate-guarded[3986874]: committed (pending cleared, .prev retained)
2026-11-10T15:51:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:50.032+01:00 level=WARN msg="selfupdate: update committed" version=0.173.0 wrapper=""
2026-11-10T15:51:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:50.128+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.173.0 removed=48
2026-11-10T15:51:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:50.128+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=48 kept=felhom-agent-0.173.0
active
NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:50:48 CET
@@ -0,0 +1,38 @@
host=demo-hp at=2026-11-10T15:00:34Z agent=felhom-agent 0.173.0
entries=1 staged=1 size_kb=16028
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-10T15:59:19 .
drwxr-x--- 10 felhom-agent felhom-agent 4096 2026-11-10T11:36:41 ..
-rwxr-xr-x 1 felhom-agent felhom-agent 16407361 2026-11-10T15:58:14 felhom-agent-0.173.0
pending.json: absent
-rwxr-xr-x 1 root root 16407361 2026-11-10T15:58:14 /usr/local/bin/felhom-agent
-rwxr-xr-x 1 root root 16393888 2026-11-07T23:11:58 /usr/local/bin/felhom-agent.prev
prev version: felhom-agent 0.172.1
bak files: 1
-- journal (host local time)
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.000+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.799+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.799+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.858+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.859+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.859+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=95
2026-11-10T15:58:14+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:58:14.583+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.173.0 version=0.173.0
2026-11-10T15:58:14+01:00 demo-hp felhom-os-apply[66717]: os-apply: AGENT-UPDATE signed by the operator: version=0.173.0 sha=752b2727fb604341 — handing the root copy to the A/B wrapper
2026-11-10T15:58:15+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:58:15.231+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.173.0 report="{\"agent_update\": {\"sha256\": \"752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea\", \"version\": \"0.17
2026-11-10T15:58:15+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:58:15.231+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=897285dcaecd3e9d op=agent_update
2026-11-10T15:58:17+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:17.331+01:00 level=INFO msg="felhom-agent daemon starting" version=0.173.0 host_id=demo-hp-bb76ea hub_url=https://hub.felhom.eu interval_s=900
2026-11-10T15:58:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:18.424+01:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
2026-11-10T15:58:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:18.424+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T15:58:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:18.424+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.207+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.207+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.388+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.389+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.173.0 prev=0.172.1 dwell=1m0s
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.389+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.389+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=95
2026-11-10T15:59:19+01:00 demo-hp sudo[72223]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-selfupdate-guarded commit
2026-11-10T15:59:19+01:00 demo-hp felhom-selfupdate-guarded[72232]: committed (pending cleared, .prev retained)
2026-11-10T15:59:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:19.427+01:00 level=WARN msg="selfupdate: update committed" version=0.173.0 wrapper=""
2026-11-10T15:59:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:19.592+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.173.0 removed=49
2026-11-10T15:59:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:19.592+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=49 kept=felhom-agent-0.173.0
active
NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:58:17 CET
@@ -0,0 +1,38 @@
host=felhom at=2026-11-10T15:00:35Z agent=felhom-agent 0.173.0
entries=1 staged=1 size_kb=16028
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-10T16:00:19 .
drwxr-x--- 10 felhom-agent felhom-agent 4096 2026-11-10T13:35:18 ..
-rwxr-xr-x 1 felhom-agent felhom-agent 16407361 2026-11-10T15:59:14 felhom-agent-0.173.0
pending.json: absent
-rwxr-xr-x 1 root root 16407361 2026-11-10T15:59:15 /usr/local/bin/felhom-agent
-rwxr-xr-x 1 root root 16393888 2026-11-07T23:12:45 /usr/local/bin/felhom-agent.prev
prev version: felhom-agent 0.172.1
bak files: 0
-- journal (host local time)
2026-11-10T15:43:45+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:45.441+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.623+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.631+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.728+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.728+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.732+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=213
2026-11-10T15:59:14+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:59:14.962+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.173.0 version=0.173.0
2026-11-10T15:59:15+01:00 felhom felhom-os-apply[28155]: os-apply: AGENT-UPDATE signed by the operator: version=0.173.0 sha=752b2727fb604341 — handing the root copy to the A/B wrapper
2026-11-10T15:59:15+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:59:15.660+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.173.0 report="{\"agent_update\": {\"sha256\": \"752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea\", \"version\": \"0.173
2026-11-10T15:59:15+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:59:15.660+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=50cce53e2c398bee op=agent_update
2026-11-10T15:59:17+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:17.757+01:00 level=INFO msg="felhom-agent daemon starting" version=0.173.0 host_id=tester-1-d70be4 hub_url=https://hub.felhom.eu interval_s=900
2026-11-10T15:59:18+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:18.689+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T15:59:18+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:18.689+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T15:59:18+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:18.689+01:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.231+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.232+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.338+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.339+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.339+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=213
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.339+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.173.0 prev=0.172.1 dwell=1m0s
2026-11-10T16:00:19+01:00 felhom sudo[30337]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-selfupdate-guarded commit
2026-11-10T16:00:19+01:00 felhom felhom-selfupdate-guarded[30345]: committed (pending cleared, .prev retained)
2026-11-10T16:00:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:00:19.390+01:00 level=WARN msg="selfupdate: update committed" version=0.173.0 wrapper=""
2026-11-10T16:00:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:00:19.607+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.173.0 removed=33
2026-11-10T16:00:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:00:19.607+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=33 kept=felhom-agent-0.173.0
active
NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:59:17 CET
@@ -0,0 +1,5 @@
== rollback proof on Tester 1: a signed agent_update to the OLD version 0.172.1, then one crash inside the 60 s dwell
2026-11-10T15:01:19Z
signed: op=agent_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=3039f82fc2b2103065da50c4015a8d11 expires=2026-11-10T15:31:19Z
uploaded signed op to the hub jobs queue
SIGN RC=0
@@ -0,0 +1,19 @@
watch start 2026-11-10T15:01:24Z: agent=felhom-agent 0.173.0 pending=absent
+788s pending marker present, binary on disk = 0.172.1, MainPID=28267 exe=/usr/local/bin/felhom-agent (deleted)
pending.json: {"old_version":"0.173.0","new_version":"0.172.1","sha256":"505a0594a2fb117f359e329a0b7ae2c7a6902b74c51f52c1883145f3c2ac72f8","applied_at":"2026-11-10T16:14:31+01:00"}
folder now: felhom-agent-0.172.1 felhom-agent-0.173.0 pending.json
os-apply running: 0; dpkg/apt running: 0
+796s 2026-11-10T15:14:40Z KILL: systemctl kill --kill-whom=main -s SEGV felhom-agent (MainPID=51118, pending=PRESENT)
kill rc=0
== after +821s 2026-11-10T15:15:05Z
agent=felhom-agent 0.173.0 active=active pending=absent
running exe version: felhom-agent 0.173.0
folder: entries=2 size_kb=32040: felhom-agent-0.172.1 felhom-agent-0.173.0
sha256 bin / prev:
752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea /usr/local/bin/felhom-agent
752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea /usr/local/bin/felhom-agent.prev
-- journal
2026-11-10T16:14:40+01:00 felhom felhom-selfupdate-guarded[51930]: felhom-selfupdate-guarded: rolled back to previous binary and restarted felhom-agent.service
failed units: 0
ROLLBACK-PROOF DONE
PROOF RC=0
@@ -0,0 +1,39 @@
host=felhom at=2026-11-10T15:15:56Z agent=felhom-agent 0.173.0
entries=2 staged=2 size_kb=32040
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-10T16:14:40 .
drwxr-x--- 10 felhom-agent felhom-agent 4096 2026-11-10T13:35:18 ..
-rwxr-xr-x 1 felhom-agent felhom-agent 16393888 2026-11-10T16:14:30 felhom-agent-0.172.1
-rwxr-xr-x 1 felhom-agent felhom-agent 16407361 2026-11-10T15:59:14 felhom-agent-0.173.0
pending.json: absent
-rwxr-xr-x 1 root root 16407361 2026-11-10T16:14:40 /usr/local/bin/felhom-agent
-rwxr-xr-x 1 root root 16407361 2026-11-10T15:59:15 /usr/local/bin/felhom-agent.prev
prev version: felhom-agent 0.173.0
bak files: 0
-- journal (host local time)
2026-11-10T16:14:31+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:14:31.582+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.172.1 report="{\"agent_update\": {\"sha256\": \"505a0594a2fb117f359e329a0b7ae2c7a6902b74c51f52c1883145f3c2ac72f8\", \"version\": \"0.17
2026-11-10T16:14:31+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:14:31.582+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=f6848443a231739a op=agent_update
2026-11-10T16:14:33+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:33.722+01:00 level=INFO msg="felhom-agent daemon starting" version=0.172.1 host_id=tester-1-d70be4 hub_url=https://hub.felhom.eu interval_s=900
2026-11-10T16:14:34+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:34.663+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T16:14:34+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:34.663+01:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
2026-11-10T16:14:34+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:34.663+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.232+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.233+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.319+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.320+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.321+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.172.1 prev=0.173.0 dwell=1m0s
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.321+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=213
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate.NewManager.func2({0xe5d130, 0x1e342d05aac0}, 0xef4820?)
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate/commit.go:54 +0xb4 fp=0x1e342d2d1dc8 sp=0x1e342d2d1d38 pc=0x880e14
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate.(*Manager).MaybeCommit(0x1e342d13b2c0, {0xe5d130, 0x1e342d05aac0})
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate/commit.go:111 +0x3aa fp=0x1e342d2d1fb8 sp=0x1e342d2d1dc8 pc=0x87e0aa
2026-11-10T16:14:40+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:40.445+01:00 level=INFO msg="felhom-agent daemon starting" version=0.173.0 host_id=tester-1-d70be4 hub_url=https://hub.felhom.eu interval_s=900
2026-11-10T16:14:41+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:41.489+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T16:14:41+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:41.489+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T16:14:41+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:41.489+01:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.072+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.072+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.172+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.173+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.173+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=213
active
NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 16:14:40 CET
@@ -0,0 +1,4 @@
2026/11/10 16:29:16 [INFO] artifact vouch: golden 0.350.0 unchanged (behind the fleet's 0.352.0 — the weekly bake) — the R-120 gate does not apply to an unchanged golden; saving agent=0.173.0 min_agent="0.158.1"
2026/11/10 16:29:16 [INFO] Artifact manifest set: agent=0.173.0 golden=0.350.0 min_agent="0.158.1" wrapper_sha=false bundle_sha="51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1"
2026/11/10 16:29:16 [INFO] artifact vouch timing (R-1346): registry probes 101ms, sha lookups 29ms, bundle sha 13ms, fleet check + save 87ms, total 232ms
read 2026-11-10T15:29:45Z (the hub log prints CET)
@@ -0,0 +1,16 @@
== N100 host clock against DooPlex and demo-hp (read-only), for R-1301
dooplex 2026-11-10T15:38:07.145528284Z
n100 2026-11-10T15:37:34.778614952Z
demo-hp 2026-11-10T15:38:07.999970091Z
dooplex 2026-11-10T15:38:08.002419987Z
-- n100 timedatectl
NTPSynchronized=no
TimeUSec=Tue 2026-11-10 16:37:35 CET
failed
Nov 08 05:35:23 demo-felhom chronyd[690]: Loaded seccomp filter (level 1)
Nov 08 05:35:23 demo-felhom chronyd[690]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Nov 08 05:35:23 demo-felhom chronyd[690]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Nov 08 05:35:23 demo-felhom systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE
Nov 08 05:35:23 demo-felhom systemd[1]: chrony.service: Failed with result 'exit-code'.
Nov 08 05:35:23 demo-felhom systemd[1]: Failed to start chrony.service - chrony, an NTP client/server.
2026-11-08 05:35:18
@@ -0,0 +1,8 @@
Fill watch BEFORE the release (controller 0.351.0 on the three test boxes, 0.350.0 on scratch guest 9202), read 2026-11-10 14:24–14:51 UTC.
The line counts TARGETS. On a one-volume box the two built-in targets "/" and /mnt/sys_drive are one filesystem
(felhom.eu documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt).
demo-felhom (N100) guest 9201, 0.351.0: [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok (14:24:05, 14:34:05)
demo-hp guest 9201, 0.351.0: 2026/11/10 14:45:00 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
demo-hp scratch guest 9202, 0.350.0: 2026/11/10 14:44:49 [INFO] [fillwatch] checked 11 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: all ok
Tester 1 guest 9201, 0.351.0: 2026/11/10 14:45:41 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
@@ -0,0 +1,41 @@
== The warning band on scratch guest 9202, OLD controller 0.350.0 (the 'before': how many warnings does one fill send?)
== 2026-11-10T14:53:04Z guest config and the state before
hostname: demo-hp-scratch
mp0: nvme-scratch:9202/vm-9202-disk-3.raw,mp=/var/lib/felhom,backup=1,size=70G
mp8: /mnt/hdd_1/scratch-drives/scratch_hdd,mp=/mnt/felhom-drives/scratch_hdd
mp9: /var/lib/felhom-agent/guests/9202/bootstrap,mp=/etc/felhom-bootstrap,ro=1
rootfs: nvme-scratch:9202/vm-9202-disk-2.raw,size=32G
/dev/loop1 73391005696 13985423360 55630708736 21% /var/lib/felhom
total 32
drwxr-xr-x 5 root root 4096 Oct 20 08:44 .
drwxr-xr-x 24 root root 4096 Oct 4 07:13 ..
drwx--x--- 12 root root 4096 Nov 10 14:43 docker
drwx------ 2 nobody nogroup 16384 Oct 4 07:11 lost+found
drwxr-xr-x 4 root root 4096 Oct 26 19:52 sys_drive
2026/11/10 14:44:49 fillwatch.go:284: [INFO] [fillwatch] checked 11 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: all ok
== 2026-11-10T14:53:15Z the test file: one file, exact name, 45902462976 bytes (42.75 GiB), made with fallocate (no data written); target 86 % used, about 9 GiB still free; the ext4 reserve is not touched (the file is smaller than 'avail')
df: /mnt/nvme-1tb: No such file or directory
fallocate_rc=0
-rw-r--r-- 1 root root 45902462976 Nov 10 14:53 /var/lib/felhom/FILLTEST-2026-11-10-r1371.bin
/dev/loop1 73391005696 59887894528 9728237568 87% /var/lib/felhom
/dev/loop1 69G 56G 9.1G 87% /var/lib/felhom
df: /mnt/nvme-1tb: No such file or directory
== 2026-11-10T14:54:01Z the controller's own lines (0.350.0): ONE fill, TWO warnings
gitea.dooplex.hu/admin/felhom-controller:0.350.0 Up 10 minutes (healthy)
2026/11/10 14:53:19 fillwatch.go:262: [WARN] [fillwatch] / ("Alkalmazások területe"): ok → warning — 86% used, 9.1 GB free of 68.4 GB; notifying the customer
2026/11/10 14:53:19 notifier.go:1471: [WARN] notifier disabled (no hub configured): DROPPED event disk_warning (severity warning) — further disk_warning events are logged at DEBUG only
2026/11/10 14:53:19 fillwatch.go:262: [WARN] [fillwatch] /mnt/sys_drive ("Rendszer- és mentési terület"): ok → warning — 86% used, 9.1 GB free of 68.4 GB; notifying the customer
2026/11/10 14:53:19 notifier.go:1475: [DEBUG] notifier disabled: dropped event disk_warning (severity warning)
2026/11/10 14:53:19 fillwatch.go:284: [INFO] [fillwatch] checked 11 filesystem(s), 2 unreadable/skipped, 2 notification(s); bands: /=warning /mnt/sys_drive=warning
-- remove the test file by its exact name
removed '/var/lib/felhom/FILLTEST-2026-11-10-r1371.bin'
total 32
drwxr-xr-x 5 root root 4096 Nov 10 14:54 .
drwxr-xr-x 24 root root 4096 Oct 4 07:13 ..
drwx--x--- 12 root root 4096 Nov 10 14:43 docker
drwx------ 2 nobody nogroup 16384 Oct 4 07:11 lost+found
drwxr-xr-x 4 root root 4096 Oct 26 19:52 sys_drive
/dev/loop1 73391005696 13985468416 55630663680 21% /var/lib/felhom
@@ -0,0 +1,85 @@
== 2026-11-10T14:54:02Z controller 0.352.0-rc.1 onto scratch guest 9202 (by hand, the skill's step 3; digest sha256:d9b47ad5f3fc666ba06bb8542b3b49ff056f25643871021ef073fdb924194e7d)
gitea.dooplex.hu/admin/felhom-controller:0.350.0
gitea.dooplex.hu/admin/felhom-controller:0.352.0-rc.1
restart_rc=0
gitea.dooplex.hu/admin/felhom-controller:0.352.0-rc.1 Up 12 seconds (healthy)
sha256:9298978a8206200a32e9e13ab4910ac136e81775ee9f9337e4e33ae20e092edc
gitea.dooplex.hu/admin/felhom-controller@sha256:d9b47ad5f3fc666ba06bb8542b3b49ff056f25643871021ef073fdb924194e7d
== 2026-11-10T14:56:10Z first fill check on 0.352.0-rc.1 (the old state held two warnings; the file was already removed)
2026/11/10 14:54:05 main.go:335: [INFO] felhom-controller 0.352.0-rc.1 starting (customer: demo-hp, domain: enkisfelhom.hu)
2026/11/10 14:55:35 fillwatch.go:250: [INFO] [fillwatch] /: warning → ok (20% used, 51.8 GB free) — cleared silently, re-armed
2026/11/10 14:55:35 fillwatch.go:233: [DEBUG] [fillwatch] /mnt/felhom-drives/scratch_hdd/userdata/grimmory: usage unreadable — skipped (an unreadable filesystem is not a full one)
2026/11/10 14:55:35 fillwatch.go:233: [DEBUG] [fillwatch] /mnt/felhom-drives/scratch_hdd/userdata/metube: usage unreadable — skipped (an unreadable filesystem is not a full one)
2026/11/10 14:55:35 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: /=ok
-- state file
{
"bands": {}
}
== the test file again: same exact name, same size
2026-11-10T14:56:11Z
fallocate_rc=0
-rw-r--r-- 1 root root 45902462976 Nov 10 14:56 /var/lib/felhom/FILLTEST-2026-11-10-r1371.bin
/dev/loop1 73391005696 59928866816 9687265280 87% /var/lib/felhom
== 2026-11-10T15:04:16Z the next fill check ran (+471s)
gitea.dooplex.hu/admin/felhom-controller:0.352.0-rc.1 Up 10 minutes (healthy)
2026/11/10 15:04:06 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: /=ok
-- state file
{
"bands": {}
}
-- remove the test file by its exact name
removed '/var/lib/felhom/FILLTEST-2026-11-10-r1371.bin'
total 32
drwxr-xr-x 5 root root 4096 Nov 10 15:04 .
drwxr-xr-x 24 root root 4096 Oct 4 07:13 ..
drwx--x--- 12 root root 4096 Nov 10 14:43 docker
drwx------ 2 nobody nogroup 16384 Oct 4 07:11 lost+found
drwxr-xr-x 4 root root 4096 Oct 26 19:52 sys_drive
/dev/loop1 73391005696 12412989440 57203142656 18% /var/lib/felhom
CROSSWAIT DONE 2026-11-10T15:04:18Z
== NOTE 2026-11-10T15:05:00Z: the 15:04:06 check read ok and that is CORRECT. Between 14:56 and 15:04 the box freed 1.6 GB by itself
(other use 14,026,403,840 -> 12,412,993,536 bytes), so the volume was at 83.8 % with 10.5 GiB free: under the 85 % line. The file was too small.
== second try: a bigger file of the same exact name, target 88 %
2026-11-10T15:05:00Z
fallocate_rc=0
-rw-r--r-- 1 root root 48850000000 Nov 10 15:05 /var/lib/felhom/FILLTEST-2026-11-10-r1371.bin
/dev/loop1 73391005696 61263015936 8353116160 89% /var/lib/felhom
== 2026-11-10T15:14:06Z the next fill check ran (+540s)
gitea.dooplex.hu/admin/felhom-controller:0.352.0-rc.1 Up 20 minutes (healthy)
2026/11/10 15:14:06 fillwatch.go:262: [WARN] [fillwatch] / ("Alkalmazások területe"): ok → warning — 88% used, 7.8 GB free of 68.4 GB; notifying the customer
2026/11/10 15:14:06 notifier.go:1471: [WARN] notifier disabled (no hub configured): DROPPED event disk_warning (severity warning) — further disk_warning events are logged at DEBUG only
2026/11/10 15:14:06 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 2 unreadable/skipped, 1 notification(s); bands: /=warning
-- state file
{
"bands": {
"/": "warning"
}
}
-- remove the test file by its exact name
removed '/var/lib/felhom/FILLTEST-2026-11-10-r1371.bin'
total 32
drwxr-xr-x 5 root root 4096 Nov 10 15:14 .
drwxr-xr-x 24 root root 4096 Oct 4 07:13 ..
drwx--x--- 12 root root 4096 Nov 10 14:43 docker
drwx------ 2 nobody nogroup 16384 Oct 4 07:11 lost+found
drwxr-xr-x 4 root root 4096 Oct 26 19:52 sys_drive
/dev/loop1 73391005696 12413243392 57202888704 18% /var/lib/felhom
CROSSWAIT DONE 2026-11-10T15:14:07Z
== 2026-11-10T15:32:45Z after the file was removed: the next check (15:24:06Z) cleared it without a notification; the test file is gone
total 32
drwxr-xr-x 5 root root 4096 Nov 10 15:14 .
drwxr-xr-x 24 root root 4096 Oct 4 07:13 ..
drwx--x--- 12 root root 4096 Nov 10 14:43 docker
drwx------ 2 nobody nogroup 16384 Oct 4 07:11 lost+found
drwxr-xr-x 4 root root 4096 Oct 26 19:52 sys_drive
/dev/loop1 73391005696 12413661184 57202470912 18% /var/lib/felhom
2026/11/10 15:24:06 fillwatch.go:250: [INFO] [fillwatch] /: warning → ok (18% used, 53.3 GB free) — cleared silently, re-armed
2026/11/10 15:24:06 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: /=ok
{
"bands": {}
}== 2026-11-10T15:32:46Z 9202 moves from rc.1 to the release tag by hand (it has no hub, so no floor moves it)
gitea.dooplex.hu/admin/felhom-controller:0.352.0
restart_rc=0
@@ -0,0 +1,40 @@
### root@192.168.0.154 2026-11-10T15:02:11Z
containers=10 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.351.0 Up 18 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.351.0
started=2026-11-10T14:44:09.016454308Z
errors=0 warn=0 info=457
2026/11/10 14:44:10 [INFO] felhom-controller 0.351.0 starting (customer: tester-1, domain: enkicsifelhom.hu)
2026/11/10 14:44:11 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
2026/11/10 14:44:31 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.351.0 (we are 0.351.0), no managed update running
2026/11/10 14:44:47 [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/10 14:45:41 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 14:54:12 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
### demo-hp 2026-11-10T15:02:12Z
containers=26 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.351.0 Up 18 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.351.0
started=2026-11-10T14:43:25.514593512Z
errors=0 warn=6 info=519
2026/11/10 14:44:15 updater.go:104: [DEBUG] [selfupdate] queryRegistry: anonymous tags/list with Bearer HTTP 200
2026/11/10 14:44:15 updater.go:104: [DEBUG] [selfupdate] queryRegistry: 38 tags returned: [0.331.0 0.341.0 0.341.0-rc.1 0.342.0 0.342.0-rc.1 0.342.0-rc.2 0.343.0 0.343.0-rc.1 0.343.0-rc.2 0.343.0-rc.3 0.343.0-rc.4 0.343.0-rc.5 0.3
2026/11/10 14:44:15 updater.go:196: [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/10 14:44:15 updater.go:104: [DEBUG] [selfupdate] version comparison: current=0.351.0 (0.351.0), latest=0.351.0 (0.351.0), cmp=0, updateAvailable=false
2026/11/10 14:58:33 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.351.0 >= floor 0.351.0 — no action
2026/11/10 14:45:00 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 14:53:31 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
### felhom-pve 2026-11-10T15:02:14Z
containers=9 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.351.0 Up 27 hours (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.351.0
started=2026-11-09T12:14:03.775366821Z
errors=0 warn=6 info=27865
2026/11/09 12:14:36 [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/09 18:14:05 [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/10 00:14:05 [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/10 06:14:05 [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/10 12:14:05 [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/10 14:24:05 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 14:34:05 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 14:44:05 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 14:54:05 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
@@ -0,0 +1,14 @@
== 2026-11-10T15:16:52Z per-box floors 0.352.0 / MinAgent 0.158.1 (image sha256:3c785dd396b091f19d4c777db9d7a206e75ab018b6238b5d887e1a4c6cfd3b29)
-- demo-hp
2026-11-10T15:16:52Z POST /customers/demo-hp/floor min_controller_version='0.352.0' min_agent='0.158.1'
HTTP/1.1 303 See Other
Location: /customers/demo-hp?flash=floor_set
-- demo-felhom
2026-11-10T15:16:52Z POST /customers/demo-felhom/floor min_controller_version='0.352.0' min_agent='0.158.1'
HTTP/1.1 303 See Other
Location: /customers/demo-felhom?flash=floor_set
-- tester-1
2026-11-10T15:16:52Z POST /customers/tester-1/floor min_controller_version='0.352.0' min_agent='0.158.1'
HTTP/1.1 303 See Other
Location: /customers/tester-1?flash=floor_set
== 2026-11-10T15:16:52Z done
@@ -0,0 +1,39 @@
### root@192.168.0.154 2026-11-10T15:30:01Z
containers=10 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 12 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
started=2026-11-10T15:17:03.705107516Z
errors=0 warn=0 info=335
2026/11/10 15:17:04 [INFO] felhom-controller 0.352.0 starting (customer: tester-1, domain: enkicsifelhom.hu)
2026/11/10 15:17:04 [INFO] [selfupdate] Post-update startup: update successful (0.351.0 → 0.352.0)
2026/11/10 15:17:04 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
2026/11/10 15:17:14 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.352.0 (we are 0.352.0), no managed update running
2026/11/10 15:17:36 [INFO] [selfupdate] Current version 0.352.0 is up to date
2026/11/10 15:18:34 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:27:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
### demo-hp 2026-11-10T15:30:02Z
containers=26 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 13 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
started=2026-11-10T15:17:03.918469562Z
errors=0 warn=3 info=345
2026/11/10 15:17:37 updater.go:104: [DEBUG] [selfupdate] queryRegistry: anonymous plain GET HTTP 401
2026/11/10 15:17:37 updater.go:104: [DEBUG] [selfupdate] queryRegistry: anonymous tags/list with Bearer HTTP 200
2026/11/10 15:17:37 updater.go:104: [DEBUG] [selfupdate] queryRegistry: 40 tags returned: [0.331.0 0.341.0 0.341.0-rc.1 0.342.0 0.342.0-rc.1 0.342.0-rc.2 0.343.0 0.343.0-rc.1 0.343.0-rc.2 0.343.0-rc.3 0.343.0-rc.4 0.343.0-rc.5 0.3
2026/11/10 15:17:37 updater.go:196: [INFO] [selfupdate] Current version 0.352.0 is up to date
2026/11/10 15:17:37 updater.go:104: [DEBUG] [selfupdate] version comparison: current=0.352.0 (0.352.0), latest=0.352.0 (0.352.0), cmp=0, updateAvailable=false
2026/11/10 15:18:35 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:27:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
### felhom-pve 2026-11-10T15:30:04Z
containers=9 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 13 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
started=2026-11-10T15:16:28.700768259Z
errors=0 warn=0 info=342
2026/11/10 15:16:29 [INFO] felhom-controller 0.352.0 starting (customer: demo-felhom, domain: demo-felhom.eu)
2026/11/10 15:16:29 [INFO] [selfupdate] Post-update startup: update successful (0.351.0 → 0.352.0)
2026/11/10 15:16:29 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
2026/11/10 15:16:39 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.352.0 (we are 0.352.0), no managed update running
2026/11/10 15:17:01 [INFO] [selfupdate] Current version 0.352.0 is up to date
2026/11/10 15:17:59 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:26:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
@@ -0,0 +1,43 @@
# after 0.352.0: what the controller container sees, and what the fill watch says — read-only, 2026-11-10T15:30:48Z
# channel 1 = the kernel (stat -f / df inside the container); channel 2 = the fill watch's own log line
== root@192.168.0.154:9201
/ type=overlayfs fsid=b4d48bcf2747c72b blocks=18016108 free=15822196 dev=74
/mnt/felhom-drives type=ext2/ext3 fsid=322b724225b4b36d blocks=15357389 free=10191290 dev=64513
/mnt/sys_drive type=ext2/ext3 fsid=65ddc34fe9988fc8 blocks=18016108 free=15822196 dev=64519
-- distinct filesystems by fsid, the overlay root counted as the volume below it:
2
2026/11/10 15:18:34 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:27:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
state: { "bands": {}}
== demo-hp:9201
/ type=overlayfs fsid=71b91e3d7cadcd4f blocks=18016108 free=10674045 dev=49
/mnt/felhom-drives type=ext2/ext3 fsid=c04f01ca42f6f3e9 blocks=10113344 free=6060989 dev=64513
/mnt/sizing-bench type=ext2/ext3 fsid=85fbdcb4db8310a7 blocks=8179140 free=7847629 dev=64518
/mnt/sys_drive type=ext2/ext3 fsid=e1508f4b7cb1b4b0 blocks=18016108 free=10674045 dev=64519
-- distinct filesystems by fsid, the overlay root counted as the volume below it:
3
2026/11/10 15:18:35 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:27:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
state: { "bands": {}}
== felhom-pve:9201
/ type=overlayfs fsid=87e8f2c78a7b090c blocks=64230001 free=63627374 dev=88
/mnt/felhom-drives type=ext2/ext3 fsid=813814bd69e4dd6e blocks=24624445 free=15508719 dev=64513
/mnt/sys_drive type=ext2/ext3 fsid=6968e12ca6ac967a blocks=64230001 free=63627374 dev=64519
-- distinct filesystems by fsid, the overlay root counted as the volume below it:
2
2026/11/10 15:17:59 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:26:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
state:
# the drives below /mnt/felhom-drives (the folder itself is on the guest's root disk, which nobody watches — R-1371), 2026-11-10T15:31:05Z
== root@192.168.0.154:9201
/mnt/felhom-drives/adatlemez type=ext2/ext3 fsid=a25b481c3dbf279a blocks=12822648 free=9621417 dev=2064
/mnt/felhom-drives/media_a type=ext2/ext3 fsid=322b724225b4b36d blocks=15357389 free=10191292 dev=64513
/mnt/felhom-drives/media_b type=ext2/ext3 fsid=322b724225b4b36d blocks=15357389 free=10191292 dev=64513
/mnt/felhom-drives/media_c type=ext2/ext3 fsid=322b724225b4b36d blocks=15357389 free=10191292 dev=64513
/mnt/felhom-drives/media_d type=ext2/ext3 fsid=322b724225b4b36d blocks=15357389 free=10191292 dev=64513
== demo-hp:9201
/mnt/felhom-drives/hdd_1 type=ext2/ext3 fsid=3ce20f0a870807d7 blocks=245844925 free=167050197 dev=66304
== felhom-pve:9201
/mnt/felhom-drives/hdd_1 type=ext2/ext3 fsid=ef113ee179770dd1 blocks=240075962 free=238087780 dev=2064
# count per box = the apps' volume (/ and /mnt/sys_drive are one filesystem) + the drives that are a filesystem of their own
@@ -0,0 +1,36 @@
== 2026-11-10T15:31:16Z python3 scripts/https_only_check.py
control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -)
== tester-1 (enkicsifelhom.hu): 6 public name(s)
OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1
OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1
OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1
OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1
OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1
OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1
== demo-hp (enkisfelhom.hu): 15 public name(s)
OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1
OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1
OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1
OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1
OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1
OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1
OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1
OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1
OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1
OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1
OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1
OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1
OK http://time.enkisfelhom.hu/r1033-check?q=1 -> 301 https://time.enkisfelhom.hu/r1033-check?q=1
OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1
OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1
== demo-felhom (demo-felhom.eu): 8 public name(s)
OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1
OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1
OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1
OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1
OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1
OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1
OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1
OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
== every public name redirects http -> https
HTTPS RC=0
@@ -0,0 +1,14 @@
== 2026-11-10T15:31:33Z before
2026-11-10T15:31:34Z GET /configuration -> 200
Effective floor: v0.351.0 — source: DB (hub_settings) ; env fallback would be v0.120.0 Save global floor… DB override: v0.351.0 … Type the version again to confirm (or CLEAR to remove the DB override): Confirm & apply Cancel function confirmGlobalFloor() { var v = document.g
.. MinAgent: . The hub HOLDS the floor for any box whose agent is below this — blank = uncoupled release, no gating. PBS wrapper sha256 of configs/felhom-pbs-apply (R-50b)
== 2026-11-10T15:31:34Z the global floor, in its own save
2026-11-10T15:31:34Z POST /configuration/global-floor min_controller_version='0.352.0' min_agent='0.158.1'
HTTP/1.1 303 See Other
Location: /configuration?flash=floor_set
== 2026-11-10T15:31:34Z read back
2026-11-10T15:31:34Z GET /configuration -> 200
Effective floor: v0.352.0 — source: DB (hub_settings) ; env fallback would be v0.120.0 Save global floor… DB override: v0.352.0 … Type the version again to confirm (or CLEAR to remove the DB override): Confirm & apply Cancel function confirmGlobalFloor() { var v = document.g
.. MinAgent: . The hub HOLDS the floor for any box whose agent is below this — blank = uncoupled release, no gating. PBS wrapper sha256 of configs/felhom-pbs-apply (R-50b)
2026/11/10 16:31:34 [INFO] Global controller-version floor set to "0.352.0" (declared MinAgent "0.158.1")
2026/11/10 16:31:34 [INFO] access token "cc-releases-2": POST /configuration/global-floor → 303
@@ -0,0 +1,91 @@
== controller watch (watch), hub log since 2026-11-10T15:31:30Z; read 2026-11-10T16:02:15Z
-- the three boxes (read on the box, over SSH)
### root@192.168.0.154 2026-11-10T16:02:15Z
containers=10 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 45 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
started=2026-11-10T15:17:03.705107516Z
errors=0 warn=0 info=945
2026/11/10 15:17:04 [INFO] felhom-controller 0.352.0 starting (customer: tester-1, domain: enkicsifelhom.hu)
2026/11/10 15:17:04 [INFO] [selfupdate] Post-update startup: update successful (0.351.0 → 0.352.0)
2026/11/10 15:17:04 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
2026/11/10 15:17:14 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.352.0 (we are 0.352.0), no managed update running
2026/11/10 15:17:36 [INFO] [selfupdate] Current version 0.352.0 is up to date
2026/11/10 15:27:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:37:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:47:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:57:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
### demo-hp 2026-11-10T16:02:16Z
containers=26 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 45 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
started=2026-11-10T15:17:03.918469562Z
errors=0 warn=3 info=962
2026/11/10 15:17:37 updater.go:104: [DEBUG] [selfupdate] version comparison: current=0.352.0 (0.352.0), latest=0.352.0 (0.352.0), cmp=0, updateAvailable=false
2026/11/10 15:31:38 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
2026/11/10 15:32:07 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
2026/11/10 15:47:07 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
2026/11/10 16:02:07 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
2026/11/10 15:17:05 deploy.go:1470: [WARN] [stacks] Stack plex: new field PLEX_CLAIM (type=text) requires manual configuration
2026/11/10 15:27:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:37:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:47:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:57:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
### felhom-pve 2026-11-10T16:02:18Z
containers=9 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 45 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
started=2026-11-10T15:16:28.700768259Z
errors=0 warn=0 info=906
2026/11/10 15:16:29 [INFO] felhom-controller 0.352.0 starting (customer: demo-felhom, domain: demo-felhom.eu)
2026/11/10 15:16:29 [INFO] [selfupdate] Post-update startup: update successful (0.351.0 → 0.352.0)
2026/11/10 15:16:29 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
2026/11/10 15:16:39 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.352.0 (we are 0.352.0), no managed update running
2026/11/10 15:17:01 [INFO] [selfupdate] Current version 0.352.0 is up to date
2026/11/10 15:26:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:36:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:46:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:56:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
-- hub log since 2026-11-10T15:31:30Z: lines / [ERROR] lines
152
0
-- [WARN] lines by kind (the log prints CET)
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers → 403
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers/demo-hp →
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers/tester-1 →
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /events → 403
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /notifications → 403
-- floor lines
2026/11/10 16:31:34 [INFO] Global controller-version floor set to "0.352.0" (declared MinAgent "0.158.1")
2026/11/10 16:31:34 [INFO] access token "cc-releases-2": POST /configuration/global-floor → 303
-- events from boxes (lines that name an event)
(count: 0)
-- alarm checks and mails
-- lines that name a mail send
0
-- positive signs (the hub works, it is not only quiet): reports received in the window, by box
4 demo-felhom
4 demo-hp
4 tester-1
-- host reports, by host
2 demo-felhom-8363b5
2 demo-hp-bb76ea
2 tester-1-d70be4
-- pod
hub-658c847c65-d8zbk phase=Running restarts=0 image=gitea.dooplex.hu/admin/felhom-hub:0.180.0
-- the hub's hosts page (GET /hosts with CC's token), read 2026-11-10T16:02:58Z -> 200
demo-felhom-8363b5 | agent 0.173.0 | 7.0.14-28-pve | ONLINE | cloudflared running (connected)
demo-hp-bb76ea | agent 0.173.0 | 7.0.14-28-pve | ONLINE | cloudflared running (connected)
tester-1-d70be4 | agent 0.173.0 | 7.0.14-22-pve | ONLINE | cloudflared running (connected)
(the page holds these three hosts and no other)
-- the effective global floor, read back 2026-11-10T16:02:59Z (GET /configuration -> 200)
Effective floor: v0.352.0 — source: DB (hub_settings)
-- demo-hp's three [WARN] lines since its controller start (the walk's secret filter hid two; field NAMES only, no value)
15:17:05 [WARN] [stacks] Stack calibre-web: new field ADMIN_PASSWORD (type=password) requires manual configuration
15:17:05 [WARN] [stacks] Stack plex: new field PLEX_CLAIM (type=text) requires manual configuration
15:17:05 [WARN] [stacks] Stack bookstack: new field ADMIN_PASSWORD (type=password) requires manual configuration
(the controller's own start note for a field added to an installed app; this release did not change that code)
-- second channel, not the hub: the mailbox that gets admin@ and tester1@ mail, read by the mail connector 2026-11-10T16:02Z
search 'newer_than:3h' -> 0 threads (no mail since 13:02 UTC; the run's first release step was 14:51 UTC)
positive control, search 'newer_than:1d' -> 1 thread: the security watch's mail of 2026-11-10T03:41:40Z to admin@ (so the search does find mail)
-- verdict: every box that reports is on 0.352.0 and healthy; no event, no alarm and no mail in the 30 minutes after the global floor
@@ -0,0 +1,32 @@
== 2026-11-10T16:03:11Z clear the per-box floors (both fields empty); the global floor stays
-- demo-hp
2026-11-10T16:03:11Z POST /customers/demo-hp/floor min_controller_version='' min_agent=''
HTTP/1.1 303 See Other
Location: /customers/demo-hp?flash=floor_set
-- demo-felhom
2026-11-10T16:03:11Z POST /customers/demo-felhom/floor min_controller_version='' min_agent=''
HTTP/1.1 303 See Other
Location: /customers/demo-felhom?flash=floor_set
-- tester-1
2026-11-10T16:03:11Z POST /customers/tester-1/floor min_controller_version='' min_agent=''
HTTP/1.1 303 See Other
Location: /customers/tester-1?flash=floor_set
== 2026-11-10T16:03:11Z done
-- the hub's log for the clear (the log prints CET), and the first report of each box after it
2026/11/10 17:03:11 [INFO] Customer demo-hp controller-version floor override set to "" (declared MinAgent "")
2026/11/10 17:03:11 [INFO] access token "cc-releases-2": POST /customers/demo-hp/floor → 303
2026/11/10 17:03:11 [INFO] Customer demo-felhom controller-version floor override set to "" (declared MinAgent "")
2026/11/10 17:03:11 [INFO] access token "cc-releases-2": POST /customers/demo-felhom/floor → 303
2026/11/10 17:03:11 [INFO] Customer tester-1 controller-version floor override set to "" (declared MinAgent "")
2026/11/10 17:03:11 [INFO] access token "cc-releases-2": POST /customers/tester-1/floor → 303
2026/11/10 17:03:14 [INFO] Received report from tester-1 (5252 bytes)
2026/11/10 17:03:15 [INFO] Received report from demo-felhom (5563 bytes)
2026/11/10 17:03:15 [INFO] Received report from demo-hp (10292 bytes)
-- the boxes, read on the box 2026-11-10T16:08Z, 5 minutes after the clear
tester-1: gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 50 minutes (healthy)
demo-hp: gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 50 minutes (healthy)
demo-felhom: gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 51 minutes (healthy)
-- the floor a box got AFTER the clear (demo-hp logs at DEBUG; the other two do not print this line)
2026/11/10 16:03:15 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
-- verdict: the per-box floors are gone; the boxes stay on 0.352.0; the floor demo-hp got 4 seconds after the clear is the global one
-- not read: the hub's page of each customer (it refuses a token, 403); the proof of the clear is the hub's own log line
@@ -0,0 +1,5 @@
--- PASS: TestD68_SeedHoldsNoOperatorOnlyType (0.00s)
--- PASS: TestD68_HostDiskIsOperatorOnly_EvenWhenTheHouseholdSavedIt (0.07s)
--- PASS: TestD68_StorageFillReachesAHouseholdOnlyForADriveItCanFree (0.34s)
--- PASS: TestD68_RegisterAndGate (0.00s)
D68 RC=0
@@ -0,0 +1,4 @@
14:59:29Z felhom.eu 97b7e47114: run 2616 gates #1418 status running
14:59:49Z felhom.eu 97b7e47114: run 2616 gates #1418 status running
15:00:10Z felhom.eu 97b7e47114: run 2616 gates #1418 status success
CI RC=0
@@ -0,0 +1,41 @@
== 2026-11-10T15:12:34Z before: image gitea.dooplex.hu/admin/felhom-hub:0.179.1
app: sync=OutOfSync health=Healthy rev=d0a54c2f8e308ae959900fd8068bc2b434d28664
== OutOfSync resources
Deployment felhom-system hub OutOfSync
count: 1
== 2026-11-10T15:12:48Z sync ONLY Deployment/hub at d0a54c2f8e308ae959900fd8068bc2b434d28664
application.argoproj.io/felhom patched
Waiting for deployment "hub" rollout to finish: 0 of 1 updated replicas are available...
deployment "hub" successfully rolled out
rollout rc=0
== 2026-11-10T15:13:10Z after
image: gitea.dooplex.hu/admin/felhom-hub:0.180.0
hub-658c847c65-d8zbk Running ready=true restarts=0 imageID=gitea.dooplex.hu/admin/felhom-hub@sha256:768e73ab1ab502a8c3fe23a9d42cc038443a6456a1f40dcec853a2c588d04a8f started=2026-11-10T15:12:49Z
operation: phase=Succeeded msg=successfully synced (all tasks run) rev=d0a54c2f8e308ae959900fd8068bc2b434d28664
app: sync=Synced health=Healthy rev=d0a54c2f8e308ae959900fd8068bc2b434d28664
== startup log (first 25 lines of the new pod)
2026/11/10 16:12:52 [INFO] felhom-hub 0.180.0 starting
2026/11/10 16:12:52 [INFO] Database opened at /data/hub.db
2026/11/10 16:12:52 [INFO] Default controller-version floor: 0.120.0
2026/11/10 16:12:52 [INFO] Template fetcher started (every 1h)
2026/11/10 16:12:52 [DEBUG] Template fetched (5881 bytes)
2026/11/10 16:12:53 [INFO] Asset seed: all 263 files up-to-date
2026/11/10 16:12:53 [INFO] Asset manifest built: 288 files
2026/11/10 16:12:53 [INFO] App-email relay enabled (limit 30/min/customer, From domains [felhom.eu])
2026/11/10 16:12:53 [INFO] Gitea artifact browser enabled (Day-0 version dropdowns) via http://gitea.gitea-system.svc.cluster.local:3000
2026/11/10 16:12:53 [INFO] Offsite provisioning enabled (pool_box=611714, location=fsn1)
2026/11/10 16:12:53 [INFO] Offsite pool-box checker initialized: box=611714 fill warn=80% crit=90%, oversub warn=2.00x, unreachable after 3 consecutive failed reads, refresh 15m0s
2026/11/10 16:12:54 [INFO] osupdates: approval rule = every ring-0 box healthy for 24h0m0s and 1 night run(s)
2026/11/10 16:12:54 [INFO] osupdates: the Docker engine set is approved only by the operator, after 2 healthy ring-0 night(s)
2026/11/10 16:12:54 [INFO] osupdates: alarms = no OS leg 168h0m0s, reboot needed 336h0m0s, ring 0 stalled 168h0m0s, not covered 336h0m0s
2026/11/10 16:12:54 [INFO] Off-site key registrar enabled; daily key check at 07:10 Budapest
2026/11/10 16:12:54 [INFO] access log: /data/access-log, one file per UTC day, kept 90 days
2026/11/10 16:12:54 [INFO] Registry version checker started (every 6h)
2026/11/10 16:12:54 [INFO] offsite-key-audit: next run at 2026-11-11 07:10 CET (in 14h57m6s)
2026/11/10 16:12:54 [INFO] WG peer-sync enabled (endpoint 167.233.158.164:22, user felhom-peersync)
2026/11/10 16:12:54 [INFO] PBS DR tenantsync enabled (endpoint 167.233.158.164:22, user felhom-peersync; WG-registration auto-provision hook armed)
2026/11/10 16:12:54 [INFO] PBS-DR box checker initialized: fill warn=80% crit=90%, unreachable after 3 consecutive failed reads, refresh 15m0s
2026/11/10 16:12:54 [INFO] agent-plane poke enabled (endpoint 167.233.158.164:22, user felhom-peersync; web + api admin seams armed)
2026/11/10 16:12:54 [INFO] PBS-DR self-heal reconciler started (interval 5m)
2026/11/10 16:12:54 [INFO] offsite credential self-heal reconciler started (interval 5m, debounce 2 reports)
2026/11/10 16:12:54 [DEBUG] Registry version check: latest = 0.351.0
@@ -0,0 +1 @@
15:15:35Z felhom.eu d0a54c2f8e: run 2617 gates #1419 status success
@@ -0,0 +1,8 @@
# what the running hub lets CC's token read about a household's saved alarm list — GET only, 2026-11-10T15:32:36Z
2026/11/10 16:32:12 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers → 403
2026/11/10 16:32:16 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers/tester-1 → 403
2026/11/10 16:32:17 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers/demo-hp → 403
2026/11/10 16:32:17 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /events → 403
2026/11/10 16:32:17 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /notifications → 403
# result: every page that shows a household's list is 'session only — never by a token' (by design, 05 §17).
# no copy of the hub DB was taken: the live file holds mail addresses and old token bytes (05 §16.2).
@@ -0,0 +1,37 @@
== hub watch, 30 minutes after the deploy (pod started 2026-11-10T15:12:49Z); read 2026-11-10T15:43:01Z
-- argocd app
sync=Synced health=Healthy revision=d0a54c2f8e308ae959900fd8068bc2b434d28664
-- pod
hub-658c847c65-d8zbk phase=Running restarts=0 started=2026-11-10T15:12:52Z image=gitea.dooplex.hu/admin/felhom-hub:0.180.0
-- hub log since the start: lines / [ERROR] lines
260
0
-- [WARN] lines by kind (the log prints CET)
3 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /cu
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /ev
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /no
2 [WARN] slow statement: Nms (over 1s): DELETE …
8 [WARN] slow statement: Nms (over 1s): SELECT …
1 [WARN] slow statement: Nms (over 1s): UPDATE …
-- events from boxes, by box / type / severity
1 demo-felhom controller_started info
1 demo-felhom controller_updated info
1 demo-hp controller_started info
1 demo-hp controller_updated info
1 tester-1 controller_started info
1 tester-1 controller_updated info
-- alarm checks and mails
1 [INFO] osupdates: alarm check done in N, 0 alarm(s) sent []
-- lines that name a mail send
0
-- positive signs (the hub works, it is not only quiet): reports received in the window, by box
6 demo-felhom
6 demo-hp
6 tester-1
-- host reports, by host
3 demo-felhom-8363b5
3 demo-hp-bb76ea
5 tester-1-d70be4
-- startup line
2026/11/10 16:12:52 [INFO] felhom-hub 0.180.0 starting
-- second channel (not the hub): the operator's mailbox, read by the mail connector — see ../controller/12-watch.txt
+4 -4
View File
@@ -278,10 +278,10 @@ stopping line that lies.
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` | OPEN — filed 2026-10-26 | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC |
| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` | OPEN — filed 2026-10-26 **2026-11-10:** still so. The host clock is 32 s behind DooPlex (it was 14 s on 2026-10-26); `chrony` failed again at the 2026-11-08 05:35 boot with the same line; `NTPSynchronized=no`. Seen in a release run, where the box's log times read half a minute early (`documentation/audits/release-space-fixes-2026-11-10/agent/13-n100-clock.txt`). Not fixed by hand: the N100 takes no hand step in a release run. | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC |
| **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC |
| **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H7: the N100 (4.28 GB, 74 files) and Tester 1 (4.29 GB, 81 files) sit at the journal's own cap; Tester 1 writes about 150 MB a day. Lines written by `sudo` are 92 / 92 / 90 % of all lines in one hour — this counts three lines for each call (the command and the session pair), while this row's 61 % counts the session pair only: the same fact, counted two ways. No rank change is proposed: the journal stops at its own cap (about 4.3 GB), so it costs space and log history, not safety. | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC |
| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC |
| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. **2026-11-10 (release run) — part (2) RELEASED, VOUCHED and PROVEN on three test boxes** (agent 0.173.0, vouched by the operator 15:29 UTC; `documentation/audits/release-space-fixes-2026-11-10/agent/`): the folder went 49 → 1, 48 → 1 and 33 → 1 files (about 700 / 690 / 485 MB back) at the commit of THIS update, not at the next one (the earlier sentence here and in the agent's CHANGELOG was wrong, and is corrected there). A rollback after the clean-up worked on Tester 1: a signed update to the older version was crashed once, 9 seconds into the 60-second dwell, and the wrapper put the released version back in the same second; the failed version's file stays until the next good update (by design). The kept older kernel on demo-hp booted to healthy in 313 s and the box went back (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). **Still open:** (1) the kernel clean-up as a program (`11-os-updates.md` §5.11.1, not built) and a second box; (3) template archives; (4) the `.bak-*` copies — the release does not reach them; (5) the failed-run logs. A household box gets part (2) at its first good agent update after the vouch. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC |
| **R-1369** | Box system & updates | P3 | **When a disk is full there is no written way out: no runbook and no self-repair, for the host system disk, the thin pool, the guest disks or a data drive.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 7). One missing runbook, seen from 21 places; the search of the runbooks folder for a disk-full procedure gave 0 hits (`evidence/searches.txt` S6). The alarms exist (hub: 90 / 95 % host disk, 85 / 90 % thin pool; controller: 85 / 95 % on the data volume and each drive) but they end in a mail, not in a step. For three small places nobody wrote what fails when they are full (EFI partition, host and guest temp folders). **Members (answers marked N):** H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7. | OPEN — filed 2026-11-10 (space list) **2026-11-10 (afternoon) — the runbook is written:** `documentation/runbooks/disk-full.md`, five parts, each hand step marked PROVEN (date and evidence; all on demo-hp) or NOT PROVEN. Still open: one rehearsal of the whole runbook on demo-hp; the NOT PROVEN steps (copies on a former target, template archives, the `.bak-*` binaries, the guest root disk); what fails when the EFI partition or a temp folder is full. | — | Docs: one runbook „a disk is full” with one section per disk (host system disk, thin pool, guest root, data volume, data drive): what to look at first, what is safe to delete (the places of R-1368 and R-1367), what is never deleted by hand (household data, the kept-data rulings). Rehearse it once on demo-hp. | CC |
| **R-1370** | Box system & updates | P3 | **The host's system disk has no space budget: no document says how much the host's own files may take, and the installer's sizing rule covers the guest only.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` place H1, answer H1-3; search S5 — 0 hits). Today: demo-hp 58 % of 41 GB, N100 38 % of 101 GB, Tester 1 34 % of 63 GB — and the parts that grow are the journal (R-1367), old kernels and agent binaries (R-1368), old backup copies (R-1087) and the local whole-guest copy (R-548). Second half, known but with no row until now: on a small disk the guest's own guards read the guest's 70 GiB while the thin pool behind it holds 54 GiB (demo-hp: 102 GiB promised on a 53.9 GiB pool, 70.67 % used), so only the hub's thin-pool alarm stands before a full pool. **Member:** H1-3. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D69 = yes; `09` §3 decision 288):** the section goes inside `16-storage-physical-identity.md`, no new document. **2026-11-10 (afternoon) — the section is written:** `16-storage-physical-identity.md` §14 (45 places, the list „no limit, by ruling”, the budget, the open design points); `PROMPT-TEMPLATE.md` §N.5 keeps it true. The budget from measured numbers: the root needs about 14 GB when the whole-box copies go to a backup drive, and 28–54 GB at keep 3 when they do not. **D75 (for the operator):** the smallest system disk Felhom accepts. CC's pick: 128 GB only with a backup drive, 200 GB without one. The other option, 128 GB always, lets a box with many apps and no backup drive stop its local copies. With no ruling the installer accepts any disk. Open after the ruling: the installer's refusal. **2026-11-10 15:19 — RULED (D75 = one number, 200 GB, with or without a backup drive; `09` §3 decision 295).** This replaces CC's two-step pick. **Open, not ruled:** what the installer does below 200 GB — refuse, or warn. demo-hp (128 GB) stays as a test box. `16` §14.3 carries the ruling. | — | Design first: what the installer does below 200 GB (refuse, or warn) — a later session. Then the installer. | CC |
| **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC |
@@ -326,7 +326,7 @@ stopping line that lies.
| **R-1361** | Monitoring & notifications | P2 | **An event the box sends while the hub cannot be reached is lost: 3 tries over 9 s, then dropped.** Walked 2026-11-03 on Tester 1 (hub blackholed 13 min): the box ran normally, an app install worked, a crash loop stopped the app and held it, and `app_stopped_unhealthy` and `app_held` failed after 3 attempts (`notifier.go`). Afterwards NOTHING was re-sent: the hub's log shows no event from the outage, its „Held apps" had no row (`a row existed: false`), and the hold reporter had already recorded the app as reported (`holdreport.go`, the push is asynchronous) so only its periodic re-send can repair it. Any DooPlex restart (2026-11-01 was one) can swallow a household's and the operator's mails this way. `audits/security-2026-11-02/part5/outage-hub.log` | OPEN — filed 2026-11-03 **2026-11-03, a second cost seen:** the walk's cut (01:17–01:39 UTC) covered Tester 1's 01:30 dump, its `db_dump_completed` was lost, and at 04:00 UTC the hub mailed `expected_dbdump_missed` „No DB dump completed or failed since midnight", while the box's own report read `last_db_dump` 03:15 UTC. A lost event becomes a false alarm. | — | a small on-disk outbox for events, sent in order when the hub answers (a design: which types, how long kept); the hold reporter marks „reported" only after a 2xx. Test red first | CC |
| **R-1363** | Monitoring & notifications | P3 | **A household's mail ends with raw JSON: `Megjegyzés: {"avail_gb":5.192043304443359,"band":"warning",...}`.** Seen in the walks of 2026-11-02 (`audits/security-2026-11-02/part5/mail-fill-household.txt`, `mail-crash-household.txt`). `api/handler.go` adds the details as the „Megjegyzés" line; a golden pins it (`mail_golden_test.go`), so it was once wanted. The sentence itself is right and Hungarian. | OPEN — filed 2026-11-03 | — | drop the line from a household's mail, or render known fields in words; the operator's mail keeps it | CC |
| **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) |
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
| **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC |
## Hub & operator — 16 rows (P2 1, P3 4, P4 11)
@@ -346,7 +346,7 @@ stopping line that lies.
| **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator |
| **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC |
| **R-1345** | Hub & operator | P4 | **An access token's scopes cannot be changed after it is made — widening or narrowing one means a new token (and a new secret).** The operator, 2026-10-29: to give CC `hosts:read` + `hosts:write` for the live proofs he had to make `CC_HOSTS` (now in `sec`) beside `cc-releases-2`. Wanted: edit a token's scopes in the session (an audit line per change), so a scope can also be taken away after a session without retiring the secret. | OPEN | — | design + build the scope edit (session only), red-first test that an unticked scope is refused at once | CC |
| **R-1353** | Hub & operator | P4 | **Statements still over 1 s, named by the new slow-statement line (hub 0.178.0) in its first hour:** a background deadline/restore-test check reading days of full host-report JSON (`GetHostReportsSince`, `store.go`; 1.5–9.2 s cold, from `monitor/deadline.go` and `monitor/restoretest.go` — no page waits on it); the start-up prune of `events` and `app_log_issues` (1.1 s and 1.8 s; no index on `created_at` / `last_seen`); the customer page's newest-report read (`GetCustomer`, 1.3 s cold in the pod probe). With the 1 GiB limit most of this should come from the cache once warm. Evidence: the hub log 2026-10-30 09:22–10:05 UTC; `audits/release-2026-10-30/p2-r1346/`. | OPEN **2026-11-02 read (`audits/security-2026-11-02/part4/hub-slow-lines-2026-11-01-02.txt`; the log covers only the 31 h since DooPlex restarted on 2026-11-01 16:11 CET, the lines from 10-30 are gone with the old pod):** 35 lines, 15 statements. What repeats: `VACUUM INTO` 5× (avg 110 s, the 6-hourly snapshot, expected); `GetHostReportsSince` 9× (avg 7.9 s, max 26.6 s); `GetCustomer`'s newest report 3× (1.1 s). Once, at the cold start: the prune of `events` (10.4 s) and `app_log_issues` (21 s, again 04:30), the OS-release and host lists (2–22 s). No fix before the full week (11-06). | — | read a week of slow-statement lines; index or narrow what still repeats (the history read needs only a few fields, not the whole JSON) | CC |
| **R-1353** | Hub & operator | P4 | **Statements still over 1 s, named by the new slow-statement line (hub 0.178.0) in its first hour:** a background deadline/restore-test check reading days of full host-report JSON (`GetHostReportsSince`, `store.go`; 1.5–9.2 s cold, from `monitor/deadline.go` and `monitor/restoretest.go` — no page waits on it); the start-up prune of `events` and `app_log_issues` (1.1 s and 1.8 s; no index on `created_at` / `last_seen`); the customer page's newest-report read (`GetCustomer`, 1.3 s cold in the pod probe). With the 1 GiB limit most of this should come from the cache once warm. Evidence: the hub log 2026-10-30 09:22–10:05 UTC; `audits/release-2026-10-30/p2-r1346/`. | OPEN **2026-11-02 read (`audits/security-2026-11-02/part4/hub-slow-lines-2026-11-01-02.txt`; the log covers only the 31 h since DooPlex restarted on 2026-11-01 16:11 CET, the lines from 10-30 are gone with the old pod):** 35 lines, 15 statements. What repeats: `VACUUM INTO` 5× (avg 110 s, the 6-hourly snapshot, expected); `GetHostReportsSince` 9× (avg 7.9 s, max 26.6 s); `GetCustomer`'s newest report 3× (1.1 s). Once, at the cold start: the prune of `events` (10.4 s) and `app_log_issues` (21 s, again 04:30), the OS-release and host lists (2–22 s). No fix before the full week (11-06). **2026-11-10 read (the hub's start after a deploy, `audits/release-space-fixes-2026-11-10/hub/10-watch.txt`):** 11 slow lines in the first 2 minutes of the new pod, the longest 19.1 s (`GetHostReportsSince`), then the OS-report reads (3–4 s) and the two prunes (2.7 s, 4.2 s); none in the 28 minutes after. The 13:02 UTC snapshot's `VACUUM INTO` took 127.8 s. The same statements as the 11-02 read; no new one. | — | read a week of slow-statement lines; index or narrow what still repeats (the history read needs only a few fields, not the whole JSON) | CC |
| **R-1375** | Hub & operator | P2 | **The outside part of three endings is done by hand or not at all — the worst: an off-site account can stay at the provider with no record in the hub.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH06, EH01, EA21 and the ending "a tester ends"; read in source and documents; no account was called). (1) **Unticking the off-site tier** of a household takes nothing down and leaves no reminder (`hub/internal/web/configs.go:1737-1743`). A later RESET takes the account down only when the tier is ticked (`hub/internal/web/customer_reset.go:203-245`), so the sub-account and its data stay, are paid for, and the hub holds no record that they exist. (2) **Switching the second off-site tier off** takes nothing down: storage entry, token and copies stay; the source calls the teardown 'a deliberate future op' (agent `internal/pbsdr/manager.go:274`). (3) **A tester ends**: no handler and no runbook; done by hand once (2026-09-25); no list says what to collect. **Members (answers marked N):** EH06-4, EH01-4, EA21-4, and 'a tester ends' in the table's head. Related: R-810 (a household leaves), R-32. | OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D71 = yes, D74 = yes; `09` §3 decisions 291, 294).** Not built. | — | Hub (D71): keep the account record after an untick; RESET and delete take the account down, ticked or not. Docs (D74): the runbook 'a household or a tester leaves', from the five code doors, in a later documents-only session. Parts (2) and (3) of this row have no ruling of their own. | CC |
| **R-1376** | Hub & operator | P3 | **Records of a thing that is gone stay in six places.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH00, EH02, EH21, EA00, EA11, EA27; read in source, `documentation/audits/endings-list-2026-11-10/evidence/hub-delete-statements.txt`; no hub database was read). (1) **Hub:** thirteen tables have no delete statement anywhere. Deleting a box leaves its update settings, its update reports and its kernel notices; deleting a household leaves its off-site key rows, off-site window rows and abandon requests. Most are small; the update reports of each box grow with time. (2) **Host uninstall** leaves the mount units and host mounts of household drives (said when R-179 closed; no open row carried it). (3) **A drive wiped by a signed job** keeps a record and a mount unit that name the old filesystem (agent `internal/signedjobs/wipe.go:61-110`). (4) **The token of a destroyed guest** stays valid until a new guest takes its number: the token store has no remove (agent `internal/localapi/tokenstore.go:127-210`). **Members (answers marked N):** EH00-4, EH02-4, EH21-4, EA00-4, EA11-4, EA27-4. | OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D73 = yes, with one exception; `09` §3 decision 293).** Not built. | — | Hub (D73): add the tables with no delete to the two purges and to the daily clean-up; the delete journal (`host_deletions`, `customer_resets`) stays on purpose, and the privacy text says so. Agent: remove a guest's token when the guest is destroyed. Parts (2) and (3) have no ruling of their own. | CC |
+4
View File
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
## Moved from STATUS on 2026-11-10 (evening)
**Updated 2026-11-10 (late afternoon).** You ruled D71–D75 at 15:19 (`documentation/architecture/09-update-architecture.md` §3, decisions 291–296); none is built. The smallest system disk is 200 GB, and what the installer does below 200 GB (refuse, or warn) is open (R-1370). The report of this run is `REPORT-release-space-fixes-2026-11-10.md`.
## Moved from STATUS on 2026-11-10 (late afternoon)
**Updated 2026-11-10 (afternoon).** CC listed every way a thing ends on a box or in the hub (74 endings, eight questions each): 22 answers were „nobody asked”, now in six new rows (R-1373…R-1378) and four questions for you (D71–D74). CC removed six old kernels from the HP box by hand with the keep rule you ruled (8 → 2, 6.4 GB back, one restart, all apps healthy after 5 minutes); nothing is built for it yet. Three first space fixes wait in source for a release, the disk-space design (`documentation/architecture/16-storage-physical-identity.md` §14) and the runbook `documentation/runbooks/disk-full.md` (R-1369) are written, and D75 is one more question for you; the report is `REPORT-endings-kernels-fixes-2026-11-10.md`.
+3 -2
View File
@@ -34,7 +34,8 @@ pvesm list local
dpkg -l 'proxmox-kernel-*-pve-signed' | grep ^ii
```
**The product does by itself:** it empties the package cache in each OS update. It keeps the newest N whole-box copies
**The product does by itself:** it empties the package cache in each OS update. It removes old agent binaries from the
self-update folder after each good agent update. It keeps the newest N whole-box copies
(default 3) and removes half-written ones. journald caps the journal (10 % of the disk, at most 4 GB). It refuses an OS
update under 500 MB free. It skips a local whole-box copy that cannot fit, and says so on the backup page (R-548). The
hub alarms at 90 % and 95 %.
@@ -47,7 +48,7 @@ hub alarms at 90 % and 95 %.
| The host journal | Save it first, then `journalctl --vacuum-size=500M` | **PROVEN 2026-11-07**: 3.8 GB → 468 MB (`audits/demo-check-2026-11-07/README.md`; the record is off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-space-2026-11-07/cleanup.txt`). It grows back (R-1367) |
| An installer ISO that a newer one replaced; a test archive of a guest that is gone | `pvesm free <volid>`, after `pct list` and `qm list` show that no guest uses it, and after its checksum matches a copy elsewhere | **PROVEN 2026-10-23** (three ISOs, `audits/day-2026-10-23/09-demohp-space.txt`), **2026-11-07** (one archive of a removed guest), **2026-11-09** (one ISO). Test boxes only |
| Whole-box copies on a former target (H4) | — | **NOT PROVEN.** Needs the operator's word (R-1087) |
| Old binaries in the agent's self-update folder (H12) | — | **NOT PROVEN.** The agent cleans the folder after the release that carries the fix (R-1368) |
| Old binaries in the agent's self-update folder (H12) | No hand step. The agent removes them at the end of each good update, and keeps the running version's file | **PROVEN 2026-11-10** as a product path on three test boxes: 49 → 1, 48 → 1 and 33 → 1 files; a rollback worked after it (`audits/release-space-fixes-2026-11-10/agent/`). **NOT PROVEN:** a removal by hand |
| Extra guest template archives (H5); `/usr/local/bin/felhom-agent.bak-*` (H13) | — | **NOT PROVEN** (R-1368) |
**Never by hand:**