Space fixes released: agent 0.173.0 vouched, controller 0.352.0 global floor, hub 0.180.0 live; evidence, rows, status, report
gates / gates (push) Successful in 2m1s

The close-out of the attended release run of 2026-11-10 (afternoon).
- Evidence: documentation/audits/release-space-fixes-2026-11-10/ (agent, controller, hub) and the hub row after the
  fallback-kernel test.
- Rows: dated lines on R-1368, R-1371, R-1301, R-1353. 236 open before and after; opened 0, closed 0.
- Capability map: dated notes under C13 and G5; no status changed.
- disk-full runbook and 16 §14: the self-update folder is PROVEN as a product path on three test boxes.
- STATUS: new "Updated" paragraph; the old one moved to STATUS-HISTORY.
- REPORT-release-space-fixes-2026-11-10.md: wrong claims first, one deviation (the hub deploy ran before CI for the
  manifest-bump commit was read), what stays not proven.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-10 17:17:04 +01:00
1 parent d0a54c2f8e
commit bcc73b428d
40 files changed
+1037 -59

No files matched your search

@@ -278,7 +278,7 @@ The alarms, by short name:
| H9 | old kernels | each approved kernel | none today. Ruled: keep the running one, the one before it and the newest (decision 286, `11-os-updates.md` §5.11.1). Proven by hand, not built | nobody today | host disk; later `os_update_stale` |
| H10 | the EFI partition | a boot-loader update | 1.07 GB, 1 % used | none needed | none (R-1371) |
| H11 | the host package cache | each OS update | emptied in each run | the update wrapper | host disk |
| H12 | the agent's self-update folder | each agent update | none on a released agent; one binary after the next agent release (in source, R-1368) | the agent, after that release | host disk |
| H12 | the agent's self-update folder | each agent update | the running version's binary, and the file of a failed update until the next good one (about 16 MB each; measured 2026-11-10 on three boxes, R-1368) | the agent, at the end of each good update | host disk |
| H13 | old agent binaries | `.prev`: one file. `.bak-*`: each re-run of the installer | none for `.bak-*` (R-1368) | nobody | host disk |
| H14 | wrapper state and previous config bundles | each bundle | the last 3 | the wrapper | host disk |
| H15 | host `/tmp` and `/var/tmp` | nothing that lasts | `/tmp` is in memory: half of memory | systemd, after 10 and 30 days | none for `/tmp` (R-1371) |
@@ -408,7 +408,7 @@ below 200 GB — refuse, or warn (R-1370). demo-hp (128 GB) stays as a test box.
| Nobody watches the guest's root disk (G1, G13, G16). It is 4–5 % used today; app definitions lie on it | R-1371 |
| No alarm for the EFI partition (H10) or for the places in memory (H15, G15, G17) | R-1371 |
| A drive in the backup role reports as `usb` or `local-dir`. So the household gets the `storage_fill` mail for a drive it must not clean by hand | R-1371 |
| The hub's first list for a household still holds `storage_fill_critical`. It is replaced when the box sends its own list. Not measured on the running hub in the 2026-11-10 run | R-1371 |
| The hub's first list for a household still holds `storage_fill_critical`. It is replaced when the box sends its own list. **Not measured on the running hub:** the release run of 2026-11-10 deployed the hub that blocks the host-disk alarm at send time, but the pages that show a household's list refuse a token („session only"), and no test alarm was made. The proof is the tests and the deployed commit (`audits/release-space-fixes-2026-11-10/hub/`) | R-1371 |
| Helper images carry no version; the guest's journal has no stated cap; off-site restores have no limit | R-1372 |
| Old kernels: the keep rule is ruled and proven by hand, not built | `11-os-updates.md` §5.11.1 |
| The parts of the host disk with no limit and no cleaner (H4, H5, H6, H13) | R-1087, R-1368 |
@@ -229,6 +229,8 @@
**Gap / roadmap (as written):** **Nothing warned before this.** The only prior signal was the healthcheck's generic `health_degraded` at 90%, for REGISTERED STORAGE PATHS ONLY — it never looked at the docker area or the system-data area, never gave a free-byte figure and never named a drive. **The two event types already existed with NO PRODUCER** (`disk_warning`/`disk_critical`: allowlisted, copy'd, in `DefaultEnabledEvents`, checkbox'd) — the **sixth** *built-but-never-wired* instance here; this ships their producer rather than a seventh near-duplicate type. **Two threshold terms, whichever trips first, and the live proof vindicated the design:** the critical crossing fired on the FREE-BYTE term (1.7 GB) at only **91%** used — a percentage-only rule would have missed it. The hub's generic `customerMessages` entries were REMOVED, because `FormatCustomerEmail` prefers the entry over the message and would discard the label and figures. **Known gap → R-177:** there is no operator-triggerable run-now path; the check is daily 03:30 + once at startup, so confirming a cleared warning on a support call needs a controller restart or a wait
**Noted 2026-11-10 (release run):** on a box with one data volume the watch counted that volume twice. Seen on scratch guest 9202 with one real fill: the old controller logged TWO warnings for it, under two names (`audits/release-space-fixes-2026-11-10/controller/03-crossing-on-0.350.0.txt`); the fixed controller logged ONE warning, and cleared it without a message when the file was removed (`04-rc1-on-9202.txt`). On the three test boxes the watch now checks 2 filesystems, was 3 (`09-fs-identity-after.txt`). 9202 has no hub, so no mail was sent; one mail for one fill is proven by the box's own record, not by a mailbox. Still open: the guest's root disk has no watcher (R-1371).
---
<a id="c14"></a>
@@ -91,6 +91,8 @@
**Gap / roadmap (as written):** Remote-customer proof pending → R-1
**Noted 2026-11-10 (release run):** the agent now removes old downloaded binaries after a good update: 49, 48 and 33 files became 1 on demo-hp, the N100 and Tester 1. A rollback after that clean-up was proven on Tester 1: a signed update to the older version, one crash 9 seconds into the 60-second dwell, and the wrapper put the previous binary back in the same second (`audits/release-space-fixes-2026-11-10/agent/07-rollback-proof.txt`). This is one crash, not a crash loop.
---
<a id="g6"></a>
@@ -0,0 +1 @@
MATCH 2026-11-10T14:58:14Z: demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 48% 42% running (connected) 71% local-lvm
@@ -0,0 +1,61 @@
# Evidence — the release of the three Space fixes (2026-11-10, attended release run)
**What this is.** The files that prove what the release run did on the boxes and on the hub. The report is
`REPORT-release-space-fixes-2026-11-10.md` at the repository root. The fallback-kernel test of the same run has its own
folder: `audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`.
**Method.** No browser. CC read the boxes over SSH, read the hub's log from its pod, and called the hub's own
addresses with CC's token (the same addresses the hub's pages call). Times are UTC, except lines copied from a host's
journal or the hub's log, which print CET (UTC+1). The N100's host clock is about 32 seconds behind (R-1301); its
lines read early by that much.
**Secrets.** The files were scanned before the commit. They hold no token, no password and no key. The signed job
envelopes are not here.
## `agent/` — the agent release (R-1368 part 2)
| File | What it shows |
|---|---|
| `01-folder-before.txt` | The self-update folder on the three test boxes before: 49 / 48 / 33 entries |
| `03-sign-upload.txt` | One signed `agent_update` per box, uploaded to the hub |
| `04-delivery.txt` | Each box took the job, committed, and removed the old files (`removed=49`, `48`, `33`) |
| `05-after-demo-hp.txt`, `05-after-demo-felhom.txt`, `05-after-tester1.txt` | The folder after: 1 entry each; no pending marker; the unit active |
| `06-rollback-sign.txt` | The signed job to the older version, for the rollback proof on Tester 1 |
| `07-rollback-proof.txt` | One crash 9 seconds into the dwell; the wrapper put the released version back in the same second |
| `08-after-rollback-tester1.txt` | After the rollback: the released version runs; 2 files in the folder (the failed one stays, by design) |
| `12-vouch-readback.txt` | The operator's vouch, read from the hub's log |
| `13-n100-clock.txt` | The N100's host clock against DooPlex and demo-hp; `chrony` failed at boot (R-1301) |
## `controller/` — the controller release (R-1371 part 2)
| File | What it shows |
|---|---|
| `00-fill-before.txt` | The fill watch on the boxes before the release: 3 filesystems checked, two of them the same one |
| `03-crossing-on-0.350.0.txt` | **Red on a box.** Scratch guest 9202, the version before, one test file: TWO warnings |
| `04-rc1-on-9202.txt` | **Green on a box.** 9202 on the release candidate, the same fill: ONE warning; cleared silently after the file was removed by its exact name |
| `05-boxwalk-before.txt` | The three test boxes before the per-box floors |
| `07-per-box-floors.txt` | The three per-box floors, each `303 … flash=floor_set` |
| `08-boxwalk-after.txt` | The three boxes on the release: healthy, containers 10 / 26 / 9, none unhealthy |
| `09-fs-identity-after.txt` | The watch's target list on each box after: 2 filesystems, one target each |
| `10-https-only.txt` | `scripts/https_only_check.py`, exit 0 |
| `11-global-floor.txt` | The global floor, in its own save; the hub's read-back |
| `12-watch.txt` | 30 minutes after the global floor: every box on the release, no new alarm; the mailbox as a second channel |
| `13-per-box-floors-cleared.txt` | The three per-box floors cleared; the boxes stay on the release by the global floor |
## `hub/` — the hub release (R-1371 part 3, D68)
| File | What it shows |
|---|---|
| `02-d68-tests.txt` | The `TestD68_*` tests, green, on the released commit |
| `03-ci-heading.txt` | CI for the commit the image was built from, read by commit hash |
| `07-deploy.txt` | Before and after the deploy: only the hub's Deployment was out of sync; Synced/Healthy; the image id |
| `08-ci-bump.txt` | CI for the manifest-bump commit. **Read 3 minutes after the deploy, not before** (a deviation; see the report) |
| `09-first-list-read.txt` | The pages that show a household's alarm list refuse a token (403, by design). So the open point of `16` §14.4 is not measured on the running hub |
| `10-watch.txt` | 30 minutes after the deploy: Synced/Healthy, no restart, no error line, 6 reports from each box, no alarm sent |
## Not proven by this folder
- That a household which saved its own alarm list before no longer gets the host-disk alarm. The proof is the tests
and the deployed commit. No test alarm was made, by the brief.
- A real fill on a household's box. A fill at the critical line (95 %) on any box.
- A removal by hand in the agent's self-update folder; the `.bak-*` copies of the agent (R-1368 part 4).
@@ -0,0 +1,27 @@
== demo-hp 2026-11-10T14:34:22Z
demo-hp
felhom-agent 0.172.1
entries=49 staged=49 size_kb=722596
felhom-agent-0.171.0
felhom-agent-0.172.0
felhom-agent-0.172.1
1
-rwxr-xr-x 1 root root 16393888 Nov 3 08:56 /usr/local/bin/felhom-agent.prev
== felhom-pve 2026-11-10T14:34:22Z
demo-felhom
felhom-agent 0.172.1
entries=48 staged=48 size_kb=706968
felhom-agent-0.171.0
felhom-agent-0.172.0
felhom-agent-0.172.1
11
-rwxr-xr-x 1 root root 16393888 Nov 3 08:49 /usr/local/bin/felhom-agent.prev
== root@192.168.0.154 2026-11-10T14:34:22Z
felhom
felhom-agent 0.172.1
entries=33 staged=33 size_kb=497244
felhom-agent-0.171.0
felhom-agent-0.172.0
felhom-agent-0.172.1
0
-rwxr-xr-x 1 root root 16393888 Nov 3 08:57 /usr/local/bin/felhom-agent.prev
@@ -0,0 +1,12 @@
== tester-1-d70be4 2026-11-10T14:46:49Z
signed: op=agent_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=fef383cc866be77372a2c7421c0f70ac expires=2026-11-10T15:16:49Z
uploaded signed op to the hub jobs queue
rc=0
== demo-hp-bb76ea 2026-11-10T14:46:49Z
signed: op=agent_update host=demo-hp-bb76ea guest="" key_id=felhom-op-1 nonce=c230909e8a25f4e33c0ac9c6e5b3b1fe expires=2026-11-10T15:16:49Z
uploaded signed op to the hub jobs queue
rc=0
== demo-felhom-8363b5 2026-11-10T14:46:49Z
signed: op=agent_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=481078428450ad9576174647ba6db78b expires=2026-11-10T15:16:50Z
uploaded signed op to the hub jobs queue
rc=0
@@ -0,0 +1,5 @@
signed ops uploaded 2026-11-10T14:46:49Z
+330s felhom-pve: version 0.173.0, committed
+753s demo-hp: version 0.173.0, committed
+815s root@192.168.0.154: version 0.173.0, committed
AGWATCH RESULT=ok all three committed at 2026-11-10T15:00:37Z
@@ -0,0 +1,31 @@
host=demo-felhom at=2026-11-10T14:53:03Z agent=felhom-agent 0.173.0
entries=1 staged=1 size_kb=16028
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-10T15:51:50 .
drwxr-x--- 9 felhom-agent felhom-agent 4096 2026-11-10T10:22:07 ..
-rwxr-xr-x 1 felhom-agent felhom-agent 16407361 2026-11-10T15:50:45 felhom-agent-0.173.0
pending.json: absent
-rwxr-xr-x 1 root root 16407361 2026-11-10T15:50:46 /usr/local/bin/felhom-agent
-rwxr-xr-x 1 root root 16393888 2026-11-07T23:04:48 /usr/local/bin/felhom-agent.prev
prev version: felhom-agent 0.172.1
bak files: 11
-- journal (host local time)
2026-11-10T15:50:46+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T15:50:46.007+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.173.0 version=0.173.0
2026-11-10T15:50:46+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T15:50:46.327+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.173.0 report="{\"agent_update\": {\"sha256\": \"752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea\", \"version\": \"
2026-11-10T15:50:46+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T15:50:46.327+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=77bd95f4bec54d8e op=agent_update
2026-11-10T15:50:48+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:48.455+01:00 level=INFO msg="felhom-agent daemon starting" version=0.173.0 host_id=demo-felhom-8363b5 hub_url=https://hub.felhom.eu interval_s=900
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.046+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.046+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.047+01:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.938+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.938+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.173.0 prev=0.172.1 dwell=1m0s
2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=20
2026-11-10T15:51:50+01:00 demo-felhom sudo[3986865]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-selfupdate-guarded commit
2026-11-10T15:51:50+01:00 demo-felhom felhom-selfupdate-guarded[3986874]: committed (pending cleared, .prev retained)
2026-11-10T15:51:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:50.032+01:00 level=WARN msg="selfupdate: update committed" version=0.173.0 wrapper=""
2026-11-10T15:51:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:50.128+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.173.0 removed=48
2026-11-10T15:51:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:50.128+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=48 kept=felhom-agent-0.173.0
active
NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:50:48 CET
@@ -0,0 +1,38 @@
host=demo-hp at=2026-11-10T15:00:34Z agent=felhom-agent 0.173.0
entries=1 staged=1 size_kb=16028
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-10T15:59:19 .
drwxr-x--- 10 felhom-agent felhom-agent 4096 2026-11-10T11:36:41 ..
-rwxr-xr-x 1 felhom-agent felhom-agent 16407361 2026-11-10T15:58:14 felhom-agent-0.173.0
pending.json: absent
-rwxr-xr-x 1 root root 16407361 2026-11-10T15:58:14 /usr/local/bin/felhom-agent
-rwxr-xr-x 1 root root 16393888 2026-11-07T23:11:58 /usr/local/bin/felhom-agent.prev
prev version: felhom-agent 0.172.1
bak files: 1
-- journal (host local time)
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.000+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.799+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.799+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.858+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.859+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.859+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=95
2026-11-10T15:58:14+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:58:14.583+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.173.0 version=0.173.0
2026-11-10T15:58:14+01:00 demo-hp felhom-os-apply[66717]: os-apply: AGENT-UPDATE signed by the operator: version=0.173.0 sha=752b2727fb604341 — handing the root copy to the A/B wrapper
2026-11-10T15:58:15+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:58:15.231+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.173.0 report="{\"agent_update\": {\"sha256\": \"752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea\", \"version\": \"0.17
2026-11-10T15:58:15+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:58:15.231+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=897285dcaecd3e9d op=agent_update
2026-11-10T15:58:17+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:17.331+01:00 level=INFO msg="felhom-agent daemon starting" version=0.173.0 host_id=demo-hp-bb76ea hub_url=https://hub.felhom.eu interval_s=900
2026-11-10T15:58:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:18.424+01:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
2026-11-10T15:58:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:18.424+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T15:58:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:18.424+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.207+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.207+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.388+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.389+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.173.0 prev=0.172.1 dwell=1m0s
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.389+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.389+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=95
2026-11-10T15:59:19+01:00 demo-hp sudo[72223]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-selfupdate-guarded commit
2026-11-10T15:59:19+01:00 demo-hp felhom-selfupdate-guarded[72232]: committed (pending cleared, .prev retained)
2026-11-10T15:59:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:19.427+01:00 level=WARN msg="selfupdate: update committed" version=0.173.0 wrapper=""
2026-11-10T15:59:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:19.592+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.173.0 removed=49
2026-11-10T15:59:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:19.592+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=49 kept=felhom-agent-0.173.0
active
NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:58:17 CET
@@ -0,0 +1,38 @@
host=felhom at=2026-11-10T15:00:35Z agent=felhom-agent 0.173.0
entries=1 staged=1 size_kb=16028
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-10T16:00:19 .
drwxr-x--- 10 felhom-agent felhom-agent 4096 2026-11-10T13:35:18 ..
-rwxr-xr-x 1 felhom-agent felhom-agent 16407361 2026-11-10T15:59:14 felhom-agent-0.173.0
pending.json: absent
-rwxr-xr-x 1 root root 16407361 2026-11-10T15:59:15 /usr/local/bin/felhom-agent
-rwxr-xr-x 1 root root 16393888 2026-11-07T23:12:45 /usr/local/bin/felhom-agent.prev
prev version: felhom-agent 0.172.1
bak files: 0
-- journal (host local time)
2026-11-10T15:43:45+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:45.441+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.623+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.631+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.728+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.728+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.732+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=213
2026-11-10T15:59:14+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:59:14.962+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.173.0 version=0.173.0
2026-11-10T15:59:15+01:00 felhom felhom-os-apply[28155]: os-apply: AGENT-UPDATE signed by the operator: version=0.173.0 sha=752b2727fb604341 — handing the root copy to the A/B wrapper
2026-11-10T15:59:15+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:59:15.660+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.173.0 report="{\"agent_update\": {\"sha256\": \"752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea\", \"version\": \"0.173
2026-11-10T15:59:15+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:59:15.660+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=50cce53e2c398bee op=agent_update
2026-11-10T15:59:17+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:17.757+01:00 level=INFO msg="felhom-agent daemon starting" version=0.173.0 host_id=tester-1-d70be4 hub_url=https://hub.felhom.eu interval_s=900
2026-11-10T15:59:18+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:18.689+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T15:59:18+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:18.689+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T15:59:18+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:18.689+01:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.231+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.232+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.338+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.339+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.339+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=213
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.339+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.173.0 prev=0.172.1 dwell=1m0s
2026-11-10T16:00:19+01:00 felhom sudo[30337]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-selfupdate-guarded commit
2026-11-10T16:00:19+01:00 felhom felhom-selfupdate-guarded[30345]: committed (pending cleared, .prev retained)
2026-11-10T16:00:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:00:19.390+01:00 level=WARN msg="selfupdate: update committed" version=0.173.0 wrapper=""
2026-11-10T16:00:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:00:19.607+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.173.0 removed=33
2026-11-10T16:00:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:00:19.607+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=33 kept=felhom-agent-0.173.0
active
NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:59:17 CET
@@ -0,0 +1,5 @@
== rollback proof on Tester 1: a signed agent_update to the OLD version 0.172.1, then one crash inside the 60 s dwell
2026-11-10T15:01:19Z
signed: op=agent_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=3039f82fc2b2103065da50c4015a8d11 expires=2026-11-10T15:31:19Z
uploaded signed op to the hub jobs queue
SIGN RC=0
@@ -0,0 +1,19 @@
watch start 2026-11-10T15:01:24Z: agent=felhom-agent 0.173.0 pending=absent
+788s pending marker present, binary on disk = 0.172.1, MainPID=28267 exe=/usr/local/bin/felhom-agent (deleted)
pending.json: {"old_version":"0.173.0","new_version":"0.172.1","sha256":"505a0594a2fb117f359e329a0b7ae2c7a6902b74c51f52c1883145f3c2ac72f8","applied_at":"2026-11-10T16:14:31+01:00"}
folder now: felhom-agent-0.172.1 felhom-agent-0.173.0 pending.json
os-apply running: 0; dpkg/apt running: 0
+796s 2026-11-10T15:14:40Z KILL: systemctl kill --kill-whom=main -s SEGV felhom-agent (MainPID=51118, pending=PRESENT)
kill rc=0
== after +821s 2026-11-10T15:15:05Z
agent=felhom-agent 0.173.0 active=active pending=absent
running exe version: felhom-agent 0.173.0
folder: entries=2 size_kb=32040: felhom-agent-0.172.1 felhom-agent-0.173.0
sha256 bin / prev:
752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea /usr/local/bin/felhom-agent
752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea /usr/local/bin/felhom-agent.prev
-- journal
2026-11-10T16:14:40+01:00 felhom felhom-selfupdate-guarded[51930]: felhom-selfupdate-guarded: rolled back to previous binary and restarted felhom-agent.service
failed units: 0
ROLLBACK-PROOF DONE
PROOF RC=0
@@ -0,0 +1,39 @@
host=felhom at=2026-11-10T15:15:56Z agent=felhom-agent 0.173.0
entries=2 staged=2 size_kb=32040
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-10T16:14:40 .
drwxr-x--- 10 felhom-agent felhom-agent 4096 2026-11-10T13:35:18 ..
-rwxr-xr-x 1 felhom-agent felhom-agent 16393888 2026-11-10T16:14:30 felhom-agent-0.172.1
-rwxr-xr-x 1 felhom-agent felhom-agent 16407361 2026-11-10T15:59:14 felhom-agent-0.173.0
pending.json: absent
-rwxr-xr-x 1 root root 16407361 2026-11-10T16:14:40 /usr/local/bin/felhom-agent
-rwxr-xr-x 1 root root 16407361 2026-11-10T15:59:15 /usr/local/bin/felhom-agent.prev
prev version: felhom-agent 0.173.0
bak files: 0
-- journal (host local time)
2026-11-10T16:14:31+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:14:31.582+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.172.1 report="{\"agent_update\": {\"sha256\": \"505a0594a2fb117f359e329a0b7ae2c7a6902b74c51f52c1883145f3c2ac72f8\", \"version\": \"0.17
2026-11-10T16:14:31+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:14:31.582+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=f6848443a231739a op=agent_update
2026-11-10T16:14:33+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:33.722+01:00 level=INFO msg="felhom-agent daemon starting" version=0.172.1 host_id=tester-1-d70be4 hub_url=https://hub.felhom.eu interval_s=900
2026-11-10T16:14:34+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:34.663+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T16:14:34+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:34.663+01:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
2026-11-10T16:14:34+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:34.663+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.232+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.233+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.319+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.320+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.321+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.172.1 prev=0.173.0 dwell=1m0s
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.321+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=213
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate.NewManager.func2({0xe5d130, 0x1e342d05aac0}, 0xef4820?)
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate/commit.go:54 +0xb4 fp=0x1e342d2d1dc8 sp=0x1e342d2d1d38 pc=0x880e14
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate.(*Manager).MaybeCommit(0x1e342d13b2c0, {0xe5d130, 0x1e342d05aac0})
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate/commit.go:111 +0x3aa fp=0x1e342d2d1fb8 sp=0x1e342d2d1dc8 pc=0x87e0aa
2026-11-10T16:14:40+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:40.445+01:00 level=INFO msg="felhom-agent daemon starting" version=0.173.0 host_id=tester-1-d70be4 hub_url=https://hub.felhom.eu interval_s=900
2026-11-10T16:14:41+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:41.489+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T16:14:41+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:41.489+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T16:14:41+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:41.489+01:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.072+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.072+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.172+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.173+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.173+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=213
active
NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 16:14:40 CET
@@ -0,0 +1,4 @@
2026/11/10 16:29:16 [INFO] artifact vouch: golden 0.350.0 unchanged (behind the fleet's 0.352.0 — the weekly bake) — the R-120 gate does not apply to an unchanged golden; saving agent=0.173.0 min_agent="0.158.1"
2026/11/10 16:29:16 [INFO] Artifact manifest set: agent=0.173.0 golden=0.350.0 min_agent="0.158.1" wrapper_sha=false bundle_sha="51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1"
2026/11/10 16:29:16 [INFO] artifact vouch timing (R-1346): registry probes 101ms, sha lookups 29ms, bundle sha 13ms, fleet check + save 87ms, total 232ms
read 2026-11-10T15:29:45Z (the hub log prints CET)
@@ -0,0 +1,16 @@
== N100 host clock against DooPlex and demo-hp (read-only), for R-1301
dooplex 2026-11-10T15:38:07.145528284Z
n100 2026-11-10T15:37:34.778614952Z
demo-hp 2026-11-10T15:38:07.999970091Z
dooplex 2026-11-10T15:38:08.002419987Z
-- n100 timedatectl
NTPSynchronized=no
TimeUSec=Tue 2026-11-10 16:37:35 CET
failed
Nov 08 05:35:23 demo-felhom chronyd[690]: Loaded seccomp filter (level 1)
Nov 08 05:35:23 demo-felhom chronyd[690]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Nov 08 05:35:23 demo-felhom chronyd[690]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Nov 08 05:35:23 demo-felhom systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE
Nov 08 05:35:23 demo-felhom systemd[1]: chrony.service: Failed with result 'exit-code'.
Nov 08 05:35:23 demo-felhom systemd[1]: Failed to start chrony.service - chrony, an NTP client/server.
2026-11-08 05:35:18
@@ -0,0 +1,8 @@
Fill watch BEFORE the release (controller 0.351.0 on the three test boxes, 0.350.0 on scratch guest 9202), read 2026-11-10 14:24–14:51 UTC.
The line counts TARGETS. On a one-volume box the two built-in targets "/" and /mnt/sys_drive are one filesystem
(felhom.eu documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt).
demo-felhom (N100) guest 9201, 0.351.0: [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok (14:24:05, 14:34:05)
demo-hp guest 9201, 0.351.0: 2026/11/10 14:45:00 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
demo-hp scratch guest 9202, 0.350.0: 2026/11/10 14:44:49 [INFO] [fillwatch] checked 11 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: all ok
Tester 1 guest 9201, 0.351.0: 2026/11/10 14:45:41 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
@@ -0,0 +1,41 @@
== The warning band on scratch guest 9202, OLD controller 0.350.0 (the 'before': how many warnings does one fill send?)
== 2026-11-10T14:53:04Z guest config and the state before
hostname: demo-hp-scratch
mp0: nvme-scratch:9202/vm-9202-disk-3.raw,mp=/var/lib/felhom,backup=1,size=70G
mp8: /mnt/hdd_1/scratch-drives/scratch_hdd,mp=/mnt/felhom-drives/scratch_hdd
mp9: /var/lib/felhom-agent/guests/9202/bootstrap,mp=/etc/felhom-bootstrap,ro=1
rootfs: nvme-scratch:9202/vm-9202-disk-2.raw,size=32G
/dev/loop1 73391005696 13985423360 55630708736 21% /var/lib/felhom
total 32
drwxr-xr-x 5 root root 4096 Oct 20 08:44 .
drwxr-xr-x 24 root root 4096 Oct 4 07:13 ..
drwx--x--- 12 root root 4096 Nov 10 14:43 docker
drwx------ 2 nobody nogroup 16384 Oct 4 07:11 lost+found
drwxr-xr-x 4 root root 4096 Oct 26 19:52 sys_drive
2026/11/10 14:44:49 fillwatch.go:284: [INFO] [fillwatch] checked 11 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: all ok
== 2026-11-10T14:53:15Z the test file: one file, exact name, 45902462976 bytes (42.75 GiB), made with fallocate (no data written); target 86 % used, about 9 GiB still free; the ext4 reserve is not touched (the file is smaller than 'avail')
df: /mnt/nvme-1tb: No such file or directory
fallocate_rc=0
-rw-r--r-- 1 root root 45902462976 Nov 10 14:53 /var/lib/felhom/FILLTEST-2026-11-10-r1371.bin
/dev/loop1 73391005696 59887894528 9728237568 87% /var/lib/felhom
/dev/loop1 69G 56G 9.1G 87% /var/lib/felhom
df: /mnt/nvme-1tb: No such file or directory
== 2026-11-10T14:54:01Z the controller's own lines (0.350.0): ONE fill, TWO warnings
gitea.dooplex.hu/admin/felhom-controller:0.350.0 Up 10 minutes (healthy)
2026/11/10 14:53:19 fillwatch.go:262: [WARN] [fillwatch] / ("Alkalmazások területe"): ok → warning — 86% used, 9.1 GB free of 68.4 GB; notifying the customer
2026/11/10 14:53:19 notifier.go:1471: [WARN] notifier disabled (no hub configured): DROPPED event disk_warning (severity warning) — further disk_warning events are logged at DEBUG only
2026/11/10 14:53:19 fillwatch.go:262: [WARN] [fillwatch] /mnt/sys_drive ("Rendszer- és mentési terület"): ok → warning — 86% used, 9.1 GB free of 68.4 GB; notifying the customer
2026/11/10 14:53:19 notifier.go:1475: [DEBUG] notifier disabled: dropped event disk_warning (severity warning)
2026/11/10 14:53:19 fillwatch.go:284: [INFO] [fillwatch] checked 11 filesystem(s), 2 unreadable/skipped, 2 notification(s); bands: /=warning /mnt/sys_drive=warning
-- remove the test file by its exact name
removed '/var/lib/felhom/FILLTEST-2026-11-10-r1371.bin'
total 32
drwxr-xr-x 5 root root 4096 Nov 10 14:54 .
drwxr-xr-x 24 root root 4096 Oct 4 07:13 ..
drwx--x--- 12 root root 4096 Nov 10 14:43 docker
drwx------ 2 nobody nogroup 16384 Oct 4 07:11 lost+found
drwxr-xr-x 4 root root 4096 Oct 26 19:52 sys_drive
/dev/loop1 73391005696 13985468416 55630663680 21% /var/lib/felhom
@@ -0,0 +1,85 @@
== 2026-11-10T14:54:02Z controller 0.352.0-rc.1 onto scratch guest 9202 (by hand, the skill's step 3; digest sha256:d9b47ad5f3fc666ba06bb8542b3b49ff056f25643871021ef073fdb924194e7d)
gitea.dooplex.hu/admin/felhom-controller:0.350.0
gitea.dooplex.hu/admin/felhom-controller:0.352.0-rc.1
restart_rc=0
gitea.dooplex.hu/admin/felhom-controller:0.352.0-rc.1 Up 12 seconds (healthy)
sha256:9298978a8206200a32e9e13ab4910ac136e81775ee9f9337e4e33ae20e092edc
gitea.dooplex.hu/admin/felhom-controller@sha256:d9b47ad5f3fc666ba06bb8542b3b49ff056f25643871021ef073fdb924194e7d
== 2026-11-10T14:56:10Z first fill check on 0.352.0-rc.1 (the old state held two warnings; the file was already removed)
2026/11/10 14:54:05 main.go:335: [INFO] felhom-controller 0.352.0-rc.1 starting (customer: demo-hp, domain: enkisfelhom.hu)
2026/11/10 14:55:35 fillwatch.go:250: [INFO] [fillwatch] /: warning → ok (20% used, 51.8 GB free) — cleared silently, re-armed
2026/11/10 14:55:35 fillwatch.go:233: [DEBUG] [fillwatch] /mnt/felhom-drives/scratch_hdd/userdata/grimmory: usage unreadable — skipped (an unreadable filesystem is not a full one)
2026/11/10 14:55:35 fillwatch.go:233: [DEBUG] [fillwatch] /mnt/felhom-drives/scratch_hdd/userdata/metube: usage unreadable — skipped (an unreadable filesystem is not a full one)
2026/11/10 14:55:35 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: /=ok
-- state file
{
"bands": {}
}
== the test file again: same exact name, same size
2026-11-10T14:56:11Z
fallocate_rc=0
-rw-r--r-- 1 root root 45902462976 Nov 10 14:56 /var/lib/felhom/FILLTEST-2026-11-10-r1371.bin
/dev/loop1 73391005696 59928866816 9687265280 87% /var/lib/felhom
== 2026-11-10T15:04:16Z the next fill check ran (+471s)
gitea.dooplex.hu/admin/felhom-controller:0.352.0-rc.1 Up 10 minutes (healthy)
2026/11/10 15:04:06 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: /=ok
-- state file
{
"bands": {}
}
-- remove the test file by its exact name
removed '/var/lib/felhom/FILLTEST-2026-11-10-r1371.bin'
total 32
drwxr-xr-x 5 root root 4096 Nov 10 15:04 .
drwxr-xr-x 24 root root 4096 Oct 4 07:13 ..
drwx--x--- 12 root root 4096 Nov 10 14:43 docker
drwx------ 2 nobody nogroup 16384 Oct 4 07:11 lost+found
drwxr-xr-x 4 root root 4096 Oct 26 19:52 sys_drive
/dev/loop1 73391005696 12412989440 57203142656 18% /var/lib/felhom
CROSSWAIT DONE 2026-11-10T15:04:18Z
== NOTE 2026-11-10T15:05:00Z: the 15:04:06 check read ok and that is CORRECT. Between 14:56 and 15:04 the box freed 1.6 GB by itself
(other use 14,026,403,840 -> 12,412,993,536 bytes), so the volume was at 83.8 % with 10.5 GiB free: under the 85 % line. The file was too small.
== second try: a bigger file of the same exact name, target 88 %
2026-11-10T15:05:00Z
fallocate_rc=0
-rw-r--r-- 1 root root 48850000000 Nov 10 15:05 /var/lib/felhom/FILLTEST-2026-11-10-r1371.bin
/dev/loop1 73391005696 61263015936 8353116160 89% /var/lib/felhom
== 2026-11-10T15:14:06Z the next fill check ran (+540s)
gitea.dooplex.hu/admin/felhom-controller:0.352.0-rc.1 Up 20 minutes (healthy)
2026/11/10 15:14:06 fillwatch.go:262: [WARN] [fillwatch] / ("Alkalmazások területe"): ok → warning — 88% used, 7.8 GB free of 68.4 GB; notifying the customer
2026/11/10 15:14:06 notifier.go:1471: [WARN] notifier disabled (no hub configured): DROPPED event disk_warning (severity warning) — further disk_warning events are logged at DEBUG only
2026/11/10 15:14:06 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 2 unreadable/skipped, 1 notification(s); bands: /=warning
-- state file
{
"bands": {
"/": "warning"
}
}
-- remove the test file by its exact name
removed '/var/lib/felhom/FILLTEST-2026-11-10-r1371.bin'
total 32
drwxr-xr-x 5 root root 4096 Nov 10 15:14 .
drwxr-xr-x 24 root root 4096 Oct 4 07:13 ..
drwx--x--- 12 root root 4096 Nov 10 14:43 docker
drwx------ 2 nobody nogroup 16384 Oct 4 07:11 lost+found
drwxr-xr-x 4 root root 4096 Oct 26 19:52 sys_drive
/dev/loop1 73391005696 12413243392 57202888704 18% /var/lib/felhom
CROSSWAIT DONE 2026-11-10T15:14:07Z
== 2026-11-10T15:32:45Z after the file was removed: the next check (15:24:06Z) cleared it without a notification; the test file is gone
total 32
drwxr-xr-x 5 root root 4096 Nov 10 15:14 .
drwxr-xr-x 24 root root 4096 Oct 4 07:13 ..
drwx--x--- 12 root root 4096 Nov 10 14:43 docker
drwx------ 2 nobody nogroup 16384 Oct 4 07:11 lost+found
drwxr-xr-x 4 root root 4096 Oct 26 19:52 sys_drive
/dev/loop1 73391005696 12413661184 57202470912 18% /var/lib/felhom
2026/11/10 15:24:06 fillwatch.go:250: [INFO] [fillwatch] /: warning → ok (18% used, 53.3 GB free) — cleared silently, re-armed
2026/11/10 15:24:06 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: /=ok
{
"bands": {}
}== 2026-11-10T15:32:46Z 9202 moves from rc.1 to the release tag by hand (it has no hub, so no floor moves it)
gitea.dooplex.hu/admin/felhom-controller:0.352.0
restart_rc=0
@@ -0,0 +1,40 @@
### root@192.168.0.154 2026-11-10T15:02:11Z
containers=10 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.351.0 Up 18 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.351.0
started=2026-11-10T14:44:09.016454308Z
errors=0 warn=0 info=457
2026/11/10 14:44:10 [INFO] felhom-controller 0.351.0 starting (customer: tester-1, domain: enkicsifelhom.hu)
2026/11/10 14:44:11 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
2026/11/10 14:44:31 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.351.0 (we are 0.351.0), no managed update running
2026/11/10 14:44:47 [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/10 14:45:41 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 14:54:12 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
### demo-hp 2026-11-10T15:02:12Z
containers=26 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.351.0 Up 18 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.351.0
started=2026-11-10T14:43:25.514593512Z
errors=0 warn=6 info=519
2026/11/10 14:44:15 updater.go:104: [DEBUG] [selfupdate] queryRegistry: anonymous tags/list with Bearer HTTP 200
2026/11/10 14:44:15 updater.go:104: [DEBUG] [selfupdate] queryRegistry: 38 tags returned: [0.331.0 0.341.0 0.341.0-rc.1 0.342.0 0.342.0-rc.1 0.342.0-rc.2 0.343.0 0.343.0-rc.1 0.343.0-rc.2 0.343.0-rc.3 0.343.0-rc.4 0.343.0-rc.5 0.3
2026/11/10 14:44:15 updater.go:196: [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/10 14:44:15 updater.go:104: [DEBUG] [selfupdate] version comparison: current=0.351.0 (0.351.0), latest=0.351.0 (0.351.0), cmp=0, updateAvailable=false
2026/11/10 14:58:33 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.351.0 >= floor 0.351.0 — no action
2026/11/10 14:45:00 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 14:53:31 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
### felhom-pve 2026-11-10T15:02:14Z
containers=9 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.351.0 Up 27 hours (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.351.0
started=2026-11-09T12:14:03.775366821Z
errors=0 warn=6 info=27865
2026/11/09 12:14:36 [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/09 18:14:05 [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/10 00:14:05 [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/10 06:14:05 [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/10 12:14:05 [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/10 14:24:05 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 14:34:05 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 14:44:05 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 14:54:05 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
@@ -0,0 +1,14 @@
== 2026-11-10T15:16:52Z per-box floors 0.352.0 / MinAgent 0.158.1 (image sha256:3c785dd396b091f19d4c777db9d7a206e75ab018b6238b5d887e1a4c6cfd3b29)
-- demo-hp
2026-11-10T15:16:52Z POST /customers/demo-hp/floor min_controller_version='0.352.0' min_agent='0.158.1'
HTTP/1.1 303 See Other
Location: /customers/demo-hp?flash=floor_set
-- demo-felhom
2026-11-10T15:16:52Z POST /customers/demo-felhom/floor min_controller_version='0.352.0' min_agent='0.158.1'
HTTP/1.1 303 See Other
Location: /customers/demo-felhom?flash=floor_set
-- tester-1
2026-11-10T15:16:52Z POST /customers/tester-1/floor min_controller_version='0.352.0' min_agent='0.158.1'
HTTP/1.1 303 See Other
Location: /customers/tester-1?flash=floor_set
== 2026-11-10T15:16:52Z done
@@ -0,0 +1,39 @@
### root@192.168.0.154 2026-11-10T15:30:01Z
containers=10 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 12 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
started=2026-11-10T15:17:03.705107516Z
errors=0 warn=0 info=335
2026/11/10 15:17:04 [INFO] felhom-controller 0.352.0 starting (customer: tester-1, domain: enkicsifelhom.hu)
2026/11/10 15:17:04 [INFO] [selfupdate] Post-update startup: update successful (0.351.0 → 0.352.0)
2026/11/10 15:17:04 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
2026/11/10 15:17:14 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.352.0 (we are 0.352.0), no managed update running
2026/11/10 15:17:36 [INFO] [selfupdate] Current version 0.352.0 is up to date
2026/11/10 15:18:34 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:27:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
### demo-hp 2026-11-10T15:30:02Z
containers=26 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 13 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
started=2026-11-10T15:17:03.918469562Z
errors=0 warn=3 info=345
2026/11/10 15:17:37 updater.go:104: [DEBUG] [selfupdate] queryRegistry: anonymous plain GET HTTP 401
2026/11/10 15:17:37 updater.go:104: [DEBUG] [selfupdate] queryRegistry: anonymous tags/list with Bearer HTTP 200
2026/11/10 15:17:37 updater.go:104: [DEBUG] [selfupdate] queryRegistry: 40 tags returned: [0.331.0 0.341.0 0.341.0-rc.1 0.342.0 0.342.0-rc.1 0.342.0-rc.2 0.343.0 0.343.0-rc.1 0.343.0-rc.2 0.343.0-rc.3 0.343.0-rc.4 0.343.0-rc.5 0.3
2026/11/10 15:17:37 updater.go:196: [INFO] [selfupdate] Current version 0.352.0 is up to date
2026/11/10 15:17:37 updater.go:104: [DEBUG] [selfupdate] version comparison: current=0.352.0 (0.352.0), latest=0.352.0 (0.352.0), cmp=0, updateAvailable=false
2026/11/10 15:18:35 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:27:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
### felhom-pve 2026-11-10T15:30:04Z
containers=9 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 13 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
started=2026-11-10T15:16:28.700768259Z
errors=0 warn=0 info=342
2026/11/10 15:16:29 [INFO] felhom-controller 0.352.0 starting (customer: demo-felhom, domain: demo-felhom.eu)
2026/11/10 15:16:29 [INFO] [selfupdate] Post-update startup: update successful (0.351.0 → 0.352.0)
2026/11/10 15:16:29 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
2026/11/10 15:16:39 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.352.0 (we are 0.352.0), no managed update running
2026/11/10 15:17:01 [INFO] [selfupdate] Current version 0.352.0 is up to date
2026/11/10 15:17:59 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:26:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
@@ -0,0 +1,43 @@
# after 0.352.0: what the controller container sees, and what the fill watch says — read-only, 2026-11-10T15:30:48Z
# channel 1 = the kernel (stat -f / df inside the container); channel 2 = the fill watch's own log line
== root@192.168.0.154:9201
/ type=overlayfs fsid=b4d48bcf2747c72b blocks=18016108 free=15822196 dev=74
/mnt/felhom-drives type=ext2/ext3 fsid=322b724225b4b36d blocks=15357389 free=10191290 dev=64513
/mnt/sys_drive type=ext2/ext3 fsid=65ddc34fe9988fc8 blocks=18016108 free=15822196 dev=64519
-- distinct filesystems by fsid, the overlay root counted as the volume below it:
2
2026/11/10 15:18:34 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:27:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
state: { "bands": {}}
== demo-hp:9201
/ type=overlayfs fsid=71b91e3d7cadcd4f blocks=18016108 free=10674045 dev=49
/mnt/felhom-drives type=ext2/ext3 fsid=c04f01ca42f6f3e9 blocks=10113344 free=6060989 dev=64513
/mnt/sizing-bench type=ext2/ext3 fsid=85fbdcb4db8310a7 blocks=8179140 free=7847629 dev=64518
/mnt/sys_drive type=ext2/ext3 fsid=e1508f4b7cb1b4b0 blocks=18016108 free=10674045 dev=64519
-- distinct filesystems by fsid, the overlay root counted as the volume below it:
3
2026/11/10 15:18:35 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:27:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
state: { "bands": {}}
== felhom-pve:9201
/ type=overlayfs fsid=87e8f2c78a7b090c blocks=64230001 free=63627374 dev=88
/mnt/felhom-drives type=ext2/ext3 fsid=813814bd69e4dd6e blocks=24624445 free=15508719 dev=64513
/mnt/sys_drive type=ext2/ext3 fsid=6968e12ca6ac967a blocks=64230001 free=63627374 dev=64519
-- distinct filesystems by fsid, the overlay root counted as the volume below it:
2
2026/11/10 15:17:59 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:26:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
state:
# the drives below /mnt/felhom-drives (the folder itself is on the guest's root disk, which nobody watches — R-1371), 2026-11-10T15:31:05Z
== root@192.168.0.154:9201
/mnt/felhom-drives/adatlemez type=ext2/ext3 fsid=a25b481c3dbf279a blocks=12822648 free=9621417 dev=2064
/mnt/felhom-drives/media_a type=ext2/ext3 fsid=322b724225b4b36d blocks=15357389 free=10191292 dev=64513
/mnt/felhom-drives/media_b type=ext2/ext3 fsid=322b724225b4b36d blocks=15357389 free=10191292 dev=64513
/mnt/felhom-drives/media_c type=ext2/ext3 fsid=322b724225b4b36d blocks=15357389 free=10191292 dev=64513
/mnt/felhom-drives/media_d type=ext2/ext3 fsid=322b724225b4b36d blocks=15357389 free=10191292 dev=64513
== demo-hp:9201
/mnt/felhom-drives/hdd_1 type=ext2/ext3 fsid=3ce20f0a870807d7 blocks=245844925 free=167050197 dev=66304
== felhom-pve:9201
/mnt/felhom-drives/hdd_1 type=ext2/ext3 fsid=ef113ee179770dd1 blocks=240075962 free=238087780 dev=2064
# count per box = the apps' volume (/ and /mnt/sys_drive are one filesystem) + the drives that are a filesystem of their own
@@ -0,0 +1,36 @@
== 2026-11-10T15:31:16Z python3 scripts/https_only_check.py
control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -)
== tester-1 (enkicsifelhom.hu): 6 public name(s)
OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1
OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1
OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1
OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1
OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1
OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1
== demo-hp (enkisfelhom.hu): 15 public name(s)
OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1
OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1
OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1
OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1
OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1
OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1
OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1
OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1
OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1
OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1
OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1
OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1
OK http://time.enkisfelhom.hu/r1033-check?q=1 -> 301 https://time.enkisfelhom.hu/r1033-check?q=1
OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1
OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1
== demo-felhom (demo-felhom.eu): 8 public name(s)
OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1
OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1
OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1
OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1
OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1
OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1
OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1
OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
== every public name redirects http -> https
HTTPS RC=0
@@ -0,0 +1,14 @@
== 2026-11-10T15:31:33Z before
2026-11-10T15:31:34Z GET /configuration -> 200
Effective floor: v0.351.0 — source: DB (hub_settings) ; env fallback would be v0.120.0 Save global floor… DB override: v0.351.0 … Type the version again to confirm (or CLEAR to remove the DB override): Confirm & apply Cancel function confirmGlobalFloor() { var v = document.g
.. MinAgent: . The hub HOLDS the floor for any box whose agent is below this — blank = uncoupled release, no gating. PBS wrapper sha256 of configs/felhom-pbs-apply (R-50b)
== 2026-11-10T15:31:34Z the global floor, in its own save
2026-11-10T15:31:34Z POST /configuration/global-floor min_controller_version='0.352.0' min_agent='0.158.1'
HTTP/1.1 303 See Other
Location: /configuration?flash=floor_set
== 2026-11-10T15:31:34Z read back
2026-11-10T15:31:34Z GET /configuration -> 200
Effective floor: v0.352.0 — source: DB (hub_settings) ; env fallback would be v0.120.0 Save global floor… DB override: v0.352.0 … Type the version again to confirm (or CLEAR to remove the DB override): Confirm & apply Cancel function confirmGlobalFloor() { var v = document.g
.. MinAgent: . The hub HOLDS the floor for any box whose agent is below this — blank = uncoupled release, no gating. PBS wrapper sha256 of configs/felhom-pbs-apply (R-50b)
2026/11/10 16:31:34 [INFO] Global controller-version floor set to "0.352.0" (declared MinAgent "0.158.1")
2026/11/10 16:31:34 [INFO] access token "cc-releases-2": POST /configuration/global-floor → 303
@@ -0,0 +1,91 @@
== controller watch (watch), hub log since 2026-11-10T15:31:30Z; read 2026-11-10T16:02:15Z
-- the three boxes (read on the box, over SSH)
### root@192.168.0.154 2026-11-10T16:02:15Z
containers=10 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 45 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
started=2026-11-10T15:17:03.705107516Z
errors=0 warn=0 info=945
2026/11/10 15:17:04 [INFO] felhom-controller 0.352.0 starting (customer: tester-1, domain: enkicsifelhom.hu)
2026/11/10 15:17:04 [INFO] [selfupdate] Post-update startup: update successful (0.351.0 → 0.352.0)
2026/11/10 15:17:04 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
2026/11/10 15:17:14 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.352.0 (we are 0.352.0), no managed update running
2026/11/10 15:17:36 [INFO] [selfupdate] Current version 0.352.0 is up to date
2026/11/10 15:27:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:37:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:47:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:57:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
### demo-hp 2026-11-10T16:02:16Z
containers=26 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 45 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
started=2026-11-10T15:17:03.918469562Z
errors=0 warn=3 info=962
2026/11/10 15:17:37 updater.go:104: [DEBUG] [selfupdate] version comparison: current=0.352.0 (0.352.0), latest=0.352.0 (0.352.0), cmp=0, updateAvailable=false
2026/11/10 15:31:38 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
2026/11/10 15:32:07 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
2026/11/10 15:47:07 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
2026/11/10 16:02:07 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
2026/11/10 15:17:05 deploy.go:1470: [WARN] [stacks] Stack plex: new field PLEX_CLAIM (type=text) requires manual configuration
2026/11/10 15:27:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:37:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:47:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:57:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
### felhom-pve 2026-11-10T16:02:18Z
containers=9 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 45 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
started=2026-11-10T15:16:28.700768259Z
errors=0 warn=0 info=906
2026/11/10 15:16:29 [INFO] felhom-controller 0.352.0 starting (customer: demo-felhom, domain: demo-felhom.eu)
2026/11/10 15:16:29 [INFO] [selfupdate] Post-update startup: update successful (0.351.0 → 0.352.0)
2026/11/10 15:16:29 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
2026/11/10 15:16:39 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.352.0 (we are 0.352.0), no managed update running
2026/11/10 15:17:01 [INFO] [selfupdate] Current version 0.352.0 is up to date
2026/11/10 15:26:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:36:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:46:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:56:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
-- hub log since 2026-11-10T15:31:30Z: lines / [ERROR] lines
152
0
-- [WARN] lines by kind (the log prints CET)
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers → 403
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers/demo-hp →
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers/tester-1 →
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /events → 403
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /notifications → 403
-- floor lines
2026/11/10 16:31:34 [INFO] Global controller-version floor set to "0.352.0" (declared MinAgent "0.158.1")
2026/11/10 16:31:34 [INFO] access token "cc-releases-2": POST /configuration/global-floor → 303
-- events from boxes (lines that name an event)
(count: 0)
-- alarm checks and mails
-- lines that name a mail send
0
-- positive signs (the hub works, it is not only quiet): reports received in the window, by box
4 demo-felhom
4 demo-hp
4 tester-1
-- host reports, by host
2 demo-felhom-8363b5
2 demo-hp-bb76ea
2 tester-1-d70be4
-- pod
hub-658c847c65-d8zbk phase=Running restarts=0 image=gitea.dooplex.hu/admin/felhom-hub:0.180.0
-- the hub's hosts page (GET /hosts with CC's token), read 2026-11-10T16:02:58Z -> 200
demo-felhom-8363b5 | agent 0.173.0 | 7.0.14-28-pve | ONLINE | cloudflared running (connected)
demo-hp-bb76ea | agent 0.173.0 | 7.0.14-28-pve | ONLINE | cloudflared running (connected)
tester-1-d70be4 | agent 0.173.0 | 7.0.14-22-pve | ONLINE | cloudflared running (connected)
(the page holds these three hosts and no other)
-- the effective global floor, read back 2026-11-10T16:02:59Z (GET /configuration -> 200)
Effective floor: v0.352.0 — source: DB (hub_settings)
-- demo-hp's three [WARN] lines since its controller start (the walk's secret filter hid two; field NAMES only, no value)
15:17:05 [WARN] [stacks] Stack calibre-web: new field ADMIN_PASSWORD (type=password) requires manual configuration
15:17:05 [WARN] [stacks] Stack plex: new field PLEX_CLAIM (type=text) requires manual configuration
15:17:05 [WARN] [stacks] Stack bookstack: new field ADMIN_PASSWORD (type=password) requires manual configuration
(the controller's own start note for a field added to an installed app; this release did not change that code)
-- second channel, not the hub: the mailbox that gets admin@ and tester1@ mail, read by the mail connector 2026-11-10T16:02Z
search 'newer_than:3h' -> 0 threads (no mail since 13:02 UTC; the run's first release step was 14:51 UTC)
positive control, search 'newer_than:1d' -> 1 thread: the security watch's mail of 2026-11-10T03:41:40Z to admin@ (so the search does find mail)
-- verdict: every box that reports is on 0.352.0 and healthy; no event, no alarm and no mail in the 30 minutes after the global floor
@@ -0,0 +1,32 @@
== 2026-11-10T16:03:11Z clear the per-box floors (both fields empty); the global floor stays
-- demo-hp
2026-11-10T16:03:11Z POST /customers/demo-hp/floor min_controller_version='' min_agent=''
HTTP/1.1 303 See Other
Location: /customers/demo-hp?flash=floor_set
-- demo-felhom
2026-11-10T16:03:11Z POST /customers/demo-felhom/floor min_controller_version='' min_agent=''
HTTP/1.1 303 See Other
Location: /customers/demo-felhom?flash=floor_set
-- tester-1
2026-11-10T16:03:11Z POST /customers/tester-1/floor min_controller_version='' min_agent=''
HTTP/1.1 303 See Other
Location: /customers/tester-1?flash=floor_set
== 2026-11-10T16:03:11Z done
-- the hub's log for the clear (the log prints CET), and the first report of each box after it
2026/11/10 17:03:11 [INFO] Customer demo-hp controller-version floor override set to "" (declared MinAgent "")
2026/11/10 17:03:11 [INFO] access token "cc-releases-2": POST /customers/demo-hp/floor → 303
2026/11/10 17:03:11 [INFO] Customer demo-felhom controller-version floor override set to "" (declared MinAgent "")
2026/11/10 17:03:11 [INFO] access token "cc-releases-2": POST /customers/demo-felhom/floor → 303
2026/11/10 17:03:11 [INFO] Customer tester-1 controller-version floor override set to "" (declared MinAgent "")
2026/11/10 17:03:11 [INFO] access token "cc-releases-2": POST /customers/tester-1/floor → 303
2026/11/10 17:03:14 [INFO] Received report from tester-1 (5252 bytes)
2026/11/10 17:03:15 [INFO] Received report from demo-felhom (5563 bytes)
2026/11/10 17:03:15 [INFO] Received report from demo-hp (10292 bytes)
-- the boxes, read on the box 2026-11-10T16:08Z, 5 minutes after the clear
tester-1: gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 50 minutes (healthy)
demo-hp: gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 50 minutes (healthy)
demo-felhom: gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 51 minutes (healthy)
-- the floor a box got AFTER the clear (demo-hp logs at DEBUG; the other two do not print this line)
2026/11/10 16:03:15 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
-- verdict: the per-box floors are gone; the boxes stay on 0.352.0; the floor demo-hp got 4 seconds after the clear is the global one
-- not read: the hub's page of each customer (it refuses a token, 403); the proof of the clear is the hub's own log line
@@ -0,0 +1,5 @@
--- PASS: TestD68_SeedHoldsNoOperatorOnlyType (0.00s)
--- PASS: TestD68_HostDiskIsOperatorOnly_EvenWhenTheHouseholdSavedIt (0.07s)
--- PASS: TestD68_StorageFillReachesAHouseholdOnlyForADriveItCanFree (0.34s)
--- PASS: TestD68_RegisterAndGate (0.00s)
D68 RC=0
@@ -0,0 +1,4 @@
14:59:29Z felhom.eu 97b7e47114: run 2616 gates #1418 status running
14:59:49Z felhom.eu 97b7e47114: run 2616 gates #1418 status running
15:00:10Z felhom.eu 97b7e47114: run 2616 gates #1418 status success
CI RC=0
@@ -0,0 +1,41 @@
== 2026-11-10T15:12:34Z before: image gitea.dooplex.hu/admin/felhom-hub:0.179.1
app: sync=OutOfSync health=Healthy rev=d0a54c2f8e308ae959900fd8068bc2b434d28664
== OutOfSync resources
Deployment felhom-system hub OutOfSync
count: 1
== 2026-11-10T15:12:48Z sync ONLY Deployment/hub at d0a54c2f8e308ae959900fd8068bc2b434d28664
application.argoproj.io/felhom patched
Waiting for deployment "hub" rollout to finish: 0 of 1 updated replicas are available...
deployment "hub" successfully rolled out
rollout rc=0
== 2026-11-10T15:13:10Z after
image: gitea.dooplex.hu/admin/felhom-hub:0.180.0
hub-658c847c65-d8zbk Running ready=true restarts=0 imageID=gitea.dooplex.hu/admin/felhom-hub@sha256:768e73ab1ab502a8c3fe23a9d42cc038443a6456a1f40dcec853a2c588d04a8f started=2026-11-10T15:12:49Z
operation: phase=Succeeded msg=successfully synced (all tasks run) rev=d0a54c2f8e308ae959900fd8068bc2b434d28664
app: sync=Synced health=Healthy rev=d0a54c2f8e308ae959900fd8068bc2b434d28664
== startup log (first 25 lines of the new pod)
2026/11/10 16:12:52 [INFO] felhom-hub 0.180.0 starting
2026/11/10 16:12:52 [INFO] Database opened at /data/hub.db
2026/11/10 16:12:52 [INFO] Default controller-version floor: 0.120.0
2026/11/10 16:12:52 [INFO] Template fetcher started (every 1h)
2026/11/10 16:12:52 [DEBUG] Template fetched (5881 bytes)
2026/11/10 16:12:53 [INFO] Asset seed: all 263 files up-to-date
2026/11/10 16:12:53 [INFO] Asset manifest built: 288 files
2026/11/10 16:12:53 [INFO] App-email relay enabled (limit 30/min/customer, From domains [felhom.eu])
2026/11/10 16:12:53 [INFO] Gitea artifact browser enabled (Day-0 version dropdowns) via http://gitea.gitea-system.svc.cluster.local:3000
2026/11/10 16:12:53 [INFO] Offsite provisioning enabled (pool_box=611714, location=fsn1)
2026/11/10 16:12:53 [INFO] Offsite pool-box checker initialized: box=611714 fill warn=80% crit=90%, oversub warn=2.00x, unreachable after 3 consecutive failed reads, refresh 15m0s
2026/11/10 16:12:54 [INFO] osupdates: approval rule = every ring-0 box healthy for 24h0m0s and 1 night run(s)
2026/11/10 16:12:54 [INFO] osupdates: the Docker engine set is approved only by the operator, after 2 healthy ring-0 night(s)
2026/11/10 16:12:54 [INFO] osupdates: alarms = no OS leg 168h0m0s, reboot needed 336h0m0s, ring 0 stalled 168h0m0s, not covered 336h0m0s
2026/11/10 16:12:54 [INFO] Off-site key registrar enabled; daily key check at 07:10 Budapest
2026/11/10 16:12:54 [INFO] access log: /data/access-log, one file per UTC day, kept 90 days
2026/11/10 16:12:54 [INFO] Registry version checker started (every 6h)
2026/11/10 16:12:54 [INFO] offsite-key-audit: next run at 2026-11-11 07:10 CET (in 14h57m6s)
2026/11/10 16:12:54 [INFO] WG peer-sync enabled (endpoint 167.233.158.164:22, user felhom-peersync)
2026/11/10 16:12:54 [INFO] PBS DR tenantsync enabled (endpoint 167.233.158.164:22, user felhom-peersync; WG-registration auto-provision hook armed)
2026/11/10 16:12:54 [INFO] PBS-DR box checker initialized: fill warn=80% crit=90%, unreachable after 3 consecutive failed reads, refresh 15m0s
2026/11/10 16:12:54 [INFO] agent-plane poke enabled (endpoint 167.233.158.164:22, user felhom-peersync; web + api admin seams armed)
2026/11/10 16:12:54 [INFO] PBS-DR self-heal reconciler started (interval 5m)
2026/11/10 16:12:54 [INFO] offsite credential self-heal reconciler started (interval 5m, debounce 2 reports)
2026/11/10 16:12:54 [DEBUG] Registry version check: latest = 0.351.0
@@ -0,0 +1 @@
15:15:35Z felhom.eu d0a54c2f8e: run 2617 gates #1419 status success
@@ -0,0 +1,8 @@
# what the running hub lets CC's token read about a household's saved alarm list — GET only, 2026-11-10T15:32:36Z
2026/11/10 16:32:12 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers → 403
2026/11/10 16:32:16 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers/tester-1 → 403
2026/11/10 16:32:17 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers/demo-hp → 403
2026/11/10 16:32:17 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /events → 403
2026/11/10 16:32:17 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /notifications → 403
# result: every page that shows a household's list is 'session only — never by a token' (by design, 05 §17).
# no copy of the hub DB was taken: the live file holds mail addresses and old token bytes (05 §16.2).
@@ -0,0 +1,37 @@
== hub watch, 30 minutes after the deploy (pod started 2026-11-10T15:12:49Z); read 2026-11-10T15:43:01Z
-- argocd app
sync=Synced health=Healthy revision=d0a54c2f8e308ae959900fd8068bc2b434d28664
-- pod
hub-658c847c65-d8zbk phase=Running restarts=0 started=2026-11-10T15:12:52Z image=gitea.dooplex.hu/admin/felhom-hub:0.180.0
-- hub log since the start: lines / [ERROR] lines
260
0
-- [WARN] lines by kind (the log prints CET)
3 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /cu
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /ev
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /no
2 [WARN] slow statement: Nms (over 1s): DELETE …
8 [WARN] slow statement: Nms (over 1s): SELECT …
1 [WARN] slow statement: Nms (over 1s): UPDATE …
-- events from boxes, by box / type / severity
1 demo-felhom controller_started info
1 demo-felhom controller_updated info
1 demo-hp controller_started info
1 demo-hp controller_updated info
1 tester-1 controller_started info
1 tester-1 controller_updated info
-- alarm checks and mails
1 [INFO] osupdates: alarm check done in N, 0 alarm(s) sent []
-- lines that name a mail send
0
-- positive signs (the hub works, it is not only quiet): reports received in the window, by box
6 demo-felhom
6 demo-hp
6 tester-1
-- host reports, by host
3 demo-felhom-8363b5
3 demo-hp-bb76ea
5 tester-1-d70be4
-- startup line
2026/11/10 16:12:52 [INFO] felhom-hub 0.180.0 starting
-- second channel (not the hub): the operator's mailbox, read by the mail connector — see ../controller/12-watch.txt
+4 -4
View File
@@ -278,10 +278,10 @@ stopping line that lies.
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` | OPEN — filed 2026-10-26 | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC |
| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` | OPEN — filed 2026-10-26 **2026-11-10:** still so. The host clock is 32 s behind DooPlex (it was 14 s on 2026-10-26); `chrony` failed again at the 2026-11-08 05:35 boot with the same line; `NTPSynchronized=no`. Seen in a release run, where the box's log times read half a minute early (`documentation/audits/release-space-fixes-2026-11-10/agent/13-n100-clock.txt`). Not fixed by hand: the N100 takes no hand step in a release run. | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC |
| **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC |
| **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H7: the N100 (4.28 GB, 74 files) and Tester 1 (4.29 GB, 81 files) sit at the journal's own cap; Tester 1 writes about 150 MB a day. Lines written by `sudo` are 92 / 92 / 90 % of all lines in one hour — this counts three lines for each call (the command and the session pair), while this row's 61 % counts the session pair only: the same fact, counted two ways. No rank change is proposed: the journal stops at its own cap (about 4.3 GB), so it costs space and log history, not safety. | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC |
| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC |
| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. **2026-11-10 (release run) — part (2) RELEASED, VOUCHED and PROVEN on three test boxes** (agent 0.173.0, vouched by the operator 15:29 UTC; `documentation/audits/release-space-fixes-2026-11-10/agent/`): the folder went 49 → 1, 48 → 1 and 33 → 1 files (about 700 / 690 / 485 MB back) at the commit of THIS update, not at the next one (the earlier sentence here and in the agent's CHANGELOG was wrong, and is corrected there). A rollback after the clean-up worked on Tester 1: a signed update to the older version was crashed once, 9 seconds into the 60-second dwell, and the wrapper put the released version back in the same second; the failed version's file stays until the next good update (by design). The kept older kernel on demo-hp booted to healthy in 313 s and the box went back (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). **Still open:** (1) the kernel clean-up as a program (`11-os-updates.md` §5.11.1, not built) and a second box; (3) template archives; (4) the `.bak-*` copies — the release does not reach them; (5) the failed-run logs. A household box gets part (2) at its first good agent update after the vouch. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC |
| **R-1369** | Box system & updates | P3 | **When a disk is full there is no written way out: no runbook and no self-repair, for the host system disk, the thin pool, the guest disks or a data drive.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 7). One missing runbook, seen from 21 places; the search of the runbooks folder for a disk-full procedure gave 0 hits (`evidence/searches.txt` S6). The alarms exist (hub: 90 / 95 % host disk, 85 / 90 % thin pool; controller: 85 / 95 % on the data volume and each drive) but they end in a mail, not in a step. For three small places nobody wrote what fails when they are full (EFI partition, host and guest temp folders). **Members (answers marked N):** H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7. | OPEN — filed 2026-11-10 (space list) **2026-11-10 (afternoon) — the runbook is written:** `documentation/runbooks/disk-full.md`, five parts, each hand step marked PROVEN (date and evidence; all on demo-hp) or NOT PROVEN. Still open: one rehearsal of the whole runbook on demo-hp; the NOT PROVEN steps (copies on a former target, template archives, the `.bak-*` binaries, the guest root disk); what fails when the EFI partition or a temp folder is full. | — | Docs: one runbook „a disk is full” with one section per disk (host system disk, thin pool, guest root, data volume, data drive): what to look at first, what is safe to delete (the places of R-1368 and R-1367), what is never deleted by hand (household data, the kept-data rulings). Rehearse it once on demo-hp. | CC |
| **R-1370** | Box system & updates | P3 | **The host's system disk has no space budget: no document says how much the host's own files may take, and the installer's sizing rule covers the guest only.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` place H1, answer H1-3; search S5 — 0 hits). Today: demo-hp 58 % of 41 GB, N100 38 % of 101 GB, Tester 1 34 % of 63 GB — and the parts that grow are the journal (R-1367), old kernels and agent binaries (R-1368), old backup copies (R-1087) and the local whole-guest copy (R-548). Second half, known but with no row until now: on a small disk the guest's own guards read the guest's 70 GiB while the thin pool behind it holds 54 GiB (demo-hp: 102 GiB promised on a 53.9 GiB pool, 70.67 % used), so only the hub's thin-pool alarm stands before a full pool. **Member:** H1-3. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D69 = yes; `09` §3 decision 288):** the section goes inside `16-storage-physical-identity.md`, no new document. **2026-11-10 (afternoon) — the section is written:** `16-storage-physical-identity.md` §14 (45 places, the list „no limit, by ruling”, the budget, the open design points); `PROMPT-TEMPLATE.md` §N.5 keeps it true. The budget from measured numbers: the root needs about 14 GB when the whole-box copies go to a backup drive, and 28–54 GB at keep 3 when they do not. **D75 (for the operator):** the smallest system disk Felhom accepts. CC's pick: 128 GB only with a backup drive, 200 GB without one. The other option, 128 GB always, lets a box with many apps and no backup drive stop its local copies. With no ruling the installer accepts any disk. Open after the ruling: the installer's refusal. **2026-11-10 15:19 — RULED (D75 = one number, 200 GB, with or without a backup drive; `09` §3 decision 295).** This replaces CC's two-step pick. **Open, not ruled:** what the installer does below 200 GB — refuse, or warn. demo-hp (128 GB) stays as a test box. `16` §14.3 carries the ruling. | — | Design first: what the installer does below 200 GB (refuse, or warn) — a later session. Then the installer. | CC |
| **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC |
@@ -326,7 +326,7 @@ stopping line that lies.
| **R-1361** | Monitoring & notifications | P2 | **An event the box sends while the hub cannot be reached is lost: 3 tries over 9 s, then dropped.** Walked 2026-11-03 on Tester 1 (hub blackholed 13 min): the box ran normally, an app install worked, a crash loop stopped the app and held it, and `app_stopped_unhealthy` and `app_held` failed after 3 attempts (`notifier.go`). Afterwards NOTHING was re-sent: the hub's log shows no event from the outage, its „Held apps" had no row (`a row existed: false`), and the hold reporter had already recorded the app as reported (`holdreport.go`, the push is asynchronous) so only its periodic re-send can repair it. Any DooPlex restart (2026-11-01 was one) can swallow a household's and the operator's mails this way. `audits/security-2026-11-02/part5/outage-hub.log` | OPEN — filed 2026-11-03 **2026-11-03, a second cost seen:** the walk's cut (01:17–01:39 UTC) covered Tester 1's 01:30 dump, its `db_dump_completed` was lost, and at 04:00 UTC the hub mailed `expected_dbdump_missed` „No DB dump completed or failed since midnight", while the box's own report read `last_db_dump` 03:15 UTC. A lost event becomes a false alarm. | — | a small on-disk outbox for events, sent in order when the hub answers (a design: which types, how long kept); the hold reporter marks „reported" only after a 2xx. Test red first | CC |
| **R-1363** | Monitoring & notifications | P3 | **A household's mail ends with raw JSON: `Megjegyzés: {"avail_gb":5.192043304443359,"band":"warning",...}`.** Seen in the walks of 2026-11-02 (`audits/security-2026-11-02/part5/mail-fill-household.txt`, `mail-crash-household.txt`). `api/handler.go` adds the details as the „Megjegyzés" line; a golden pins it (`mail_golden_test.go`), so it was once wanted. The sentence itself is right and Hungarian. | OPEN — filed 2026-11-03 | — | drop the line from a household's mail, or render known fields in words; the operator's mail keeps it | CC |
| **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) |
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
| **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC |
## Hub & operator — 16 rows (P2 1, P3 4, P4 11)
@@ -346,7 +346,7 @@ stopping line that lies.
| **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator |
| **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC |
| **R-1345** | Hub & operator | P4 | **An access token's scopes cannot be changed after it is made — widening or narrowing one means a new token (and a new secret).** The operator, 2026-10-29: to give CC `hosts:read` + `hosts:write` for the live proofs he had to make `CC_HOSTS` (now in `sec`) beside `cc-releases-2`. Wanted: edit a token's scopes in the session (an audit line per change), so a scope can also be taken away after a session without retiring the secret. | OPEN | — | design + build the scope edit (session only), red-first test that an unticked scope is refused at once | CC |
| **R-1353** | Hub & operator | P4 | **Statements still over 1 s, named by the new slow-statement line (hub 0.178.0) in its first hour:** a background deadline/restore-test check reading days of full host-report JSON (`GetHostReportsSince`, `store.go`; 1.5–9.2 s cold, from `monitor/deadline.go` and `monitor/restoretest.go` — no page waits on it); the start-up prune of `events` and `app_log_issues` (1.1 s and 1.8 s; no index on `created_at` / `last_seen`); the customer page's newest-report read (`GetCustomer`, 1.3 s cold in the pod probe). With the 1 GiB limit most of this should come from the cache once warm. Evidence: the hub log 2026-10-30 09:22–10:05 UTC; `audits/release-2026-10-30/p2-r1346/`. | OPEN **2026-11-02 read (`audits/security-2026-11-02/part4/hub-slow-lines-2026-11-01-02.txt`; the log covers only the 31 h since DooPlex restarted on 2026-11-01 16:11 CET, the lines from 10-30 are gone with the old pod):** 35 lines, 15 statements. What repeats: `VACUUM INTO` 5× (avg 110 s, the 6-hourly snapshot, expected); `GetHostReportsSince` 9× (avg 7.9 s, max 26.6 s); `GetCustomer`'s newest report 3× (1.1 s). Once, at the cold start: the prune of `events` (10.4 s) and `app_log_issues` (21 s, again 04:30), the OS-release and host lists (2–22 s). No fix before the full week (11-06). | — | read a week of slow-statement lines; index or narrow what still repeats (the history read needs only a few fields, not the whole JSON) | CC |
| **R-1353** | Hub & operator | P4 | **Statements still over 1 s, named by the new slow-statement line (hub 0.178.0) in its first hour:** a background deadline/restore-test check reading days of full host-report JSON (`GetHostReportsSince`, `store.go`; 1.5–9.2 s cold, from `monitor/deadline.go` and `monitor/restoretest.go` — no page waits on it); the start-up prune of `events` and `app_log_issues` (1.1 s and 1.8 s; no index on `created_at` / `last_seen`); the customer page's newest-report read (`GetCustomer`, 1.3 s cold in the pod probe). With the 1 GiB limit most of this should come from the cache once warm. Evidence: the hub log 2026-10-30 09:22–10:05 UTC; `audits/release-2026-10-30/p2-r1346/`. | OPEN **2026-11-02 read (`audits/security-2026-11-02/part4/hub-slow-lines-2026-11-01-02.txt`; the log covers only the 31 h since DooPlex restarted on 2026-11-01 16:11 CET, the lines from 10-30 are gone with the old pod):** 35 lines, 15 statements. What repeats: `VACUUM INTO` 5× (avg 110 s, the 6-hourly snapshot, expected); `GetHostReportsSince` 9× (avg 7.9 s, max 26.6 s); `GetCustomer`'s newest report 3× (1.1 s). Once, at the cold start: the prune of `events` (10.4 s) and `app_log_issues` (21 s, again 04:30), the OS-release and host lists (2–22 s). No fix before the full week (11-06). **2026-11-10 read (the hub's start after a deploy, `audits/release-space-fixes-2026-11-10/hub/10-watch.txt`):** 11 slow lines in the first 2 minutes of the new pod, the longest 19.1 s (`GetHostReportsSince`), then the OS-report reads (3–4 s) and the two prunes (2.7 s, 4.2 s); none in the 28 minutes after. The 13:02 UTC snapshot's `VACUUM INTO` took 127.8 s. The same statements as the 11-02 read; no new one. | — | read a week of slow-statement lines; index or narrow what still repeats (the history read needs only a few fields, not the whole JSON) | CC |
| **R-1375** | Hub & operator | P2 | **The outside part of three endings is done by hand or not at all — the worst: an off-site account can stay at the provider with no record in the hub.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH06, EH01, EA21 and the ending "a tester ends"; read in source and documents; no account was called). (1) **Unticking the off-site tier** of a household takes nothing down and leaves no reminder (`hub/internal/web/configs.go:1737-1743`). A later RESET takes the account down only when the tier is ticked (`hub/internal/web/customer_reset.go:203-245`), so the sub-account and its data stay, are paid for, and the hub holds no record that they exist. (2) **Switching the second off-site tier off** takes nothing down: storage entry, token and copies stay; the source calls the teardown 'a deliberate future op' (agent `internal/pbsdr/manager.go:274`). (3) **A tester ends**: no handler and no runbook; done by hand once (2026-09-25); no list says what to collect. **Members (answers marked N):** EH06-4, EH01-4, EA21-4, and 'a tester ends' in the table's head. Related: R-810 (a household leaves), R-32. | OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D71 = yes, D74 = yes; `09` §3 decisions 291, 294).** Not built. | — | Hub (D71): keep the account record after an untick; RESET and delete take the account down, ticked or not. Docs (D74): the runbook 'a household or a tester leaves', from the five code doors, in a later documents-only session. Parts (2) and (3) of this row have no ruling of their own. | CC |
| **R-1376** | Hub & operator | P3 | **Records of a thing that is gone stay in six places.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH00, EH02, EH21, EA00, EA11, EA27; read in source, `documentation/audits/endings-list-2026-11-10/evidence/hub-delete-statements.txt`; no hub database was read). (1) **Hub:** thirteen tables have no delete statement anywhere. Deleting a box leaves its update settings, its update reports and its kernel notices; deleting a household leaves its off-site key rows, off-site window rows and abandon requests. Most are small; the update reports of each box grow with time. (2) **Host uninstall** leaves the mount units and host mounts of household drives (said when R-179 closed; no open row carried it). (3) **A drive wiped by a signed job** keeps a record and a mount unit that name the old filesystem (agent `internal/signedjobs/wipe.go:61-110`). (4) **The token of a destroyed guest** stays valid until a new guest takes its number: the token store has no remove (agent `internal/localapi/tokenstore.go:127-210`). **Members (answers marked N):** EH00-4, EH02-4, EH21-4, EA00-4, EA11-4, EA27-4. | OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D73 = yes, with one exception; `09` §3 decision 293).** Not built. | — | Hub (D73): add the tables with no delete to the two purges and to the daily clean-up; the delete journal (`host_deletions`, `customer_resets`) stays on purpose, and the privacy text says so. Agent: remove a guest's token when the guest is destroyed. Parts (2) and (3) have no ruling of their own. | CC |
+4
View File
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
## Moved from STATUS on 2026-11-10 (evening)
**Updated 2026-11-10 (late afternoon).** You ruled D71–D75 at 15:19 (`documentation/architecture/09-update-architecture.md` §3, decisions 291–296); none is built. The smallest system disk is 200 GB, and what the installer does below 200 GB (refuse, or warn) is open (R-1370). The report of this run is `REPORT-release-space-fixes-2026-11-10.md`.
## Moved from STATUS on 2026-11-10 (late afternoon)
**Updated 2026-11-10 (afternoon).** CC listed every way a thing ends on a box or in the hub (74 endings, eight questions each): 22 answers were „nobody asked”, now in six new rows (R-1373…R-1378) and four questions for you (D71–D74). CC removed six old kernels from the HP box by hand with the keep rule you ruled (8 → 2, 6.4 GB back, one restart, all apps healthy after 5 minutes); nothing is built for it yet. Three first space fixes wait in source for a release, the disk-space design (`documentation/architecture/16-storage-physical-identity.md` §14) and the runbook `documentation/runbooks/disk-full.md` (R-1369) are written, and D75 is one more question for you; the report is `REPORT-endings-kernels-fixes-2026-11-10.md`.
+3 -2
View File
@@ -34,7 +34,8 @@ pvesm list local
dpkg -l 'proxmox-kernel-*-pve-signed' | grep ^ii
```
**The product does by itself:** it empties the package cache in each OS update. It keeps the newest N whole-box copies
**The product does by itself:** it empties the package cache in each OS update. It removes old agent binaries from the
self-update folder after each good agent update. It keeps the newest N whole-box copies
(default 3) and removes half-written ones. journald caps the journal (10 % of the disk, at most 4 GB). It refuses an OS
update under 500 MB free. It skips a local whole-box copy that cannot fit, and says so on the backup page (R-548). The
hub alarms at 90 % and 95 %.
@@ -47,7 +48,7 @@ hub alarms at 90 % and 95 %.
| The host journal | Save it first, then `journalctl --vacuum-size=500M` | **PROVEN 2026-11-07**: 3.8 GB → 468 MB (`audits/demo-check-2026-11-07/README.md`; the record is off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-space-2026-11-07/cleanup.txt`). It grows back (R-1367) |
| An installer ISO that a newer one replaced; a test archive of a guest that is gone | `pvesm free <volid>`, after `pct list` and `qm list` show that no guest uses it, and after its checksum matches a copy elsewhere | **PROVEN 2026-10-23** (three ISOs, `audits/day-2026-10-23/09-demohp-space.txt`), **2026-11-07** (one archive of a removed guest), **2026-11-09** (one ISO). Test boxes only |
| Whole-box copies on a former target (H4) | — | **NOT PROVEN.** Needs the operator's word (R-1087) |
| Old binaries in the agent's self-update folder (H12) | — | **NOT PROVEN.** The agent cleans the folder after the release that carries the fix (R-1368) |
| Old binaries in the agent's self-update folder (H12) | No hand step. The agent removes them at the end of each good update, and keeps the running version's file | **PROVEN 2026-11-10** as a product path on three test boxes: 49 → 1, 48 → 1 and 33 → 1 files; a rollback worked after it (`audits/release-space-fixes-2026-11-10/agent/`). **NOT PROVEN:** a removal by hand |
| Extra guest template archives (H5); `/usr/local/bin/felhom-agent.bak-*` (H13) | — | **NOT PROVEN** (R-1368) |
**Never by hand:**