Space fixes released: agent 0.173.0 vouched, controller 0.352.0 global floor, hub 0.180.0 live; evidence, rows, status, report
gates / gates (push) Successful in 2m1s

The close-out of the attended release run of 2026-11-10 (afternoon).
- Evidence: documentation/audits/release-space-fixes-2026-11-10/ (agent, controller, hub) and the hub row after the
  fallback-kernel test.
- Rows: dated lines on R-1368, R-1371, R-1301, R-1353. 236 open before and after; opened 0, closed 0.
- Capability map: dated notes under C13 and G5; no status changed.
- disk-full runbook and 16 §14: the self-update folder is PROVEN as a product path on three test boxes.
- STATUS: new "Updated" paragraph; the old one moved to STATUS-HISTORY.
- REPORT-release-space-fixes-2026-11-10.md: wrong claims first, one deviation (the hub deploy ran before CI for the
  manifest-bump commit was read), what stays not proven.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-10 17:17:04 +01:00
1 parent d0a54c2f8e
commit bcc73b428d
40 files changed
+1037 -59

No files matched your search

@@ -0,0 +1 @@
MATCH 2026-11-10T14:58:14Z: demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 48% 42% running (connected) 71% local-lvm
@@ -0,0 +1,61 @@
# Evidence — the release of the three Space fixes (2026-11-10, attended release run)
**What this is.** The files that prove what the release run did on the boxes and on the hub. The report is
`REPORT-release-space-fixes-2026-11-10.md` at the repository root. The fallback-kernel test of the same run has its own
folder: `audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`.
**Method.** No browser. CC read the boxes over SSH, read the hub's log from its pod, and called the hub's own
addresses with CC's token (the same addresses the hub's pages call). Times are UTC, except lines copied from a host's
journal or the hub's log, which print CET (UTC+1). The N100's host clock is about 32 seconds behind (R-1301); its
lines read early by that much.
**Secrets.** The files were scanned before the commit. They hold no token, no password and no key. The signed job
envelopes are not here.
## `agent/` — the agent release (R-1368 part 2)
| File | What it shows |
|---|---|
| `01-folder-before.txt` | The self-update folder on the three test boxes before: 49 / 48 / 33 entries |
| `03-sign-upload.txt` | One signed `agent_update` per box, uploaded to the hub |
| `04-delivery.txt` | Each box took the job, committed, and removed the old files (`removed=49`, `48`, `33`) |
| `05-after-demo-hp.txt`, `05-after-demo-felhom.txt`, `05-after-tester1.txt` | The folder after: 1 entry each; no pending marker; the unit active |
| `06-rollback-sign.txt` | The signed job to the older version, for the rollback proof on Tester 1 |
| `07-rollback-proof.txt` | One crash 9 seconds into the dwell; the wrapper put the released version back in the same second |
| `08-after-rollback-tester1.txt` | After the rollback: the released version runs; 2 files in the folder (the failed one stays, by design) |
| `12-vouch-readback.txt` | The operator's vouch, read from the hub's log |
| `13-n100-clock.txt` | The N100's host clock against DooPlex and demo-hp; `chrony` failed at boot (R-1301) |
## `controller/` — the controller release (R-1371 part 2)
| File | What it shows |
|---|---|
| `00-fill-before.txt` | The fill watch on the boxes before the release: 3 filesystems checked, two of them the same one |
| `03-crossing-on-0.350.0.txt` | **Red on a box.** Scratch guest 9202, the version before, one test file: TWO warnings |
| `04-rc1-on-9202.txt` | **Green on a box.** 9202 on the release candidate, the same fill: ONE warning; cleared silently after the file was removed by its exact name |
| `05-boxwalk-before.txt` | The three test boxes before the per-box floors |
| `07-per-box-floors.txt` | The three per-box floors, each `303 … flash=floor_set` |
| `08-boxwalk-after.txt` | The three boxes on the release: healthy, containers 10 / 26 / 9, none unhealthy |
| `09-fs-identity-after.txt` | The watch's target list on each box after: 2 filesystems, one target each |
| `10-https-only.txt` | `scripts/https_only_check.py`, exit 0 |
| `11-global-floor.txt` | The global floor, in its own save; the hub's read-back |
| `12-watch.txt` | 30 minutes after the global floor: every box on the release, no new alarm; the mailbox as a second channel |
| `13-per-box-floors-cleared.txt` | The three per-box floors cleared; the boxes stay on the release by the global floor |
## `hub/` — the hub release (R-1371 part 3, D68)
| File | What it shows |
|---|---|
| `02-d68-tests.txt` | The `TestD68_*` tests, green, on the released commit |
| `03-ci-heading.txt` | CI for the commit the image was built from, read by commit hash |
| `07-deploy.txt` | Before and after the deploy: only the hub's Deployment was out of sync; Synced/Healthy; the image id |
| `08-ci-bump.txt` | CI for the manifest-bump commit. **Read 3 minutes after the deploy, not before** (a deviation; see the report) |
| `09-first-list-read.txt` | The pages that show a household's alarm list refuse a token (403, by design). So the open point of `16` §14.4 is not measured on the running hub |
| `10-watch.txt` | 30 minutes after the deploy: Synced/Healthy, no restart, no error line, 6 reports from each box, no alarm sent |
## Not proven by this folder
- That a household which saved its own alarm list before no longer gets the host-disk alarm. The proof is the tests
and the deployed commit. No test alarm was made, by the brief.
- A real fill on a household's box. A fill at the critical line (95 %) on any box.
- A removal by hand in the agent's self-update folder; the `.bak-*` copies of the agent (R-1368 part 4).
@@ -0,0 +1,27 @@
== demo-hp 2026-11-10T14:34:22Z
demo-hp
felhom-agent 0.172.1
entries=49 staged=49 size_kb=722596
felhom-agent-0.171.0
felhom-agent-0.172.0
felhom-agent-0.172.1
1
-rwxr-xr-x 1 root root 16393888 Nov 3 08:56 /usr/local/bin/felhom-agent.prev
== felhom-pve 2026-11-10T14:34:22Z
demo-felhom
felhom-agent 0.172.1
entries=48 staged=48 size_kb=706968
felhom-agent-0.171.0
felhom-agent-0.172.0
felhom-agent-0.172.1
11
-rwxr-xr-x 1 root root 16393888 Nov 3 08:49 /usr/local/bin/felhom-agent.prev
== root@192.168.0.154 2026-11-10T14:34:22Z
felhom
felhom-agent 0.172.1
entries=33 staged=33 size_kb=497244
felhom-agent-0.171.0
felhom-agent-0.172.0
felhom-agent-0.172.1
0
-rwxr-xr-x 1 root root 16393888 Nov 3 08:57 /usr/local/bin/felhom-agent.prev
@@ -0,0 +1,12 @@
== tester-1-d70be4 2026-11-10T14:46:49Z
signed: op=agent_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=fef383cc866be77372a2c7421c0f70ac expires=2026-11-10T15:16:49Z
uploaded signed op to the hub jobs queue
rc=0
== demo-hp-bb76ea 2026-11-10T14:46:49Z
signed: op=agent_update host=demo-hp-bb76ea guest="" key_id=felhom-op-1 nonce=c230909e8a25f4e33c0ac9c6e5b3b1fe expires=2026-11-10T15:16:49Z
uploaded signed op to the hub jobs queue
rc=0
== demo-felhom-8363b5 2026-11-10T14:46:49Z
signed: op=agent_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=481078428450ad9576174647ba6db78b expires=2026-11-10T15:16:50Z
uploaded signed op to the hub jobs queue
rc=0
@@ -0,0 +1,5 @@
signed ops uploaded 2026-11-10T14:46:49Z
+330s felhom-pve: version 0.173.0, committed
+753s demo-hp: version 0.173.0, committed
+815s root@192.168.0.154: version 0.173.0, committed
AGWATCH RESULT=ok all three committed at 2026-11-10T15:00:37Z
@@ -0,0 +1,31 @@
host=demo-felhom at=2026-11-10T14:53:03Z agent=felhom-agent 0.173.0
entries=1 staged=1 size_kb=16028
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-10T15:51:50 .
drwxr-x--- 9 felhom-agent felhom-agent 4096 2026-11-10T10:22:07 ..
-rwxr-xr-x 1 felhom-agent felhom-agent 16407361 2026-11-10T15:50:45 felhom-agent-0.173.0
pending.json: absent
-rwxr-xr-x 1 root root 16407361 2026-11-10T15:50:46 /usr/local/bin/felhom-agent
-rwxr-xr-x 1 root root 16393888 2026-11-07T23:04:48 /usr/local/bin/felhom-agent.prev
prev version: felhom-agent 0.172.1
bak files: 11
-- journal (host local time)
2026-11-10T15:50:46+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T15:50:46.007+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.173.0 version=0.173.0
2026-11-10T15:50:46+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T15:50:46.327+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.173.0 report="{\"agent_update\": {\"sha256\": \"752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea\", \"version\": \"
2026-11-10T15:50:46+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T15:50:46.327+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=77bd95f4bec54d8e op=agent_update
2026-11-10T15:50:48+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:48.455+01:00 level=INFO msg="felhom-agent daemon starting" version=0.173.0 host_id=demo-felhom-8363b5 hub_url=https://hub.felhom.eu interval_s=900
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.046+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.046+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.047+01:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.938+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.938+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.173.0 prev=0.172.1 dwell=1m0s
2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=20
2026-11-10T15:51:50+01:00 demo-felhom sudo[3986865]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-selfupdate-guarded commit
2026-11-10T15:51:50+01:00 demo-felhom felhom-selfupdate-guarded[3986874]: committed (pending cleared, .prev retained)
2026-11-10T15:51:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:50.032+01:00 level=WARN msg="selfupdate: update committed" version=0.173.0 wrapper=""
2026-11-10T15:51:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:50.128+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.173.0 removed=48
2026-11-10T15:51:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:50.128+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=48 kept=felhom-agent-0.173.0
active
NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:50:48 CET
@@ -0,0 +1,38 @@
host=demo-hp at=2026-11-10T15:00:34Z agent=felhom-agent 0.173.0
entries=1 staged=1 size_kb=16028
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-10T15:59:19 .
drwxr-x--- 10 felhom-agent felhom-agent 4096 2026-11-10T11:36:41 ..
-rwxr-xr-x 1 felhom-agent felhom-agent 16407361 2026-11-10T15:58:14 felhom-agent-0.173.0
pending.json: absent
-rwxr-xr-x 1 root root 16407361 2026-11-10T15:58:14 /usr/local/bin/felhom-agent
-rwxr-xr-x 1 root root 16393888 2026-11-07T23:11:58 /usr/local/bin/felhom-agent.prev
prev version: felhom-agent 0.172.1
bak files: 1
-- journal (host local time)
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.000+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.799+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.799+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.858+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.859+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.859+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=95
2026-11-10T15:58:14+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:58:14.583+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.173.0 version=0.173.0
2026-11-10T15:58:14+01:00 demo-hp felhom-os-apply[66717]: os-apply: AGENT-UPDATE signed by the operator: version=0.173.0 sha=752b2727fb604341 — handing the root copy to the A/B wrapper
2026-11-10T15:58:15+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:58:15.231+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.173.0 report="{\"agent_update\": {\"sha256\": \"752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea\", \"version\": \"0.17
2026-11-10T15:58:15+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:58:15.231+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=897285dcaecd3e9d op=agent_update
2026-11-10T15:58:17+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:17.331+01:00 level=INFO msg="felhom-agent daemon starting" version=0.173.0 host_id=demo-hp-bb76ea hub_url=https://hub.felhom.eu interval_s=900
2026-11-10T15:58:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:18.424+01:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
2026-11-10T15:58:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:18.424+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T15:58:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:18.424+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.207+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.207+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.388+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.389+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.173.0 prev=0.172.1 dwell=1m0s
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.389+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.389+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=95
2026-11-10T15:59:19+01:00 demo-hp sudo[72223]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-selfupdate-guarded commit
2026-11-10T15:59:19+01:00 demo-hp felhom-selfupdate-guarded[72232]: committed (pending cleared, .prev retained)
2026-11-10T15:59:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:19.427+01:00 level=WARN msg="selfupdate: update committed" version=0.173.0 wrapper=""
2026-11-10T15:59:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:19.592+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.173.0 removed=49
2026-11-10T15:59:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:19.592+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=49 kept=felhom-agent-0.173.0
active
NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:58:17 CET
@@ -0,0 +1,38 @@
host=felhom at=2026-11-10T15:00:35Z agent=felhom-agent 0.173.0
entries=1 staged=1 size_kb=16028
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-10T16:00:19 .
drwxr-x--- 10 felhom-agent felhom-agent 4096 2026-11-10T13:35:18 ..
-rwxr-xr-x 1 felhom-agent felhom-agent 16407361 2026-11-10T15:59:14 felhom-agent-0.173.0
pending.json: absent
-rwxr-xr-x 1 root root 16407361 2026-11-10T15:59:15 /usr/local/bin/felhom-agent
-rwxr-xr-x 1 root root 16393888 2026-11-07T23:12:45 /usr/local/bin/felhom-agent.prev
prev version: felhom-agent 0.172.1
bak files: 0
-- journal (host local time)
2026-11-10T15:43:45+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:45.441+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.623+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.631+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.728+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.728+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.732+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=213
2026-11-10T15:59:14+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:59:14.962+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.173.0 version=0.173.0
2026-11-10T15:59:15+01:00 felhom felhom-os-apply[28155]: os-apply: AGENT-UPDATE signed by the operator: version=0.173.0 sha=752b2727fb604341 — handing the root copy to the A/B wrapper
2026-11-10T15:59:15+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:59:15.660+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.173.0 report="{\"agent_update\": {\"sha256\": \"752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea\", \"version\": \"0.173
2026-11-10T15:59:15+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:59:15.660+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=50cce53e2c398bee op=agent_update
2026-11-10T15:59:17+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:17.757+01:00 level=INFO msg="felhom-agent daemon starting" version=0.173.0 host_id=tester-1-d70be4 hub_url=https://hub.felhom.eu interval_s=900
2026-11-10T15:59:18+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:18.689+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T15:59:18+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:18.689+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T15:59:18+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:18.689+01:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.231+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.232+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.338+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.339+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.339+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=213
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.339+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.173.0 prev=0.172.1 dwell=1m0s
2026-11-10T16:00:19+01:00 felhom sudo[30337]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-selfupdate-guarded commit
2026-11-10T16:00:19+01:00 felhom felhom-selfupdate-guarded[30345]: committed (pending cleared, .prev retained)
2026-11-10T16:00:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:00:19.390+01:00 level=WARN msg="selfupdate: update committed" version=0.173.0 wrapper=""
2026-11-10T16:00:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:00:19.607+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.173.0 removed=33
2026-11-10T16:00:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:00:19.607+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=33 kept=felhom-agent-0.173.0
active
NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:59:17 CET
@@ -0,0 +1,5 @@
== rollback proof on Tester 1: a signed agent_update to the OLD version 0.172.1, then one crash inside the 60 s dwell
2026-11-10T15:01:19Z
signed: op=agent_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=3039f82fc2b2103065da50c4015a8d11 expires=2026-11-10T15:31:19Z
uploaded signed op to the hub jobs queue
SIGN RC=0
@@ -0,0 +1,19 @@
watch start 2026-11-10T15:01:24Z: agent=felhom-agent 0.173.0 pending=absent
+788s pending marker present, binary on disk = 0.172.1, MainPID=28267 exe=/usr/local/bin/felhom-agent (deleted)
pending.json: {"old_version":"0.173.0","new_version":"0.172.1","sha256":"505a0594a2fb117f359e329a0b7ae2c7a6902b74c51f52c1883145f3c2ac72f8","applied_at":"2026-11-10T16:14:31+01:00"}
folder now: felhom-agent-0.172.1 felhom-agent-0.173.0 pending.json
os-apply running: 0; dpkg/apt running: 0
+796s 2026-11-10T15:14:40Z KILL: systemctl kill --kill-whom=main -s SEGV felhom-agent (MainPID=51118, pending=PRESENT)
kill rc=0
== after +821s 2026-11-10T15:15:05Z
agent=felhom-agent 0.173.0 active=active pending=absent
running exe version: felhom-agent 0.173.0
folder: entries=2 size_kb=32040: felhom-agent-0.172.1 felhom-agent-0.173.0
sha256 bin / prev:
752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea /usr/local/bin/felhom-agent
752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea /usr/local/bin/felhom-agent.prev
-- journal
2026-11-10T16:14:40+01:00 felhom felhom-selfupdate-guarded[51930]: felhom-selfupdate-guarded: rolled back to previous binary and restarted felhom-agent.service
failed units: 0
ROLLBACK-PROOF DONE
PROOF RC=0
@@ -0,0 +1,39 @@
host=felhom at=2026-11-10T15:15:56Z agent=felhom-agent 0.173.0
entries=2 staged=2 size_kb=32040
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-10T16:14:40 .
drwxr-x--- 10 felhom-agent felhom-agent 4096 2026-11-10T13:35:18 ..
-rwxr-xr-x 1 felhom-agent felhom-agent 16393888 2026-11-10T16:14:30 felhom-agent-0.172.1
-rwxr-xr-x 1 felhom-agent felhom-agent 16407361 2026-11-10T15:59:14 felhom-agent-0.173.0
pending.json: absent
-rwxr-xr-x 1 root root 16407361 2026-11-10T16:14:40 /usr/local/bin/felhom-agent
-rwxr-xr-x 1 root root 16407361 2026-11-10T15:59:15 /usr/local/bin/felhom-agent.prev
prev version: felhom-agent 0.173.0
bak files: 0
-- journal (host local time)
2026-11-10T16:14:31+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:14:31.582+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.172.1 report="{\"agent_update\": {\"sha256\": \"505a0594a2fb117f359e329a0b7ae2c7a6902b74c51f52c1883145f3c2ac72f8\", \"version\": \"0.17
2026-11-10T16:14:31+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:14:31.582+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=f6848443a231739a op=agent_update
2026-11-10T16:14:33+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:33.722+01:00 level=INFO msg="felhom-agent daemon starting" version=0.172.1 host_id=tester-1-d70be4 hub_url=https://hub.felhom.eu interval_s=900
2026-11-10T16:14:34+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:34.663+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T16:14:34+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:34.663+01:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
2026-11-10T16:14:34+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:34.663+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.232+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.233+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.319+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.320+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.321+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.172.1 prev=0.173.0 dwell=1m0s
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.321+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=213
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate.NewManager.func2({0xe5d130, 0x1e342d05aac0}, 0xef4820?)
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate/commit.go:54 +0xb4 fp=0x1e342d2d1dc8 sp=0x1e342d2d1d38 pc=0x880e14
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate.(*Manager).MaybeCommit(0x1e342d13b2c0, {0xe5d130, 0x1e342d05aac0})
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate/commit.go:111 +0x3aa fp=0x1e342d2d1fb8 sp=0x1e342d2d1dc8 pc=0x87e0aa
2026-11-10T16:14:40+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:40.445+01:00 level=INFO msg="felhom-agent daemon starting" version=0.173.0 host_id=tester-1-d70be4 hub_url=https://hub.felhom.eu interval_s=900
2026-11-10T16:14:41+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:41.489+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T16:14:41+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:41.489+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T16:14:41+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:41.489+01:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.072+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.072+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.172+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.173+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.173+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=213
active
NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 16:14:40 CET
@@ -0,0 +1,4 @@
2026/11/10 16:29:16 [INFO] artifact vouch: golden 0.350.0 unchanged (behind the fleet's 0.352.0 — the weekly bake) — the R-120 gate does not apply to an unchanged golden; saving agent=0.173.0 min_agent="0.158.1"
2026/11/10 16:29:16 [INFO] Artifact manifest set: agent=0.173.0 golden=0.350.0 min_agent="0.158.1" wrapper_sha=false bundle_sha="51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1"
2026/11/10 16:29:16 [INFO] artifact vouch timing (R-1346): registry probes 101ms, sha lookups 29ms, bundle sha 13ms, fleet check + save 87ms, total 232ms
read 2026-11-10T15:29:45Z (the hub log prints CET)
@@ -0,0 +1,16 @@
== N100 host clock against DooPlex and demo-hp (read-only), for R-1301
dooplex 2026-11-10T15:38:07.145528284Z
n100 2026-11-10T15:37:34.778614952Z
demo-hp 2026-11-10T15:38:07.999970091Z
dooplex 2026-11-10T15:38:08.002419987Z
-- n100 timedatectl
NTPSynchronized=no
TimeUSec=Tue 2026-11-10 16:37:35 CET
failed
Nov 08 05:35:23 demo-felhom chronyd[690]: Loaded seccomp filter (level 1)
Nov 08 05:35:23 demo-felhom chronyd[690]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Nov 08 05:35:23 demo-felhom chronyd[690]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Nov 08 05:35:23 demo-felhom systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE
Nov 08 05:35:23 demo-felhom systemd[1]: chrony.service: Failed with result 'exit-code'.
Nov 08 05:35:23 demo-felhom systemd[1]: Failed to start chrony.service - chrony, an NTP client/server.
2026-11-08 05:35:18
@@ -0,0 +1,8 @@
Fill watch BEFORE the release (controller 0.351.0 on the three test boxes, 0.350.0 on scratch guest 9202), read 2026-11-10 14:24–14:51 UTC.
The line counts TARGETS. On a one-volume box the two built-in targets "/" and /mnt/sys_drive are one filesystem
(felhom.eu documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt).
demo-felhom (N100) guest 9201, 0.351.0: [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok (14:24:05, 14:34:05)
demo-hp guest 9201, 0.351.0: 2026/11/10 14:45:00 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
demo-hp scratch guest 9202, 0.350.0: 2026/11/10 14:44:49 [INFO] [fillwatch] checked 11 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: all ok
Tester 1 guest 9201, 0.351.0: 2026/11/10 14:45:41 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
@@ -0,0 +1,41 @@
== The warning band on scratch guest 9202, OLD controller 0.350.0 (the 'before': how many warnings does one fill send?)
== 2026-11-10T14:53:04Z guest config and the state before
hostname: demo-hp-scratch
mp0: nvme-scratch:9202/vm-9202-disk-3.raw,mp=/var/lib/felhom,backup=1,size=70G
mp8: /mnt/hdd_1/scratch-drives/scratch_hdd,mp=/mnt/felhom-drives/scratch_hdd
mp9: /var/lib/felhom-agent/guests/9202/bootstrap,mp=/etc/felhom-bootstrap,ro=1
rootfs: nvme-scratch:9202/vm-9202-disk-2.raw,size=32G
/dev/loop1 73391005696 13985423360 55630708736 21% /var/lib/felhom
total 32
drwxr-xr-x 5 root root 4096 Oct 20 08:44 .
drwxr-xr-x 24 root root 4096 Oct 4 07:13 ..
drwx--x--- 12 root root 4096 Nov 10 14:43 docker
drwx------ 2 nobody nogroup 16384 Oct 4 07:11 lost+found
drwxr-xr-x 4 root root 4096 Oct 26 19:52 sys_drive
2026/11/10 14:44:49 fillwatch.go:284: [INFO] [fillwatch] checked 11 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: all ok
== 2026-11-10T14:53:15Z the test file: one file, exact name, 45902462976 bytes (42.75 GiB), made with fallocate (no data written); target 86 % used, about 9 GiB still free; the ext4 reserve is not touched (the file is smaller than 'avail')
df: /mnt/nvme-1tb: No such file or directory
fallocate_rc=0
-rw-r--r-- 1 root root 45902462976 Nov 10 14:53 /var/lib/felhom/FILLTEST-2026-11-10-r1371.bin
/dev/loop1 73391005696 59887894528 9728237568 87% /var/lib/felhom
/dev/loop1 69G 56G 9.1G 87% /var/lib/felhom
df: /mnt/nvme-1tb: No such file or directory
== 2026-11-10T14:54:01Z the controller's own lines (0.350.0): ONE fill, TWO warnings
gitea.dooplex.hu/admin/felhom-controller:0.350.0 Up 10 minutes (healthy)
2026/11/10 14:53:19 fillwatch.go:262: [WARN] [fillwatch] / ("Alkalmazások területe"): ok → warning — 86% used, 9.1 GB free of 68.4 GB; notifying the customer
2026/11/10 14:53:19 notifier.go:1471: [WARN] notifier disabled (no hub configured): DROPPED event disk_warning (severity warning) — further disk_warning events are logged at DEBUG only
2026/11/10 14:53:19 fillwatch.go:262: [WARN] [fillwatch] /mnt/sys_drive ("Rendszer- és mentési terület"): ok → warning — 86% used, 9.1 GB free of 68.4 GB; notifying the customer
2026/11/10 14:53:19 notifier.go:1475: [DEBUG] notifier disabled: dropped event disk_warning (severity warning)
2026/11/10 14:53:19 fillwatch.go:284: [INFO] [fillwatch] checked 11 filesystem(s), 2 unreadable/skipped, 2 notification(s); bands: /=warning /mnt/sys_drive=warning
-- remove the test file by its exact name
removed '/var/lib/felhom/FILLTEST-2026-11-10-r1371.bin'
total 32
drwxr-xr-x 5 root root 4096 Nov 10 14:54 .
drwxr-xr-x 24 root root 4096 Oct 4 07:13 ..
drwx--x--- 12 root root 4096 Nov 10 14:43 docker
drwx------ 2 nobody nogroup 16384 Oct 4 07:11 lost+found
drwxr-xr-x 4 root root 4096 Oct 26 19:52 sys_drive
/dev/loop1 73391005696 13985468416 55630663680 21% /var/lib/felhom
@@ -0,0 +1,85 @@
== 2026-11-10T14:54:02Z controller 0.352.0-rc.1 onto scratch guest 9202 (by hand, the skill's step 3; digest sha256:d9b47ad5f3fc666ba06bb8542b3b49ff056f25643871021ef073fdb924194e7d)
gitea.dooplex.hu/admin/felhom-controller:0.350.0
gitea.dooplex.hu/admin/felhom-controller:0.352.0-rc.1
restart_rc=0
gitea.dooplex.hu/admin/felhom-controller:0.352.0-rc.1 Up 12 seconds (healthy)
sha256:9298978a8206200a32e9e13ab4910ac136e81775ee9f9337e4e33ae20e092edc
gitea.dooplex.hu/admin/felhom-controller@sha256:d9b47ad5f3fc666ba06bb8542b3b49ff056f25643871021ef073fdb924194e7d
== 2026-11-10T14:56:10Z first fill check on 0.352.0-rc.1 (the old state held two warnings; the file was already removed)
2026/11/10 14:54:05 main.go:335: [INFO] felhom-controller 0.352.0-rc.1 starting (customer: demo-hp, domain: enkisfelhom.hu)
2026/11/10 14:55:35 fillwatch.go:250: [INFO] [fillwatch] /: warning → ok (20% used, 51.8 GB free) — cleared silently, re-armed
2026/11/10 14:55:35 fillwatch.go:233: [DEBUG] [fillwatch] /mnt/felhom-drives/scratch_hdd/userdata/grimmory: usage unreadable — skipped (an unreadable filesystem is not a full one)
2026/11/10 14:55:35 fillwatch.go:233: [DEBUG] [fillwatch] /mnt/felhom-drives/scratch_hdd/userdata/metube: usage unreadable — skipped (an unreadable filesystem is not a full one)
2026/11/10 14:55:35 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: /=ok
-- state file
{
"bands": {}
}
== the test file again: same exact name, same size
2026-11-10T14:56:11Z
fallocate_rc=0
-rw-r--r-- 1 root root 45902462976 Nov 10 14:56 /var/lib/felhom/FILLTEST-2026-11-10-r1371.bin
/dev/loop1 73391005696 59928866816 9687265280 87% /var/lib/felhom
== 2026-11-10T15:04:16Z the next fill check ran (+471s)
gitea.dooplex.hu/admin/felhom-controller:0.352.0-rc.1 Up 10 minutes (healthy)
2026/11/10 15:04:06 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: /=ok
-- state file
{
"bands": {}
}
-- remove the test file by its exact name
removed '/var/lib/felhom/FILLTEST-2026-11-10-r1371.bin'
total 32
drwxr-xr-x 5 root root 4096 Nov 10 15:04 .
drwxr-xr-x 24 root root 4096 Oct 4 07:13 ..
drwx--x--- 12 root root 4096 Nov 10 14:43 docker
drwx------ 2 nobody nogroup 16384 Oct 4 07:11 lost+found
drwxr-xr-x 4 root root 4096 Oct 26 19:52 sys_drive
/dev/loop1 73391005696 12412989440 57203142656 18% /var/lib/felhom
CROSSWAIT DONE 2026-11-10T15:04:18Z
== NOTE 2026-11-10T15:05:00Z: the 15:04:06 check read ok and that is CORRECT. Between 14:56 and 15:04 the box freed 1.6 GB by itself
(other use 14,026,403,840 -> 12,412,993,536 bytes), so the volume was at 83.8 % with 10.5 GiB free: under the 85 % line. The file was too small.
== second try: a bigger file of the same exact name, target 88 %
2026-11-10T15:05:00Z
fallocate_rc=0
-rw-r--r-- 1 root root 48850000000 Nov 10 15:05 /var/lib/felhom/FILLTEST-2026-11-10-r1371.bin
/dev/loop1 73391005696 61263015936 8353116160 89% /var/lib/felhom
== 2026-11-10T15:14:06Z the next fill check ran (+540s)
gitea.dooplex.hu/admin/felhom-controller:0.352.0-rc.1 Up 20 minutes (healthy)
2026/11/10 15:14:06 fillwatch.go:262: [WARN] [fillwatch] / ("Alkalmazások területe"): ok → warning — 88% used, 7.8 GB free of 68.4 GB; notifying the customer
2026/11/10 15:14:06 notifier.go:1471: [WARN] notifier disabled (no hub configured): DROPPED event disk_warning (severity warning) — further disk_warning events are logged at DEBUG only
2026/11/10 15:14:06 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 2 unreadable/skipped, 1 notification(s); bands: /=warning
-- state file
{
"bands": {
"/": "warning"
}
}
-- remove the test file by its exact name
removed '/var/lib/felhom/FILLTEST-2026-11-10-r1371.bin'
total 32
drwxr-xr-x 5 root root 4096 Nov 10 15:14 .
drwxr-xr-x 24 root root 4096 Oct 4 07:13 ..
drwx--x--- 12 root root 4096 Nov 10 14:43 docker
drwx------ 2 nobody nogroup 16384 Oct 4 07:11 lost+found
drwxr-xr-x 4 root root 4096 Oct 26 19:52 sys_drive
/dev/loop1 73391005696 12413243392 57202888704 18% /var/lib/felhom
CROSSWAIT DONE 2026-11-10T15:14:07Z
== 2026-11-10T15:32:45Z after the file was removed: the next check (15:24:06Z) cleared it without a notification; the test file is gone
total 32
drwxr-xr-x 5 root root 4096 Nov 10 15:14 .
drwxr-xr-x 24 root root 4096 Oct 4 07:13 ..
drwx--x--- 12 root root 4096 Nov 10 14:43 docker
drwx------ 2 nobody nogroup 16384 Oct 4 07:11 lost+found
drwxr-xr-x 4 root root 4096 Oct 26 19:52 sys_drive
/dev/loop1 73391005696 12413661184 57202470912 18% /var/lib/felhom
2026/11/10 15:24:06 fillwatch.go:250: [INFO] [fillwatch] /: warning → ok (18% used, 53.3 GB free) — cleared silently, re-armed
2026/11/10 15:24:06 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: /=ok
{
"bands": {}
}== 2026-11-10T15:32:46Z 9202 moves from rc.1 to the release tag by hand (it has no hub, so no floor moves it)
gitea.dooplex.hu/admin/felhom-controller:0.352.0
restart_rc=0
@@ -0,0 +1,40 @@
### root@192.168.0.154 2026-11-10T15:02:11Z
containers=10 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.351.0 Up 18 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.351.0
started=2026-11-10T14:44:09.016454308Z
errors=0 warn=0 info=457
2026/11/10 14:44:10 [INFO] felhom-controller 0.351.0 starting (customer: tester-1, domain: enkicsifelhom.hu)
2026/11/10 14:44:11 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
2026/11/10 14:44:31 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.351.0 (we are 0.351.0), no managed update running
2026/11/10 14:44:47 [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/10 14:45:41 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 14:54:12 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
### demo-hp 2026-11-10T15:02:12Z
containers=26 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.351.0 Up 18 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.351.0
started=2026-11-10T14:43:25.514593512Z
errors=0 warn=6 info=519
2026/11/10 14:44:15 updater.go:104: [DEBUG] [selfupdate] queryRegistry: anonymous tags/list with Bearer HTTP 200
2026/11/10 14:44:15 updater.go:104: [DEBUG] [selfupdate] queryRegistry: 38 tags returned: [0.331.0 0.341.0 0.341.0-rc.1 0.342.0 0.342.0-rc.1 0.342.0-rc.2 0.343.0 0.343.0-rc.1 0.343.0-rc.2 0.343.0-rc.3 0.343.0-rc.4 0.343.0-rc.5 0.3
2026/11/10 14:44:15 updater.go:196: [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/10 14:44:15 updater.go:104: [DEBUG] [selfupdate] version comparison: current=0.351.0 (0.351.0), latest=0.351.0 (0.351.0), cmp=0, updateAvailable=false
2026/11/10 14:58:33 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.351.0 >= floor 0.351.0 — no action
2026/11/10 14:45:00 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 14:53:31 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
### felhom-pve 2026-11-10T15:02:14Z
containers=9 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.351.0 Up 27 hours (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.351.0
started=2026-11-09T12:14:03.775366821Z
errors=0 warn=6 info=27865
2026/11/09 12:14:36 [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/09 18:14:05 [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/10 00:14:05 [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/10 06:14:05 [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/10 12:14:05 [INFO] [selfupdate] Current version 0.351.0 is up to date
2026/11/10 14:24:05 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 14:34:05 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 14:44:05 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 14:54:05 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
@@ -0,0 +1,14 @@
== 2026-11-10T15:16:52Z per-box floors 0.352.0 / MinAgent 0.158.1 (image sha256:3c785dd396b091f19d4c777db9d7a206e75ab018b6238b5d887e1a4c6cfd3b29)
-- demo-hp
2026-11-10T15:16:52Z POST /customers/demo-hp/floor min_controller_version='0.352.0' min_agent='0.158.1'
HTTP/1.1 303 See Other
Location: /customers/demo-hp?flash=floor_set
-- demo-felhom
2026-11-10T15:16:52Z POST /customers/demo-felhom/floor min_controller_version='0.352.0' min_agent='0.158.1'
HTTP/1.1 303 See Other
Location: /customers/demo-felhom?flash=floor_set
-- tester-1
2026-11-10T15:16:52Z POST /customers/tester-1/floor min_controller_version='0.352.0' min_agent='0.158.1'
HTTP/1.1 303 See Other
Location: /customers/tester-1?flash=floor_set
== 2026-11-10T15:16:52Z done
@@ -0,0 +1,39 @@
### root@192.168.0.154 2026-11-10T15:30:01Z
containers=10 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 12 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
started=2026-11-10T15:17:03.705107516Z
errors=0 warn=0 info=335
2026/11/10 15:17:04 [INFO] felhom-controller 0.352.0 starting (customer: tester-1, domain: enkicsifelhom.hu)
2026/11/10 15:17:04 [INFO] [selfupdate] Post-update startup: update successful (0.351.0 → 0.352.0)
2026/11/10 15:17:04 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
2026/11/10 15:17:14 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.352.0 (we are 0.352.0), no managed update running
2026/11/10 15:17:36 [INFO] [selfupdate] Current version 0.352.0 is up to date
2026/11/10 15:18:34 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:27:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
### demo-hp 2026-11-10T15:30:02Z
containers=26 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 13 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
started=2026-11-10T15:17:03.918469562Z
errors=0 warn=3 info=345
2026/11/10 15:17:37 updater.go:104: [DEBUG] [selfupdate] queryRegistry: anonymous plain GET HTTP 401
2026/11/10 15:17:37 updater.go:104: [DEBUG] [selfupdate] queryRegistry: anonymous tags/list with Bearer HTTP 200
2026/11/10 15:17:37 updater.go:104: [DEBUG] [selfupdate] queryRegistry: 40 tags returned: [0.331.0 0.341.0 0.341.0-rc.1 0.342.0 0.342.0-rc.1 0.342.0-rc.2 0.343.0 0.343.0-rc.1 0.343.0-rc.2 0.343.0-rc.3 0.343.0-rc.4 0.343.0-rc.5 0.3
2026/11/10 15:17:37 updater.go:196: [INFO] [selfupdate] Current version 0.352.0 is up to date
2026/11/10 15:17:37 updater.go:104: [DEBUG] [selfupdate] version comparison: current=0.352.0 (0.352.0), latest=0.352.0 (0.352.0), cmp=0, updateAvailable=false
2026/11/10 15:18:35 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:27:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
### felhom-pve 2026-11-10T15:30:04Z
containers=9 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 13 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
started=2026-11-10T15:16:28.700768259Z
errors=0 warn=0 info=342
2026/11/10 15:16:29 [INFO] felhom-controller 0.352.0 starting (customer: demo-felhom, domain: demo-felhom.eu)
2026/11/10 15:16:29 [INFO] [selfupdate] Post-update startup: update successful (0.351.0 → 0.352.0)
2026/11/10 15:16:29 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
2026/11/10 15:16:39 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.352.0 (we are 0.352.0), no managed update running
2026/11/10 15:17:01 [INFO] [selfupdate] Current version 0.352.0 is up to date
2026/11/10 15:17:59 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:26:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
@@ -0,0 +1,43 @@
# after 0.352.0: what the controller container sees, and what the fill watch says — read-only, 2026-11-10T15:30:48Z
# channel 1 = the kernel (stat -f / df inside the container); channel 2 = the fill watch's own log line
== root@192.168.0.154:9201
/ type=overlayfs fsid=b4d48bcf2747c72b blocks=18016108 free=15822196 dev=74
/mnt/felhom-drives type=ext2/ext3 fsid=322b724225b4b36d blocks=15357389 free=10191290 dev=64513
/mnt/sys_drive type=ext2/ext3 fsid=65ddc34fe9988fc8 blocks=18016108 free=15822196 dev=64519
-- distinct filesystems by fsid, the overlay root counted as the volume below it:
2
2026/11/10 15:18:34 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:27:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
state: { "bands": {}}
== demo-hp:9201
/ type=overlayfs fsid=71b91e3d7cadcd4f blocks=18016108 free=10674045 dev=49
/mnt/felhom-drives type=ext2/ext3 fsid=c04f01ca42f6f3e9 blocks=10113344 free=6060989 dev=64513
/mnt/sizing-bench type=ext2/ext3 fsid=85fbdcb4db8310a7 blocks=8179140 free=7847629 dev=64518
/mnt/sys_drive type=ext2/ext3 fsid=e1508f4b7cb1b4b0 blocks=18016108 free=10674045 dev=64519
-- distinct filesystems by fsid, the overlay root counted as the volume below it:
3
2026/11/10 15:18:35 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:27:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
state: { "bands": {}}
== felhom-pve:9201
/ type=overlayfs fsid=87e8f2c78a7b090c blocks=64230001 free=63627374 dev=88
/mnt/felhom-drives type=ext2/ext3 fsid=813814bd69e4dd6e blocks=24624445 free=15508719 dev=64513
/mnt/sys_drive type=ext2/ext3 fsid=6968e12ca6ac967a blocks=64230001 free=63627374 dev=64519
-- distinct filesystems by fsid, the overlay root counted as the volume below it:
2
2026/11/10 15:17:59 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:26:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
state:
# the drives below /mnt/felhom-drives (the folder itself is on the guest's root disk, which nobody watches — R-1371), 2026-11-10T15:31:05Z
== root@192.168.0.154:9201
/mnt/felhom-drives/adatlemez type=ext2/ext3 fsid=a25b481c3dbf279a blocks=12822648 free=9621417 dev=2064
/mnt/felhom-drives/media_a type=ext2/ext3 fsid=322b724225b4b36d blocks=15357389 free=10191292 dev=64513
/mnt/felhom-drives/media_b type=ext2/ext3 fsid=322b724225b4b36d blocks=15357389 free=10191292 dev=64513
/mnt/felhom-drives/media_c type=ext2/ext3 fsid=322b724225b4b36d blocks=15357389 free=10191292 dev=64513
/mnt/felhom-drives/media_d type=ext2/ext3 fsid=322b724225b4b36d blocks=15357389 free=10191292 dev=64513
== demo-hp:9201
/mnt/felhom-drives/hdd_1 type=ext2/ext3 fsid=3ce20f0a870807d7 blocks=245844925 free=167050197 dev=66304
== felhom-pve:9201
/mnt/felhom-drives/hdd_1 type=ext2/ext3 fsid=ef113ee179770dd1 blocks=240075962 free=238087780 dev=2064
# count per box = the apps' volume (/ and /mnt/sys_drive are one filesystem) + the drives that are a filesystem of their own
@@ -0,0 +1,36 @@
== 2026-11-10T15:31:16Z python3 scripts/https_only_check.py
control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -)
== tester-1 (enkicsifelhom.hu): 6 public name(s)
OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1
OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1
OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1
OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1
OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1
OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1
== demo-hp (enkisfelhom.hu): 15 public name(s)
OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1
OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1
OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1
OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1
OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1
OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1
OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1
OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1
OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1
OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1
OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1
OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1
OK http://time.enkisfelhom.hu/r1033-check?q=1 -> 301 https://time.enkisfelhom.hu/r1033-check?q=1
OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1
OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1
== demo-felhom (demo-felhom.eu): 8 public name(s)
OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1
OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1
OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1
OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1
OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1
OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1
OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1
OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
== every public name redirects http -> https
HTTPS RC=0
@@ -0,0 +1,14 @@
== 2026-11-10T15:31:33Z before
2026-11-10T15:31:34Z GET /configuration -> 200
Effective floor: v0.351.0 — source: DB (hub_settings) ; env fallback would be v0.120.0 Save global floor… DB override: v0.351.0 … Type the version again to confirm (or CLEAR to remove the DB override): Confirm & apply Cancel function confirmGlobalFloor() { var v = document.g
.. MinAgent: . The hub HOLDS the floor for any box whose agent is below this — blank = uncoupled release, no gating. PBS wrapper sha256 of configs/felhom-pbs-apply (R-50b)
== 2026-11-10T15:31:34Z the global floor, in its own save
2026-11-10T15:31:34Z POST /configuration/global-floor min_controller_version='0.352.0' min_agent='0.158.1'
HTTP/1.1 303 See Other
Location: /configuration?flash=floor_set
== 2026-11-10T15:31:34Z read back
2026-11-10T15:31:34Z GET /configuration -> 200
Effective floor: v0.352.0 — source: DB (hub_settings) ; env fallback would be v0.120.0 Save global floor… DB override: v0.352.0 … Type the version again to confirm (or CLEAR to remove the DB override): Confirm & apply Cancel function confirmGlobalFloor() { var v = document.g
.. MinAgent: . The hub HOLDS the floor for any box whose agent is below this — blank = uncoupled release, no gating. PBS wrapper sha256 of configs/felhom-pbs-apply (R-50b)
2026/11/10 16:31:34 [INFO] Global controller-version floor set to "0.352.0" (declared MinAgent "0.158.1")
2026/11/10 16:31:34 [INFO] access token "cc-releases-2": POST /configuration/global-floor → 303
@@ -0,0 +1,91 @@
== controller watch (watch), hub log since 2026-11-10T15:31:30Z; read 2026-11-10T16:02:15Z
-- the three boxes (read on the box, over SSH)
### root@192.168.0.154 2026-11-10T16:02:15Z
containers=10 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 45 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
started=2026-11-10T15:17:03.705107516Z
errors=0 warn=0 info=945
2026/11/10 15:17:04 [INFO] felhom-controller 0.352.0 starting (customer: tester-1, domain: enkicsifelhom.hu)
2026/11/10 15:17:04 [INFO] [selfupdate] Post-update startup: update successful (0.351.0 → 0.352.0)
2026/11/10 15:17:04 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
2026/11/10 15:17:14 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.352.0 (we are 0.352.0), no managed update running
2026/11/10 15:17:36 [INFO] [selfupdate] Current version 0.352.0 is up to date
2026/11/10 15:27:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:37:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:47:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:57:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
### demo-hp 2026-11-10T16:02:16Z
containers=26 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 45 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
started=2026-11-10T15:17:03.918469562Z
errors=0 warn=3 info=962
2026/11/10 15:17:37 updater.go:104: [DEBUG] [selfupdate] version comparison: current=0.352.0 (0.352.0), latest=0.352.0 (0.352.0), cmp=0, updateAvailable=false
2026/11/10 15:31:38 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
2026/11/10 15:32:07 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
2026/11/10 15:47:07 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
2026/11/10 16:02:07 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
2026/11/10 15:17:05 deploy.go:1470: [WARN] [stacks] Stack plex: new field PLEX_CLAIM (type=text) requires manual configuration
2026/11/10 15:27:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:37:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:47:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:57:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
### felhom-pve 2026-11-10T16:02:18Z
containers=9 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 45 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
started=2026-11-10T15:16:28.700768259Z
errors=0 warn=0 info=906
2026/11/10 15:16:29 [INFO] felhom-controller 0.352.0 starting (customer: demo-felhom, domain: demo-felhom.eu)
2026/11/10 15:16:29 [INFO] [selfupdate] Post-update startup: update successful (0.351.0 → 0.352.0)
2026/11/10 15:16:29 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
2026/11/10 15:16:39 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.352.0 (we are 0.352.0), no managed update running
2026/11/10 15:17:01 [INFO] [selfupdate] Current version 0.352.0 is up to date
2026/11/10 15:26:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:36:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:46:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 15:56:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
-- hub log since 2026-11-10T15:31:30Z: lines / [ERROR] lines
152
0
-- [WARN] lines by kind (the log prints CET)
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers → 403
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers/demo-hp →
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers/tester-1 →
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /events → 403
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /notifications → 403
-- floor lines
2026/11/10 16:31:34 [INFO] Global controller-version floor set to "0.352.0" (declared MinAgent "0.158.1")
2026/11/10 16:31:34 [INFO] access token "cc-releases-2": POST /configuration/global-floor → 303
-- events from boxes (lines that name an event)
(count: 0)
-- alarm checks and mails
-- lines that name a mail send
0
-- positive signs (the hub works, it is not only quiet): reports received in the window, by box
4 demo-felhom
4 demo-hp
4 tester-1
-- host reports, by host
2 demo-felhom-8363b5
2 demo-hp-bb76ea
2 tester-1-d70be4
-- pod
hub-658c847c65-d8zbk phase=Running restarts=0 image=gitea.dooplex.hu/admin/felhom-hub:0.180.0
-- the hub's hosts page (GET /hosts with CC's token), read 2026-11-10T16:02:58Z -> 200
demo-felhom-8363b5 | agent 0.173.0 | 7.0.14-28-pve | ONLINE | cloudflared running (connected)
demo-hp-bb76ea | agent 0.173.0 | 7.0.14-28-pve | ONLINE | cloudflared running (connected)
tester-1-d70be4 | agent 0.173.0 | 7.0.14-22-pve | ONLINE | cloudflared running (connected)
(the page holds these three hosts and no other)
-- the effective global floor, read back 2026-11-10T16:02:59Z (GET /configuration -> 200)
Effective floor: v0.352.0 — source: DB (hub_settings)
-- demo-hp's three [WARN] lines since its controller start (the walk's secret filter hid two; field NAMES only, no value)
15:17:05 [WARN] [stacks] Stack calibre-web: new field ADMIN_PASSWORD (type=password) requires manual configuration
15:17:05 [WARN] [stacks] Stack plex: new field PLEX_CLAIM (type=text) requires manual configuration
15:17:05 [WARN] [stacks] Stack bookstack: new field ADMIN_PASSWORD (type=password) requires manual configuration
(the controller's own start note for a field added to an installed app; this release did not change that code)
-- second channel, not the hub: the mailbox that gets admin@ and tester1@ mail, read by the mail connector 2026-11-10T16:02Z
search 'newer_than:3h' -> 0 threads (no mail since 13:02 UTC; the run's first release step was 14:51 UTC)
positive control, search 'newer_than:1d' -> 1 thread: the security watch's mail of 2026-11-10T03:41:40Z to admin@ (so the search does find mail)
-- verdict: every box that reports is on 0.352.0 and healthy; no event, no alarm and no mail in the 30 minutes after the global floor
@@ -0,0 +1,32 @@
== 2026-11-10T16:03:11Z clear the per-box floors (both fields empty); the global floor stays
-- demo-hp
2026-11-10T16:03:11Z POST /customers/demo-hp/floor min_controller_version='' min_agent=''
HTTP/1.1 303 See Other
Location: /customers/demo-hp?flash=floor_set
-- demo-felhom
2026-11-10T16:03:11Z POST /customers/demo-felhom/floor min_controller_version='' min_agent=''
HTTP/1.1 303 See Other
Location: /customers/demo-felhom?flash=floor_set
-- tester-1
2026-11-10T16:03:11Z POST /customers/tester-1/floor min_controller_version='' min_agent=''
HTTP/1.1 303 See Other
Location: /customers/tester-1?flash=floor_set
== 2026-11-10T16:03:11Z done
-- the hub's log for the clear (the log prints CET), and the first report of each box after it
2026/11/10 17:03:11 [INFO] Customer demo-hp controller-version floor override set to "" (declared MinAgent "")
2026/11/10 17:03:11 [INFO] access token "cc-releases-2": POST /customers/demo-hp/floor → 303
2026/11/10 17:03:11 [INFO] Customer demo-felhom controller-version floor override set to "" (declared MinAgent "")
2026/11/10 17:03:11 [INFO] access token "cc-releases-2": POST /customers/demo-felhom/floor → 303
2026/11/10 17:03:11 [INFO] Customer tester-1 controller-version floor override set to "" (declared MinAgent "")
2026/11/10 17:03:11 [INFO] access token "cc-releases-2": POST /customers/tester-1/floor → 303
2026/11/10 17:03:14 [INFO] Received report from tester-1 (5252 bytes)
2026/11/10 17:03:15 [INFO] Received report from demo-felhom (5563 bytes)
2026/11/10 17:03:15 [INFO] Received report from demo-hp (10292 bytes)
-- the boxes, read on the box 2026-11-10T16:08Z, 5 minutes after the clear
tester-1: gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 50 minutes (healthy)
demo-hp: gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 50 minutes (healthy)
demo-felhom: gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 51 minutes (healthy)
-- the floor a box got AFTER the clear (demo-hp logs at DEBUG; the other two do not print this line)
2026/11/10 16:03:15 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
-- verdict: the per-box floors are gone; the boxes stay on 0.352.0; the floor demo-hp got 4 seconds after the clear is the global one
-- not read: the hub's page of each customer (it refuses a token, 403); the proof of the clear is the hub's own log line
@@ -0,0 +1,5 @@
--- PASS: TestD68_SeedHoldsNoOperatorOnlyType (0.00s)
--- PASS: TestD68_HostDiskIsOperatorOnly_EvenWhenTheHouseholdSavedIt (0.07s)
--- PASS: TestD68_StorageFillReachesAHouseholdOnlyForADriveItCanFree (0.34s)
--- PASS: TestD68_RegisterAndGate (0.00s)
D68 RC=0
@@ -0,0 +1,4 @@
14:59:29Z felhom.eu 97b7e47114: run 2616 gates #1418 status running
14:59:49Z felhom.eu 97b7e47114: run 2616 gates #1418 status running
15:00:10Z felhom.eu 97b7e47114: run 2616 gates #1418 status success
CI RC=0
@@ -0,0 +1,41 @@
== 2026-11-10T15:12:34Z before: image gitea.dooplex.hu/admin/felhom-hub:0.179.1
app: sync=OutOfSync health=Healthy rev=d0a54c2f8e308ae959900fd8068bc2b434d28664
== OutOfSync resources
Deployment felhom-system hub OutOfSync
count: 1
== 2026-11-10T15:12:48Z sync ONLY Deployment/hub at d0a54c2f8e308ae959900fd8068bc2b434d28664
application.argoproj.io/felhom patched
Waiting for deployment "hub" rollout to finish: 0 of 1 updated replicas are available...
deployment "hub" successfully rolled out
rollout rc=0
== 2026-11-10T15:13:10Z after
image: gitea.dooplex.hu/admin/felhom-hub:0.180.0
hub-658c847c65-d8zbk Running ready=true restarts=0 imageID=gitea.dooplex.hu/admin/felhom-hub@sha256:768e73ab1ab502a8c3fe23a9d42cc038443a6456a1f40dcec853a2c588d04a8f started=2026-11-10T15:12:49Z
operation: phase=Succeeded msg=successfully synced (all tasks run) rev=d0a54c2f8e308ae959900fd8068bc2b434d28664
app: sync=Synced health=Healthy rev=d0a54c2f8e308ae959900fd8068bc2b434d28664
== startup log (first 25 lines of the new pod)
2026/11/10 16:12:52 [INFO] felhom-hub 0.180.0 starting
2026/11/10 16:12:52 [INFO] Database opened at /data/hub.db
2026/11/10 16:12:52 [INFO] Default controller-version floor: 0.120.0
2026/11/10 16:12:52 [INFO] Template fetcher started (every 1h)
2026/11/10 16:12:52 [DEBUG] Template fetched (5881 bytes)
2026/11/10 16:12:53 [INFO] Asset seed: all 263 files up-to-date
2026/11/10 16:12:53 [INFO] Asset manifest built: 288 files
2026/11/10 16:12:53 [INFO] App-email relay enabled (limit 30/min/customer, From domains [felhom.eu])
2026/11/10 16:12:53 [INFO] Gitea artifact browser enabled (Day-0 version dropdowns) via http://gitea.gitea-system.svc.cluster.local:3000
2026/11/10 16:12:53 [INFO] Offsite provisioning enabled (pool_box=611714, location=fsn1)
2026/11/10 16:12:53 [INFO] Offsite pool-box checker initialized: box=611714 fill warn=80% crit=90%, oversub warn=2.00x, unreachable after 3 consecutive failed reads, refresh 15m0s
2026/11/10 16:12:54 [INFO] osupdates: approval rule = every ring-0 box healthy for 24h0m0s and 1 night run(s)
2026/11/10 16:12:54 [INFO] osupdates: the Docker engine set is approved only by the operator, after 2 healthy ring-0 night(s)
2026/11/10 16:12:54 [INFO] osupdates: alarms = no OS leg 168h0m0s, reboot needed 336h0m0s, ring 0 stalled 168h0m0s, not covered 336h0m0s
2026/11/10 16:12:54 [INFO] Off-site key registrar enabled; daily key check at 07:10 Budapest
2026/11/10 16:12:54 [INFO] access log: /data/access-log, one file per UTC day, kept 90 days
2026/11/10 16:12:54 [INFO] Registry version checker started (every 6h)
2026/11/10 16:12:54 [INFO] offsite-key-audit: next run at 2026-11-11 07:10 CET (in 14h57m6s)
2026/11/10 16:12:54 [INFO] WG peer-sync enabled (endpoint 167.233.158.164:22, user felhom-peersync)
2026/11/10 16:12:54 [INFO] PBS DR tenantsync enabled (endpoint 167.233.158.164:22, user felhom-peersync; WG-registration auto-provision hook armed)
2026/11/10 16:12:54 [INFO] PBS-DR box checker initialized: fill warn=80% crit=90%, unreachable after 3 consecutive failed reads, refresh 15m0s
2026/11/10 16:12:54 [INFO] agent-plane poke enabled (endpoint 167.233.158.164:22, user felhom-peersync; web + api admin seams armed)
2026/11/10 16:12:54 [INFO] PBS-DR self-heal reconciler started (interval 5m)
2026/11/10 16:12:54 [INFO] offsite credential self-heal reconciler started (interval 5m, debounce 2 reports)
2026/11/10 16:12:54 [DEBUG] Registry version check: latest = 0.351.0
@@ -0,0 +1 @@
15:15:35Z felhom.eu d0a54c2f8e: run 2617 gates #1419 status success
@@ -0,0 +1,8 @@
# what the running hub lets CC's token read about a household's saved alarm list — GET only, 2026-11-10T15:32:36Z
2026/11/10 16:32:12 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers → 403
2026/11/10 16:32:16 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers/tester-1 → 403
2026/11/10 16:32:17 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers/demo-hp → 403
2026/11/10 16:32:17 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /events → 403
2026/11/10 16:32:17 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /notifications → 403
# result: every page that shows a household's list is 'session only — never by a token' (by design, 05 §17).
# no copy of the hub DB was taken: the live file holds mail addresses and old token bytes (05 §16.2).
@@ -0,0 +1,37 @@
== hub watch, 30 minutes after the deploy (pod started 2026-11-10T15:12:49Z); read 2026-11-10T15:43:01Z
-- argocd app
sync=Synced health=Healthy revision=d0a54c2f8e308ae959900fd8068bc2b434d28664
-- pod
hub-658c847c65-d8zbk phase=Running restarts=0 started=2026-11-10T15:12:52Z image=gitea.dooplex.hu/admin/felhom-hub:0.180.0
-- hub log since the start: lines / [ERROR] lines
260
0
-- [WARN] lines by kind (the log prints CET)
3 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /cu
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /ev
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /no
2 [WARN] slow statement: Nms (over 1s): DELETE …
8 [WARN] slow statement: Nms (over 1s): SELECT …
1 [WARN] slow statement: Nms (over 1s): UPDATE …
-- events from boxes, by box / type / severity
1 demo-felhom controller_started info
1 demo-felhom controller_updated info
1 demo-hp controller_started info
1 demo-hp controller_updated info
1 tester-1 controller_started info
1 tester-1 controller_updated info
-- alarm checks and mails
1 [INFO] osupdates: alarm check done in N, 0 alarm(s) sent []
-- lines that name a mail send
0
-- positive signs (the hub works, it is not only quiet): reports received in the window, by box
6 demo-felhom
6 demo-hp
6 tester-1
-- host reports, by host
3 demo-felhom-8363b5
3 demo-hp-bb76ea
5 tester-1-d70be4
-- startup line
2026/11/10 16:12:52 [INFO] felhom-hub 0.180.0 starting
-- second channel (not the hub): the operator's mailbox, read by the mail connector — see ../controller/12-watch.txt