Space fixes released: agent 0.173.0 vouched, controller 0.352.0 global floor, hub 0.180.0 live; evidence, rows, status, report
gates / gates (push) Successful in 2m1s
gates / gates (push) Successful in 2m1s
The close-out of the attended release run of 2026-11-10 (afternoon). - Evidence: documentation/audits/release-space-fixes-2026-11-10/ (agent, controller, hub) and the hub row after the fallback-kernel test. - Rows: dated lines on R-1368, R-1371, R-1301, R-1353. 236 open before and after; opened 0, closed 0. - Capability map: dated notes under C13 and G5; no status changed. - disk-full runbook and 16 §14: the self-update folder is PROVEN as a product path on three test boxes. - STATUS: new "Updated" paragraph; the old one moved to STATUS-HISTORY. - REPORT-release-space-fixes-2026-11-10.md: wrong claims first, one deviation (the hub deploy ran before CI for the manifest-bump commit was read), what stays not proven. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
1 parent
d0a54c2f8e
commit
bcc73b428d
40 files changed
+1037
-59
No files matched your search
@@ -278,7 +278,7 @@ The alarms, by short name:
|
||||
| H9 | old kernels | each approved kernel | none today. Ruled: keep the running one, the one before it and the newest (decision 286, `11-os-updates.md` §5.11.1). Proven by hand, not built | nobody today | host disk; later `os_update_stale` |
|
||||
| H10 | the EFI partition | a boot-loader update | 1.07 GB, 1 % used | none needed | none (R-1371) |
|
||||
| H11 | the host package cache | each OS update | emptied in each run | the update wrapper | host disk |
|
||||
| H12 | the agent's self-update folder | each agent update | none on a released agent; one binary after the next agent release (in source, R-1368) | the agent, after that release | host disk |
|
||||
| H12 | the agent's self-update folder | each agent update | the running version's binary, and the file of a failed update until the next good one (about 16 MB each; measured 2026-11-10 on three boxes, R-1368) | the agent, at the end of each good update | host disk |
|
||||
| H13 | old agent binaries | `.prev`: one file. `.bak-*`: each re-run of the installer | none for `.bak-*` (R-1368) | nobody | host disk |
|
||||
| H14 | wrapper state and previous config bundles | each bundle | the last 3 | the wrapper | host disk |
|
||||
| H15 | host `/tmp` and `/var/tmp` | nothing that lasts | `/tmp` is in memory: half of memory | systemd, after 10 and 30 days | none for `/tmp` (R-1371) |
|
||||
@@ -408,7 +408,7 @@ below 200 GB — refuse, or warn (R-1370). demo-hp (128 GB) stays as a test box.
|
||||
| Nobody watches the guest's root disk (G1, G13, G16). It is 4–5 % used today; app definitions lie on it | R-1371 |
|
||||
| No alarm for the EFI partition (H10) or for the places in memory (H15, G15, G17) | R-1371 |
|
||||
| A drive in the backup role reports as `usb` or `local-dir`. So the household gets the `storage_fill` mail for a drive it must not clean by hand | R-1371 |
|
||||
| The hub's first list for a household still holds `storage_fill_critical`. It is replaced when the box sends its own list. Not measured on the running hub in the 2026-11-10 run | R-1371 |
|
||||
| The hub's first list for a household still holds `storage_fill_critical`. It is replaced when the box sends its own list. **Not measured on the running hub:** the release run of 2026-11-10 deployed the hub that blocks the host-disk alarm at send time, but the pages that show a household's list refuse a token („session only"), and no test alarm was made. The proof is the tests and the deployed commit (`audits/release-space-fixes-2026-11-10/hub/`) | R-1371 |
|
||||
| Helper images carry no version; the guest's journal has no stated cap; off-site restores have no limit | R-1372 |
|
||||
| Old kernels: the keep rule is ruled and proven by hand, not built | `11-os-updates.md` §5.11.1 |
|
||||
| The parts of the host disk with no limit and no cleaner (H4, H5, H6, H13) | R-1087, R-1368 |
|
||||
|
||||
@@ -229,6 +229,8 @@
|
||||
|
||||
**Gap / roadmap (as written):** **Nothing warned before this.** The only prior signal was the healthcheck's generic `health_degraded` at 90%, for REGISTERED STORAGE PATHS ONLY — it never looked at the docker area or the system-data area, never gave a free-byte figure and never named a drive. **The two event types already existed with NO PRODUCER** (`disk_warning`/`disk_critical`: allowlisted, copy'd, in `DefaultEnabledEvents`, checkbox'd) — the **sixth** *built-but-never-wired* instance here; this ships their producer rather than a seventh near-duplicate type. **Two threshold terms, whichever trips first, and the live proof vindicated the design:** the critical crossing fired on the FREE-BYTE term (1.7 GB) at only **91%** used — a percentage-only rule would have missed it. The hub's generic `customerMessages` entries were REMOVED, because `FormatCustomerEmail` prefers the entry over the message and would discard the label and figures. **Known gap → R-177:** there is no operator-triggerable run-now path; the check is daily 03:30 + once at startup, so confirming a cleared warning on a support call needs a controller restart or a wait
|
||||
|
||||
**Noted 2026-11-10 (release run):** on a box with one data volume the watch counted that volume twice. Seen on scratch guest 9202 with one real fill: the old controller logged TWO warnings for it, under two names (`audits/release-space-fixes-2026-11-10/controller/03-crossing-on-0.350.0.txt`); the fixed controller logged ONE warning, and cleared it without a message when the file was removed (`04-rc1-on-9202.txt`). On the three test boxes the watch now checks 2 filesystems, was 3 (`09-fs-identity-after.txt`). 9202 has no hub, so no mail was sent; one mail for one fill is proven by the box's own record, not by a mailbox. Still open: the guest's root disk has no watcher (R-1371).
|
||||
|
||||
---
|
||||
|
||||
<a id="c14"></a>
|
||||
|
||||
@@ -91,6 +91,8 @@
|
||||
|
||||
**Gap / roadmap (as written):** Remote-customer proof pending → R-1
|
||||
|
||||
**Noted 2026-11-10 (release run):** the agent now removes old downloaded binaries after a good update: 49, 48 and 33 files became 1 on demo-hp, the N100 and Tester 1. A rollback after that clean-up was proven on Tester 1: a signed update to the older version, one crash 9 seconds into the 60-second dwell, and the wrapper put the previous binary back in the same second (`audits/release-space-fixes-2026-11-10/agent/07-rollback-proof.txt`). This is one crash, not a crash loop.
|
||||
|
||||
---
|
||||
|
||||
<a id="g6"></a>
|
||||
|
||||
+1
@@ -0,0 +1 @@
|
||||
MATCH 2026-11-10T14:58:14Z: demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 48% 42% running (connected) 71% local-lvm
|
||||
@@ -0,0 +1,61 @@
|
||||
# Evidence — the release of the three Space fixes (2026-11-10, attended release run)
|
||||
|
||||
**What this is.** The files that prove what the release run did on the boxes and on the hub. The report is
|
||||
`REPORT-release-space-fixes-2026-11-10.md` at the repository root. The fallback-kernel test of the same run has its own
|
||||
folder: `audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`.
|
||||
|
||||
**Method.** No browser. CC read the boxes over SSH, read the hub's log from its pod, and called the hub's own
|
||||
addresses with CC's token (the same addresses the hub's pages call). Times are UTC, except lines copied from a host's
|
||||
journal or the hub's log, which print CET (UTC+1). The N100's host clock is about 32 seconds behind (R-1301); its
|
||||
lines read early by that much.
|
||||
|
||||
**Secrets.** The files were scanned before the commit. They hold no token, no password and no key. The signed job
|
||||
envelopes are not here.
|
||||
|
||||
## `agent/` — the agent release (R-1368 part 2)
|
||||
|
||||
| File | What it shows |
|
||||
|---|---|
|
||||
| `01-folder-before.txt` | The self-update folder on the three test boxes before: 49 / 48 / 33 entries |
|
||||
| `03-sign-upload.txt` | One signed `agent_update` per box, uploaded to the hub |
|
||||
| `04-delivery.txt` | Each box took the job, committed, and removed the old files (`removed=49`, `48`, `33`) |
|
||||
| `05-after-demo-hp.txt`, `05-after-demo-felhom.txt`, `05-after-tester1.txt` | The folder after: 1 entry each; no pending marker; the unit active |
|
||||
| `06-rollback-sign.txt` | The signed job to the older version, for the rollback proof on Tester 1 |
|
||||
| `07-rollback-proof.txt` | One crash 9 seconds into the dwell; the wrapper put the released version back in the same second |
|
||||
| `08-after-rollback-tester1.txt` | After the rollback: the released version runs; 2 files in the folder (the failed one stays, by design) |
|
||||
| `12-vouch-readback.txt` | The operator's vouch, read from the hub's log |
|
||||
| `13-n100-clock.txt` | The N100's host clock against DooPlex and demo-hp; `chrony` failed at boot (R-1301) |
|
||||
|
||||
## `controller/` — the controller release (R-1371 part 2)
|
||||
|
||||
| File | What it shows |
|
||||
|---|---|
|
||||
| `00-fill-before.txt` | The fill watch on the boxes before the release: 3 filesystems checked, two of them the same one |
|
||||
| `03-crossing-on-0.350.0.txt` | **Red on a box.** Scratch guest 9202, the version before, one test file: TWO warnings |
|
||||
| `04-rc1-on-9202.txt` | **Green on a box.** 9202 on the release candidate, the same fill: ONE warning; cleared silently after the file was removed by its exact name |
|
||||
| `05-boxwalk-before.txt` | The three test boxes before the per-box floors |
|
||||
| `07-per-box-floors.txt` | The three per-box floors, each `303 … flash=floor_set` |
|
||||
| `08-boxwalk-after.txt` | The three boxes on the release: healthy, containers 10 / 26 / 9, none unhealthy |
|
||||
| `09-fs-identity-after.txt` | The watch's target list on each box after: 2 filesystems, one target each |
|
||||
| `10-https-only.txt` | `scripts/https_only_check.py`, exit 0 |
|
||||
| `11-global-floor.txt` | The global floor, in its own save; the hub's read-back |
|
||||
| `12-watch.txt` | 30 minutes after the global floor: every box on the release, no new alarm; the mailbox as a second channel |
|
||||
| `13-per-box-floors-cleared.txt` | The three per-box floors cleared; the boxes stay on the release by the global floor |
|
||||
|
||||
## `hub/` — the hub release (R-1371 part 3, D68)
|
||||
|
||||
| File | What it shows |
|
||||
|---|---|
|
||||
| `02-d68-tests.txt` | The `TestD68_*` tests, green, on the released commit |
|
||||
| `03-ci-heading.txt` | CI for the commit the image was built from, read by commit hash |
|
||||
| `07-deploy.txt` | Before and after the deploy: only the hub's Deployment was out of sync; Synced/Healthy; the image id |
|
||||
| `08-ci-bump.txt` | CI for the manifest-bump commit. **Read 3 minutes after the deploy, not before** (a deviation; see the report) |
|
||||
| `09-first-list-read.txt` | The pages that show a household's alarm list refuse a token (403, by design). So the open point of `16` §14.4 is not measured on the running hub |
|
||||
| `10-watch.txt` | 30 minutes after the deploy: Synced/Healthy, no restart, no error line, 6 reports from each box, no alarm sent |
|
||||
|
||||
## Not proven by this folder
|
||||
|
||||
- That a household which saved its own alarm list before no longer gets the host-disk alarm. The proof is the tests
|
||||
and the deployed commit. No test alarm was made, by the brief.
|
||||
- A real fill on a household's box. A fill at the critical line (95 %) on any box.
|
||||
- A removal by hand in the agent's self-update folder; the `.bak-*` copies of the agent (R-1368 part 4).
|
||||
@@ -0,0 +1,27 @@
|
||||
== demo-hp 2026-11-10T14:34:22Z
|
||||
demo-hp
|
||||
felhom-agent 0.172.1
|
||||
entries=49 staged=49 size_kb=722596
|
||||
felhom-agent-0.171.0
|
||||
felhom-agent-0.172.0
|
||||
felhom-agent-0.172.1
|
||||
1
|
||||
-rwxr-xr-x 1 root root 16393888 Nov 3 08:56 /usr/local/bin/felhom-agent.prev
|
||||
== felhom-pve 2026-11-10T14:34:22Z
|
||||
demo-felhom
|
||||
felhom-agent 0.172.1
|
||||
entries=48 staged=48 size_kb=706968
|
||||
felhom-agent-0.171.0
|
||||
felhom-agent-0.172.0
|
||||
felhom-agent-0.172.1
|
||||
11
|
||||
-rwxr-xr-x 1 root root 16393888 Nov 3 08:49 /usr/local/bin/felhom-agent.prev
|
||||
== root@192.168.0.154 2026-11-10T14:34:22Z
|
||||
felhom
|
||||
felhom-agent 0.172.1
|
||||
entries=33 staged=33 size_kb=497244
|
||||
felhom-agent-0.171.0
|
||||
felhom-agent-0.172.0
|
||||
felhom-agent-0.172.1
|
||||
0
|
||||
-rwxr-xr-x 1 root root 16393888 Nov 3 08:57 /usr/local/bin/felhom-agent.prev
|
||||
@@ -0,0 +1,12 @@
|
||||
== tester-1-d70be4 2026-11-10T14:46:49Z
|
||||
signed: op=agent_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=fef383cc866be77372a2c7421c0f70ac expires=2026-11-10T15:16:49Z
|
||||
uploaded signed op to the hub jobs queue
|
||||
rc=0
|
||||
== demo-hp-bb76ea 2026-11-10T14:46:49Z
|
||||
signed: op=agent_update host=demo-hp-bb76ea guest="" key_id=felhom-op-1 nonce=c230909e8a25f4e33c0ac9c6e5b3b1fe expires=2026-11-10T15:16:49Z
|
||||
uploaded signed op to the hub jobs queue
|
||||
rc=0
|
||||
== demo-felhom-8363b5 2026-11-10T14:46:49Z
|
||||
signed: op=agent_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=481078428450ad9576174647ba6db78b expires=2026-11-10T15:16:50Z
|
||||
uploaded signed op to the hub jobs queue
|
||||
rc=0
|
||||
@@ -0,0 +1,5 @@
|
||||
signed ops uploaded 2026-11-10T14:46:49Z
|
||||
+330s felhom-pve: version 0.173.0, committed
|
||||
+753s demo-hp: version 0.173.0, committed
|
||||
+815s root@192.168.0.154: version 0.173.0, committed
|
||||
AGWATCH RESULT=ok all three committed at 2026-11-10T15:00:37Z
|
||||
@@ -0,0 +1,31 @@
|
||||
host=demo-felhom at=2026-11-10T14:53:03Z agent=felhom-agent 0.173.0
|
||||
entries=1 staged=1 size_kb=16028
|
||||
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-10T15:51:50 .
|
||||
drwxr-x--- 9 felhom-agent felhom-agent 4096 2026-11-10T10:22:07 ..
|
||||
-rwxr-xr-x 1 felhom-agent felhom-agent 16407361 2026-11-10T15:50:45 felhom-agent-0.173.0
|
||||
pending.json: absent
|
||||
-rwxr-xr-x 1 root root 16407361 2026-11-10T15:50:46 /usr/local/bin/felhom-agent
|
||||
-rwxr-xr-x 1 root root 16393888 2026-11-07T23:04:48 /usr/local/bin/felhom-agent.prev
|
||||
prev version: felhom-agent 0.172.1
|
||||
bak files: 11
|
||||
-- journal (host local time)
|
||||
2026-11-10T15:50:46+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T15:50:46.007+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.173.0 version=0.173.0
|
||||
2026-11-10T15:50:46+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T15:50:46.327+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.173.0 report="{\"agent_update\": {\"sha256\": \"752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea\", \"version\": \"
|
||||
2026-11-10T15:50:46+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T15:50:46.327+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=77bd95f4bec54d8e op=agent_update
|
||||
2026-11-10T15:50:48+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:48.455+01:00 level=INFO msg="felhom-agent daemon starting" version=0.173.0 host_id=demo-felhom-8363b5 hub_url=https://hub.felhom.eu interval_s=900
|
||||
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.046+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
|
||||
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.046+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
|
||||
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.047+01:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
|
||||
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.938+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
|
||||
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.938+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
|
||||
2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
|
||||
2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.173.0 prev=0.172.1 dwell=1m0s
|
||||
2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
|
||||
2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=20
|
||||
2026-11-10T15:51:50+01:00 demo-felhom sudo[3986865]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-selfupdate-guarded commit
|
||||
2026-11-10T15:51:50+01:00 demo-felhom felhom-selfupdate-guarded[3986874]: committed (pending cleared, .prev retained)
|
||||
2026-11-10T15:51:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:50.032+01:00 level=WARN msg="selfupdate: update committed" version=0.173.0 wrapper=""
|
||||
2026-11-10T15:51:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:50.128+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.173.0 removed=48
|
||||
2026-11-10T15:51:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:50.128+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=48 kept=felhom-agent-0.173.0
|
||||
active
|
||||
NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:50:48 CET
|
||||
@@ -0,0 +1,38 @@
|
||||
host=demo-hp at=2026-11-10T15:00:34Z agent=felhom-agent 0.173.0
|
||||
entries=1 staged=1 size_kb=16028
|
||||
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-10T15:59:19 .
|
||||
drwxr-x--- 10 felhom-agent felhom-agent 4096 2026-11-10T11:36:41 ..
|
||||
-rwxr-xr-x 1 felhom-agent felhom-agent 16407361 2026-11-10T15:58:14 felhom-agent-0.173.0
|
||||
pending.json: absent
|
||||
-rwxr-xr-x 1 root root 16407361 2026-11-10T15:58:14 /usr/local/bin/felhom-agent
|
||||
-rwxr-xr-x 1 root root 16393888 2026-11-07T23:11:58 /usr/local/bin/felhom-agent.prev
|
||||
prev version: felhom-agent 0.172.1
|
||||
bak files: 1
|
||||
-- journal (host local time)
|
||||
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.000+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
|
||||
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.799+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
|
||||
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.799+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
|
||||
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.858+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
|
||||
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.859+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
|
||||
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.859+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=95
|
||||
2026-11-10T15:58:14+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:58:14.583+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.173.0 version=0.173.0
|
||||
2026-11-10T15:58:14+01:00 demo-hp felhom-os-apply[66717]: os-apply: AGENT-UPDATE signed by the operator: version=0.173.0 sha=752b2727fb604341 — handing the root copy to the A/B wrapper
|
||||
2026-11-10T15:58:15+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:58:15.231+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.173.0 report="{\"agent_update\": {\"sha256\": \"752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea\", \"version\": \"0.17
|
||||
2026-11-10T15:58:15+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:58:15.231+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=897285dcaecd3e9d op=agent_update
|
||||
2026-11-10T15:58:17+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:17.331+01:00 level=INFO msg="felhom-agent daemon starting" version=0.173.0 host_id=demo-hp-bb76ea hub_url=https://hub.felhom.eu interval_s=900
|
||||
2026-11-10T15:58:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:18.424+01:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
|
||||
2026-11-10T15:58:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:18.424+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
|
||||
2026-11-10T15:58:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:18.424+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
|
||||
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.207+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
|
||||
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.207+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
|
||||
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.388+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
|
||||
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.389+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.173.0 prev=0.172.1 dwell=1m0s
|
||||
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.389+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
|
||||
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.389+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=95
|
||||
2026-11-10T15:59:19+01:00 demo-hp sudo[72223]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-selfupdate-guarded commit
|
||||
2026-11-10T15:59:19+01:00 demo-hp felhom-selfupdate-guarded[72232]: committed (pending cleared, .prev retained)
|
||||
2026-11-10T15:59:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:19.427+01:00 level=WARN msg="selfupdate: update committed" version=0.173.0 wrapper=""
|
||||
2026-11-10T15:59:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:19.592+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.173.0 removed=49
|
||||
2026-11-10T15:59:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:19.592+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=49 kept=felhom-agent-0.173.0
|
||||
active
|
||||
NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:58:17 CET
|
||||
@@ -0,0 +1,38 @@
|
||||
host=felhom at=2026-11-10T15:00:35Z agent=felhom-agent 0.173.0
|
||||
entries=1 staged=1 size_kb=16028
|
||||
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-10T16:00:19 .
|
||||
drwxr-x--- 10 felhom-agent felhom-agent 4096 2026-11-10T13:35:18 ..
|
||||
-rwxr-xr-x 1 felhom-agent felhom-agent 16407361 2026-11-10T15:59:14 felhom-agent-0.173.0
|
||||
pending.json: absent
|
||||
-rwxr-xr-x 1 root root 16407361 2026-11-10T15:59:15 /usr/local/bin/felhom-agent
|
||||
-rwxr-xr-x 1 root root 16393888 2026-11-07T23:12:45 /usr/local/bin/felhom-agent.prev
|
||||
prev version: felhom-agent 0.172.1
|
||||
bak files: 0
|
||||
-- journal (host local time)
|
||||
2026-11-10T15:43:45+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:45.441+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
|
||||
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.623+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
|
||||
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.631+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
|
||||
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.728+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
|
||||
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.728+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
|
||||
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.732+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=213
|
||||
2026-11-10T15:59:14+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:59:14.962+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.173.0 version=0.173.0
|
||||
2026-11-10T15:59:15+01:00 felhom felhom-os-apply[28155]: os-apply: AGENT-UPDATE signed by the operator: version=0.173.0 sha=752b2727fb604341 — handing the root copy to the A/B wrapper
|
||||
2026-11-10T15:59:15+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:59:15.660+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.173.0 report="{\"agent_update\": {\"sha256\": \"752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea\", \"version\": \"0.173
|
||||
2026-11-10T15:59:15+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:59:15.660+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=50cce53e2c398bee op=agent_update
|
||||
2026-11-10T15:59:17+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:17.757+01:00 level=INFO msg="felhom-agent daemon starting" version=0.173.0 host_id=tester-1-d70be4 hub_url=https://hub.felhom.eu interval_s=900
|
||||
2026-11-10T15:59:18+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:18.689+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
|
||||
2026-11-10T15:59:18+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:18.689+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
|
||||
2026-11-10T15:59:18+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:18.689+01:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
|
||||
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.231+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
|
||||
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.232+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
|
||||
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.338+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
|
||||
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.339+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
|
||||
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.339+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=213
|
||||
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.339+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.173.0 prev=0.172.1 dwell=1m0s
|
||||
2026-11-10T16:00:19+01:00 felhom sudo[30337]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-selfupdate-guarded commit
|
||||
2026-11-10T16:00:19+01:00 felhom felhom-selfupdate-guarded[30345]: committed (pending cleared, .prev retained)
|
||||
2026-11-10T16:00:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:00:19.390+01:00 level=WARN msg="selfupdate: update committed" version=0.173.0 wrapper=""
|
||||
2026-11-10T16:00:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:00:19.607+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.173.0 removed=33
|
||||
2026-11-10T16:00:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:00:19.607+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=33 kept=felhom-agent-0.173.0
|
||||
active
|
||||
NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:59:17 CET
|
||||
@@ -0,0 +1,5 @@
|
||||
== rollback proof on Tester 1: a signed agent_update to the OLD version 0.172.1, then one crash inside the 60 s dwell
|
||||
2026-11-10T15:01:19Z
|
||||
signed: op=agent_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=3039f82fc2b2103065da50c4015a8d11 expires=2026-11-10T15:31:19Z
|
||||
uploaded signed op to the hub jobs queue
|
||||
SIGN RC=0
|
||||
@@ -0,0 +1,19 @@
|
||||
watch start 2026-11-10T15:01:24Z: agent=felhom-agent 0.173.0 pending=absent
|
||||
+788s pending marker present, binary on disk = 0.172.1, MainPID=28267 exe=/usr/local/bin/felhom-agent (deleted)
|
||||
pending.json: {"old_version":"0.173.0","new_version":"0.172.1","sha256":"505a0594a2fb117f359e329a0b7ae2c7a6902b74c51f52c1883145f3c2ac72f8","applied_at":"2026-11-10T16:14:31+01:00"}
|
||||
folder now: felhom-agent-0.172.1 felhom-agent-0.173.0 pending.json
|
||||
os-apply running: 0; dpkg/apt running: 0
|
||||
+796s 2026-11-10T15:14:40Z KILL: systemctl kill --kill-whom=main -s SEGV felhom-agent (MainPID=51118, pending=PRESENT)
|
||||
kill rc=0
|
||||
== after +821s 2026-11-10T15:15:05Z
|
||||
agent=felhom-agent 0.173.0 active=active pending=absent
|
||||
running exe version: felhom-agent 0.173.0
|
||||
folder: entries=2 size_kb=32040: felhom-agent-0.172.1 felhom-agent-0.173.0
|
||||
sha256 bin / prev:
|
||||
752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea /usr/local/bin/felhom-agent
|
||||
752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea /usr/local/bin/felhom-agent.prev
|
||||
-- journal
|
||||
2026-11-10T16:14:40+01:00 felhom felhom-selfupdate-guarded[51930]: felhom-selfupdate-guarded: rolled back to previous binary and restarted felhom-agent.service
|
||||
failed units: 0
|
||||
ROLLBACK-PROOF DONE
|
||||
PROOF RC=0
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
host=felhom at=2026-11-10T15:15:56Z agent=felhom-agent 0.173.0
|
||||
entries=2 staged=2 size_kb=32040
|
||||
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-10T16:14:40 .
|
||||
drwxr-x--- 10 felhom-agent felhom-agent 4096 2026-11-10T13:35:18 ..
|
||||
-rwxr-xr-x 1 felhom-agent felhom-agent 16393888 2026-11-10T16:14:30 felhom-agent-0.172.1
|
||||
-rwxr-xr-x 1 felhom-agent felhom-agent 16407361 2026-11-10T15:59:14 felhom-agent-0.173.0
|
||||
pending.json: absent
|
||||
-rwxr-xr-x 1 root root 16407361 2026-11-10T16:14:40 /usr/local/bin/felhom-agent
|
||||
-rwxr-xr-x 1 root root 16407361 2026-11-10T15:59:15 /usr/local/bin/felhom-agent.prev
|
||||
prev version: felhom-agent 0.173.0
|
||||
bak files: 0
|
||||
-- journal (host local time)
|
||||
2026-11-10T16:14:31+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:14:31.582+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.172.1 report="{\"agent_update\": {\"sha256\": \"505a0594a2fb117f359e329a0b7ae2c7a6902b74c51f52c1883145f3c2ac72f8\", \"version\": \"0.17
|
||||
2026-11-10T16:14:31+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:14:31.582+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=f6848443a231739a op=agent_update
|
||||
2026-11-10T16:14:33+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:33.722+01:00 level=INFO msg="felhom-agent daemon starting" version=0.172.1 host_id=tester-1-d70be4 hub_url=https://hub.felhom.eu interval_s=900
|
||||
2026-11-10T16:14:34+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:34.663+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
|
||||
2026-11-10T16:14:34+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:34.663+01:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
|
||||
2026-11-10T16:14:34+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:34.663+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
|
||||
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.232+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
|
||||
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.233+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
|
||||
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.319+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
|
||||
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.320+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
|
||||
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.321+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.172.1 prev=0.173.0 dwell=1m0s
|
||||
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.321+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=213
|
||||
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate.NewManager.func2({0xe5d130, 0x1e342d05aac0}, 0xef4820?)
|
||||
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate/commit.go:54 +0xb4 fp=0x1e342d2d1dc8 sp=0x1e342d2d1d38 pc=0x880e14
|
||||
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate.(*Manager).MaybeCommit(0x1e342d13b2c0, {0xe5d130, 0x1e342d05aac0})
|
||||
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate/commit.go:111 +0x3aa fp=0x1e342d2d1fb8 sp=0x1e342d2d1dc8 pc=0x87e0aa
|
||||
2026-11-10T16:14:40+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:40.445+01:00 level=INFO msg="felhom-agent daemon starting" version=0.173.0 host_id=tester-1-d70be4 hub_url=https://hub.felhom.eu interval_s=900
|
||||
2026-11-10T16:14:41+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:41.489+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
|
||||
2026-11-10T16:14:41+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:41.489+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
|
||||
2026-11-10T16:14:41+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:41.489+01:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
|
||||
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.072+01:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
|
||||
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.072+01:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
|
||||
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.172+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
|
||||
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.173+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
|
||||
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.173+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update" slot_minute=213
|
||||
active
|
||||
NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 16:14:40 CET
|
||||
@@ -0,0 +1,4 @@
|
||||
2026/11/10 16:29:16 [INFO] artifact vouch: golden 0.350.0 unchanged (behind the fleet's 0.352.0 — the weekly bake) — the R-120 gate does not apply to an unchanged golden; saving agent=0.173.0 min_agent="0.158.1"
|
||||
2026/11/10 16:29:16 [INFO] Artifact manifest set: agent=0.173.0 golden=0.350.0 min_agent="0.158.1" wrapper_sha=false bundle_sha="51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1"
|
||||
2026/11/10 16:29:16 [INFO] artifact vouch timing (R-1346): registry probes 101ms, sha lookups 29ms, bundle sha 13ms, fleet check + save 87ms, total 232ms
|
||||
read 2026-11-10T15:29:45Z (the hub log prints CET)
|
||||
@@ -0,0 +1,16 @@
|
||||
== N100 host clock against DooPlex and demo-hp (read-only), for R-1301
|
||||
dooplex 2026-11-10T15:38:07.145528284Z
|
||||
n100 2026-11-10T15:37:34.778614952Z
|
||||
demo-hp 2026-11-10T15:38:07.999970091Z
|
||||
dooplex 2026-11-10T15:38:08.002419987Z
|
||||
-- n100 timedatectl
|
||||
NTPSynchronized=no
|
||||
TimeUSec=Tue 2026-11-10 16:37:35 CET
|
||||
failed
|
||||
Nov 08 05:35:23 demo-felhom chronyd[690]: Loaded seccomp filter (level 1)
|
||||
Nov 08 05:35:23 demo-felhom chronyd[690]: Fatal error : Could not send notification to $NOTIFY_SOCKET
|
||||
Nov 08 05:35:23 demo-felhom chronyd[690]: Fatal error : Could not send notification to $NOTIFY_SOCKET
|
||||
Nov 08 05:35:23 demo-felhom systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE
|
||||
Nov 08 05:35:23 demo-felhom systemd[1]: chrony.service: Failed with result 'exit-code'.
|
||||
Nov 08 05:35:23 demo-felhom systemd[1]: Failed to start chrony.service - chrony, an NTP client/server.
|
||||
2026-11-08 05:35:18
|
||||
@@ -0,0 +1,8 @@
|
||||
Fill watch BEFORE the release (controller 0.351.0 on the three test boxes, 0.350.0 on scratch guest 9202), read 2026-11-10 14:24–14:51 UTC.
|
||||
The line counts TARGETS. On a one-volume box the two built-in targets "/" and /mnt/sys_drive are one filesystem
|
||||
(felhom.eu documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt).
|
||||
|
||||
demo-felhom (N100) guest 9201, 0.351.0: [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok (14:24:05, 14:34:05)
|
||||
demo-hp guest 9201, 0.351.0: 2026/11/10 14:45:00 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
demo-hp scratch guest 9202, 0.350.0: 2026/11/10 14:44:49 [INFO] [fillwatch] checked 11 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
Tester 1 guest 9201, 0.351.0: 2026/11/10 14:45:41 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
+41
@@ -0,0 +1,41 @@
|
||||
== The warning band on scratch guest 9202, OLD controller 0.350.0 (the 'before': how many warnings does one fill send?)
|
||||
== 2026-11-10T14:53:04Z guest config and the state before
|
||||
hostname: demo-hp-scratch
|
||||
mp0: nvme-scratch:9202/vm-9202-disk-3.raw,mp=/var/lib/felhom,backup=1,size=70G
|
||||
mp8: /mnt/hdd_1/scratch-drives/scratch_hdd,mp=/mnt/felhom-drives/scratch_hdd
|
||||
mp9: /var/lib/felhom-agent/guests/9202/bootstrap,mp=/etc/felhom-bootstrap,ro=1
|
||||
rootfs: nvme-scratch:9202/vm-9202-disk-2.raw,size=32G
|
||||
/dev/loop1 73391005696 13985423360 55630708736 21% /var/lib/felhom
|
||||
total 32
|
||||
drwxr-xr-x 5 root root 4096 Oct 20 08:44 .
|
||||
drwxr-xr-x 24 root root 4096 Oct 4 07:13 ..
|
||||
drwx--x--- 12 root root 4096 Nov 10 14:43 docker
|
||||
drwx------ 2 nobody nogroup 16384 Oct 4 07:11 lost+found
|
||||
drwxr-xr-x 4 root root 4096 Oct 26 19:52 sys_drive
|
||||
2026/11/10 14:44:49 fillwatch.go:284: [INFO] [fillwatch] checked 11 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
|
||||
== 2026-11-10T14:53:15Z the test file: one file, exact name, 45902462976 bytes (42.75 GiB), made with fallocate (no data written); target 86 % used, about 9 GiB still free; the ext4 reserve is not touched (the file is smaller than 'avail')
|
||||
df: /mnt/nvme-1tb: No such file or directory
|
||||
fallocate_rc=0
|
||||
-rw-r--r-- 1 root root 45902462976 Nov 10 14:53 /var/lib/felhom/FILLTEST-2026-11-10-r1371.bin
|
||||
/dev/loop1 73391005696 59887894528 9728237568 87% /var/lib/felhom
|
||||
/dev/loop1 69G 56G 9.1G 87% /var/lib/felhom
|
||||
df: /mnt/nvme-1tb: No such file or directory
|
||||
|
||||
== 2026-11-10T14:54:01Z the controller's own lines (0.350.0): ONE fill, TWO warnings
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.350.0 Up 10 minutes (healthy)
|
||||
2026/11/10 14:53:19 fillwatch.go:262: [WARN] [fillwatch] / ("Alkalmazások területe"): ok → warning — 86% used, 9.1 GB free of 68.4 GB; notifying the customer
|
||||
2026/11/10 14:53:19 notifier.go:1471: [WARN] notifier disabled (no hub configured): DROPPED event disk_warning (severity warning) — further disk_warning events are logged at DEBUG only
|
||||
2026/11/10 14:53:19 fillwatch.go:262: [WARN] [fillwatch] /mnt/sys_drive ("Rendszer- és mentési terület"): ok → warning — 86% used, 9.1 GB free of 68.4 GB; notifying the customer
|
||||
2026/11/10 14:53:19 notifier.go:1475: [DEBUG] notifier disabled: dropped event disk_warning (severity warning)
|
||||
2026/11/10 14:53:19 fillwatch.go:284: [INFO] [fillwatch] checked 11 filesystem(s), 2 unreadable/skipped, 2 notification(s); bands: /=warning /mnt/sys_drive=warning
|
||||
|
||||
-- remove the test file by its exact name
|
||||
removed '/var/lib/felhom/FILLTEST-2026-11-10-r1371.bin'
|
||||
total 32
|
||||
drwxr-xr-x 5 root root 4096 Nov 10 14:54 .
|
||||
drwxr-xr-x 24 root root 4096 Oct 4 07:13 ..
|
||||
drwx--x--- 12 root root 4096 Nov 10 14:43 docker
|
||||
drwx------ 2 nobody nogroup 16384 Oct 4 07:11 lost+found
|
||||
drwxr-xr-x 4 root root 4096 Oct 26 19:52 sys_drive
|
||||
/dev/loop1 73391005696 13985468416 55630663680 21% /var/lib/felhom
|
||||
@@ -0,0 +1,85 @@
|
||||
== 2026-11-10T14:54:02Z controller 0.352.0-rc.1 onto scratch guest 9202 (by hand, the skill's step 3; digest sha256:d9b47ad5f3fc666ba06bb8542b3b49ff056f25643871021ef073fdb924194e7d)
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.350.0
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.352.0-rc.1
|
||||
restart_rc=0
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.352.0-rc.1 Up 12 seconds (healthy)
|
||||
sha256:9298978a8206200a32e9e13ab4910ac136e81775ee9f9337e4e33ae20e092edc
|
||||
gitea.dooplex.hu/admin/felhom-controller@sha256:d9b47ad5f3fc666ba06bb8542b3b49ff056f25643871021ef073fdb924194e7d
|
||||
|
||||
== 2026-11-10T14:56:10Z first fill check on 0.352.0-rc.1 (the old state held two warnings; the file was already removed)
|
||||
2026/11/10 14:54:05 main.go:335: [INFO] felhom-controller 0.352.0-rc.1 starting (customer: demo-hp, domain: enkisfelhom.hu)
|
||||
2026/11/10 14:55:35 fillwatch.go:250: [INFO] [fillwatch] /: warning → ok (20% used, 51.8 GB free) — cleared silently, re-armed
|
||||
2026/11/10 14:55:35 fillwatch.go:233: [DEBUG] [fillwatch] /mnt/felhom-drives/scratch_hdd/userdata/grimmory: usage unreadable — skipped (an unreadable filesystem is not a full one)
|
||||
2026/11/10 14:55:35 fillwatch.go:233: [DEBUG] [fillwatch] /mnt/felhom-drives/scratch_hdd/userdata/metube: usage unreadable — skipped (an unreadable filesystem is not a full one)
|
||||
2026/11/10 14:55:35 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: /=ok
|
||||
-- state file
|
||||
{
|
||||
"bands": {}
|
||||
}
|
||||
== the test file again: same exact name, same size
|
||||
2026-11-10T14:56:11Z
|
||||
fallocate_rc=0
|
||||
-rw-r--r-- 1 root root 45902462976 Nov 10 14:56 /var/lib/felhom/FILLTEST-2026-11-10-r1371.bin
|
||||
/dev/loop1 73391005696 59928866816 9687265280 87% /var/lib/felhom
|
||||
== 2026-11-10T15:04:16Z the next fill check ran (+471s)
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.352.0-rc.1 Up 10 minutes (healthy)
|
||||
2026/11/10 15:04:06 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: /=ok
|
||||
-- state file
|
||||
{
|
||||
"bands": {}
|
||||
}
|
||||
-- remove the test file by its exact name
|
||||
removed '/var/lib/felhom/FILLTEST-2026-11-10-r1371.bin'
|
||||
total 32
|
||||
drwxr-xr-x 5 root root 4096 Nov 10 15:04 .
|
||||
drwxr-xr-x 24 root root 4096 Oct 4 07:13 ..
|
||||
drwx--x--- 12 root root 4096 Nov 10 14:43 docker
|
||||
drwx------ 2 nobody nogroup 16384 Oct 4 07:11 lost+found
|
||||
drwxr-xr-x 4 root root 4096 Oct 26 19:52 sys_drive
|
||||
/dev/loop1 73391005696 12412989440 57203142656 18% /var/lib/felhom
|
||||
CROSSWAIT DONE 2026-11-10T15:04:18Z
|
||||
|
||||
== NOTE 2026-11-10T15:05:00Z: the 15:04:06 check read ok and that is CORRECT. Between 14:56 and 15:04 the box freed 1.6 GB by itself
|
||||
(other use 14,026,403,840 -> 12,412,993,536 bytes), so the volume was at 83.8 % with 10.5 GiB free: under the 85 % line. The file was too small.
|
||||
== second try: a bigger file of the same exact name, target 88 %
|
||||
2026-11-10T15:05:00Z
|
||||
fallocate_rc=0
|
||||
-rw-r--r-- 1 root root 48850000000 Nov 10 15:05 /var/lib/felhom/FILLTEST-2026-11-10-r1371.bin
|
||||
/dev/loop1 73391005696 61263015936 8353116160 89% /var/lib/felhom
|
||||
== 2026-11-10T15:14:06Z the next fill check ran (+540s)
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.352.0-rc.1 Up 20 minutes (healthy)
|
||||
2026/11/10 15:14:06 fillwatch.go:262: [WARN] [fillwatch] / ("Alkalmazások területe"): ok → warning — 88% used, 7.8 GB free of 68.4 GB; notifying the customer
|
||||
2026/11/10 15:14:06 notifier.go:1471: [WARN] notifier disabled (no hub configured): DROPPED event disk_warning (severity warning) — further disk_warning events are logged at DEBUG only
|
||||
2026/11/10 15:14:06 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 2 unreadable/skipped, 1 notification(s); bands: /=warning
|
||||
-- state file
|
||||
{
|
||||
"bands": {
|
||||
"/": "warning"
|
||||
}
|
||||
}
|
||||
-- remove the test file by its exact name
|
||||
removed '/var/lib/felhom/FILLTEST-2026-11-10-r1371.bin'
|
||||
total 32
|
||||
drwxr-xr-x 5 root root 4096 Nov 10 15:14 .
|
||||
drwxr-xr-x 24 root root 4096 Oct 4 07:13 ..
|
||||
drwx--x--- 12 root root 4096 Nov 10 14:43 docker
|
||||
drwx------ 2 nobody nogroup 16384 Oct 4 07:11 lost+found
|
||||
drwxr-xr-x 4 root root 4096 Oct 26 19:52 sys_drive
|
||||
/dev/loop1 73391005696 12413243392 57202888704 18% /var/lib/felhom
|
||||
CROSSWAIT DONE 2026-11-10T15:14:07Z
|
||||
|
||||
== 2026-11-10T15:32:45Z after the file was removed: the next check (15:24:06Z) cleared it without a notification; the test file is gone
|
||||
total 32
|
||||
drwxr-xr-x 5 root root 4096 Nov 10 15:14 .
|
||||
drwxr-xr-x 24 root root 4096 Oct 4 07:13 ..
|
||||
drwx--x--- 12 root root 4096 Nov 10 14:43 docker
|
||||
drwx------ 2 nobody nogroup 16384 Oct 4 07:11 lost+found
|
||||
drwxr-xr-x 4 root root 4096 Oct 26 19:52 sys_drive
|
||||
/dev/loop1 73391005696 12413661184 57202470912 18% /var/lib/felhom
|
||||
2026/11/10 15:24:06 fillwatch.go:250: [INFO] [fillwatch] /: warning → ok (18% used, 53.3 GB free) — cleared silently, re-armed
|
||||
2026/11/10 15:24:06 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: /=ok
|
||||
{
|
||||
"bands": {}
|
||||
}== 2026-11-10T15:32:46Z 9202 moves from rc.1 to the release tag by hand (it has no hub, so no floor moves it)
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.352.0
|
||||
restart_rc=0
|
||||
@@ -0,0 +1,40 @@
|
||||
### root@192.168.0.154 2026-11-10T15:02:11Z
|
||||
containers=10 unhealthy=0 restarting=0 exited=[]
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.351.0 Up 18 minutes (healthy)
|
||||
image file: gitea.dooplex.hu/admin/felhom-controller:0.351.0
|
||||
started=2026-11-10T14:44:09.016454308Z
|
||||
errors=0 warn=0 info=457
|
||||
2026/11/10 14:44:10 [INFO] felhom-controller 0.351.0 starting (customer: tester-1, domain: enkicsifelhom.hu)
|
||||
2026/11/10 14:44:11 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
|
||||
2026/11/10 14:44:31 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.351.0 (we are 0.351.0), no managed update running
|
||||
2026/11/10 14:44:47 [INFO] [selfupdate] Current version 0.351.0 is up to date
|
||||
2026/11/10 14:45:41 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026/11/10 14:54:12 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
### demo-hp 2026-11-10T15:02:12Z
|
||||
containers=26 unhealthy=0 restarting=0 exited=[]
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.351.0 Up 18 minutes (healthy)
|
||||
image file: gitea.dooplex.hu/admin/felhom-controller:0.351.0
|
||||
started=2026-11-10T14:43:25.514593512Z
|
||||
errors=0 warn=6 info=519
|
||||
2026/11/10 14:44:15 updater.go:104: [DEBUG] [selfupdate] queryRegistry: anonymous tags/list with Bearer HTTP 200
|
||||
2026/11/10 14:44:15 updater.go:104: [DEBUG] [selfupdate] queryRegistry: 38 tags returned: [0.331.0 0.341.0 0.341.0-rc.1 0.342.0 0.342.0-rc.1 0.342.0-rc.2 0.343.0 0.343.0-rc.1 0.343.0-rc.2 0.343.0-rc.3 0.343.0-rc.4 0.343.0-rc.5 0.3
|
||||
2026/11/10 14:44:15 updater.go:196: [INFO] [selfupdate] Current version 0.351.0 is up to date
|
||||
2026/11/10 14:44:15 updater.go:104: [DEBUG] [selfupdate] version comparison: current=0.351.0 (0.351.0), latest=0.351.0 (0.351.0), cmp=0, updateAvailable=false
|
||||
2026/11/10 14:58:33 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.351.0 >= floor 0.351.0 — no action
|
||||
2026/11/10 14:45:00 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026/11/10 14:53:31 fillwatch.go:284: [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
### felhom-pve 2026-11-10T15:02:14Z
|
||||
containers=9 unhealthy=0 restarting=0 exited=[]
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.351.0 Up 27 hours (healthy)
|
||||
image file: gitea.dooplex.hu/admin/felhom-controller:0.351.0
|
||||
started=2026-11-09T12:14:03.775366821Z
|
||||
errors=0 warn=6 info=27865
|
||||
2026/11/09 12:14:36 [INFO] [selfupdate] Current version 0.351.0 is up to date
|
||||
2026/11/09 18:14:05 [INFO] [selfupdate] Current version 0.351.0 is up to date
|
||||
2026/11/10 00:14:05 [INFO] [selfupdate] Current version 0.351.0 is up to date
|
||||
2026/11/10 06:14:05 [INFO] [selfupdate] Current version 0.351.0 is up to date
|
||||
2026/11/10 12:14:05 [INFO] [selfupdate] Current version 0.351.0 is up to date
|
||||
2026/11/10 14:24:05 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026/11/10 14:34:05 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026/11/10 14:44:05 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026/11/10 14:54:05 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
@@ -0,0 +1,14 @@
|
||||
== 2026-11-10T15:16:52Z per-box floors 0.352.0 / MinAgent 0.158.1 (image sha256:3c785dd396b091f19d4c777db9d7a206e75ab018b6238b5d887e1a4c6cfd3b29)
|
||||
-- demo-hp
|
||||
2026-11-10T15:16:52Z POST /customers/demo-hp/floor min_controller_version='0.352.0' min_agent='0.158.1'
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /customers/demo-hp?flash=floor_set
|
||||
-- demo-felhom
|
||||
2026-11-10T15:16:52Z POST /customers/demo-felhom/floor min_controller_version='0.352.0' min_agent='0.158.1'
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /customers/demo-felhom?flash=floor_set
|
||||
-- tester-1
|
||||
2026-11-10T15:16:52Z POST /customers/tester-1/floor min_controller_version='0.352.0' min_agent='0.158.1'
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /customers/tester-1?flash=floor_set
|
||||
== 2026-11-10T15:16:52Z done
|
||||
@@ -0,0 +1,39 @@
|
||||
### root@192.168.0.154 2026-11-10T15:30:01Z
|
||||
containers=10 unhealthy=0 restarting=0 exited=[]
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 12 minutes (healthy)
|
||||
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
|
||||
started=2026-11-10T15:17:03.705107516Z
|
||||
errors=0 warn=0 info=335
|
||||
2026/11/10 15:17:04 [INFO] felhom-controller 0.352.0 starting (customer: tester-1, domain: enkicsifelhom.hu)
|
||||
2026/11/10 15:17:04 [INFO] [selfupdate] Post-update startup: update successful (0.351.0 → 0.352.0)
|
||||
2026/11/10 15:17:04 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
|
||||
2026/11/10 15:17:14 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.352.0 (we are 0.352.0), no managed update running
|
||||
2026/11/10 15:17:36 [INFO] [selfupdate] Current version 0.352.0 is up to date
|
||||
2026/11/10 15:18:34 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026/11/10 15:27:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
### demo-hp 2026-11-10T15:30:02Z
|
||||
containers=26 unhealthy=0 restarting=0 exited=[]
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 13 minutes (healthy)
|
||||
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
|
||||
started=2026-11-10T15:17:03.918469562Z
|
||||
errors=0 warn=3 info=345
|
||||
2026/11/10 15:17:37 updater.go:104: [DEBUG] [selfupdate] queryRegistry: anonymous plain GET HTTP 401
|
||||
2026/11/10 15:17:37 updater.go:104: [DEBUG] [selfupdate] queryRegistry: anonymous tags/list with Bearer HTTP 200
|
||||
2026/11/10 15:17:37 updater.go:104: [DEBUG] [selfupdate] queryRegistry: 40 tags returned: [0.331.0 0.341.0 0.341.0-rc.1 0.342.0 0.342.0-rc.1 0.342.0-rc.2 0.343.0 0.343.0-rc.1 0.343.0-rc.2 0.343.0-rc.3 0.343.0-rc.4 0.343.0-rc.5 0.3
|
||||
2026/11/10 15:17:37 updater.go:196: [INFO] [selfupdate] Current version 0.352.0 is up to date
|
||||
2026/11/10 15:17:37 updater.go:104: [DEBUG] [selfupdate] version comparison: current=0.352.0 (0.352.0), latest=0.352.0 (0.352.0), cmp=0, updateAvailable=false
|
||||
2026/11/10 15:18:35 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026/11/10 15:27:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
### felhom-pve 2026-11-10T15:30:04Z
|
||||
containers=9 unhealthy=0 restarting=0 exited=[]
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 13 minutes (healthy)
|
||||
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
|
||||
started=2026-11-10T15:16:28.700768259Z
|
||||
errors=0 warn=0 info=342
|
||||
2026/11/10 15:16:29 [INFO] felhom-controller 0.352.0 starting (customer: demo-felhom, domain: demo-felhom.eu)
|
||||
2026/11/10 15:16:29 [INFO] [selfupdate] Post-update startup: update successful (0.351.0 → 0.352.0)
|
||||
2026/11/10 15:16:29 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
|
||||
2026/11/10 15:16:39 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.352.0 (we are 0.352.0), no managed update running
|
||||
2026/11/10 15:17:01 [INFO] [selfupdate] Current version 0.352.0 is up to date
|
||||
2026/11/10 15:17:59 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026/11/10 15:26:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
+43
@@ -0,0 +1,43 @@
|
||||
# after 0.352.0: what the controller container sees, and what the fill watch says — read-only, 2026-11-10T15:30:48Z
|
||||
# channel 1 = the kernel (stat -f / df inside the container); channel 2 = the fill watch's own log line
|
||||
== root@192.168.0.154:9201
|
||||
/ type=overlayfs fsid=b4d48bcf2747c72b blocks=18016108 free=15822196 dev=74
|
||||
/mnt/felhom-drives type=ext2/ext3 fsid=322b724225b4b36d blocks=15357389 free=10191290 dev=64513
|
||||
/mnt/sys_drive type=ext2/ext3 fsid=65ddc34fe9988fc8 blocks=18016108 free=15822196 dev=64519
|
||||
-- distinct filesystems by fsid, the overlay root counted as the volume below it:
|
||||
2
|
||||
2026/11/10 15:18:34 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026/11/10 15:27:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
state: { "bands": {}}
|
||||
== demo-hp:9201
|
||||
/ type=overlayfs fsid=71b91e3d7cadcd4f blocks=18016108 free=10674045 dev=49
|
||||
/mnt/felhom-drives type=ext2/ext3 fsid=c04f01ca42f6f3e9 blocks=10113344 free=6060989 dev=64513
|
||||
/mnt/sizing-bench type=ext2/ext3 fsid=85fbdcb4db8310a7 blocks=8179140 free=7847629 dev=64518
|
||||
/mnt/sys_drive type=ext2/ext3 fsid=e1508f4b7cb1b4b0 blocks=18016108 free=10674045 dev=64519
|
||||
-- distinct filesystems by fsid, the overlay root counted as the volume below it:
|
||||
3
|
||||
2026/11/10 15:18:35 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026/11/10 15:27:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
state: { "bands": {}}
|
||||
== felhom-pve:9201
|
||||
/ type=overlayfs fsid=87e8f2c78a7b090c blocks=64230001 free=63627374 dev=88
|
||||
/mnt/felhom-drives type=ext2/ext3 fsid=813814bd69e4dd6e blocks=24624445 free=15508719 dev=64513
|
||||
/mnt/sys_drive type=ext2/ext3 fsid=6968e12ca6ac967a blocks=64230001 free=63627374 dev=64519
|
||||
-- distinct filesystems by fsid, the overlay root counted as the volume below it:
|
||||
2
|
||||
2026/11/10 15:17:59 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026/11/10 15:26:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
state:
|
||||
|
||||
# the drives below /mnt/felhom-drives (the folder itself is on the guest's root disk, which nobody watches — R-1371), 2026-11-10T15:31:05Z
|
||||
== root@192.168.0.154:9201
|
||||
/mnt/felhom-drives/adatlemez type=ext2/ext3 fsid=a25b481c3dbf279a blocks=12822648 free=9621417 dev=2064
|
||||
/mnt/felhom-drives/media_a type=ext2/ext3 fsid=322b724225b4b36d blocks=15357389 free=10191292 dev=64513
|
||||
/mnt/felhom-drives/media_b type=ext2/ext3 fsid=322b724225b4b36d blocks=15357389 free=10191292 dev=64513
|
||||
/mnt/felhom-drives/media_c type=ext2/ext3 fsid=322b724225b4b36d blocks=15357389 free=10191292 dev=64513
|
||||
/mnt/felhom-drives/media_d type=ext2/ext3 fsid=322b724225b4b36d blocks=15357389 free=10191292 dev=64513
|
||||
== demo-hp:9201
|
||||
/mnt/felhom-drives/hdd_1 type=ext2/ext3 fsid=3ce20f0a870807d7 blocks=245844925 free=167050197 dev=66304
|
||||
== felhom-pve:9201
|
||||
/mnt/felhom-drives/hdd_1 type=ext2/ext3 fsid=ef113ee179770dd1 blocks=240075962 free=238087780 dev=2064
|
||||
# count per box = the apps' volume (/ and /mnt/sys_drive are one filesystem) + the drives that are a filesystem of their own
|
||||
@@ -0,0 +1,36 @@
|
||||
== 2026-11-10T15:31:16Z python3 scripts/https_only_check.py
|
||||
control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -)
|
||||
== tester-1 (enkicsifelhom.hu): 6 public name(s)
|
||||
OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1
|
||||
== demo-hp (enkisfelhom.hu): 15 public name(s)
|
||||
OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://time.enkisfelhom.hu/r1033-check?q=1 -> 301 https://time.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1
|
||||
== demo-felhom (demo-felhom.eu): 8 public name(s)
|
||||
OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
|
||||
== every public name redirects http -> https
|
||||
HTTPS RC=0
|
||||
@@ -0,0 +1,14 @@
|
||||
== 2026-11-10T15:31:33Z before
|
||||
2026-11-10T15:31:34Z GET /configuration -> 200
|
||||
Effective floor: v0.351.0 — source: DB (hub_settings) ; env fallback would be v0.120.0 Save global floor… DB override: v0.351.0 … Type the version again to confirm (or CLEAR to remove the DB override): Confirm & apply Cancel function confirmGlobalFloor() { var v = document.g
|
||||
.. MinAgent: . The hub HOLDS the floor for any box whose agent is below this — blank = uncoupled release, no gating. PBS wrapper sha256 of configs/felhom-pbs-apply (R-50b)
|
||||
== 2026-11-10T15:31:34Z the global floor, in its own save
|
||||
2026-11-10T15:31:34Z POST /configuration/global-floor min_controller_version='0.352.0' min_agent='0.158.1'
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /configuration?flash=floor_set
|
||||
== 2026-11-10T15:31:34Z read back
|
||||
2026-11-10T15:31:34Z GET /configuration -> 200
|
||||
Effective floor: v0.352.0 — source: DB (hub_settings) ; env fallback would be v0.120.0 Save global floor… DB override: v0.352.0 … Type the version again to confirm (or CLEAR to remove the DB override): Confirm & apply Cancel function confirmGlobalFloor() { var v = document.g
|
||||
.. MinAgent: . The hub HOLDS the floor for any box whose agent is below this — blank = uncoupled release, no gating. PBS wrapper sha256 of configs/felhom-pbs-apply (R-50b)
|
||||
2026/11/10 16:31:34 [INFO] Global controller-version floor set to "0.352.0" (declared MinAgent "0.158.1")
|
||||
2026/11/10 16:31:34 [INFO] access token "cc-releases-2": POST /configuration/global-floor → 303
|
||||
@@ -0,0 +1,91 @@
|
||||
== controller watch (watch), hub log since 2026-11-10T15:31:30Z; read 2026-11-10T16:02:15Z
|
||||
-- the three boxes (read on the box, over SSH)
|
||||
### root@192.168.0.154 2026-11-10T16:02:15Z
|
||||
containers=10 unhealthy=0 restarting=0 exited=[]
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 45 minutes (healthy)
|
||||
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
|
||||
started=2026-11-10T15:17:03.705107516Z
|
||||
errors=0 warn=0 info=945
|
||||
2026/11/10 15:17:04 [INFO] felhom-controller 0.352.0 starting (customer: tester-1, domain: enkicsifelhom.hu)
|
||||
2026/11/10 15:17:04 [INFO] [selfupdate] Post-update startup: update successful (0.351.0 → 0.352.0)
|
||||
2026/11/10 15:17:04 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
|
||||
2026/11/10 15:17:14 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.352.0 (we are 0.352.0), no managed update running
|
||||
2026/11/10 15:17:36 [INFO] [selfupdate] Current version 0.352.0 is up to date
|
||||
2026/11/10 15:27:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026/11/10 15:37:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026/11/10 15:47:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026/11/10 15:57:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
### demo-hp 2026-11-10T16:02:16Z
|
||||
containers=26 unhealthy=0 restarting=0 exited=[]
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 45 minutes (healthy)
|
||||
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
|
||||
started=2026-11-10T15:17:03.918469562Z
|
||||
errors=0 warn=3 info=962
|
||||
2026/11/10 15:17:37 updater.go:104: [DEBUG] [selfupdate] version comparison: current=0.352.0 (0.352.0), latest=0.352.0 (0.352.0), cmp=0, updateAvailable=false
|
||||
2026/11/10 15:31:38 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
|
||||
2026/11/10 15:32:07 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
|
||||
2026/11/10 15:47:07 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
|
||||
2026/11/10 16:02:07 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
|
||||
2026/11/10 15:17:05 deploy.go:1470: [WARN] [stacks] Stack plex: new field PLEX_CLAIM (type=text) requires manual configuration
|
||||
2026/11/10 15:27:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026/11/10 15:37:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026/11/10 15:47:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026/11/10 15:57:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
### felhom-pve 2026-11-10T16:02:18Z
|
||||
containers=9 unhealthy=0 restarting=0 exited=[]
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 45 minutes (healthy)
|
||||
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.0
|
||||
started=2026-11-10T15:16:28.700768259Z
|
||||
errors=0 warn=0 info=906
|
||||
2026/11/10 15:16:29 [INFO] felhom-controller 0.352.0 starting (customer: demo-felhom, domain: demo-felhom.eu)
|
||||
2026/11/10 15:16:29 [INFO] [selfupdate] Post-update startup: update successful (0.351.0 → 0.352.0)
|
||||
2026/11/10 15:16:29 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
|
||||
2026/11/10 15:16:39 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.352.0 (we are 0.352.0), no managed update running
|
||||
2026/11/10 15:17:01 [INFO] [selfupdate] Current version 0.352.0 is up to date
|
||||
2026/11/10 15:26:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026/11/10 15:36:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026/11/10 15:46:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026/11/10 15:56:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
-- hub log since 2026-11-10T15:31:30Z: lines / [ERROR] lines
|
||||
152
|
||||
0
|
||||
-- [WARN] lines by kind (the log prints CET)
|
||||
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers → 403
|
||||
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers/demo-hp →
|
||||
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers/tester-1 →
|
||||
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /events → 403
|
||||
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /notifications → 403
|
||||
-- floor lines
|
||||
2026/11/10 16:31:34 [INFO] Global controller-version floor set to "0.352.0" (declared MinAgent "0.158.1")
|
||||
2026/11/10 16:31:34 [INFO] access token "cc-releases-2": POST /configuration/global-floor → 303
|
||||
-- events from boxes (lines that name an event)
|
||||
(count: 0)
|
||||
-- alarm checks and mails
|
||||
-- lines that name a mail send
|
||||
0
|
||||
-- positive signs (the hub works, it is not only quiet): reports received in the window, by box
|
||||
4 demo-felhom
|
||||
4 demo-hp
|
||||
4 tester-1
|
||||
-- host reports, by host
|
||||
2 demo-felhom-8363b5
|
||||
2 demo-hp-bb76ea
|
||||
2 tester-1-d70be4
|
||||
-- pod
|
||||
hub-658c847c65-d8zbk phase=Running restarts=0 image=gitea.dooplex.hu/admin/felhom-hub:0.180.0
|
||||
-- the hub's hosts page (GET /hosts with CC's token), read 2026-11-10T16:02:58Z -> 200
|
||||
demo-felhom-8363b5 | agent 0.173.0 | 7.0.14-28-pve | ONLINE | cloudflared running (connected)
|
||||
demo-hp-bb76ea | agent 0.173.0 | 7.0.14-28-pve | ONLINE | cloudflared running (connected)
|
||||
tester-1-d70be4 | agent 0.173.0 | 7.0.14-22-pve | ONLINE | cloudflared running (connected)
|
||||
(the page holds these three hosts and no other)
|
||||
-- the effective global floor, read back 2026-11-10T16:02:59Z (GET /configuration -> 200)
|
||||
Effective floor: v0.352.0 — source: DB (hub_settings)
|
||||
-- demo-hp's three [WARN] lines since its controller start (the walk's secret filter hid two; field NAMES only, no value)
|
||||
15:17:05 [WARN] [stacks] Stack calibre-web: new field ADMIN_PASSWORD (type=password) requires manual configuration
|
||||
15:17:05 [WARN] [stacks] Stack plex: new field PLEX_CLAIM (type=text) requires manual configuration
|
||||
15:17:05 [WARN] [stacks] Stack bookstack: new field ADMIN_PASSWORD (type=password) requires manual configuration
|
||||
(the controller's own start note for a field added to an installed app; this release did not change that code)
|
||||
-- second channel, not the hub: the mailbox that gets admin@ and tester1@ mail, read by the mail connector 2026-11-10T16:02Z
|
||||
search 'newer_than:3h' -> 0 threads (no mail since 13:02 UTC; the run's first release step was 14:51 UTC)
|
||||
positive control, search 'newer_than:1d' -> 1 thread: the security watch's mail of 2026-11-10T03:41:40Z to admin@ (so the search does find mail)
|
||||
-- verdict: every box that reports is on 0.352.0 and healthy; no event, no alarm and no mail in the 30 minutes after the global floor
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
== 2026-11-10T16:03:11Z clear the per-box floors (both fields empty); the global floor stays
|
||||
-- demo-hp
|
||||
2026-11-10T16:03:11Z POST /customers/demo-hp/floor min_controller_version='' min_agent=''
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /customers/demo-hp?flash=floor_set
|
||||
-- demo-felhom
|
||||
2026-11-10T16:03:11Z POST /customers/demo-felhom/floor min_controller_version='' min_agent=''
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /customers/demo-felhom?flash=floor_set
|
||||
-- tester-1
|
||||
2026-11-10T16:03:11Z POST /customers/tester-1/floor min_controller_version='' min_agent=''
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /customers/tester-1?flash=floor_set
|
||||
== 2026-11-10T16:03:11Z done
|
||||
-- the hub's log for the clear (the log prints CET), and the first report of each box after it
|
||||
2026/11/10 17:03:11 [INFO] Customer demo-hp controller-version floor override set to "" (declared MinAgent "")
|
||||
2026/11/10 17:03:11 [INFO] access token "cc-releases-2": POST /customers/demo-hp/floor → 303
|
||||
2026/11/10 17:03:11 [INFO] Customer demo-felhom controller-version floor override set to "" (declared MinAgent "")
|
||||
2026/11/10 17:03:11 [INFO] access token "cc-releases-2": POST /customers/demo-felhom/floor → 303
|
||||
2026/11/10 17:03:11 [INFO] Customer tester-1 controller-version floor override set to "" (declared MinAgent "")
|
||||
2026/11/10 17:03:11 [INFO] access token "cc-releases-2": POST /customers/tester-1/floor → 303
|
||||
2026/11/10 17:03:14 [INFO] Received report from tester-1 (5252 bytes)
|
||||
2026/11/10 17:03:15 [INFO] Received report from demo-felhom (5563 bytes)
|
||||
2026/11/10 17:03:15 [INFO] Received report from demo-hp (10292 bytes)
|
||||
-- the boxes, read on the box 2026-11-10T16:08Z, 5 minutes after the clear
|
||||
tester-1: gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 50 minutes (healthy)
|
||||
demo-hp: gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 50 minutes (healthy)
|
||||
demo-felhom: gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 51 minutes (healthy)
|
||||
-- the floor a box got AFTER the clear (demo-hp logs at DEBUG; the other two do not print this line)
|
||||
2026/11/10 16:03:15 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
|
||||
-- verdict: the per-box floors are gone; the boxes stay on 0.352.0; the floor demo-hp got 4 seconds after the clear is the global one
|
||||
-- not read: the hub's page of each customer (it refuses a token, 403); the proof of the clear is the hub's own log line
|
||||
@@ -0,0 +1,5 @@
|
||||
--- PASS: TestD68_SeedHoldsNoOperatorOnlyType (0.00s)
|
||||
--- PASS: TestD68_HostDiskIsOperatorOnly_EvenWhenTheHouseholdSavedIt (0.07s)
|
||||
--- PASS: TestD68_StorageFillReachesAHouseholdOnlyForADriveItCanFree (0.34s)
|
||||
--- PASS: TestD68_RegisterAndGate (0.00s)
|
||||
D68 RC=0
|
||||
@@ -0,0 +1,4 @@
|
||||
14:59:29Z felhom.eu 97b7e47114: run 2616 gates #1418 status running
|
||||
14:59:49Z felhom.eu 97b7e47114: run 2616 gates #1418 status running
|
||||
15:00:10Z felhom.eu 97b7e47114: run 2616 gates #1418 status success
|
||||
CI RC=0
|
||||
@@ -0,0 +1,41 @@
|
||||
== 2026-11-10T15:12:34Z before: image gitea.dooplex.hu/admin/felhom-hub:0.179.1
|
||||
app: sync=OutOfSync health=Healthy rev=d0a54c2f8e308ae959900fd8068bc2b434d28664
|
||||
== OutOfSync resources
|
||||
Deployment felhom-system hub OutOfSync
|
||||
count: 1
|
||||
== 2026-11-10T15:12:48Z sync ONLY Deployment/hub at d0a54c2f8e308ae959900fd8068bc2b434d28664
|
||||
application.argoproj.io/felhom patched
|
||||
Waiting for deployment "hub" rollout to finish: 0 of 1 updated replicas are available...
|
||||
deployment "hub" successfully rolled out
|
||||
rollout rc=0
|
||||
== 2026-11-10T15:13:10Z after
|
||||
image: gitea.dooplex.hu/admin/felhom-hub:0.180.0
|
||||
hub-658c847c65-d8zbk Running ready=true restarts=0 imageID=gitea.dooplex.hu/admin/felhom-hub@sha256:768e73ab1ab502a8c3fe23a9d42cc038443a6456a1f40dcec853a2c588d04a8f started=2026-11-10T15:12:49Z
|
||||
operation: phase=Succeeded msg=successfully synced (all tasks run) rev=d0a54c2f8e308ae959900fd8068bc2b434d28664
|
||||
app: sync=Synced health=Healthy rev=d0a54c2f8e308ae959900fd8068bc2b434d28664
|
||||
== startup log (first 25 lines of the new pod)
|
||||
2026/11/10 16:12:52 [INFO] felhom-hub 0.180.0 starting
|
||||
2026/11/10 16:12:52 [INFO] Database opened at /data/hub.db
|
||||
2026/11/10 16:12:52 [INFO] Default controller-version floor: 0.120.0
|
||||
2026/11/10 16:12:52 [INFO] Template fetcher started (every 1h)
|
||||
2026/11/10 16:12:52 [DEBUG] Template fetched (5881 bytes)
|
||||
2026/11/10 16:12:53 [INFO] Asset seed: all 263 files up-to-date
|
||||
2026/11/10 16:12:53 [INFO] Asset manifest built: 288 files
|
||||
2026/11/10 16:12:53 [INFO] App-email relay enabled (limit 30/min/customer, From domains [felhom.eu])
|
||||
2026/11/10 16:12:53 [INFO] Gitea artifact browser enabled (Day-0 version dropdowns) via http://gitea.gitea-system.svc.cluster.local:3000
|
||||
2026/11/10 16:12:53 [INFO] Offsite provisioning enabled (pool_box=611714, location=fsn1)
|
||||
2026/11/10 16:12:53 [INFO] Offsite pool-box checker initialized: box=611714 fill warn=80% crit=90%, oversub warn=2.00x, unreachable after 3 consecutive failed reads, refresh 15m0s
|
||||
2026/11/10 16:12:54 [INFO] osupdates: approval rule = every ring-0 box healthy for 24h0m0s and 1 night run(s)
|
||||
2026/11/10 16:12:54 [INFO] osupdates: the Docker engine set is approved only by the operator, after 2 healthy ring-0 night(s)
|
||||
2026/11/10 16:12:54 [INFO] osupdates: alarms = no OS leg 168h0m0s, reboot needed 336h0m0s, ring 0 stalled 168h0m0s, not covered 336h0m0s
|
||||
2026/11/10 16:12:54 [INFO] Off-site key registrar enabled; daily key check at 07:10 Budapest
|
||||
2026/11/10 16:12:54 [INFO] access log: /data/access-log, one file per UTC day, kept 90 days
|
||||
2026/11/10 16:12:54 [INFO] Registry version checker started (every 6h)
|
||||
2026/11/10 16:12:54 [INFO] offsite-key-audit: next run at 2026-11-11 07:10 CET (in 14h57m6s)
|
||||
2026/11/10 16:12:54 [INFO] WG peer-sync enabled (endpoint 167.233.158.164:22, user felhom-peersync)
|
||||
2026/11/10 16:12:54 [INFO] PBS DR tenantsync enabled (endpoint 167.233.158.164:22, user felhom-peersync; WG-registration auto-provision hook armed)
|
||||
2026/11/10 16:12:54 [INFO] PBS-DR box checker initialized: fill warn=80% crit=90%, unreachable after 3 consecutive failed reads, refresh 15m0s
|
||||
2026/11/10 16:12:54 [INFO] agent-plane poke enabled (endpoint 167.233.158.164:22, user felhom-peersync; web + api admin seams armed)
|
||||
2026/11/10 16:12:54 [INFO] PBS-DR self-heal reconciler started (interval 5m)
|
||||
2026/11/10 16:12:54 [INFO] offsite credential self-heal reconciler started (interval 5m, debounce 2 reports)
|
||||
2026/11/10 16:12:54 [DEBUG] Registry version check: latest = 0.351.0
|
||||
@@ -0,0 +1 @@
|
||||
15:15:35Z felhom.eu d0a54c2f8e: run 2617 gates #1419 status success
|
||||
@@ -0,0 +1,8 @@
|
||||
# what the running hub lets CC's token read about a household's saved alarm list — GET only, 2026-11-10T15:32:36Z
|
||||
2026/11/10 16:32:12 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers → 403
|
||||
2026/11/10 16:32:16 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers/tester-1 → 403
|
||||
2026/11/10 16:32:17 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers/demo-hp → 403
|
||||
2026/11/10 16:32:17 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /events → 403
|
||||
2026/11/10 16:32:17 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /notifications → 403
|
||||
# result: every page that shows a household's list is 'session only — never by a token' (by design, 05 §17).
|
||||
# no copy of the hub DB was taken: the live file holds mail addresses and old token bytes (05 §16.2).
|
||||
@@ -0,0 +1,37 @@
|
||||
== hub watch, 30 minutes after the deploy (pod started 2026-11-10T15:12:49Z); read 2026-11-10T15:43:01Z
|
||||
-- argocd app
|
||||
sync=Synced health=Healthy revision=d0a54c2f8e308ae959900fd8068bc2b434d28664
|
||||
-- pod
|
||||
hub-658c847c65-d8zbk phase=Running restarts=0 started=2026-11-10T15:12:52Z image=gitea.dooplex.hu/admin/felhom-hub:0.180.0
|
||||
-- hub log since the start: lines / [ERROR] lines
|
||||
260
|
||||
0
|
||||
-- [WARN] lines by kind (the log prints CET)
|
||||
3 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /cu
|
||||
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /ev
|
||||
1 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /no
|
||||
2 [WARN] slow statement: Nms (over 1s): DELETE …
|
||||
8 [WARN] slow statement: Nms (over 1s): SELECT …
|
||||
1 [WARN] slow statement: Nms (over 1s): UPDATE …
|
||||
-- events from boxes, by box / type / severity
|
||||
1 demo-felhom controller_started info
|
||||
1 demo-felhom controller_updated info
|
||||
1 demo-hp controller_started info
|
||||
1 demo-hp controller_updated info
|
||||
1 tester-1 controller_started info
|
||||
1 tester-1 controller_updated info
|
||||
-- alarm checks and mails
|
||||
1 [INFO] osupdates: alarm check done in N, 0 alarm(s) sent []
|
||||
-- lines that name a mail send
|
||||
0
|
||||
-- positive signs (the hub works, it is not only quiet): reports received in the window, by box
|
||||
6 demo-felhom
|
||||
6 demo-hp
|
||||
6 tester-1
|
||||
-- host reports, by host
|
||||
3 demo-felhom-8363b5
|
||||
3 demo-hp-bb76ea
|
||||
5 tester-1-d70be4
|
||||
-- startup line
|
||||
2026/11/10 16:12:52 [INFO] felhom-hub 0.180.0 starting
|
||||
-- second channel (not the hub): the operator's mailbox, read by the mail connector — see ../controller/12-watch.txt
|
||||
@@ -278,10 +278,10 @@ stopping line that lies.
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` | OPEN — filed 2026-10-26 | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC |
|
||||
| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` | OPEN — filed 2026-10-26 **2026-11-10:** still so. The host clock is 32 s behind DooPlex (it was 14 s on 2026-10-26); `chrony` failed again at the 2026-11-08 05:35 boot with the same line; `NTPSynchronized=no`. Seen in a release run, where the box's log times read half a minute early (`documentation/audits/release-space-fixes-2026-11-10/agent/13-n100-clock.txt`). Not fixed by hand: the N100 takes no hand step in a release run. | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC |
|
||||
| **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC |
|
||||
| **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H7: the N100 (4.28 GB, 74 files) and Tester 1 (4.29 GB, 81 files) sit at the journal's own cap; Tester 1 writes about 150 MB a day. Lines written by `sudo` are 92 / 92 / 90 % of all lines in one hour — this counts three lines for each call (the command and the session pair), while this row's 61 % counts the session pair only: the same fact, counted two ways. No rank change is proposed: the journal stops at its own cap (about 4.3 GB), so it costs space and log history, not safety. | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC |
|
||||
| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC |
|
||||
| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. **2026-11-10 (release run) — part (2) RELEASED, VOUCHED and PROVEN on three test boxes** (agent 0.173.0, vouched by the operator 15:29 UTC; `documentation/audits/release-space-fixes-2026-11-10/agent/`): the folder went 49 → 1, 48 → 1 and 33 → 1 files (about 700 / 690 / 485 MB back) at the commit of THIS update, not at the next one (the earlier sentence here and in the agent's CHANGELOG was wrong, and is corrected there). A rollback after the clean-up worked on Tester 1: a signed update to the older version was crashed once, 9 seconds into the 60-second dwell, and the wrapper put the released version back in the same second; the failed version's file stays until the next good update (by design). The kept older kernel on demo-hp booted to healthy in 313 s and the box went back (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). **Still open:** (1) the kernel clean-up as a program (`11-os-updates.md` §5.11.1, not built) and a second box; (3) template archives; (4) the `.bak-*` copies — the release does not reach them; (5) the failed-run logs. A household box gets part (2) at its first good agent update after the vouch. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC |
|
||||
| **R-1369** | Box system & updates | P3 | **When a disk is full there is no written way out: no runbook and no self-repair, for the host system disk, the thin pool, the guest disks or a data drive.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 7). One missing runbook, seen from 21 places; the search of the runbooks folder for a disk-full procedure gave 0 hits (`evidence/searches.txt` S6). The alarms exist (hub: 90 / 95 % host disk, 85 / 90 % thin pool; controller: 85 / 95 % on the data volume and each drive) but they end in a mail, not in a step. For three small places nobody wrote what fails when they are full (EFI partition, host and guest temp folders). **Members (answers marked N):** H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7. | OPEN — filed 2026-11-10 (space list) **2026-11-10 (afternoon) — the runbook is written:** `documentation/runbooks/disk-full.md`, five parts, each hand step marked PROVEN (date and evidence; all on demo-hp) or NOT PROVEN. Still open: one rehearsal of the whole runbook on demo-hp; the NOT PROVEN steps (copies on a former target, template archives, the `.bak-*` binaries, the guest root disk); what fails when the EFI partition or a temp folder is full. | — | Docs: one runbook „a disk is full” with one section per disk (host system disk, thin pool, guest root, data volume, data drive): what to look at first, what is safe to delete (the places of R-1368 and R-1367), what is never deleted by hand (household data, the kept-data rulings). Rehearse it once on demo-hp. | CC |
|
||||
| **R-1370** | Box system & updates | P3 | **The host's system disk has no space budget: no document says how much the host's own files may take, and the installer's sizing rule covers the guest only.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` place H1, answer H1-3; search S5 — 0 hits). Today: demo-hp 58 % of 41 GB, N100 38 % of 101 GB, Tester 1 34 % of 63 GB — and the parts that grow are the journal (R-1367), old kernels and agent binaries (R-1368), old backup copies (R-1087) and the local whole-guest copy (R-548). Second half, known but with no row until now: on a small disk the guest's own guards read the guest's 70 GiB while the thin pool behind it holds 54 GiB (demo-hp: 102 GiB promised on a 53.9 GiB pool, 70.67 % used), so only the hub's thin-pool alarm stands before a full pool. **Member:** H1-3. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D69 = yes; `09` §3 decision 288):** the section goes inside `16-storage-physical-identity.md`, no new document. **2026-11-10 (afternoon) — the section is written:** `16-storage-physical-identity.md` §14 (45 places, the list „no limit, by ruling”, the budget, the open design points); `PROMPT-TEMPLATE.md` §N.5 keeps it true. The budget from measured numbers: the root needs about 14 GB when the whole-box copies go to a backup drive, and 28–54 GB at keep 3 when they do not. **D75 (for the operator):** the smallest system disk Felhom accepts. CC's pick: 128 GB only with a backup drive, 200 GB without one. The other option, 128 GB always, lets a box with many apps and no backup drive stop its local copies. With no ruling the installer accepts any disk. Open after the ruling: the installer's refusal. **2026-11-10 15:19 — RULED (D75 = one number, 200 GB, with or without a backup drive; `09` §3 decision 295).** This replaces CC's two-step pick. **Open, not ruled:** what the installer does below 200 GB — refuse, or warn. demo-hp (128 GB) stays as a test box. `16` §14.3 carries the ruling. | — | Design first: what the installer does below 200 GB (refuse, or warn) — a later session. Then the installer. | CC |
|
||||
| **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC |
|
||||
@@ -326,7 +326,7 @@ stopping line that lies.
|
||||
| **R-1361** | Monitoring & notifications | P2 | **An event the box sends while the hub cannot be reached is lost: 3 tries over 9 s, then dropped.** Walked 2026-11-03 on Tester 1 (hub blackholed 13 min): the box ran normally, an app install worked, a crash loop stopped the app and held it, and `app_stopped_unhealthy` and `app_held` failed after 3 attempts (`notifier.go`). Afterwards NOTHING was re-sent: the hub's log shows no event from the outage, its „Held apps" had no row (`a row existed: false`), and the hold reporter had already recorded the app as reported (`holdreport.go`, the push is asynchronous) so only its periodic re-send can repair it. Any DooPlex restart (2026-11-01 was one) can swallow a household's and the operator's mails this way. `audits/security-2026-11-02/part5/outage-hub.log` | OPEN — filed 2026-11-03 **2026-11-03, a second cost seen:** the walk's cut (01:17–01:39 UTC) covered Tester 1's 01:30 dump, its `db_dump_completed` was lost, and at 04:00 UTC the hub mailed `expected_dbdump_missed` „No DB dump completed or failed since midnight", while the box's own report read `last_db_dump` 03:15 UTC. A lost event becomes a false alarm. | — | a small on-disk outbox for events, sent in order when the hub answers (a design: which types, how long kept); the hold reporter marks „reported" only after a 2xx. Test red first | CC |
|
||||
| **R-1363** | Monitoring & notifications | P3 | **A household's mail ends with raw JSON: `Megjegyzés: {"avail_gb":5.192043304443359,"band":"warning",...}`.** Seen in the walks of 2026-11-02 (`audits/security-2026-11-02/part5/mail-fill-household.txt`, `mail-crash-household.txt`). `api/handler.go` adds the details as the „Megjegyzés" line; a golden pins it (`mail_golden_test.go`), so it was once wanted. The sentence itself is right and Hungarian. | OPEN — filed 2026-11-03 | — | drop the line from a household's mail, or render known fields in words; the operator's mail keeps it | CC |
|
||||
| **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) |
|
||||
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
|
||||
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
|
||||
| **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC |
|
||||
|
||||
## Hub & operator — 16 rows (P2 1, P3 4, P4 11)
|
||||
@@ -346,7 +346,7 @@ stopping line that lies.
|
||||
| **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator |
|
||||
| **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC |
|
||||
| **R-1345** | Hub & operator | P4 | **An access token's scopes cannot be changed after it is made — widening or narrowing one means a new token (and a new secret).** The operator, 2026-10-29: to give CC `hosts:read` + `hosts:write` for the live proofs he had to make `CC_HOSTS` (now in `sec`) beside `cc-releases-2`. Wanted: edit a token's scopes in the session (an audit line per change), so a scope can also be taken away after a session without retiring the secret. | OPEN | — | design + build the scope edit (session only), red-first test that an unticked scope is refused at once | CC |
|
||||
| **R-1353** | Hub & operator | P4 | **Statements still over 1 s, named by the new slow-statement line (hub 0.178.0) in its first hour:** a background deadline/restore-test check reading days of full host-report JSON (`GetHostReportsSince`, `store.go`; 1.5–9.2 s cold, from `monitor/deadline.go` and `monitor/restoretest.go` — no page waits on it); the start-up prune of `events` and `app_log_issues` (1.1 s and 1.8 s; no index on `created_at` / `last_seen`); the customer page's newest-report read (`GetCustomer`, 1.3 s cold in the pod probe). With the 1 GiB limit most of this should come from the cache once warm. Evidence: the hub log 2026-10-30 09:22–10:05 UTC; `audits/release-2026-10-30/p2-r1346/`. | OPEN **2026-11-02 read (`audits/security-2026-11-02/part4/hub-slow-lines-2026-11-01-02.txt`; the log covers only the 31 h since DooPlex restarted on 2026-11-01 16:11 CET, the lines from 10-30 are gone with the old pod):** 35 lines, 15 statements. What repeats: `VACUUM INTO` 5× (avg 110 s, the 6-hourly snapshot, expected); `GetHostReportsSince` 9× (avg 7.9 s, max 26.6 s); `GetCustomer`'s newest report 3× (1.1 s). Once, at the cold start: the prune of `events` (10.4 s) and `app_log_issues` (21 s, again 04:30), the OS-release and host lists (2–22 s). No fix before the full week (11-06). | — | read a week of slow-statement lines; index or narrow what still repeats (the history read needs only a few fields, not the whole JSON) | CC |
|
||||
| **R-1353** | Hub & operator | P4 | **Statements still over 1 s, named by the new slow-statement line (hub 0.178.0) in its first hour:** a background deadline/restore-test check reading days of full host-report JSON (`GetHostReportsSince`, `store.go`; 1.5–9.2 s cold, from `monitor/deadline.go` and `monitor/restoretest.go` — no page waits on it); the start-up prune of `events` and `app_log_issues` (1.1 s and 1.8 s; no index on `created_at` / `last_seen`); the customer page's newest-report read (`GetCustomer`, 1.3 s cold in the pod probe). With the 1 GiB limit most of this should come from the cache once warm. Evidence: the hub log 2026-10-30 09:22–10:05 UTC; `audits/release-2026-10-30/p2-r1346/`. | OPEN **2026-11-02 read (`audits/security-2026-11-02/part4/hub-slow-lines-2026-11-01-02.txt`; the log covers only the 31 h since DooPlex restarted on 2026-11-01 16:11 CET, the lines from 10-30 are gone with the old pod):** 35 lines, 15 statements. What repeats: `VACUUM INTO` 5× (avg 110 s, the 6-hourly snapshot, expected); `GetHostReportsSince` 9× (avg 7.9 s, max 26.6 s); `GetCustomer`'s newest report 3× (1.1 s). Once, at the cold start: the prune of `events` (10.4 s) and `app_log_issues` (21 s, again 04:30), the OS-release and host lists (2–22 s). No fix before the full week (11-06). **2026-11-10 read (the hub's start after a deploy, `audits/release-space-fixes-2026-11-10/hub/10-watch.txt`):** 11 slow lines in the first 2 minutes of the new pod, the longest 19.1 s (`GetHostReportsSince`), then the OS-report reads (3–4 s) and the two prunes (2.7 s, 4.2 s); none in the 28 minutes after. The 13:02 UTC snapshot's `VACUUM INTO` took 127.8 s. The same statements as the 11-02 read; no new one. | — | read a week of slow-statement lines; index or narrow what still repeats (the history read needs only a few fields, not the whole JSON) | CC |
|
||||
| **R-1375** | Hub & operator | P2 | **The outside part of three endings is done by hand or not at all — the worst: an off-site account can stay at the provider with no record in the hub.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH06, EH01, EA21 and the ending "a tester ends"; read in source and documents; no account was called). (1) **Unticking the off-site tier** of a household takes nothing down and leaves no reminder (`hub/internal/web/configs.go:1737-1743`). A later RESET takes the account down only when the tier is ticked (`hub/internal/web/customer_reset.go:203-245`), so the sub-account and its data stay, are paid for, and the hub holds no record that they exist. (2) **Switching the second off-site tier off** takes nothing down: storage entry, token and copies stay; the source calls the teardown 'a deliberate future op' (agent `internal/pbsdr/manager.go:274`). (3) **A tester ends**: no handler and no runbook; done by hand once (2026-09-25); no list says what to collect. **Members (answers marked N):** EH06-4, EH01-4, EA21-4, and 'a tester ends' in the table's head. Related: R-810 (a household leaves), R-32. | OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D71 = yes, D74 = yes; `09` §3 decisions 291, 294).** Not built. | — | Hub (D71): keep the account record after an untick; RESET and delete take the account down, ticked or not. Docs (D74): the runbook 'a household or a tester leaves', from the five code doors, in a later documents-only session. Parts (2) and (3) of this row have no ruling of their own. | CC |
|
||||
| **R-1376** | Hub & operator | P3 | **Records of a thing that is gone stay in six places.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH00, EH02, EH21, EA00, EA11, EA27; read in source, `documentation/audits/endings-list-2026-11-10/evidence/hub-delete-statements.txt`; no hub database was read). (1) **Hub:** thirteen tables have no delete statement anywhere. Deleting a box leaves its update settings, its update reports and its kernel notices; deleting a household leaves its off-site key rows, off-site window rows and abandon requests. Most are small; the update reports of each box grow with time. (2) **Host uninstall** leaves the mount units and host mounts of household drives (said when R-179 closed; no open row carried it). (3) **A drive wiped by a signed job** keeps a record and a mount unit that name the old filesystem (agent `internal/signedjobs/wipe.go:61-110`). (4) **The token of a destroyed guest** stays valid until a new guest takes its number: the token store has no remove (agent `internal/localapi/tokenstore.go:127-210`). **Members (answers marked N):** EH00-4, EH02-4, EH21-4, EA00-4, EA11-4, EA27-4. | OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D73 = yes, with one exception; `09` §3 decision 293).** Not built. | — | Hub (D73): add the tables with no delete to the two purges and to the daily clean-up; the delete journal (`host_deletions`, `customer_resets`) stays on purpose, and the privacy text says so. Agent: remove a guest's token when the guest is destroyed. Parts (2) and (3) have no ruling of their own. | CC |
|
||||
|
||||
|
||||
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
|
||||
|
||||
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
|
||||
|
||||
## Moved from STATUS on 2026-11-10 (evening)
|
||||
|
||||
**Updated 2026-11-10 (late afternoon).** You ruled D71–D75 at 15:19 (`documentation/architecture/09-update-architecture.md` §3, decisions 291–296); none is built. The smallest system disk is 200 GB, and what the installer does below 200 GB (refuse, or warn) is open (R-1370). The report of this run is `REPORT-release-space-fixes-2026-11-10.md`.
|
||||
|
||||
## Moved from STATUS on 2026-11-10 (late afternoon)
|
||||
|
||||
**Updated 2026-11-10 (afternoon).** CC listed every way a thing ends on a box or in the hub (74 endings, eight questions each): 22 answers were „nobody asked”, now in six new rows (R-1373…R-1378) and four questions for you (D71–D74). CC removed six old kernels from the HP box by hand with the keep rule you ruled (8 → 2, 6.4 GB back, one restart, all apps healthy after 5 minutes); nothing is built for it yet. Three first space fixes wait in source for a release, the disk-space design (`documentation/architecture/16-storage-physical-identity.md` §14) and the runbook `documentation/runbooks/disk-full.md` (R-1369) are written, and D75 is one more question for you; the report is `REPORT-endings-kernels-fixes-2026-11-10.md`.
|
||||
|
||||
@@ -34,7 +34,8 @@ pvesm list local
|
||||
dpkg -l 'proxmox-kernel-*-pve-signed' | grep ^ii
|
||||
```
|
||||
|
||||
**The product does by itself:** it empties the package cache in each OS update. It keeps the newest N whole-box copies
|
||||
**The product does by itself:** it empties the package cache in each OS update. It removes old agent binaries from the
|
||||
self-update folder after each good agent update. It keeps the newest N whole-box copies
|
||||
(default 3) and removes half-written ones. journald caps the journal (10 % of the disk, at most 4 GB). It refuses an OS
|
||||
update under 500 MB free. It skips a local whole-box copy that cannot fit, and says so on the backup page (R-548). The
|
||||
hub alarms at 90 % and 95 %.
|
||||
@@ -47,7 +48,7 @@ hub alarms at 90 % and 95 %.
|
||||
| The host journal | Save it first, then `journalctl --vacuum-size=500M` | **PROVEN 2026-11-07**: 3.8 GB → 468 MB (`audits/demo-check-2026-11-07/README.md`; the record is off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-space-2026-11-07/cleanup.txt`). It grows back (R-1367) |
|
||||
| An installer ISO that a newer one replaced; a test archive of a guest that is gone | `pvesm free <volid>`, after `pct list` and `qm list` show that no guest uses it, and after its checksum matches a copy elsewhere | **PROVEN 2026-10-23** (three ISOs, `audits/day-2026-10-23/09-demohp-space.txt`), **2026-11-07** (one archive of a removed guest), **2026-11-09** (one ISO). Test boxes only |
|
||||
| Whole-box copies on a former target (H4) | — | **NOT PROVEN.** Needs the operator's word (R-1087) |
|
||||
| Old binaries in the agent's self-update folder (H12) | — | **NOT PROVEN.** The agent cleans the folder after the release that carries the fix (R-1368) |
|
||||
| Old binaries in the agent's self-update folder (H12) | No hand step. The agent removes them at the end of each good update, and keeps the running version's file | **PROVEN 2026-11-10** as a product path on three test boxes: 49 → 1, 48 → 1 and 33 → 1 files; a rollback worked after it (`audits/release-space-fixes-2026-11-10/agent/`). **NOT PROVEN:** a removal by hand |
|
||||
| Extra guest template archives (H5); `/usr/local/bin/felhom-agent.bak-*` (H13) | — | **NOT PROVEN** (R-1368) |
|
||||
|
||||
**Never by hand:**
|
||||
|
||||
Reference in new issue
Block a user