kernel night 7->8 read back: demo-felhom passed (7.0.14-20 default, ~1.5 min), demo-hp no step (R-899 filed); 126 -> 127
gates / gates (push) Successful in 2m58s
gates / gates (push) Successful in 2m58s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -16,6 +16,11 @@
|
||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||
|
||||
|
||||
> **2026-10-08 (morning) — the first real kernel night read back.** demo-felhom: staged exactly the told 7.0.14-20 at
|
||||
> 04:39:04, healthy 04:40:38, default now 7.0.14-20, apps ~1.5 min away, no alarm. demo-hp: no whole-guest backup that
|
||||
> night (yesterday's 08:49 press + 24 h cadence > window end) → no step; R-899 filed. Reply-To proven by the operator's
|
||||
> reply. Next: both boxes due 7.0.14-22 tonight; read back 2026-10-09. Register 126 → 127.
|
||||
|
||||
> **2026-10-07 (late evening) — pre-night fixes, the night moved to 7→8 (`09` §3 174–176).** Agent v0.153.0 (ring 0 stages
|
||||
> exactly the told kernel — R-898 closed), controller v0.303.0 (`backup/driveready.go`: captures wait ≤10 min after start
|
||||
> for a live drive mount — R-897 closed), hub v0.143.1 (`reply_to` = operator on every household mail). Armed: demo-felhom
|
||||
|
||||
@@ -1,34 +1,30 @@
|
||||
# REPORT — three fixes before the first real kernel night, 2026-10-07 (evening)
|
||||
# REPORT — read-back of the first real kernel night (2026-10-07 → 08)
|
||||
|
||||
The brief said 2026-10-08; at 18:34 it was 2026-10-07. Asked; the operator answered *"Start today, and do both boxes this
|
||||
night"* (`09` §3 decision 176). So the first real kernel night is **7→8**, read back on the morning of 2026-10-08.
|
||||
|
||||
| Part | Result |
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| Rulings | Decisions 174 (mail text stays + a reply address), 175 (window 09–20, max 3 mails: keep), 176 (today, both boxes tonight) recorded FIRST (`a0ff737e`). |
|
||||
| A — the told kernel boots (R-898) | **Done, delivered, closed.** Agent v0.153.0: ring 0 stages EXACTLY the told kernel (select `listed`, the set from the kver). Told 20 / sources offer 22 → 20 installs (wrapper test); 20 gone → R7 before any change (wrapper test); the hub keeps R7 temporary and tells the household again for the newer kernel (hub test). Red-proofs `audits/kernel-night-2026-10-07/A/redproof.txt`. |
|
||||
| B — first backup after a restart waits (R-897) | **Done, delivered, closed.** The reviewer's pick, taken: controller v0.303.0 — for 10 min after the controller starts, a capture for an app on a drive runs only once that drive is a LIVE mount in the controller's own namespace (`/proc/self/mountinfo`, the signal the startup app gate already uses); the 5-min refresh skips, a data run waits; after 10 min it runs and logs once. The agent's bind order is unchanged. Red test: not-bound → no capture; bound → capture. |
|
||||
| C — a reply reaches a person | **Built and delivered (hub v0.143.1); the header read-back is NOT done.** Every mail to a household now carries `reply_to` = `admin@felhom.eu` (all household templates invite contact: the kernel notice, the event sign-off "contact your operator", the setup/link mails); a mail to the operator carries none. Test pinned and red-proved. One test mail sent through the mail service with the hub's exact fields to `tester1@felhom.eu` — it arrived in Gmail (16:54 UTC). **Read-back tried:** the Gmail tool returns no headers (METADATA_ONLY, no RAW); the mail service's read API refused (`restricted_api_key` — send-only, correctly). Note: the demo households' own address IS `admin@felhom.eu`, so their mails carry no Reply-To by design. |
|
||||
| D — deliver, arm the night | **Done.** Hub 0.143.1 deployed 18:54 local (operator attending), agent 0.153.0 on all three boxes 19:03 (binary only — no root file changed), controller 0.303.0 on all three 18:50 (per-customer floors; global untouched). **Armed:** demo-felhom → **7.0.14-20-pve**, household mail **18:12**; demo-hp → **7.0.14-22-pve**, household mail **18:38**. No reboot by hand. |
|
||||
| demo-felhom — step ran | **PASS.** Whole-guest backup 04:35; guest/host/Proxmox steps 04:37–04:38; kernel **staged 04:39:04 as exactly the told 7.0.14-20** (the sources offered 7.0.14-22 — R-898's fix held); restart 04:39:08; boot on 7.0.14-20; judged healthy **04:40:38** (38 s after the agent started); **7.0.14-20 is the default**, flag empty, step `good`. |
|
||||
| demo-felhom — apps | Away about **1.5 min** (restart 04:39:08 → every container healthy at the 04:40:38 verdict). |
|
||||
| demo-felhom — alarms | None: no `host_stale`, no backup failure mail (demo-felhom has no drive apps, so R-897's fix was not exercised there). Crash guard armed, 0 unclean boots. Hub logged staged → judging → applied. |
|
||||
| demo-hp — step ran | **NO.** No whole-guest backup that night: the last one was yesterday's morning press at 08:49:53; with the 24 h cadence it came due at 08:49 today, after the window closed (08:30). No backup → no night leg → no kernel step. Nothing changed: running and default 7.0.14-20. Filed **R-899**. |
|
||||
| Reply-To | **Proven by you:** your reply to the test mail (2026-10-07 20:29) went to admin@felhom.eu. |
|
||||
| Your inbox overnight | Only Tester 2's expected missed-backup mails (the laptop is off). |
|
||||
|
||||
**Rows: 128 before → 126 after. Opened 0. Closed 2 (R-897, R-898).**
|
||||
**Rows: 126 before → 127 after. Opened 1 (R-899). Closed 0.**
|
||||
|
||||
**How demo-hp got told tonight:** its apt lists were a day old, so its last report named no new kernel. I switched its OS
|
||||
updates OFF on the hub for ~1 minute, ran the agent's own report-only pass (`--selftest=os-update`; with the switch off it
|
||||
installs nothing and runs no Docker/Proxmox/kernel step), and switched it back ON (`kernel-night-2026-10-07/demo-hp-inventory-pass.txt`).
|
||||
The hub mailed 20 s after the host report arrived (18:37:51 → 18:38:11).
|
||||
**What happens next, by itself:**
|
||||
- demo-hp is still due 7.0.14-22. A new household mail is allowed after 14:38 today (20 h after the last); its
|
||||
whole-guest backup is due inside tonight's window → the step should run on the night 8→9.
|
||||
- demo-felhom now sees 7.0.14-22 pending → due; its household gets a mail after 09:00; it should step to 22 tonight too.
|
||||
- After that both run 7.0.14-22, and "Approve kernel set" can appear. Read back the morning of 2026-10-09.
|
||||
|
||||
**Releases:** agent v0.153.0 (binary `b204ebe6…`, tag at `2d1e5d0`); controller v0.303.0 (`29bebbb`, MinAgent 0.131.0);
|
||||
hub v0.143.1 (`87765bfa`, deployed `d1457892`, Synced/Healthy). CI green on every push, checked by commit. No change on
|
||||
ep0, Tester 2 or DooPlex's own system.
|
||||
**Seen, not fixed:** the after-boot "judging" report carries ring 1 (the agent reads its ring from the hub's block, which
|
||||
it has not fetched yet in the first second after a boot). Cosmetic: the hub's approval reads its own ring list, not
|
||||
this field. Not fixed today because it needs an agent release and delivery for a label.
|
||||
|
||||
**What the morning read-back must check:** each demo box on its told kernel as the default; the household apps back and
|
||||
the minutes they were down; no false backup failure after the restart (R-897's fix); the hub's kernel events. The two
|
||||
boxes run DIFFERENT kernels after tonight, so "Approve kernel set" waits until both boot the same one.
|
||||
Evidence: `documentation/audits/kernel-night-2026-10-07/readback/`.
|
||||
|
||||
## Decisions for you
|
||||
|
||||
1. **Check the reply address in one click:** open „TEST — Reply-To check" in Gmail and press Reply — it should go to
|
||||
admin@felhom.eu. My pick: do it once. If you do nothing: the code and its test say it works; nobody has seen it.
|
||||
2. **The demo households' address is your own**, so their kernel mails carry no Reply-To. My pick: leave it. If you do
|
||||
nothing: it stays (a reply to them lands in your catch-all anyway).
|
||||
1. **R-899 — a daytime "back up now" press skips the next night's backup (and its updates).** My pick: count nights,
|
||||
not 24 hours (the backup is due when the last one is older than ~20 h at the window's start). If you do nothing: each
|
||||
daytime press costs one night, and the kernel step waits one more day.
|
||||
|
||||
@@ -2,8 +2,18 @@
|
||||
|
||||
**Ready for the first real tester (Tester-2): yes. Tester 2 (a laptop) is off; nothing was sent to it.**
|
||||
|
||||
**Updated 2026-10-07 19:10: hub 0.143.1; demo-hp, demo-felhom and Tester 1 run agent 0.153.0 and controller 0.303.0.
|
||||
The open-items list is at 126. Report: `REPORT.md`.**
|
||||
**Updated 2026-10-08 06:50: hub 0.143.1; demo-hp, demo-felhom and Tester 1 run agent 0.153.0 and controller 0.303.0.
|
||||
The open-items list is at 127. Report: `REPORT.md`.**
|
||||
|
||||
## Morning (2026-10-08): the first real kernel night, read back
|
||||
|
||||
- **demo-felhom passed.** It restarted at 04:39 into the new kernel it was told about, was healthy in 38 seconds, and
|
||||
kept it. Apps away about 1.5 minutes. No alarm.
|
||||
- **demo-hp did not restart.** Its nightly full backup did not run, because yesterday's daytime "back up now" press moved
|
||||
its 24-hour clock past the night. Nothing changed on it. It should go tonight (filed as a small item).
|
||||
- **The reply address works:** your test reply went to admin@.
|
||||
|
||||
**Needs you:** one choice on the backup clock (see `REPORT.md`). If nothing: a daytime press costs one night.
|
||||
|
||||
## Tonight (2026-10-07 → 08): the first real kernel night
|
||||
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
2026-10-08T04:38:31+02:00 demo-felhom felhom-agent[206012]: time=2026-10-08T04:38:31.955+02:00 level=INFO msg="osupdate: DONE" run=20261008T023722Z layer=docker vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=
|
||||
2026-10-08T04:38:31+02:00 demo-felhom felhom-agent[206012]: time=2026-10-08T04:38:31.968+02:00 level=INFO msg="osupdate: pve step holds the /etc/pve write gate — the agent's own writes wait until it ends" run=20261008T023722Z layer=pve vmid=9201 tr
|
||||
2026-10-08T04:38:31+02:00 demo-felhom felhom-agent[206012]: time=2026-10-08T04:38:31.968+02:00 level=INFO msg="osupdate: START" run=20261008T023722Z layer=pve vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261008T023722Z
|
||||
2026-10-08T04:38:32+02:00 demo-felhom felhom-os-apply[725634]: os-apply: START release=ring0-20261008T023722Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring0
|
||||
2026-10-08T04:38:47+02:00 demo-felhom felhom-os-apply[726282]: os-apply: DONE rc=0 seconds=4.1 upgraded=5 restart-needed=pmxcfs,pve-firewall,pve-ha-crm,pve-ha-lrm,pvedaemon,pvedaemon worke,pveproxy,pveproxy worker,pvescheduler,pvestatd,rrdcached,spic
|
||||
2026-10-08T04:38:52+02:00 demo-felhom felhom-agent[206012]: time=2026-10-08T04:38:52.894+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261008T023722Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 a
|
||||
2026-10-08T04:38:52+02:00 demo-felhom felhom-agent[206012]: time=2026-10-08T04:38:52.895+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=4.1 upgraded=5 restart-needed=pmxcfs,pve-firewall,pve-ha-crm,pve-ha-lrm,pvedaemon,pved
|
||||
2026-10-08T04:38:53+02:00 demo-felhom felhom-agent[206012]: time=2026-10-08T04:38:53.688+02:00 level=INFO msg="osupdate: DONE" run=20261008T023722Z layer=pve vmid=9201 ring=0 trigger=night outcome=applied healthy=true reason="" upgraded=5 pending=0 n
|
||||
2026-10-08T04:38:53+02:00 demo-felhom felhom-agent[206012]: time=2026-10-08T04:38:53.714+02:00 level=INFO msg="osupdate: pve step released the /etc/pve write gate" run=20261008T023722Z layer=pve vmid=9201 trigger=night
|
||||
2026-10-08T04:38:54+02:00 demo-felhom felhom-os-apply[726516]: os-apply: START release=ring0-20261008T023722Z layer=kernel lane=slow mode=apply select=listed authority=ring0 running=7.0.2-6-pve
|
||||
2026-10-08T04:39:02+02:00 demo-felhom felhom-os-apply[727201]: os-apply: KERNEL default = 7.0.2-6-pve (proved from grub.cfg)
|
||||
2026-10-08T04:39:04+02:00 demo-felhom felhom-os-apply[727259]: os-apply: KERNEL STAGED 7.0.2-6-pve -> 7.0.14-20-pve (default 7.0.2-6-pve, one-shot flag 7.0.14-20-pve); upgraded=1 seconds=1.9
|
||||
2026-10-08T04:39:04+02:00 demo-felhom felhom-agent[206012]: time=2026-10-08T04:39:04.532+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261008T023722Z layer=kernel lane=slow mode=apply select=listed authority=ring0 run
|
||||
2026-10-08T04:39:04+02:00 demo-felhom felhom-agent[206012]: time=2026-10-08T04:39:04.532+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL default = 7.0.2-6-pve (proved from grub.cfg)"
|
||||
2026-10-08T04:39:04+02:00 demo-felhom felhom-agent[206012]: time=2026-10-08T04:39:04.533+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL STAGED 7.0.2-6-pve -> 7.0.14-20-pve (default 7.0.2-6-pve, one-shot flag 7.0.14-20-pve); upgraded=
|
||||
2026-10-08T04:39:04+02:00 demo-felhom felhom-agent[206012]: time=2026-10-08T04:39:04.533+02:00 level=INFO msg="osupdate: DONE" run=20261008T023722Z layer=kernel vmid=9201 trigger=night ring=0 outcome=staged healthy=true reason="" upgraded=1 pending=0
|
||||
2026-10-08T04:39:08+02:00 demo-felhom felhom-os-apply[727378]: os-apply: KERNEL REBOOT — one-shot boot of 7.0.14-20-pve (the default stays 7.0.2-6-pve)
|
||||
2026-10-08T04:39:08+02:00 demo-felhom felhom-agent[206012]: time=2026-10-08T04:39:08.515+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL REBOOT — one-shot boot of 7.0.14-20-pve (the default stays 7.0.2-6-pve)"
|
||||
---BOOT
|
||||
2026-10-08T04:39:59+02:00 demo-felhom felhom-agent[1268]: time=2026-10-08T04:39:59.718+02:00 level=INFO msg="felhom-agent daemon starting" version=0.153.0 host_id=demo-felhom-8363b5 hub_url=https://hub.felhom.eu interval_s=900
|
||||
2026-10-08T04:39:59+02:00 demo-felhom felhom-agent[1268]: time=2026-10-08T04:39:59.852+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL AFTER-BOOT oneshot -> judging running=7.0.14-20-pve (7.0.2-6-pve -> 7.0.14-20-pve)"
|
||||
2026-10-08T04:39:59+02:00 demo-felhom felhom-agent[1268]: time=2026-10-08T04:39:59.853+02:00 level=INFO msg="osupdate: kernel step — judging the one-shot boot" run=boot-20261008T023959Z layer=kernel vmid=0 from=7.0.2-6-pve to=7.0.14-20-pve wait=20m
|
||||
2026-10-08T04:39:59+02:00 demo-felhom felhom-agent[1268]: time=2026-10-08T04:39:59.942+02:00 level=INFO msg="osupdate: kernel step — the box reached the hub on the new kernel" run=boot-20261008T023959Z layer=kernel vmid=0 after=0s
|
||||
2026-10-08T04:40:38+02:00 demo-felhom felhom-os-apply[4716]: os-apply: KERNEL default = 7.0.14-20-pve (proved from grub.cfg)
|
||||
2026-10-08T04:40:38+02:00 demo-felhom felhom-os-apply[4718]: os-apply: KERNEL GOOD 7.0.14-20-pve is the default now (was 7.0.2-6-pve)
|
||||
2026-10-08T04:40:38+02:00 demo-felhom felhom-agent[1268]: time=2026-10-08T04:40:38.269+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL default = 7.0.14-20-pve (proved from grub.cfg)"
|
||||
2026-10-08T04:40:38+02:00 demo-felhom felhom-agent[1268]: time=2026-10-08T04:40:38.269+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL GOOD 7.0.14-20-pve is the default now (was 7.0.2-6-pve)"
|
||||
2026-10-08T04:40:38+02:00 demo-felhom felhom-agent[1268]: time=2026-10-08T04:40:38.269+02:00 level=INFO msg="osupdate: kernel step — applied" run=boot-20261008T023959Z layer=kernel vmid=0 to=7.0.14-20-pve reason="healthy 38s after the agent started
|
||||
2026-10-08T04:40:38+02:00 demo-felhom felhom-agent[1268]: time=2026-10-08T04:40:38.269+02:00 level=INFO msg="osupdate: DONE" run=boot-20261008T023959Z layer=kernel vmid=0 outcome=applied healthy=true reason="healthy 38s after the agent started; the n
|
||||
---GUEST
|
||||
2026-10-08T04:40:00+02:00 demo-felhom systemd[1]: Starting pve-guests.service - PVE guests...
|
||||
2026-10-08T04:40:01+02:00 demo-felhom pve-guests[1451]: <root@pam> starting task UPID:demo-felhom:000005C7:000004F9:6AC70281:startall::root@pam:
|
||||
2026-10-08T04:40:01+02:00 demo-felhom pvesh[1451]: Starting CT 9201
|
||||
2026-10-08T04:40:01+02:00 demo-felhom pve-guests[1479]: <root@pam> starting task UPID:demo-felhom:000005C8:000004FA:6AC70281:vzstart:9201:root@pam:
|
||||
2026-10-08T04:40:01+02:00 demo-felhom pve-guests[1480]: starting CT 9201: UPID:demo-felhom:000005C8:000004FA:6AC70281:vzstart:9201:root@pam:
|
||||
{
|
||||
"authority": "ring0",
|
||||
"default_before": "7.0.2-6-pve",
|
||||
"from": "7.0.2-6-pve",
|
||||
"health_before": {
|
||||
"guest": {
|
||||
"containers": {
|
||||
"cloudflared": {
|
||||
"health": "healthy",
|
||||
"id": "bdbd524f10b0e4c37dab5d1faa7df84f820cb3b58ad6dbc8ab2dcb73902fe12d",
|
||||
"state": "running"
|
||||
},
|
||||
"felhom-controller": {
|
||||
"health": "healthy",
|
||||
"id": "dd0ff16d55df84742838fb39ca6e810cde6e20bc70bf36c40d16ac35bc6c2fbf",
|
||||
"state": "running"
|
||||
},
|
||||
"filebrowser": {
|
||||
"health": "healthy",
|
||||
"id": "6bda22106919fbef251af5494395af05b34e095d5e4187b90af85af69470ffa4",
|
||||
"state": "running"
|
||||
},
|
||||
"opengist": {
|
||||
"health": "healthy",
|
||||
"id": "1c0e86675b1353a094361c2519236c41e9ca6502e81be91b2ec53f9ad9de0c5d",
|
||||
"state": "running"
|
||||
},
|
||||
"traefik": {
|
||||
"health": "none",
|
||||
"id": "ddc8abb0ce45574cb0aad245db8c42641139114fe075d06d7f849ffd0441524e",
|
||||
"state": "running"
|
||||
}
|
||||
},
|
||||
"controller": "healthy",
|
||||
"controller_docker_ok": true,
|
||||
"docker_ok": true,
|
||||
"network_ok": true
|
||||
},
|
||||
"guest_running": true,
|
||||
"host_services": {
|
||||
"felhom-agent": "active",
|
||||
"pve-cluster": "active",
|
||||
"pvedaemon": "active",
|
||||
"pveproxy": "active",
|
||||
"pvestatd": "active"
|
||||
}
|
||||
},
|
||||
"judging_since": "2026-10-08T02:39:59Z",
|
||||
"packages": [
|
||||
{
|
||||
"name": "proxmox-kernel-7.0",
|
||||
"version": "7.0.14-20"
|
||||
}
|
||||
],
|
||||
"phase": "good",
|
||||
"rebooted_at": "2026-10-08T02:39:08Z",
|
||||
"result_at": "2026-10-08T02:40:38Z",
|
||||
"self_revert_used": false,
|
||||
"staged_at": "2026-10-08T02:39:04Z",
|
||||
"step_id": "ring0-20261008T023722Z",
|
||||
"to": "7.0.14-20-pve",
|
||||
"updated_at": "2026-10-08T02:40:38Z",
|
||||
"vmid": 9201
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
2026-10-07T04:40:48+02:00 demo-hp felhom-agent[297545]: time=2026-10-07T04:40:48.072+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_10_07-04_35_11.tar.zst size_b
|
||||
2026-10-07T08:49:53+02:00 demo-hp felhom-agent[297545]: time=2026-10-07T08:49:53.496+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_10_07-08_45_01.tar.zst size_b
|
||||
demo-hp last whole-guest backup 2026-10-07 08:49:53 (the morning 'Mentes most' press, R-518 read-back) + 24 h cadence = due 2026-10-08 08:49, after the window [04:30, 08:30) closed -> no backup -> no night leg -> no kernel step. No kernel state written; running and default 7.0.14-20.
|
||||
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user