Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
41d5eb8051 | ||
|
|
8bd137787e | ||
|
|
39db03065a | ||
|
|
ba409de4a2 | ||
|
|
a4f1280399 | ||
|
|
ecbb0cc479 | ||
|
|
50eeccc986 | ||
|
|
808cdb36ad | ||
|
|
84726c9f27 |
No files matched your search
+32
-1
@@ -1,4 +1,35 @@
|
||||
## Unreleased — R-1344 decided from the night's measurement: a controller that runs but does not answer is RESTARTED
|
||||
## v0.172.1 (2026-11-07) — proxmox-grub joins the boot-chain lane: the nightly pve step stops being refused R6 — RELEASED and VOUCHED 2026-11-07 (release-agent.sh from `39db030`; agent sha256 505a0594a2fb117f359e329a0b7ae2c7a6902b74c51f52c1883145f3c2ac72f8, bundle 3e7acaee4500893bfaf567b1d427790ef89bf41f08cb6a1419ee528aeb40ddc3)
|
||||
|
||||
- **The defect (measured on demo-felhom, 2026-11-05, 11-06, 11-07).** Proxmox published `proxmox-grub` 2.12-9+pmx4. It
|
||||
depends on `grub-efi-amd64` of the same version. `HOST_SLOW_RE` matches `grub` only at the START of a name, so
|
||||
`proxmox-grub` landed in the ring-0 pve plan; the simulation then pulled `grub-efi-amd64`, which is not in the plan,
|
||||
and `felhom-os-apply` refused the WHOLE pve step (R6 „the plan would touch grub-efi-amd64") every night — 34 Proxmox
|
||||
userspace upgrades held with it, and an `os_update_failed` error mail each morning. The same class as the 2026-10-09
|
||||
`proxmox-secure-boot-support` fix. Read-only proof on the box: `apt-get -s install --only-upgrade proxmox-grub` →
|
||||
`Inst grub-efi-amd64 …pmx4`.
|
||||
- **The fix.** `proxmox-grub` joins `HOST_SLOW_RE`: GRUB's meta-package now waits with the rest of the boot chain (the
|
||||
slow lane, the operator's signed step), and the pve step runs without it.
|
||||
- **Test** `test_ring0_pending_pve_leaves_proxmox_grub_out`; red seen before the fix (proxmox-grub entered the plan), green
|
||||
after; the wrapper suite 149 of 149.
|
||||
- **Delivered 2026-11-07 (operator, signed jobs):** demo-felhom, demo-hp, Tester 1 — `agent_update` committed
|
||||
(23:05/23:13/23:13 CET), then the bundle: `BUNDLE DONE agent=0.172.1 written=1 same=30 self-check=ok`, capability probe
|
||||
77/77 on each. `pinned.day0_agent` = 0.172.1 (D50).
|
||||
- **Delivery.** `configs/` changed, so a box needs BOTH signed jobs: `agent_update` 0.172.1, then `agent_config_update`
|
||||
0.172.1 with the bundle sha. Not vouched by CC. Bundle vs 0.170.0 (what the boxes run): no path added or removed, only
|
||||
`felhom-os-apply` changed — so no step bundle is needed (R-880). Both downloads re-checked by sha.
|
||||
|
||||
## v0.172.0 (2026-11-03) — R-1302: the first grant of a tier is no loss — RELEASED and VOUCHED 10:18:10 CET (release-agent.sh from a clean clone of `ecbb0cc`; agent sha256 2c0222cd01713f65c627d6d17c18b4a8b2b1e0c39881db0df7ccb17de48e24c7, config bundle cc34b0bc3346ffd6a4ba80fe9269852dced8c56c9503c8b14a50d9ded2961cce; configs/ unchanged since 0.171.0, so the signed delivery was agent_update alone)
|
||||
|
||||
- **R-1302 — the first grant of a tier is no loss.** A fresh install arms the whole-guest tier before its storage exists, so the
|
||||
first probe found no grant, repaired it and logged ERROR „GRANT WAS MISSING … investigate the loss (R-190)" plus a DEGRADED
|
||||
capability, on every new box (VM 9375, 2026-10-26). The agent now remembers which tiers have EVER held their grant
|
||||
(`<state dir>/store-grant-seen.json`, written by a healthy probe). A first grant logs INFO „granted" and reports healthy; a
|
||||
grant that existed and vanished keeps R-190's ERROR and its one degraded cycle. A doubt keeps the ERROR: an unreadable or
|
||||
corrupt file, or no file path. A box updated to this agent records its tiers at the first healthy probe, so a loss in the
|
||||
one probe before that would read as a first grant. Tests `TestR1302_*`; red seen with the first-grant branch removed and on
|
||||
three more mutants.
|
||||
|
||||
## v0.171.0 (2026-10-30) — R-1344 decided from the night's measurement: a controller that runs but does not answer is RESTARTED — RELEASED (release-agent.sh from a clean clone of `bd18347`; agent sha256 c888ca7ba22a1f689059081575814a725e2edcecbb6bcc56829e9d9de88202a5, config bundle 95f977f6dc827185f2b5a2aa9e1c829db41a9f425d49eb40f317286d13c83d2b; binary-only signed agent_update on Tester 1, demo-hp, demo-felhom; vouched by the operator 2026-10-30 10:47:36 CET; `pinned.day0_agent` = 0.171.0). Live on Tester 1: the controller frozen (SIGSTOP) 09:24:50 UTC → Docker „unhealthy" 09:26:36 → the alarm 09:29:21 → the restart 09:31:22 → healthy 09:31:24; the hub mailed `controller_unresponsive` and `controller_unresponsive_restart` (hub 0.178.0), not `controller_restarted_by_agent`
|
||||
|
||||
- **The measurement (operator brief 2026-10-29 Part 2, read 2026-10-30).** Docker's own health record of the controller
|
||||
on Tester 1 and demo-hp, every probe from 2026-10-29 11:12 to 2026-10-30 07:00 UTC: 4 733 probes through the night
|
||||
|
||||
+15
-1
@@ -1,12 +1,26 @@
|
||||
# CONTEXT — felhom-agent working state
|
||||
|
||||
> **2026-11-07 (attended) — v0.172.1 RELEASED and VOUCHED; on the three test boxes (agent + bundle, 77/77).** `proxmox-grub` joins `HOST_SLOW_RE` (the N100's ring-0 pve step was refused R6 three nights: proxmox-grub pmx4 pulls grub-efi-amd64). sha `505a0594…`, bundle `3e7acaee…` (only `felhom-os-apply` changed vs 0.170.0, no step bundle). Done: vouch, both signed jobs on the three boxes, `pinned.day0_agent` = 0.172.1. Next: the N100's pve step on its next night (positive line: `os-apply: DONE` for `layer=pve:9201`, no R6).
|
||||
|
||||
> **2026-11-03 (day, attended) — v0.172.0 RELEASED and VOUCHED (10:18:10 CET), R-1302.** `2c0222cd…`, bundle `cc34b0bc…`, `pinned.day0_agent` = 0.172.0; on the three test boxes by signed `agent_update`. The appliance path pre-positions the grant, so a plain first start logs neither line; staged live both ways (felhom.eu `audits/release-2026-11-03/part3/`).
|
||||
>
|
||||
> **2026-10-30 (day, attended) — v0.171.0 RELEASED and VOUCHED (10:47:36 CET): R-1344 decided — a controller that runs
|
||||
> but does not answer is RESTARTED.** The night's measurement (4 733 probes, slowest 303 ms, no failure) licensed it:
|
||||
> alarm at `controllerUnresponsiveConfirm` = 6 unhealthy sweeps, restart at `controllerUnresponsiveRestart` = 10 —
|
||||
> refused by a fresh/unreadable busy file (`controllerBusy`, shared with D65's reboot) or the brake
|
||||
> (`supervisorBrakeAllows`, now shared with the not-running restart; `recordSupervisorRestart` records both). The stanza
|
||||
> carries `unresponsive_restart_after_s` and `unresponsive_restart_at` (hub 0.178.0 mails
|
||||
> `controller_unresponsive_restart`). Built from `bd18347` (`c888ca7b…`, bundle `95f977f6…`), binary-only signed
|
||||
> updates on the three test boxes; live on Tester 1 (SIGSTOP → alarm 4.5 min → restart 6.5 min → healthy).
|
||||
> `pinned.day0_agent` = 0.171.0. felhom.eu `REPORT-release-2026-10-30.md`.
|
||||
|
||||
> **2026-10-29 (evening, attended) — v0.170.0 RELEASED and VOUCHED (18:57:48 CET): `reboot_guest` (D65 = A).**
|
||||
> `da8d12a0…`, bundle `03f2ff64…`, tag `v0.170.0`, from a clean clone of `ca15a31`; configs/ changed (one new read-only
|
||||
> sudo vector: `cat` of the controller's `data/ctl/busy`), so per box a signed `agent_config_update` + `agent_update`
|
||||
> (Tester 1, demo-hp, demo-felhom; capability probe 76/76). The agent's closed list is `restart_controller` +
|
||||
> `reboot_guest` (`opreboot.go`; guards in felhom.eu `03`/`04` §6.1). Proven on Tester 1: refused during an off-site run
|
||||
> (the controller's busy file), done in 67 s / 79 s, refused a second time inside the hour. `pinned.day0_agent` = 0.170.0.
|
||||
> **R-1344 (the stuck-controller restart) is still ALARM ONLY** — the night's health log is read on 2026-10-30.
|
||||
> **R-1344 decided 2026-10-30: RESTART** (see the 2026-10-30 entry above).
|
||||
|
||||
> **2026-10-29 (afternoon, attended) — v0.169.0 RELEASED and VOUCHED (14:06:09 CET).** `fc066de6…`, bundle `3ca33f9c…`,
|
||||
> from `637c92c`; `pinned.day0_agent` 0.169.0 (`db25eb0`). The agent's own closed operator-action list:
|
||||
|
||||
@@ -1,36 +1,62 @@
|
||||
# REPORT — v0.170.0 released and vouched: „Reboot guest" from the hub (D65 = A, 2026-10-29 evening, attended)
|
||||
# REPORT — v0.172.1 released, vouched and delivered (2026-11-07, attended)
|
||||
|
||||
**Plain words first.** The operator can now reboot a box's guest from the hub's host page, beside „Restart controller".
|
||||
The agent does it (`pct reboot` of that guest — never the host). It refuses during a backup, a restore, an update or a
|
||||
controller swap, more than once an hour, and under the restart brake. The result says when the guest and the controller
|
||||
were back. The controller that starts after the reboot tells the household for a day that Felhom support restarted
|
||||
the box.
|
||||
**Plain words first.** Since 2026-11-05 the N100 refused its nightly Proxmox updates. One GRUB package
|
||||
(`proxmox-grub`) was in the wrong update group, and it pulled the main GRUB package with it. A safety check then
|
||||
stopped the whole group. Agent 0.172.1 puts `proxmox-grub` with the other boot packages. The operator vouched it and
|
||||
sent both signed jobs; all three test boxes run it.
|
||||
|
||||
## Commits
|
||||
| | |
|
||||
|---|---|
|
||||
| Built from | `39db030`, `release-agent.sh` (gates green; tag `v0.172.1` pushed) |
|
||||
| Agent sha256 | `505a0594a2fb117f359e329a0b7ae2c7a6902b74c51f52c1883145f3c2ac72f8` |
|
||||
| Bundle sha256 | `3e7acaee4500893bfaf567b1d427790ef89bf41f08cb6a1419ee528aeb40ddc3` |
|
||||
| Re-checked | both downloaded again and hashed; the binary prints `felhom-agent 0.172.1`; the bundle's `felhom-os-apply` holds the fix |
|
||||
| Bundle vs 0.170.0 | no path added or removed; only `felhom-os-apply` changed — no step bundle needed (R-880) |
|
||||
| Test | `test_ring0_pending_pve_leaves_proxmox_grub_out` — red before the fix, green after; wrapper suite 149/149; `go test -count=1 ./internal/osupdate/` ok |
|
||||
| Design | `felhom.eu/documentation/architecture/11-os-updates.md` (the lanes, §5.4.2 bundle) |
|
||||
|
||||
| Commit | CI (job id) | What |
|
||||
|---|---|---|
|
||||
| `ca15a31` | 2520, 2524 green | `reboot_guest` in the closed list (`opreboot.go`): guards (provisioned, running, not locked, not parked; the heavy-operation gate held until the guest is up; `jobBlocker`; the controller's busy file, fresh ≤ 90 s, fail closed on any read error but „no file"; one per hour per guest, persisted; the restart brake counts it); `rebootInFlight` keeps the guest-power watchdog and the supervisor away; the result with both times; `GET /guest/operator-reboot` for the household line; one new read-only sudo vector (`cat` of the controller's busy file) + capability `controller-busy-read`. 19 mutants, each killed (one survivor found and fixed: a duplicate swap check). |
|
||||
| `6578a8b` | 2525 green | v0.170.0 CHANGELOG heading; `pinned.day0_agent` = 0.170.0 (D50), after the vouch. |
|
||||
**Evidence.** `felhom.eu/documentation/audits/demo-check-2026-11-07/n100-pve-step-R6.txt`: the three nights' refusals
|
||||
and a read-only `apt-get -s` that shows `proxmox-grub` pulling `grub-efi-amd64`.
|
||||
|
||||
## Release
|
||||
## For the operator — deliver it (PowerShell, on the Windows machine)
|
||||
|
||||
- `release-agent.sh 0.170.0` from a clean clone of `ca15a31`: sha256 `da8d12a05ef607c338b9ba307e7a81b18253a3188721bbd8421ba36658a22404`,
|
||||
bundle `03f2ff6442fc3ca600b1f25f26767ce4da466a770a258f3e362cf95f6e76b543`, tag `v0.170.0`, verified by download.
|
||||
- configs/ changed, so per box a signed `agent_config_update` AND `agent_update` (Tester 1, demo-hp, demo-felhom): all
|
||||
six COMPLETED; each box „config bundle INSTALLED", capability probe 76/76 ok, „update committed".
|
||||
- Vouched by the operator 2026-10-29 18:57:48 CET (the save took 38.3 s — felhom.eu R-1346).
|
||||
0. Hub → Configuration → Day-0 artifacts: vouch agent **0.172.1** (sha and bundle above).
|
||||
1. Check the key. The last word before the comment must be `…B63tlLw+` (the pinned signer `felhom-op-1`):
|
||||
```powershell
|
||||
Get-Content $HOME\.ssh\felhom-op-operational.pub
|
||||
```
|
||||
2. Set up (the hub key is the hub's global key, the `report-api` one; it is asked, not typed on a line):
|
||||
```powershell
|
||||
cd $HOME\.ssh
|
||||
$Hub = "https://hub.felhom.eu"
|
||||
$Ver = "0.172.1"
|
||||
$Sha = "505a0594a2fb117f359e329a0b7ae2c7a6902b74c51f52c1883145f3c2ac72f8"
|
||||
$Bundle = "3e7acaee4500893bfaf567b1d427790ef89bf41f08cb6a1419ee528aeb40ddc3"
|
||||
$Key = "$HOME\.ssh\felhom-op-operational"
|
||||
$Boxes = "demo-felhom-8363b5", "demo-hp-bb76ea", "tester-1-d70be4"
|
||||
$s = Read-Host "Hub global key" -AsSecureString
|
||||
$HubKey = [System.Net.NetworkCredential]::new("", $s).Password
|
||||
```
|
||||
3. The agent (each box takes it in 4–15 min):
|
||||
```powershell
|
||||
foreach ($b in $Boxes) { .\felhom-opsign.exe -op agent_update -host $b -key-id felhom-op-1 -key $Key -agent-version $Ver -sha256 $Sha -ttl 45m -upload $Hub -hub-key $HubKey }
|
||||
```
|
||||
4. Wait until the hub's System page shows agent 0.172.1 on all three (about 15 min). Then the root files:
|
||||
```powershell
|
||||
foreach ($b in $Boxes) { .\felhom-opsign.exe -op agent_config_update -host $b -key-id felhom-op-1 -key $Key -agent-version $Ver -bundle-sha256 $Bundle -ttl 45m -upload $Hub -hub-key $HubKey }
|
||||
```
|
||||
5. Clean up: `Remove-Variable HubKey, s`
|
||||
|
||||
## Live proofs (Tester 1, through the hub endpoint the button posts, CC's `cc-hosts` token; controller 0.349.0-rc.1)
|
||||
**Delivered (read back on each box):** `agent_update` committed on demo-felhom 23:05, demo-hp 23:13, Tester 1 23:13 CET;
|
||||
then `BUNDLE DONE agent=0.172.1 written=1 same=30 self-check=ok` and capability probe 77/77 on all three; the installed
|
||||
`felhom-os-apply` holds the fix. The first operator run of step 4 failed: the operator's `felhom-opsign.exe` was from July,
|
||||
before `-bundle-sha256` existed; a fresh Windows build (`/mnt/5_hdd/felhom.eu/build/opsign-2026-11-07/`, sha256
|
||||
`4527edba…`) worked. Note: the repo's own `felhom-opsign.exe` (June) is equally stale. `pinned.day0_agent` = 0.172.1 (D50).
|
||||
|
||||
- **Refused during an off-site run:** `offsite_backup_now` pressed 18:03:36 UTC → the controller's busy file 11 s later →
|
||||
„Reboot guest" → „refused: the controller is busy — a backup, a restore or an off-site run is running"; the guest kept
|
||||
its 13 h uptime.
|
||||
- **The reboot:** pressed 18:05:08 UTC → „done — rebooted; the guest was up after 67 s, the controller answered after
|
||||
79 s", on the hub 88 s after the press. Box walk after: 10 containers, none unhealthy, 0 errors, login and file manager
|
||||
200. The household dashboard: „A Felhom ügyfélszolgálata 19:05-kor újraindította a dobozt."
|
||||
- **A second press inside the hour:** „refused: guest 9201 was rebooted from the hub at 18:05 UTC — at most one reboot
|
||||
per hour (next from 19:05 UTC)".
|
||||
- The release-build reboot: see felhom.eu `REPORT-release-2026-10-29c.md`.
|
||||
**Not yet observed:** the N100's next nightly pve step. Positive observable: `os-apply: DONE` for `layer=pve:9201`, no R6.
|
||||
|
||||
Evidence: felhom.eu `documentation/audits/release-2026-10-29c/`.
|
||||
## Also in this session (no agent change)
|
||||
|
||||
- demo-hp: the whole-guest LOCAL tier had been refused by the space preflight for 9 days. With the operator's yes,
|
||||
the journal was shrunk to 500 MB (whole journal saved first, off-git on DooPlex) and the August test archive of
|
||||
guest 9100 removed: 9.5 GB → 14 GB free. Recorded under R-548 (the product half stays open).
|
||||
@@ -521,6 +521,66 @@ type storeGrantRepairer struct {
|
||||
mu sync.Mutex
|
||||
last map[string]time.Time // target id → last ATTEMPT (success or failure)
|
||||
repaired map[string]time.Time // target id → last CONFIRMED repair (drives the report latch)
|
||||
// seenPath (R-1302) is the file that remembers which tiers have EVER held the grant: only those can LOSE it. "" = no
|
||||
// memory, so every missing grant reads as a loss — the old, loud behaviour.
|
||||
seenPath string
|
||||
}
|
||||
|
||||
// everHeld reports whether this tier's grant was ever seen on this box (R-1302). The direction of a doubt is „yes": an
|
||||
// unreadable or corrupt file keeps the loss ERROR, so a real loss is never quietened by a broken record.
|
||||
func (r *storeGrantRepairer) everHeld(target string) bool {
|
||||
if r == nil || r.seenPath == "" {
|
||||
return true
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
seen, err := readGrantSeen(r.seenPath)
|
||||
return err != nil || seen[target]
|
||||
}
|
||||
|
||||
// noteHeld records that this tier holds the grant. A write that fails is logged and costs one false „first grant" at most
|
||||
// after a restart — the old file, or none, stands.
|
||||
func (r *storeGrantRepairer) noteHeld(target string) {
|
||||
if r == nil || r.seenPath == "" {
|
||||
return
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
seen, err := readGrantSeen(r.seenPath)
|
||||
if err != nil {
|
||||
seen = map[string]bool{} // a corrupt file is replaced by what is true now
|
||||
}
|
||||
if seen[target] {
|
||||
return
|
||||
}
|
||||
seen[target] = true
|
||||
b, _ := json.Marshal(seen)
|
||||
tmp := r.seenPath + ".tmp"
|
||||
if werr := os.WriteFile(tmp, b, 0o600); werr == nil {
|
||||
werr = os.Rename(tmp, r.seenPath)
|
||||
if werr == nil {
|
||||
return
|
||||
}
|
||||
err = werr
|
||||
} else {
|
||||
err = werr
|
||||
}
|
||||
repairLogger(r).Warn("store-grant: could not record that this tier holds its grant (R-1302)", "target", target, "err", err)
|
||||
}
|
||||
|
||||
func readGrantSeen(path string) (map[string]bool, error) {
|
||||
b, err := os.ReadFile(path)
|
||||
if os.IsNotExist(err) {
|
||||
return map[string]bool{}, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
if err := json.Unmarshal(b, &seen); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return seen, nil
|
||||
}
|
||||
|
||||
// noteRepaired latches a confirmed repair so it is reported for storeGrantRepairReportWindow.
|
||||
@@ -631,6 +691,7 @@ func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string,
|
||||
// host-report has certainly carried it. Without this latch the repairing probe may be a
|
||||
// log-only one and the hub never learns anything happened (measured live, see the window's
|
||||
// comment).
|
||||
repair.noteHeld(targetID) // R-1302: a tier that holds its grant can lose it from now on
|
||||
return storeGrantHealthyVerdict(targetID, critical, s, repair.recentlyRepaired(targetID, time.Now()))
|
||||
}
|
||||
|
||||
@@ -679,8 +740,25 @@ func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string,
|
||||
// produces exactly one alert pair and the operator learns of it. NOTHING NEW WAS BUILT: no wire
|
||||
// change, no hub change, no new event type. The `Feature` text carries the explanation because
|
||||
// that is the field the hub puts in the operator's e-mail (the Reason does not travel).
|
||||
return storeGrantAfterRepair(targetID, critical, repair, s)
|
||||
}
|
||||
|
||||
// storeGrantAfterRepair is the verdict once a repair is confirmed (split out for the tests). R-1302: the FIRST grant of a
|
||||
// tier that never held one is no loss — on a fresh install the whole-guest tier is armed before its storage exists, so
|
||||
// every new box logged „investigate the loss" and a DEGRADED capability at its first start, which teaches the reader to
|
||||
// skip the line. That grant logs INFO and reports healthy. A tier seen holding its grant, or one this box cannot tell
|
||||
// about, keeps R-190's ERROR and its one degraded cycle. Pinned by TestR1302_*.
|
||||
func storeGrantAfterRepair(targetID string, critical bool, repair *storeGrantRepairer, healthy capability.Status) capability.Status {
|
||||
if !repair.everHeld(targetID) {
|
||||
repair.noteHeld(targetID)
|
||||
repairLogger(repair).Info("store-grant: granted — the first grant of this tier, nothing was lost (R-1302)",
|
||||
"target", targetID, "privilege", storeGrantRequiredPriv,
|
||||
"action", "felhom-backup-target-apply grant "+targetID, "confirmed_by", "re-read")
|
||||
healthy.Status, healthy.Reason = capability.StatusOK, ""
|
||||
return healthy
|
||||
}
|
||||
repair.noteRepaired(targetID, time.Now())
|
||||
s = storeGrantRepairedVerdict(targetID, critical)
|
||||
s := storeGrantRepairedVerdict(targetID, critical)
|
||||
repairLogger(repair).Error("store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)",
|
||||
"target", targetID, "privilege", storeGrantRequiredPriv,
|
||||
"action", "felhom-backup-target-apply grant "+targetID, "confirmed_by", "re-read")
|
||||
@@ -853,8 +931,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// The runner is the DIRECT one for the same reason the sudo prober uses it: the wrapper is
|
||||
// invoked through the privileged path, which prepends sudo itself.
|
||||
grantRepairer := &storeGrantRepairer{
|
||||
run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
|
||||
log: logger,
|
||||
run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
|
||||
log: logger,
|
||||
seenPath: filepath.Join(agentStateDir(cfg), "store-grant-seen.json"), // R-1302
|
||||
}
|
||||
// R-973: the local-API backup cadence cap's refusals ride the report as a CRITICAL status, so the hub's
|
||||
// operator-only whole_guest_backup_refused event (hub v0.159.0, R-1150) tells the operator. Late-bound like
|
||||
|
||||
@@ -1,11 +1,14 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"go/ast"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -415,3 +418,101 @@ func TestGrantRepair_TransitionSurvivesALaterProbe(t *testing.T) {
|
||||
"be missed entirely", storeGrantRepairReportWindow)
|
||||
}
|
||||
}
|
||||
|
||||
// ── R-1302 — the first grant of a tier is no loss ──────────────────────────────────────
|
||||
//
|
||||
// A fresh install arms the whole-guest tier before its storage exists, so its first probe found no grant, repaired it and
|
||||
// logged „GRANT WAS MISSING … investigate the loss" with a DEGRADED capability — on every new box (VM 9375, 2026-10-26).
|
||||
//
|
||||
// RED-PROOF (2026-11-02): seen failing before storeGrantAfterRepair existed (build), and with its first-grant branch removed
|
||||
// TestR1302_TheFirstGrantIsNoLoss fails on „a first grant must not report a loss".
|
||||
func r1302Repairer(t *testing.T, path string) (*storeGrantRepairer, *bytes.Buffer) {
|
||||
t.Helper()
|
||||
var buf bytes.Buffer
|
||||
return &storeGrantRepairer{run: (&fakeRepairRunner{}).Run, seenPath: path,
|
||||
log: slog.New(slog.NewTextHandler(&buf, nil))}, &buf
|
||||
}
|
||||
|
||||
func TestR1302_TheFirstGrantIsNoLoss(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
|
||||
r, buf := r1302Repairer(t, path)
|
||||
healthy := capability.Status{Name: "pve:store-grant:felhom-pbs", Status: capability.StatusDegraded, Reason: "missing", Critical: true}
|
||||
s := storeGrantAfterRepair("felhom-pbs", true, r, healthy)
|
||||
if s.Status != capability.StatusOK || s.Reason != "" {
|
||||
t.Fatalf("a first grant must not report a loss: status %q reason %q", s.Status, s.Reason)
|
||||
}
|
||||
got := buf.String()
|
||||
if strings.Contains(got, "investigate the loss") || strings.Contains(got, "level=ERROR") {
|
||||
t.Fatalf("a first grant must not log the loss ERROR: %s", got)
|
||||
}
|
||||
if !strings.Contains(got, "level=INFO") || !strings.Contains(got, "granted") || !strings.Contains(got, "felhom-pbs") {
|
||||
t.Fatalf("a first grant says so at INFO: %s", got)
|
||||
}
|
||||
if r.recentlyRepaired("felhom-pbs", time.Now()) {
|
||||
t.Fatal("a first grant must not arm the loss report's latch")
|
||||
}
|
||||
}
|
||||
|
||||
func TestR1302_AGrantThatExistedAndVanishedKeepsTheError(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
|
||||
first, _ := r1302Repairer(t, path)
|
||||
first.noteHeld("felhom-pbs") // a healthy probe, in an earlier run of the agent
|
||||
|
||||
r, buf := r1302Repairer(t, path) // the agent restarted: the memory is the file, not the process
|
||||
s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
|
||||
if s.Status != capability.StatusDegraded || !strings.Contains(s.Feature, "R-190") {
|
||||
t.Fatalf("a grant that existed and vanished must keep its degraded cycle; got %q / %q", s.Status, s.Feature)
|
||||
}
|
||||
if got := buf.String(); !strings.Contains(got, "level=ERROR") || !strings.Contains(got, "investigate the loss") {
|
||||
t.Fatalf("a grant that existed and vanished must keep its ERROR: %s", got)
|
||||
}
|
||||
if !r.recentlyRepaired("felhom-pbs", time.Now()) {
|
||||
t.Fatal("the loss must arm the report's latch, or the hub may never hear it")
|
||||
}
|
||||
}
|
||||
|
||||
// After a first grant the tier counts as held: the NEXT disappearance is a loss.
|
||||
func TestR1302_AfterTheFirstGrantALossIsALoss(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
|
||||
r, _ := r1302Repairer(t, path)
|
||||
storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
|
||||
s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
|
||||
if s.Status != capability.StatusDegraded {
|
||||
t.Fatalf("the second disappearance is a loss; got %q", s.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// The direction of a doubt: a broken record, a repairer with no record, or no repairer at all keep the loud way.
|
||||
func TestR1302_ADoubtKeepsTheError(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
|
||||
if err := os.WriteFile(path, []byte("{not json"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r, _ := r1302Repairer(t, path)
|
||||
if s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{}); s.Status != capability.StatusDegraded {
|
||||
t.Fatalf("a corrupt record must not quieten a loss; got %q", s.Status)
|
||||
}
|
||||
noMemory, _ := r1302Repairer(t, "")
|
||||
if s := storeGrantAfterRepair("felhom-pbs", true, noMemory, capability.Status{}); s.Status != capability.StatusDegraded {
|
||||
t.Fatalf("no record must not quieten a loss; got %q", s.Status)
|
||||
}
|
||||
if s := storeGrantAfterRepair("felhom-pbs", true, nil, capability.Status{}); s.Status != capability.StatusDegraded {
|
||||
t.Fatalf("no repairer must not quieten a loss; got %q", s.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// The wiring, read from the source: production names the file, and a healthy probe records the tier.
|
||||
func TestR1302_TheMemoryIsWired(t *testing.T) {
|
||||
b, err := os.ReadFile("main.go")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
src := string(b)
|
||||
if !strings.Contains(src, `seenPath: filepath.Join(agentStateDir(cfg), "store-grant-seen.json")`) {
|
||||
t.Fatal("main.go must give the repairer its file, or every first grant reads as a loss")
|
||||
}
|
||||
start := strings.Index(src, "func storeGrantStatus(")
|
||||
if start < 0 || !strings.Contains(src[start:start+3000], "repair.noteHeld(targetID)") {
|
||||
t.Fatal("a healthy probe must record the tier, or a real loss would read as a first grant")
|
||||
}
|
||||
}
|
||||
@@ -97,9 +97,12 @@ OOMCHECK_SETTLE = 2
|
||||
# so it belongs with them. Measured 2026-10-09 on demo-hp (Secure Boot on): left in the pve lane, its upgrade pulled
|
||||
# shim-signed-common, the whole pve step was refused R6, and the night's kernel step was skipped with it
|
||||
# (`audits/kernel-night-2026-10-08/`). Pinned by test_ring0_pending_pve_leaves_the_secure_boot_meta_out.
|
||||
# proxmox-grub is GRUB's Proxmox meta-package (it depends on grub-efi-amd64 of the same version); the `grub` above is
|
||||
# anchored at the start, so it missed it. Measured 2026-11-05..07 on demo-felhom: its upgrade pulled grub-efi-amd64 and
|
||||
# the whole pve step was refused R6 every night. Pinned by test_ring0_pending_pve_leaves_proxmox_grub_out.
|
||||
HOST_SLOW_RE = re.compile(r"^(linux-(image|headers|kbuild|modules|base)|proxmox-kernel|proxmox-default-kernel|pve-kernel|"
|
||||
r"pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr|"
|
||||
r"proxmox-secure-boot-support)")
|
||||
r"proxmox-secure-boot-support|proxmox-grub)")
|
||||
# restart_needed() leaves out processes whose cgroup line matches (grep basic regex). Host: the LXC guests' own
|
||||
# processes (`0::/lxc/<vmid>/...`) -- NOT lxc-start itself, whose cgroup is `0::/lxc.monitor/<vmid>` (measured
|
||||
# 2026-10-04 on demo-felhom: the old pattern "lxc" hid lxc-start with 20 deleted maps, so "reboot needed" stayed false
|
||||
|
||||
@@ -1494,6 +1494,24 @@ class PVELane(unittest.TestCase):
|
||||
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
|
||||
self.assertEqual(f.installed["proxmox-secure-boot-support"], "9.0.1")
|
||||
|
||||
# 2026-11-05..07, demo-felhom (EFI): proxmox-grub 2.12-9+pmx2 -> pmx4 depends on grub-efi-amd64 of the same version;
|
||||
# HOST_SLOW_RE's `grub` is anchored at the start, so proxmox-grub fell into the pve plan, the simulation pulled
|
||||
# grub-efi-amd64 and the whole pve step was refused R6 every night. It is a boot-chain package: left out.
|
||||
# RED-PROOF: drop proxmox-grub from HOST_SLOW_RE -> it is in the plan -> this test fails.
|
||||
def test_ring0_pending_pve_leaves_proxmox_grub_out(self):
|
||||
f = pve_fake(ring0=True)
|
||||
f.plan["select"], f.plan["packages"] = "pending-pve", []
|
||||
f.installed.update({"proxmox-grub": "2.12-9+pmx2"})
|
||||
f.pending_sim = [
|
||||
"Inst pve-manager [9.2.2] (9.2.21 Proxmox Debian Repository:stable [amd64])",
|
||||
"Inst proxmox-grub [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64])",
|
||||
"Inst grub-efi-amd64 [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64])",
|
||||
]
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
|
||||
self.assertEqual(f.installed["proxmox-grub"], "2.12-9+pmx2")
|
||||
|
||||
def test_select_pending_pve_needs_the_pve_layer(self):
|
||||
f = Fake()
|
||||
f.plan["layer"], f.plan["select"], f.plan["packages"] = "host", "pending-pve", []
|
||||
|
||||
@@ -24,10 +24,10 @@
|
||||
"readers": [
|
||||
"scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
|
||||
],
|
||||
"recorded": "2026-10-29",
|
||||
"recorded_by": "CC 2026-10-29: day0_agent 0.170.0 (operator vouch 2026-10-29 18:57:48 CET, read from the hub log: agent=0.170.0, bundle 03f2ff64… = the release; the save took 38.3 s, R-1346); earlier 0.169.0 (operator vouch 2026-10-29 14:06:09 CET, read from the hub log: agent=0.169.0, bundle 3ca33f9c… = the release; his browser got a 504 but the save went through); earlier 0.168.0 (vouch 10:21:56 CET); 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
|
||||
"recorded": "2026-11-03",
|
||||
"recorded_by": "CC 2026-11-07: day0_agent 0.172.1 (operator vouch 2026-11-07 ~23:00 CET, read from the hub DB with its WAL: artifact_agent_version=0.172.1, bundle 3e7acaee… = the release); earlier 0.172.0 (CC 2026-11-03) (operator vouch 2026-11-03 10:18:10 CET, read from the hub log and DB: agent=0.172.0, bundle cc34b0bc… = the release); earlier 0.171.0 (CC 2026-10-30) (operator vouch 2026-10-30 10:47:36 CET, read from the hub log: agent=0.171.0, bundle 95f977f6… = the release; the save took 132 ms with hub 0.178.0, R-1346); earlier 0.170.0 (operator vouch 2026-10-29 18:57:48 CET, read from the hub log: agent=0.170.0, bundle 03f2ff64… = the release; the save took 38.3 s, R-1346); earlier 0.169.0 (operator vouch 2026-10-29 14:06:09 CET, read from the hub log: agent=0.169.0, bundle 3ca33f9c… = the release; his browser got a 504 but the save went through); earlier 0.168.0 (vouch 10:21:56 CET); 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
|
||||
"pinned": {
|
||||
"hub_min_agent": "0.158.1",
|
||||
"day0_agent": "0.170.0"
|
||||
"day0_agent": "0.172.1"
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user