Compare commits

...

18 Commits

Author SHA1 Message Date
admin 9cac3462bb osupdate: the health baseline is the start of the leg (inventory reading merged with the apply's own) — an app that stops during the run fails it (found live on demo-hp)
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 11:22:03 +02:00
admin 1bb8608e88 os-apply: tell an UPDATED conffile from a KEPT one (dpkg's two shapes, measured live on demo-hp)
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 11:08:43 +02:00
admin b84e0dd1bd selftest flag accepts os-update and wgtunnel (both dispatched, both refused); a test pins every dispatched mode
gates / gates (push) Successful in 19s
Found live 2026-10-04: the OS leg's debug action could not run.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 11:04:03 +02:00
admin 23a8ef3de4 OS updates, guest fast lane (11 §8 step 2): felhom-os-apply wrapper (R1-R13 refusals, repair first, snapshot.debian.org fallback), FELHOM_OSAPPLY sudoers, the OS leg after the primary backup, hub os_update block + os-report, --selftest=os-update
gates / gates (push) Successful in 18s
No automatic undo: a customer guest cannot be snapshotted (R-837, measured).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 10:44:29 +02:00
admin 596238cc2e CHANGELOG + REPORT: v0.139.0 released (R-834)
gates / gates (push) Successful in 17s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 08:52:57 +02:00
admin 475bdce7e4 DR bring-up refuses beside a live original (R-834): source guest present, drives bind, or unreadable config
gates / gates (push) Successful in 18s
The DR route keeps onboot 1, binds the real drives and starts the guest: right on a replaced
host, a second box on the same drives beside a live original. The restore-test's no-host-bind
half is now pinned too (measured safe live on demo-hp 2026-10-04).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 08:52:35 +02:00
admin d766666ff8 CHANGELOG: v0.138.0 vouched with golden 0.283.1
gates / gates (push) Successful in 15s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-30 11:40:20 +02:00
admin a4c09a7c11 REPORT: v0.138.0 (R-727)
gates / gates (push) Successful in 16s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-30 11:00:45 +02:00
admin 904dc20466 CHANGELOG: v0.138.0 released (R-727)
gates / gates (push) Successful in 14s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-30 10:34:25 +02:00
admin e1b8269be0 restore test takes only this box's archives (R-727): an archive encrypted with another key is another box's
gates / gates (push) Successful in 16s
Red-proof RP39. Released as v0.138.0 by release-agent.sh.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-30 10:33:52 +02:00
admin 5c68c869b6 docs: v0.137.0 vouched with golden 0.276.0 (2026-09-28)
gates / gates (push) Successful in 15s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-28 09:45:08 +02:00
admin 728d12b1a0 CHANGELOG: v0.137.0 released (tag + package verified by download), not vouched
gates / gates (push) Successful in 14s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 14:01:26 +02:00
admin dd81866b16 REPORT: v0.136.0 and v0.137.0
gates / gates (push) Successful in 15s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 14:01:02 +02:00
admin 3ef095fb71 agent: a guest outside the agent's ACL is not a known guest (R-689, v0.136.0 regression)
gates / gates (push) Successful in 14s
PVE answers 403 permission denied, not "does not exist", for a vmid outside the felhom pool;
v0.136.0 turned that into a lookup failure and the local tier read UNKNOWN every evaluation
(measured on demo-hp). Such an archive is skipped. Red-proofed; verified read-only on demo-hp
with the pre-release binary.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 14:00:42 +02:00
admin 7c986915ca CHANGELOG: v0.136.0 released (tag + package verified by download), not vouched
gates / gates (push) Successful in 14s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 13:46:22 +02:00
admin 16dbc83221 agent: the restore test takes only archives of a guest that still exists (R-689, second half)
gates / gates (push) Successful in 14s
Measured on demo-hp right after v0.135.0: with the golden skipped the pick fell to a leftover
archive of guest 9100, deleted in August. "does not exist" skips it; any other lookup error
makes the tier unknown. Red-proofed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 13:45:56 +02:00
admin 9555a7f93b REPORT: v0.135.0 released and signed-delivered to both demo hosts
gates / gates (push) Successful in 15s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 13:02:10 +02:00
admin 9ff937d8fb CHANGELOG: v0.135.0 released (tag + package verified by download), not vouched
gates / gates (push) Successful in 15s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 12:16:17 +02:00
24 changed files with 2507 additions and 27 deletions
+3
View File
@@ -9,3 +9,6 @@
# go # go
/vendor/ /vendor/
# Python bytecode written by configs/test_felhom_os_apply.py
configs/__pycache__/
+89
View File
@@ -1,3 +1,92 @@
## v0.139.0 — a DR restore never lands beside a live original (2026-10-04, R-834)
> **RELEASED 2026-10-04** by `scripts/release-agent.sh` — tag `v0.139.0` (`475bdce`), sha256
> `8534a9be368a6d24d8065db77436e86900443c5c6554c71f6fe91c2bdb9d0b9c`, verified by download. **Not vouched.**
**MinAgent impact:** none required by any controller.
- The DR bring-up (`--selftest=bring-up -mode dr`) keeps the archive's `onboot: 1`, binds the host's REAL drives
(`mp8 /mnt/felhom-drives`) and STARTS the guest — right on a replaced host, wrong beside a live original (a second
controller for the same household on the same drives). It now REFUSES, before any restore, when the archive's
source guest still exists on the host, when any guest binds the drives parent, or when a guest's config cannot be
read (fail closed). On a replaced host it proceeds and keeps its binds, unchanged.
- The restore-test was MEASURED safe live on demo-hp (onboot 0 and throwaway stand-ins for mp8/mp9 from the first
config read to teardown); a test now pins its "no host path" half beside the existing onboot test.
- No sudoers change: the restore-test sets onboot 0 through the API create call, and DR refuses rather than degrade,
so no `-onboot 0` line is needed.
- Tests: `TestRunBringUp_DRRefusesBesideALiveOriginal` (source guest present / drives bind on another guest / an
unreadable config refuse; a replaced host proceeds and keeps the drives bind), `TestRunBringUp_ProvisionNotBlockedByADrivesBind`,
`TestArchiveSourceVMID`, `TestRestoreTest_NoHostPathBindBesideTheOriginal`. Red-proofs: the DR check returning ""
→ three refusal cases restore and START; the restore-test's mp8 override set to the host path → fails.
## v0.138.0 — the restore test takes only THIS box's archives (2026-09-30, R-727, `09` §3 decision 51)
> **RELEASED 2026-09-30** by `scripts/release-agent.sh` — tag `v0.138.0` (`e1b8269`), sha256
> `55916026001790a79ebf97d32c032610cfde8e09d02979b9b9d8c2cbc5d88195`, verified by download. **Not vouched** (the golden
> keeps 0.137.0 until the next bake); delivered to the demo boxes by signed `agent_update` jobs.
>
> **VOUCHED 2026-09-30** with golden 0.283.1 (`min_agent` 0.131.0), on the operator's word: the hub logged
> `Artifact manifest set: agent=0.138.0 golden=0.283.1 min_agent="0.131.0"`. Evidence:
> `felhom.eu/documentation/audits/evidence-golden-0283-2026-09-30/`.
**MinAgent impact:** none required by any controller.
- A returning customer's PBS namespace can hold archives of EARLIER boxes: same guest id (9201), same token, written
with a different key. Measured 2026-09-30: the newest SETTLED archive was an earlier box's, and the test failed
`wrong key … manifest's key 6b:ca:5f:3f… does not match provided key de:51:7a:18…` every evaluation. The archive
carries no host id; it carries its key fingerprint (PVE content `encrypted`), and the storage carries its own
(`GET /storage` → `encryption-key`). `PickSettledRestoreCandidateOn` now skips — and logs by name, once — an archive
whose fingerprint is not the storage's own; an unencrypted storage is not filtered; a failed storage read is an
error (tier UNKNOWN), never "nothing to prove".
- Tests: `TestR727_TheRestoreTestTakesOnlyThisBoxsArchives` (the 2026-09-30 shape: nothing picked while this box's
archive settles, then exactly it), `TestR727_UnencryptedStorageIsNotFiltered`, `TestR727_KeyLookupFailureIsUnknown`.
Red-proof RP39: the skip removed → the earlier box's `2026-09-16T21:59:54Z` is picked.
## v0.137.0 — a guest outside the agent's ACL is not a known guest (2026-09-27, R-689, v0.136.0 regression)
> **RELEASED 2026-09-27** by `scripts/release-agent.sh` — tag `v0.137.0` (`3ef095f`), sha256 `766c9166916a1bd3674b0dc69081f8a7619e770f1402d8ad7705b395937e7627`, verified by download. **NOT vouched** (the operator's act).
>
> **VOUCHED 2026-09-28** with golden 0.276.0 (`min_agent` 0.131.0), on the operator's word of 2026-09-27: the hub logged
> `Artifact manifest set: agent=0.137.0 golden=0.276.0 min_agent="0.131.0"`, and a Day-0 test install fetched this binary
> through the manifest and sha-verified it. Evidence: `felhom.eu/documentation/audits/evidence-golden-0276-2026-09-28/`.
**MinAgent impact:** none required by any controller.
- v0.136.0 asked `GuestConfig` whether an archive's guest exists and treated anything but "does not exist" as a lookup
failure. PVE answers **403 "permission denied at /vms/<id>"** for a vmid outside the token's pool — so on demo-hp the
deleted guest 9100's archive made the local tier UNKNOWN every evaluation. A guest the agent cannot read is not one it
manages; its archive is skipped. `TestR689_AGuestOutsideTheAgentsACLIsNotAKnownGuest`, red-proofed. Verified read-only
on demo-hp with the pre-release binary before the release.
## v0.136.0 — … of a guest that still EXISTS (2026-09-27, R-689 second half)
> **RELEASED 2026-09-27** by `scripts/release-agent.sh` — tag `v0.136.0` (`16dbc83`), sha256 `2eb0b5ebe253defd68b322312bbac12418c051b0a7a0d2d1831310d97fa6d755`, verified by download. **NOT vouched** (the operator's act).
**MinAgent impact:** none required by any controller.
- **R-689, second half.** Read on demo-hp right after v0.135.0 with the read-only `-selftest=restore-test-due`: the
golden was skipped, and the pick fell to `vzdump-lxc-9100-2026_08_21…` — a leftover of a guest deleted in August. A
candidate's guest must now exist on this node (`GuestConfig`): "does not exist" skips the archive (INFO once per
volid); any other lookup failure is returned, so the tier reads UNKNOWN, never "nothing to prove". Tests
`TestR689_AnArchiveOfADeletedGuestIsNeverPicked` (red-proofed: without the check it picks the 9100 leftover),
`TestR689_AGuestLookupFailureIsUnknownNotEmpty`. A second agent release in one session — the first half was found
incomplete on the box.
## v0.135.0 — the restore test proves only backups OF A GUEST (2026-09-27, R-689)
> **RELEASED 2026-09-27** by `scripts/release-agent.sh` — tag `v0.135.0` (`d4be12c`), sha256 `ad4e75f16d338552f4588d3fe64c51cbf9651220b9d223b5386b85b6c37fd4c3`, verified by download. **NOT vouched** (the operator's act).
**MinAgent impact:** none required by any controller.
- **R-689** (`backup/runner.go` `guestBackupArchive`). demo-hp keeps its golden template in `local:backup/` — content
"backup", 654 MB, plausibly complete, the newest settled entry — and the scheduled restore test picked it every 6 h and
failed `extractconfig` with a 403, while the guest's real archive went untested. A restore-test candidate is now a
`vzdump-{lxc,qemu}-<vmid>-…` file or a PBS `backup/{ct,vm}/<vmid>/…` snapshot whose vmid the storage reports; anything
else is skipped with one INFO line per volid (not the "INCOMPLETE archive" WARN). Tests
`TestR689_TheRestoreTestNeverPicksTheGolden` (red-proof: without the check it picks the golden) and
`TestR689_GuestBackupArchiveShapes`; two older picker tests' fixtures moved to real archive names.
Evidence: `felhom.eu/documentation/audits/version-travel-2026-09-26/D1/`.
## v0.134.0 — a whole-box backup that cannot fit is skipped with a reason, before anything starts (2026-09-25 night, R-685) ## v0.134.0 — a whole-box backup that cannot fit is skipped with a reason, before anything starts (2026-09-25 night, R-685)
> **RELEASED 2026-09-24 night** by `scripts/release-agent.sh` — tag `v0.134.0` (`0722b2c`), sha256 `7593bebe03234c7d22f3ade384e7ed7787dc659aa8c8594b3ee19af2ce81c72d`, verified by download. **NOT vouched** (Day-0 stays on the previous version). > **RELEASED 2026-09-24 night** by `scripts/release-agent.sh` — tag `v0.134.0` (`0722b2c`), sha256 `7593bebe03234c7d22f3ade384e7ed7787dc659aa8c8594b3ee19af2ce81c72d`, verified by download. **NOT vouched** (Day-0 stays on the previous version).
+8 -15
View File
@@ -1,17 +1,10 @@
# REPORT — agents v0.133.0 + v0.134.0 delivered; v0.134.0 released (night 2026-09-25) # REPORT — 2026-10-04: v0.139.0 (R-834)
**Delivered** to demo-felhom and demo-hp by CC-signed `agent_update` jobs (ruling 1, 2026-09-16), each verified Full session report: `felhom.eu/REPORT-backup-close-os-spike-2026-10-04.md`.
against the published package's sha256 first: v0.133.0 (committed 21:35 / 21:50 CEST), then v0.134.0 (22:52 /
22:58). Hub reads 0.134.0 on both. Peti's box: nothing. Restore test back ON on both (A3: demo-felhom PASS 85 s;
demo-hp refused for space, correctly). demo-hp `local_backup_retention: 1` (operator option A).
**v0.134.0 = R-685 (agent half):** a vzdump to a LOCAL target needs free ≥ newest archive × 1.25 + 1 GiB; a - **Measured** on demo-hp: the scheduled restore-test's scratch guest has `onboot: 0` and throwaway stand-ins for
shortfall is a named skip before anything starts; fail-open on PBS / first backup / unknown usage. Tag `v0.134.0` mp8/mp9 on every config read until teardown — it was already safe. Evidence `felhom.eu/documentation/audits/backup-close-2026-10-04/partA/`.
(`0722b2c`), sha256 `7593bebe03234c7d22f3ade384e7ed7787dc659aa8c8594b3ee19af2ce81c72d`, verified by download; CI jobs - **Fixed:** the DR bring-up refuses beside a live original (source guest present, a drives bind on another guest,
975–977 success. **Not vouched** for Day-0. or an unreadable config). On a replaced host it is unchanged.
- Tests `TestRunBringUp_DRRefusesBesideALiveOriginal`, `TestRestoreTest_NoHostPathBindBesideTheOriginal` and two
**Found live before the release:** the first build read free space from `GET /storage` (no usage) and failed open — more; both rules red-proved. No sudoers change (said why in the CHANGELOG).
its own "must refuse" test started a real vzdump of demo-hp 9201, aborted by CC after 5 min 16 s, no archive left.
Fixed (`NodeStorage`), test fake made honest (`ListStorage` strips usage), two red-proofs, re-proven live with safe
builds (a hard stop before vzdump): ×10 refused with real numbers, ×1.25 passed. Evidence:
`felhom.eu/documentation/audits/night-2026-09-25/F/`.
+77 -1
View File
@@ -48,6 +48,7 @@ import (
"gitea.dooplex.hu/admin/felhom-agent/internal/pbsdr" "gitea.dooplex.hu/admin/felhom-agent/internal/pbsdr"
"gitea.dooplex.hu/admin/felhom-agent/internal/poke" "gitea.dooplex.hu/admin/felhom-agent/internal/poke"
"gitea.dooplex.hu/admin/felhom-agent/internal/provision" "gitea.dooplex.hu/admin/felhom-agent/internal/provision"
"gitea.dooplex.hu/admin/felhom-agent/internal/osupdate"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox" "gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile" "gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
"gitea.dooplex.hu/admin/felhom-agent/internal/restorespace" "gitea.dooplex.hu/admin/felhom-agent/internal/restorespace"
@@ -242,6 +243,8 @@ func main() {
os.Exit(runSelftestRestoreTest(context.Background(), cfg, logger, archive)) os.Exit(runSelftestRestoreTest(context.Background(), cfg, logger, archive))
case "restore-test-due": case "restore-test-due":
os.Exit(runSelftestRestoreTestDue(context.Background(), cfg, logger)) os.Exit(runSelftestRestoreTestDue(context.Background(), cfg, logger))
case "os-update":
os.Exit(runSelftestOSUpdate(context.Background(), cfg, logger, vmid))
case "pbs-verify": case "pbs-verify":
os.Exit(runSelftestPBSVerify(context.Background(), cfg, logger)) os.Exit(runSelftestPBSVerify(context.Background(), cfg, logger))
case "lanresolver": case "lanresolver":
@@ -839,6 +842,10 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// The "Down" channel sync hook: on each heartbeat, fetch desired-state when the generation // The "Down" channel sync hook: on each heartbeat, fetch desired-state when the generation
// advances. The loop calls it via the EnvelopeObserver seam (hub does not import desired). // advances. The loop calls it via the EnvelopeObserver seam (hub does not import desired).
desiredSyncer := desired.NewSyncer(client, desiredProvider, logger) desiredSyncer := desired.NewSyncer(client, desiredProvider, logger)
// OS updates, guest fast lane (agent v0.140.0, `11-os-updates.md` §8 step 2): the leg consumes the hub's
// os_update block and runs after each successful primary whole-guest backup (wired on the local API below).
osLeg := newOSLeg(cfg, client, logger)
desiredSyncer.AddConsumer(osLeg)
// S5: consume a host_loss restore_directive into an inspectable restore PLAN (derive + surface, // S5: consume a host_loss restore_directive into an inspectable restore PLAN (derive + surface,
// execute nothing). The recipe is fetched on-demand (rare directive) via a fresh Collect. // execute nothing). The recipe is fetched on-demand (rare directive) via a fresh Collect.
desiredSyncer.AddConsumer(dr.NewConsumer(func(ctx context.Context) *hub.DRRecipeHostHalf { desiredSyncer.AddConsumer(dr.NewConsumer(func(ctx context.Context) *hub.DRRecipeHostHalf {
@@ -1112,6 +1119,17 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
}, },
} }
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens) localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
if localSrv != nil {
localSrv.SetAfterPrimaryBackup(func(ctx context.Context, vmid int) {
// Let the controller finish bringing its apps back after the backup, then run (still under the gate).
select {
case <-ctx.Done():
return
case <-time.After(90 * time.Second):
}
osLeg.Run(ctx, vmid, "night")
})
}
if localTokens != nil { if localTokens != nil {
defer localTokens.Close() defer localTokens.Close()
} }
@@ -3478,8 +3496,66 @@ func (f *selftestFlag) Set(v string) error {
f.mode = "identity-consume" f.mode = "identity-consume"
case "controller-swap": case "controller-swap":
f.mode = "controller-swap" f.mode = "controller-swap"
case "os-update":
f.mode = "os-update"
case "wgtunnel": // dispatched since S3 but refused here until 2026-10-04 (TestSelftestFlag_AcceptsEveryDispatchedMode)
f.mode = "wgtunnel"
default: default:
return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test|restore-test-due|pbs-verify|bring-up|provision|escrow-create|escrow-consume|identity-consume|controller-swap)", v) return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test|restore-test-due|pbs-verify|lanresolver|wgtunnel|bring-up|provision|escrow-create|escrow-consume|identity-consume|controller-swap|os-update)", v)
} }
return nil return nil
} }
// newOSLeg builds the OS-update leg (agent v0.140.0). The wrapper runs through sudo (FELHOM_OSAPPLY); the plan and
// the once-per-night marker live in the agent's own os/ dir.
func newOSLeg(cfg config.Config, client *hub.Client, logger *slog.Logger) *osupdate.Leg {
mode := proxmox.RunnerMode(cfg.Privileged.Mode)
if mode == "" {
mode = proxmox.RunnerSudo
}
l := &osupdate.Leg{
Runner: &proxmox.ExecRunner{Mode: mode, SudoPath: cfg.Privileged.SudoPath},
Logger: logger,
PlanDir: osupdate.DefaultPlanDir,
StatePath: filepath.Join(osupdate.DefaultPlanDir, "last-night-run"),
}
if client != nil {
l.Hub = client
}
return l
}
// runSelftestOSUpdate is the OS leg's DEBUG ACTION (agent v0.140.0): one pass for -vmid, now, exactly as the night
// runs it after a backup — the hub's os_update block (fetched fresh), the wrapper via sudo, the health wait, the
// report to the hub — with trigger "debug" (never throttled, and it does NOT count as a night run for approval).
// Run it as the agent user: sudo -u felhom-agent felhom-agent --config … --selftest=os-update -vmid 9201
func runSelftestOSUpdate(ctx context.Context, cfg config.Config, logger *slog.Logger, vmid int) int {
if vmid <= 0 {
fmt.Fprintln(os.Stderr, "selftest=os-update: -vmid is required")
return 2
}
client, err := hub.NewClient(cfg.Hub, logger)
if err != nil {
fmt.Fprintln(os.Stderr, "selftest=os-update: hub client:", err)
return 1
}
leg := newOSLeg(cfg, client, logger)
resp, err := client.FetchDesiredState(ctx)
if err != nil {
fmt.Fprintln(os.Stderr, "selftest=os-update: desired state:", err)
return 1
}
leg.SetBlock(resp.DesiredState.OSUpdate)
b := leg.Block()
fmt.Printf("=== felhom-agent %s selftest=os-update vmid=%d ring=%d enabled=%v release=%v ===\n", version, vmid, b.Ring, b.Enabled, b.Release != nil)
rep := leg.Run(ctx, vmid, "debug")
printJSON("os-update report", map[string]any{"run_id": rep.RunID, "ring": rep.Ring, "release_id": rep.ReleaseID,
"mode": rep.Mode, "outcome": rep.Outcome, "healthy": rep.Healthy, "health_reason": rep.HealthReason,
"upgraded": rep.Upgraded, "pending": len(rep.Pending), "not_covered": rep.NotCovered,
"restart_needed": rep.RestartNeeded, "docker_restart_needed": rep.DockerRestartNeeded, "refused": rep.Refused})
switch rep.Outcome {
case "applied", "nothing", "inventory", "skipped":
return 0
}
return 1
}
+39
View File
@@ -0,0 +1,39 @@
package main
import (
"os"
"regexp"
"strings"
"testing"
)
// Every mode the dispatcher (`switch selftest.mode`) runs must be ACCEPTED by the --selftest flag. Found live
// 2026-10-04: --selftest=os-update had a dispatch case and a function but the flag's allow-list refused it, so the
// debug action could not run. Red-proof: drop the "os-update" case from selftestFlag.Set and this fails.
func TestSelftestFlag_AcceptsEveryDispatchedMode(t *testing.T) {
src, err := os.ReadFile("main.go")
if err != nil {
t.Fatal(err)
}
s := string(src)
i := strings.Index(s, "switch selftest.mode {")
if i < 0 {
t.Fatal("dispatch switch not found")
}
block := s[i:]
block = block[:strings.Index(block, "\n\t}\n")]
modes := regexp.MustCompile(`(?m)^\tcase "([a-z-]+)":`).FindAllStringSubmatch(block, -1)
if len(modes) < 5 {
t.Fatalf("parsed only %d dispatch cases — the parser is wrong", len(modes))
}
var bad []string
for _, m := range modes {
var f selftestFlag
if err := f.Set(m[1]); err != nil {
bad = append(bad, m[1])
}
}
if len(bad) > 0 {
t.Fatalf("dispatched but refused by --selftest: %v", bad)
}
}
+8 -1
View File
@@ -299,10 +299,17 @@ Cmnd_Alias FELHOM_SELFHEAL = \
# argument after the numeric vmid is a literal, so the grant cannot be widened by anything the guest or # argument after the numeric vmid is a literal, so the grant cannot be widened by anything the guest or
# the hub says. The address read is deliberately NOT duplicated here — it is already FELHOM_DNSMASQ's, # the hub says. The address read is deliberately NOT duplicated here — it is already FELHOM_DNSMASQ's,
# and the same command must not be granted twice under two names. # and the same command must not be granted twice under two names.
# OS updates, guest fast lane (`11-os-updates.md` §5.4.1, agent v0.140.0). The ONLY entry: the root wrapper with
# one plan file in the agent's own os/ dir. Every safety rule (no removal, no downgrade, no new or unlisted package,
# Debian origin only, the box's own customer guest only) lives in the wrapper, red-proved per rule
# (configs/test_felhom_os_apply.py). The agent gets NO apt grant of its own.
Cmnd_Alias FELHOM_OSAPPLY = \
/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json
Cmnd_Alias FELHOM_GUESTNET = \ Cmnd_Alias FELHOM_GUESTNET = \
/usr/sbin/pct exec [0-9]* -- ip route show default, \ /usr/sbin/pct exec [0-9]* -- ip route show default, \
/usr/sbin/pct exec [0-9]* -- cat /etc/network/interfaces, \ /usr/sbin/pct exec [0-9]* -- cat /etc/network/interfaces, \
/usr/sbin/pct exec [0-9]* -- pgrep -x dhclient, \ /usr/sbin/pct exec [0-9]* -- pgrep -x dhclient, \
/usr/sbin/pct exec [0-9]* -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0 /usr/sbin/pct exec [0-9]* -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN, FELHOM_OSAPPLY
+486
View File
@@ -0,0 +1,486 @@
#!/usr/bin/python3
# felhom-os-apply — the ROOT half of the agent's operating-system update leg (`11-os-updates.md` §5.4.1).
#
# Install as /usr/local/sbin/felhom-os-apply (0755 root:root). The non-root agent invokes it via `sudo -n`
# (FELHOM_OSAPPLY alias) with EXACTLY: felhom-os-apply --plan /var/lib/felhom-agent/os/plan-<id>.json
# Nothing else on the command line is accepted. Python 3, standard library only (a JSON plan cannot be parsed
# safely in sh). Tests: configs/test_felhom_os_apply.py (a fake runner; nothing real is executed).
#
# THE TRUST MODEL. The plan is written by the agent, so a broken-into agent writes whatever plan it likes. The
# protection is therefore what this file REFUSES, not where the plan came from: no removal, no downgrade, no new
# package, no package outside the plan, only Debian origin in the fast lane, only the box's own customer guest.
# Package signatures stay Debian's: apt checks every Release file against the guest's keyring, including the
# snapshot.debian.org fallback (decision 79). Nothing here is overridable from the environment.
#
# THIS RELEASE: layer "guest", lane "fast" only. The host layer and the slow lane exist in the interface and are
# REFUSED (R3, R12) until `11` §8 steps 3 and 5 enable them.
#
# Modes (plan field "mode"):
# inventory read-only for packages: `apt-get update` in the guest, then report what is installed (with origin),
# what is pending, restart-needed and health. Installs nothing.
# apply repair first, check every refusal on an `apt-get -s` simulation of EXACTLY name=version, then
# install, clean, and report the same as inventory.
# health report health only (the agent polls it after a run).
# Output: log lines on stderr and the journal (tag felhom-os-apply); the LAST stdout line is
# OSAPPLY-REPORT <one JSON object>
# which is what the agent parses. Exit 0 = done; 2 = refused (nothing changed); 3 = failed during install.
import json
import os
import re
import stat
import subprocess
import sys
import time
PLAN_DIR = "/var/lib/felhom-agent/os"
PLAN_RE = re.compile(r"^plan-[A-Za-z0-9._-]{1,80}\.json$")
AGENT_USER = "felhom-agent"
FAST_ORIGINS = ("Debian", "Debian-Security")
# Debian package name and version grammar (Debian policy §5.6.1, §5.6.12).
NAME_RE = re.compile(r"^[a-z0-9][a-z0-9+.-]+$")
VERSION_RE = re.compile(r"^(?:[0-9]+:)?[0-9][A-Za-z0-9.+~-]*$")
SNAP_RE = re.compile(r"^[0-9]{8}T[0-9]{6}Z$")
RESERVED_VMIDS = set(range(990000, 990010)) | {9999}
DRIVES_PARENT = "/mnt/felhom-drives"
SNAPSHOT_LIST = "/etc/apt/sources.list.d/felhom-os-snapshot.list"
APT_ENV = ["env", "DEBIAN_FRONTEND=noninteractive", "APT_LISTCHANGES_FRONTEND=none", "NEEDRESTART_MODE=l", "LC_ALL=C"]
DPKG_OPTS = ["-o", "Dpkg::Options::=--force-confold", "-o", "Dpkg::Options::=--force-confdef"]
MIN_FREE = 500 * 1024 * 1024
class Refused(Exception):
def __init__(self, code, reason):
super().__init__(f"{code} {reason}")
self.code, self.reason = code, reason
class Runner:
"""Runs commands for real. Tests replace it with a fake. `guest` runs inside the container via pct exec."""
def host(self, argv, timeout=600):
p = subprocess.run(argv, capture_output=True, text=True, timeout=timeout)
return p.returncode, p.stdout, p.stderr
def guest(self, vmid, argv, timeout=1800):
return self.host(["/usr/sbin/pct", "exec", str(vmid), "--"] + argv, timeout)
def read_file(self, path):
with open(path) as f:
return f.read()
def stat(self, path):
return os.lstat(path)
def agent_uid(self):
import pwd
return pwd.getpwnam(AGENT_USER).pw_uid
def log(self, line):
print(line, file=sys.stderr, flush=True)
try:
subprocess.run(["logger", "-t", "felhom-os-apply", line], timeout=10)
except Exception:
pass
class Apply:
def __init__(self, runner, plan_path):
self.r = runner
self.plan_path = plan_path
self.report = {"refused": None, "mode": None}
# ---------- checks ----------
def load_plan(self):
p = self.plan_path
d, base = os.path.dirname(p), os.path.basename(p)
if d != PLAN_DIR or not PLAN_RE.match(base) or ".." in p:
raise Refused("R1", f"the plan must be {PLAN_DIR}/plan-<id>.json, got {p!r}")
try:
st = self.r.stat(p)
except OSError as e:
raise Refused("R1", f"cannot stat the plan: {e}")
if not stat.S_ISREG(st.st_mode):
raise Refused("R1", "the plan is not a regular file (a symlink or a device is refused)")
if st.st_uid != self.r.agent_uid():
raise Refused("R1", f"the plan is not owned by {AGENT_USER}")
if st.st_size > 2 * 1024 * 1024:
raise Refused("R1", "the plan is larger than 2 MB")
try:
plan = json.loads(self.r.read_file(p))
except (OSError, ValueError) as e:
raise Refused("R1", f"the plan is not valid JSON: {e}")
if not isinstance(plan, dict):
raise Refused("R1", "the plan is not a JSON object")
return plan
def check_plan(self, plan):
mode = plan.get("mode", "apply")
if mode not in ("apply", "inventory", "health"):
raise Refused("R11", f"unknown mode {mode!r}")
if plan.get("layer") != "guest":
raise Refused("R12", f"layer {plan.get('layer')!r} is refused in this release (guest only)")
if plan.get("lane", "fast") != "fast":
raise Refused("R3", "the slow lane is refused in this release")
vmid = plan.get("vmid")
if not isinstance(vmid, int) or isinstance(vmid, bool) or vmid <= 0:
raise Refused("R11", f"vmid must be a positive integer, got {vmid!r}")
rid = plan.get("release_id", "")
if not isinstance(rid, str) or not re.match(r"^[A-Za-z0-9._:-]{1,80}$", rid):
raise Refused("R11", f"release_id {rid!r} is not a plain id")
if plan.get("allow_new"):
raise Refused("R6", "allow_new is a slow-lane field; the fast lane never adds a package")
pk = plan.get("packages", [])
if not isinstance(pk, list) or (mode == "apply" and not pk):
raise Refused("R11", "packages must be a non-empty list in apply mode")
seen = set()
for e in pk:
if not isinstance(e, dict):
raise Refused("R11", "every package entry must be an object")
n, v, o = e.get("name"), e.get("version"), e.get("origin")
if not isinstance(n, str) or not NAME_RE.match(n):
raise Refused("R11", f"package name {n!r} is not a Debian package name")
if not isinstance(v, str) or not VERSION_RE.match(v):
raise Refused("R11", f"version {v!r} of {n} is not a Debian version string")
if n in seen:
raise Refused("R11", f"package {n} is named twice")
seen.add(n)
if o not in FAST_ORIGINS:
raise Refused("R2", f"{n}: origin {o!r} is not Debian / Debian-Security (the fast lane, `11` C3)")
snap = plan.get("snapshot", "")
if snap and not SNAP_RE.match(snap):
raise Refused("R11", f"snapshot {snap!r} is not YYYYMMDDTHHMMSSZ")
return mode, vmid
def check_guest(self, vmid):
if vmid in RESERVED_VMIDS:
raise Refused("R10", f"vmid {vmid} is a reserved scratch vmid")
try:
conf = self.r.read_file(f"/etc/pve/lxc/{vmid}.conf")
except OSError:
raise Refused("R10", f"vmid {vmid} is not a container on this host")
cur = conf.split("\n[", 1)[0] # the current config, not a snapshot section
binds = [l for l in cur.splitlines() if re.match(r"^mp[0-9]+: " + re.escape(DRIVES_PARENT) + r",", l)]
if not binds:
raise Refused("R10", f"vmid {vmid} does not bind {DRIVES_PARENT} — it is not this box's customer guest")
lock = [l for l in cur.splitlines() if l.startswith("lock:")]
if lock:
raise Refused("R9", f"vmid {vmid} is locked ({lock[0].split(':', 1)[1].strip()}) — a backup or restore is running")
rc, out, _ = self.r.host(["/usr/sbin/pct", "status", str(vmid)])
if rc != 0 or "running" not in out:
raise Refused("R10", f"vmid {vmid} is not running")
# ---------- guest helpers ----------
def g(self, argv, timeout=1800):
return self.r.guest(self.vmid, argv, timeout)
def installed(self):
rc, out, _ = self.g(["dpkg-query", "-W", "-f", "${Package}\t${Version}\t${db:Status-Abbrev}\n"])
res = {}
for l in out.splitlines():
parts = l.split("\t")
if len(parts) == 3 and parts[2].startswith("ii"):
res[parts[0]] = parts[1]
return res
def dpkg_cmp(self, a, op, b):
rc, _, _ = self.g(["dpkg", "--compare-versions", a, op, b])
return rc == 0
def madison(self, name):
rc, out, _ = self.g(["apt-cache", "madison", name])
vs = set()
for l in out.splitlines():
f = [x.strip() for x in l.split("|")]
if len(f) >= 3 and f[0] == name:
vs.add(f[1])
return vs
def simulate(self, args):
rc, out, err = self.g(APT_ENV + ["apt-get", "-s", "-q"] + args)
inst, remv = [], []
for l in out.splitlines():
m = re.match(r"^Inst (\S+) (?:\[([^]]*)\] )?\((\S+) (.*?) \[[a-z0-9]+\]\)", l)
if m:
inst.append({"name": m.group(1), "from": m.group(2), "to": m.group(3), "origin": m.group(4)})
m = re.match(r"^Remv (\S+)", l)
if m:
remv.append(m.group(1))
return rc, inst, remv, out + err
@staticmethod
def origin_name(origin):
# "Debian:13.7/stable, Debian-Security:13/stable-security" -> {"Debian", "Debian-Security"}
return {o.strip().split(":")[0] for o in origin.split(",") if o.strip()}
def free_bytes(self):
rc, out, _ = self.g(["df", "-B1", "--output=avail", "/"])
try:
return int(out.strip().splitlines()[-1])
except (ValueError, IndexError):
return -1
def apt_lock_held(self):
rc, out, _ = self.g(["fuser", "/var/lib/dpkg/lock-frontend", "/var/lib/dpkg/lock"])
return rc == 0 and out.strip() != ""
def health(self):
"""The guest's signals: every container's state + health, the controller's own health, the network."""
rc, out, _ = self.g(["docker", "ps", "-a", "--format", "{{.Names}}\t{{.State}}\t{{.Status}}"], timeout=60)
cont = {}
for l in out.splitlines():
p = l.split("\t")
if len(p) == 3:
h = "healthy" if "(healthy)" in p[2] else "unhealthy" if "(unhealthy)" in p[2] else \
"starting" if "(health: starting)" in p[2] else "none"
cont[p[0]] = {"state": p[1], "health": h}
nrc, _, _ = self.g(["getent", "hosts", "deb.debian.org"], timeout=30)
return {"docker_ok": rc == 0, "containers": cont,
"controller": cont.get("felhom-controller", {}).get("health", "absent"),
"network_ok": nrc == 0}
def restart_needed(self):
"""Processes still mapping deleted files, OUTSIDE docker containers (C11)."""
script = ('for p in /proc/[0-9]*; do grep -q "(deleted)" $p/maps 2>/dev/null || continue; '
'grep -q "docker" $p/cgroup 2>/dev/null && continue; echo "${p#/proc/} $(cat $p/comm 2>/dev/null)"; done')
rc, out, _ = self.g(["sh", "-c", script], timeout=120)
procs = sorted({l.split(" ", 1)[1] for l in out.splitlines() if " " in l})
pid1 = any(l.split(" ", 1)[0] == "1" for l in out.splitlines())
return procs, pid1
def inventory(self):
inst = self.installed()
names = sorted(inst)
origins = {}
for i in range(0, len(names), 200):
rc, out, _ = self.g(["apt-cache", "policy"] + names[i:i + 200])
cur, star = None, False
for l in out.splitlines():
if not l.startswith(" "):
cur, star = l.rstrip(":"), False
continue
s = l.strip()
if s.startswith("*** "):
star = True
continue
if star and cur and re.match(r"^[0-9-]+ ", s):
if "/var/lib/dpkg/status" in s:
origins.setdefault(cur, "local")
else:
origins[cur] = s
continue
if star and not re.match(r"^[0-9-]+ ", s):
star = False
# Map an index URL to an origin name the hub understands.
def oname(src):
if src in (None, "local"):
return "unknown"
if "security" in src and "debian" in src:
return "Debian-Security"
if "docker.com" in src:
return "Docker"
if "debian" in src:
return "Debian"
return "other"
rc, pend, remv, _ = self.simulate(["dist-upgrade"])
return {
"installed": [{"name": n, "version": inst[n], "origin": oname(origins.get(n))} for n in names],
"pending": [{"name": p["name"], "from": p["from"], "to": p["to"],
"origin": sorted(self.origin_name(p["origin"]))} for p in pend],
}
# ---------- the run ----------
def run(self):
plan = self.load_plan()
self.mode, self.vmid = self.check_plan(plan)
self.report.update(mode=self.mode, release_id=plan.get("release_id"), vmid=self.vmid)
self.check_guest(self.vmid)
log = self.r.log
if self.mode == "health":
self.report["health"] = self.health()
return 0
log(f"os-apply: START release={plan.get('release_id')} layer=guest:{self.vmid} lane=fast mode={self.mode} packages={len(plan.get('packages', []))}")
if self.apt_lock_held():
raise Refused("R9", "another apt/dpkg holds the lock in the guest")
self.report["health_before"] = self.health()
if self.mode == "apply":
self.repair()
rc, out, err = self.g(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
if rc != 0:
raise Refused("R7", f"apt-get update failed in the guest: {(out + err).strip().splitlines()[-1:]}")
if self.mode == "apply":
rc = self.apply(plan)
if rc:
return rc
procs, pid1 = self.restart_needed()
self.report.update(self.inventory())
self.report["restart_needed"] = procs
self.report["docker_restart_needed"] = any(p in ("dockerd", "containerd") for p in procs)
self.report["reboot_needed"] = pid1
self.report["health_after"] = self.health()
return 0
def repair(self):
rc, before, _ = self.g(["dpkg", "--audit"])
self.g(APT_ENV + ["dpkg", "--configure", "-a", "--force-confold"])
rc2, out, err = self.g(APT_ENV + ["apt-get", "-f", "install", "-y", "-q"] + DPKG_OPTS)
_, after, _ = self.g(["dpkg", "--audit"])
configured = len([l for l in before.splitlines() if l.startswith(" ")])
fixed = len(re.findall(r"^Setting up ", out, re.M))
self.report["repair"] = {"half_configured_before": configured, "fixed": fixed, "clean_after": after.strip() == ""}
self.r.log(f"os-apply: REPAIR configured={configured} fixed={fixed}")
if after.strip():
raise Refused("R13", "dpkg is still broken after the repair: " + after.strip().splitlines()[0])
def apply(self, plan):
log = self.r.log
inst = self.installed()
upgrade, already, notinst = [], 0, 0
for e in plan["packages"]:
n, v = e["name"], e["version"]
if n not in inst:
notinst += 1
continue
if not self.dpkg_cmp(v, "gt", inst[n]):
already += 1
continue
upgrade.append((n, v))
from_snap = 0
missing = [(n, v) for n, v in upgrade if v not in self.madison(n)]
if missing:
snap = plan.get("snapshot", "")
if not snap:
raise Refused("R7", f"{missing[0][0]}={missing[0][1]} is not downloadable and the plan names no snapshot")
self.add_snapshot_sources(snap)
still = [(n, v) for n, v in missing if v not in self.madison(n)]
if still:
self.remove_snapshot_sources()
raise Refused("R7", f"{still[0][0]}={still[0][1]} is not downloadable, not even from snapshot {snap}")
from_snap = len(missing)
try:
log(f"os-apply: PLAN upgrade={len(upgrade)} already={already} not-installed={notinst} from-snapshot={from_snap}")
self.report["plan"] = {"upgrade": len(upgrade), "already": already, "not_installed": notinst, "from_snapshot": from_snap}
if not upgrade:
self.report["upgraded"] = []
log("os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)")
return 0
args = ["install", "--only-upgrade", "--no-install-recommends"] + [f"{n}={v}" for n, v in upgrade]
rc, sim, remv, text = self.simulate(args)
if rc != 0:
tail = text.strip().splitlines()[-1] if text.strip() else ""
raise Refused("R7", "the simulation failed: " + tail)
if remv:
raise Refused("R4", f"the plan would remove {', '.join(remv[:5])}")
want = dict(upgrade)
for p in sim:
if p["from"] is None:
raise Refused("R6", f"the plan would add a package that is not installed: {p['name']}")
if p["name"] not in want:
raise Refused("R6", f"the plan would touch {p['name']}, which is not in the plan")
if p["to"] != want[p["name"]]:
raise Refused("R6", f"{p['name']} would go to {p['to']}, not the approved {want[p['name']]}")
if not self.dpkg_cmp(p["to"], "gt", p["from"]):
raise Refused("R5", f"{p['name']} would be downgraded {p['from']} -> {p['to']}")
if not self.origin_name(p["origin"]) & set(FAST_ORIGINS):
raise Refused("R2", f"{p['name']} would come from {p['origin']}, not Debian")
need = self.download_bytes(args)
free = self.free_bytes()
if free >= 0 and free < max(MIN_FREE, 3 * need):
raise Refused("R8", f"free space {free} B is below max(500 MB, 3 x download {need} B)")
t0 = time.time()
rc, out, err = self.g(APT_ENV + ["apt-get", "-y", "-q"] + DPKG_OPTS + args)
secs = time.time() - t0
# dpkg says "Installing new version of config file X" when X was NOT changed locally (the package's new
# version is taken), and "Configuration file 'X'" + "Keeping old config file" when it was (--force-confold
# keeps the local one; the package's version lands as X.dpkg-dist). Measured live 2026-10-04 (debian_version).
conflict = None
for l in (out + err).splitlines():
m = re.search(r"Installing new version of config file (\S+?)\s*\.\.\.", l)
if m:
log(f"os-apply: CONFFILE updated {m.group(1)} (it was not changed locally)")
m = re.search(r"Configuration file '([^']+)'", l)
if m:
conflict = m.group(1)
if conflict and "Keeping old config file" in l:
log(f"os-apply: CONFFILE kept {conflict} (changed locally; the package's version is {conflict}.dpkg-dist)")
self.report.setdefault("conffiles_kept", []).append(conflict)
conflict = None
self.g(["apt-get", "clean"])
if rc != 0:
_, aud, _ = self.g(["dpkg", "--audit"])
first = aud.strip().splitlines()[0] if aud.strip() else "clean"
log(f"os-apply: FAILED rc={rc} step=install — dpkg state: {first}")
self.report["failed"] = {"rc": rc, "dpkg_audit": first, "tail": (out + err).strip().splitlines()[-3:]}
return 3
self.report["upgraded"] = [{"name": n, "version": v} for n, v in upgrade]
self.report["seconds"] = round(secs, 1)
log(f"os-apply: DONE rc=0 seconds={secs:.1f} upgraded={len(upgrade)}")
return 0
finally:
if from_snap:
self.remove_snapshot_sources()
def download_bytes(self, args):
rc, out, _ = self.g(APT_ENV + ["apt-get", "-s", "-o", "Debug::NoLocking=1", "--print-uris", "-q"] + args)
total = 0
for l in out.splitlines():
m = re.match(r"^'[^']+' \S+ ([0-9]+) ", l)
if m:
total += int(m.group(1))
return total
def add_snapshot_sources(self, snap):
rc, out, _ = self.g(["sh", "-c", ". /etc/os-release && echo $VERSION_CODENAME"])
code = out.strip()
if not re.match(r"^[a-z]+$", code):
raise Refused("R7", f"cannot read the guest's Debian codename ({code!r})")
body = (f"deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/{snap} {code} main\n"
f"deb [check-valid-until=no] http://snapshot.debian.org/archive/debian-security/{snap} {code}-security main\n")
self.r.guest_write(self.vmid, SNAPSHOT_LIST, body)
self.r.log(f"os-apply: SNAPSHOT using snapshot.debian.org/{snap} for versions no longer published (decision 79)")
rc, out, err = self.g(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
if rc != 0:
self.remove_snapshot_sources()
raise Refused("R7", "apt-get update against snapshot.debian.org failed")
def remove_snapshot_sources(self):
self.g(["rm", "-f", SNAPSHOT_LIST])
self.g(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
def guest_write(self, vmid, path, body):
"""Write a small text file inside the guest via `pct exec … tee` (stdin), never via a shell string."""
p = subprocess.run(["/usr/sbin/pct", "exec", str(vmid), "--", "tee", path], input=body,
capture_output=True, text=True, timeout=60)
if p.returncode != 0:
raise Refused("R7", f"could not write {path} in the guest")
Runner.guest_write = guest_write
def main(argv, runner=None):
r = runner or Runner()
if len(argv) != 3 or argv[1] != "--plan":
r.log("os-apply: REFUSED: R1 usage: felhom-os-apply --plan /var/lib/felhom-agent/os/plan-<id>.json")
print("OSAPPLY-REPORT " + json.dumps({"refused": {"code": "R1", "reason": "usage"}}))
return 2
a = Apply(r, argv[2])
try:
rc = a.run()
except Refused as e:
r.log(f"os-apply: REFUSED: {e.code} {e.reason}")
a.report["refused"] = {"code": e.code, "reason": e.reason}
rc = 2
except subprocess.TimeoutExpired as e:
r.log(f"os-apply: FAILED rc=124 step=timeout — {e.cmd}")
a.report["failed"] = {"rc": 124, "timeout": str(e.cmd)[:200]}
rc = 3
print("OSAPPLY-REPORT " + json.dumps(a.report, sort_keys=True))
return rc
if __name__ == "__main__":
if os.geteuid() != 0:
print("felhom-os-apply: must run as root (via sudo)", file=sys.stderr)
sys.exit(2)
sys.exit(main(sys.argv))
+467
View File
@@ -0,0 +1,467 @@
#!/usr/bin/env python3
"""Tests for configs/felhom-os-apply (`11` §5.4.1). A fake runner plays the host and the guest: nothing is executed
for real except the local `dpkg --compare-versions` (pure, no network). Every refusal R1–R13 has a test; the red-proof
(each test fails when its rule is removed) is `audits/os-guest-lane-2026-10-04/partB/redproof.txt`.
Run: python3 configs/test_felhom_os_apply.py (also run by internal/osupdate's Go test)
"""
import importlib.machinery
import importlib.util
import json
import os
import pathlib
import re
import stat as statmod
import subprocess
import unittest
HERE = pathlib.Path(__file__).resolve().parent
_loader = importlib.machinery.SourceFileLoader("osapply", os.environ.get("OSAPPLY_UNDER_TEST", str(HERE / "felhom-os-apply"))) # red-proof seam
_spec = importlib.util.spec_from_loader("osapply", _loader)
osapply = importlib.util.module_from_spec(_spec)
_loader.exec_module(osapply)
PLAN = "/var/lib/felhom-agent/os/plan-t1.json"
CONF_OK = ("arch: amd64\nmp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G\n"
"mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives\nrootfs: local-lvm:vm-9201-disk-0,size=32G\n")
DEB = "Debian:13.7/stable"
SEC = "Debian-Security:13/stable-security"
def dpkg_cmp(a, op, b):
return subprocess.run(["dpkg", "--compare-versions", a, op, b]).returncode == 0
class St:
def __init__(self, mode=statmod.S_IFREG | 0o600, uid=999, size=100):
self.st_mode, self.st_uid, self.st_size = mode, uid, size
class Fake:
"""The host + one guest. `installed` / `live` (name -> versions in the live archive) / `snapshot` (versions
the snapshot archive adds) / `extra_sim` (lines the simulation adds) / `dpkg_audit` / `free`."""
def __init__(self):
self.plan = {"release_id": "os-t1", "layer": "guest", "lane": "fast", "vmid": 9201, "mode": "apply",
"snapshot": "20261004T080000Z",
"packages": [{"name": "libc6", "version": "2.41-12+deb13u4", "origin": "Debian"},
{"name": "openssl", "version": "3.5.7-1~deb13u3", "origin": "Debian-Security"}]}
self.files = {"/etc/pve/lxc/9201.conf": CONF_OK}
self.stats = {PLAN: St()}
self.installed = {"libc6": "2.41-12+deb13u3", "openssl": "3.5.6-1~deb13u1", "bash": "5.2.37-2+b9"}
self.live = {"libc6": {"2.41-12+deb13u4"}, "openssl": {"3.5.7-1~deb13u3"}}
self.snapshot = {}
self.snap_active = False
self.extra_sim = []
self.dpkg_audit = ""
self.free = 10 * 1024 ** 3
self.install_rc = 0
self.calls = []
self.logs = []
self.written = {}
self.status = "status: running"
self.lock_held = False
# Runner interface
def read_file(self, p):
if p == PLAN:
return json.dumps(self.plan)
if p not in self.files:
raise OSError("no such file")
return self.files[p]
def stat(self, p):
if p not in self.stats:
raise OSError("no such file")
return self.stats[p]
def agent_uid(self):
return 999
def log(self, line):
self.logs.append(line)
def host(self, argv, timeout=600):
self.calls.append(("host", argv))
if argv[1] == "status":
return 0, self.status + "\n", ""
return 1, "", "unexpected host call"
def guest_write(self, vmid, path, body):
self.written[path] = body
if path == osapply.SNAPSHOT_LIST:
self.snap_active = True
def avail(self, n):
v = set(self.live.get(n, set()))
if self.snap_active:
v |= self.snapshot.get(n, set())
return v
def guest(self, vmid, argv, timeout=1800):
self.calls.append(("guest", vmid, argv))
a = [x for x in argv if not re.match(r"^[A-Z_]+=", x) and x != "env"]
cmd = a[0]
if cmd == "dpkg-query":
return 0, "".join(f"{n}\t{v}\tii \n" for n, v in self.installed.items()), ""
if cmd == "dpkg" and a[1] == "--compare-versions":
return (0 if dpkg_cmp(a[2], a[3], a[4]) else 1), "", ""
if cmd == "dpkg" and a[1] == "--audit":
return 0, self.dpkg_audit, ""
if cmd == "dpkg" and a[1] == "--configure":
return 0, "", ""
if cmd == "fuser":
return (0, " 123", "") if self.lock_held else (1, "", "")
if cmd == "apt-cache" and a[1] == "madison":
return 0, "".join(f" {a[2]} | {v} | http://deb.debian.org trixie/main amd64 Packages\n" for v in self.avail(a[2])), ""
if cmd == "apt-cache" and a[1] == "policy":
out = ""
for n in a[2:]:
out += f"{n}:\n Installed: {self.installed.get(n)}\n Version table:\n *** {self.installed.get(n)} 500\n 500 http://deb.debian.org/debian trixie/main amd64 Packages\n"
return 0, out, ""
if cmd == "apt-get":
if "update" in a:
return 0, "", ""
if "clean" in a:
return 0, "", ""
if "-f" in a:
self.dpkg_audit = ""
return 0, "Setting up x (1) ...\n" if getattr(self, "repaired", False) else "", ""
if "-s" in a:
return self.sim(a)
if "install" in a:
if self.install_rc:
return self.install_rc, "", "E: boom"
for x in a:
if "=" in x and not x.startswith("-") and "::" not in x:
n, v = x.split("=", 1)
self.installed[n] = v
return 0, getattr(self, "install_out", "Setting up libc6 ...\n"), ""
if cmd == "df":
return 0, f"Avail\n{self.free}\n", ""
if cmd == "docker":
return 0, "felhom-controller\trunning\tUp 1 hour (healthy)\napp\trunning\tUp 1 hour (healthy)\n", ""
if cmd == "getent":
return 0, "1.2.3.4 deb.debian.org\n", ""
if cmd == "sh":
if "os-release" in a[2]:
return 0, "trixie\n", ""
return 0, "", ""
if cmd == "rm":
self.snap_active = False
return 0, "", ""
return 1, "", f"unexpected guest call {a}"
def sim(self, a):
if "--print-uris" in a:
return 0, "'http://x/libc6.deb' libc6.deb 4000000 SHA256:x\n", ""
if "dist-upgrade" in a:
return 0, "Inst bash [5.2.37-2+b9] (5.2.37-2+b10 Debian:13.7/stable [amd64])\n", ""
out = ""
for x in a:
if "=" in x and not x.startswith("-") and "::" not in x:
n, v = x.split("=", 1)
if v not in self.avail(n):
return 100, "", f"E: Version '{v}' for '{n}' was not found"
origin = SEC if n == "openssl" else DEB
out += f"Inst {n} [{self.installed[n]}] ({v} {origin} [amd64])\n"
out += "".join(l + "\n" for l in self.extra_sim)
return 0, out, ""
def run(f):
import io
import contextlib
buf = io.StringIO()
with contextlib.redirect_stdout(buf):
rc = osapply.main(["felhom-os-apply", "--plan", PLAN], runner=f)
line = [l for l in buf.getvalue().splitlines() if l.startswith("OSAPPLY-REPORT ")][-1]
return rc, json.loads(line[len("OSAPPLY-REPORT "):])
class Happy(unittest.TestCase):
def test_apply_installs_exactly_the_plan(self):
f = Fake()
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(f.installed["libc6"], "2.41-12+deb13u4")
self.assertEqual(f.installed["openssl"], "3.5.7-1~deb13u3")
self.assertEqual(f.installed["bash"], "5.2.37-2+b9", "a package outside the plan was changed")
self.assertEqual(rep["plan"]["upgrade"], 2)
self.assertIn("installed", rep)
self.assertEqual(rep["pending"][0]["name"], "bash")
self.assertTrue(any(l.startswith("os-apply: REPAIR ") for l in f.logs), "the repair line must always print")
self.assertTrue(any(l.startswith("os-apply: DONE rc=0") for l in f.logs))
inst = [c for c in f.calls if c[0] == "guest" and "install" in c[2] and "-s" not in c[2] and "-f" not in c[2]]
self.assertTrue(inst and "Dpkg::Options::=--force-confold" in inst[0][2], "must keep existing config files")
def test_already_current_is_a_no_op(self):
f = Fake()
f.installed.update(libc6="2.41-12+deb13u4", openssl="3.5.7-1~deb13u3")
rc, rep = run(f)
self.assertEqual(rc, 0)
self.assertEqual(rep["plan"]["upgrade"], 0)
def test_inventory_installs_nothing(self):
f = Fake()
f.plan["mode"] = "inventory"
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(f.installed["libc6"], "2.41-12+deb13u3")
self.assertIn("installed", rep)
self.assertIn("restart_needed", rep)
def test_health_mode(self):
f = Fake()
f.plan["mode"] = "health"
rc, rep = run(f)
self.assertEqual(rc, 0)
self.assertEqual(rep["health"]["controller"], "healthy")
class Repair(unittest.TestCase):
def test_repair_runs_first_and_is_reported(self):
f = Fake()
f.dpkg_audit = "The following packages have been unpacked but not yet configured.\n perl Larry Wall's\n"
f.repaired = True
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["repair"]["half_configured_before"], 1)
self.assertEqual(rep["repair"]["fixed"], 1)
order = [i for i, c in enumerate(f.calls) if c[0] == "guest" and c[2][-1:] != ["update"]]
first_cfg = next(i for i, c in enumerate(f.calls) if c[0] == "guest" and "--configure" in c[2])
first_upd = next(i for i, c in enumerate(f.calls) if c[0] == "guest" and "update" in c[2])
self.assertLess(first_cfg, first_upd, "the repair must run before anything else touches apt")
self.assertTrue(order)
class Snapshot(unittest.TestCase):
def test_a_replaced_version_comes_from_the_snapshot(self):
f = Fake()
f.live["openssl"] = {"3.5.7-1~deb13u4"} # Debian moved on
f.snapshot["openssl"] = {"3.5.7-1~deb13u3"}
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["plan"]["from_snapshot"], 1)
self.assertEqual(f.installed["openssl"], "3.5.7-1~deb13u3", "must install the APPROVED version, not the newer one")
body = f.written[osapply.SNAPSHOT_LIST]
self.assertIn("snapshot.debian.org/archive/debian/20261004T080000Z trixie main", body)
self.assertIn("debian-security/20261004T080000Z trixie-security main", body)
self.assertFalse(f.snap_active, "the temporary snapshot sources must be removed after the run")
def test_snapshot_does_not_have_it_either(self):
f = Fake()
f.live["openssl"] = set()
rc, rep = run(f)
self.assertEqual((rc, rep["refused"]["code"]), (2, "R7"))
self.assertFalse(f.snap_active)
class Refusals(unittest.TestCase):
def refused(self, f, code):
rc, rep = run(f)
self.assertEqual(rc, 2, rep)
self.assertEqual(rep["refused"]["code"], code, rep)
self.assertTrue(any(l.startswith(f"os-apply: REFUSED: {code} ") for l in f.logs), f.logs)
inst = [c for c in f.calls if c[0] == "guest" and "install" in c[2] and "-s" not in c[2] and "-f" not in c[2]]
self.assertEqual(inst, [], "a refusal must install nothing")
return rep
def test_R1_usage(self):
import io
import contextlib
f = Fake()
with contextlib.redirect_stdout(io.StringIO()):
self.assertEqual(osapply.main(["felhom-os-apply", "--plan", PLAN, "--extra"], runner=f), 2)
self.assertEqual(osapply.main(["felhom-os-apply", "--plan"], runner=f), 2)
def test_R1_path_outside_the_plan_dir(self):
import io
import contextlib
f = Fake()
with contextlib.redirect_stdout(io.StringIO()):
rc = osapply.main(["felhom-os-apply", "--plan", "/tmp/plan-x.json"], runner=f)
self.assertEqual(rc, 2)
self.assertTrue(any("R1" in l for l in f.logs))
def test_R1_symlink(self):
f = Fake()
f.stats[PLAN] = St(mode=statmod.S_IFLNK | 0o777)
self.refused(f, "R1")
def test_R1_not_owned_by_the_agent(self):
f = Fake()
f.stats[PLAN] = St(uid=0)
self.refused(f, "R1")
def test_R2_non_debian_origin_in_the_plan(self):
f = Fake()
f.plan["packages"][0]["origin"] = "Proxmox"
self.refused(f, "R2")
def test_R2_non_debian_origin_in_the_simulation(self):
f = Fake()
f.installed["libc6"] = "2.41-12+deb13u3"
orig = f.sim
def sim(a):
rc, out, err = orig(a)
return rc, out.replace("Debian:13.7/stable", "Proxmox Debian Repository:stable"), err
f.sim = sim
self.refused(f, "R2")
def test_R3_slow_lane(self):
f = Fake()
f.plan["lane"] = "slow"
self.refused(f, "R3")
def test_R4_removal(self):
f = Fake()
f.extra_sim = ["Remv bash [5.2.37-2+b9]"]
self.refused(f, "R4")
def test_R5_downgrade(self):
f = Fake()
f.installed["libc6"] = "2.41-12+deb13u4"
f.plan["packages"] = [{"name": "openssl", "version": "3.5.7-1~deb13u3", "origin": "Debian-Security"}]
f.extra_sim = ["Inst openssl [3.5.6-1~deb13u1] (3.5.5-1 Debian:13.7/stable [amd64])"]
orig = f.sim
def sim(a): # the simulation answers with a LOWER version than installed
rc, out, err = orig(a)
return rc, "\n".join(l for l in out.splitlines() if not l.startswith("Inst openssl [3.5.6-1~deb13u1] (3.5.7")) + "\n", err
f.sim = sim
f.plan["packages"][0]["version"] = "3.5.7-1~deb13u3"
rep = run(f)[1]
# The plan asks 3.5.7; the simulation goes to 3.5.5: that is BOTH a wrong version (R6) and a downgrade.
self.assertIn(rep["refused"]["code"], ("R5", "R6"))
def test_R5_downgrade_exact(self):
f = Fake()
f.plan["packages"] = [{"name": "openssl", "version": "3.5.7-1~deb13u3", "origin": "Debian-Security"}]
f.installed["openssl"] = "3.5.6-1~deb13u1"
orig = f.sim
def sim(a):
rc, out, err = orig(a)
return rc, out.replace("[3.5.6-1~deb13u1]", "[3.5.8-1]"), err
f.sim = sim
self.refused(f, "R5")
def test_R6_new_package(self):
f = Fake()
f.extra_sim = ["Inst newthing (1.0 Debian:13.7/stable [amd64])"]
self.refused(f, "R6")
def test_R6_unlisted_package(self):
f = Fake()
f.extra_sim = ["Inst bash [5.2.37-2+b9] (5.2.37-2+b10 Debian:13.7/stable [amd64])"]
self.refused(f, "R6")
def test_R6_allow_new_is_slow_lane(self):
f = Fake()
f.plan["allow_new"] = ["proxmox-kernel-x"]
self.refused(f, "R6")
def test_R7_not_downloadable_and_no_snapshot(self):
f = Fake()
f.live["openssl"] = set()
f.plan["snapshot"] = ""
self.refused(f, "R7")
def test_R8_free_space(self):
f = Fake()
f.free = 100 * 1024 * 1024
self.refused(f, "R8")
def test_R9_guest_locked_by_a_backup(self):
f = Fake()
f.files["/etc/pve/lxc/9201.conf"] = CONF_OK + "lock: backup\n"
self.refused(f, "R9")
def test_R9_apt_lock_held(self):
f = Fake()
f.lock_held = True
self.refused(f, "R9")
def test_R10_not_the_boxs_own_guest(self):
f = Fake()
f.files["/etc/pve/lxc/9201.conf"] = CONF_OK.replace("mp8: /mnt/felhom-drives,", "mp8: /mnt/hdd_1/scratch,")
self.refused(f, "R10")
def test_R10_reserved_vmid(self):
f = Fake()
f.plan["vmid"] = 990003
self.refused(f, "R10")
def test_R10_bind_only_in_a_snapshot_section(self):
f = Fake()
f.files["/etc/pve/lxc/9201.conf"] = "rootfs: x\n[snap1]\nmp8: /mnt/felhom-drives,mp=/mnt/felhom-drives\n"
self.refused(f, "R10")
def test_R10_not_running(self):
f = Fake()
f.status = "status: stopped"
self.refused(f, "R10")
def test_R11_duplicate(self):
f = Fake()
f.plan["packages"].append(dict(f.plan["packages"][0]))
self.refused(f, "R11")
def test_R11_bad_version_string(self):
f = Fake()
f.plan["packages"][0]["version"] = "1.0; rm -rf /"
self.refused(f, "R11")
def test_R11_bad_name(self):
f = Fake()
f.plan["packages"][0]["name"] = "--purge"
self.refused(f, "R11")
def test_R12_host_layer(self):
f = Fake()
f.plan["layer"] = "host"
self.refused(f, "R12")
def test_R13_repair_does_not_fix_it(self):
f = Fake()
f.dpkg_audit = "The following packages are broken\n perl\n"
orig = f.guest
def guest(vmid, argv, timeout=1800):
rc, out, err = orig(vmid, argv, timeout)
if "-f" in argv:
f.dpkg_audit = "The following packages are broken\n perl\n"
return rc, out, err
f.guest = guest
self.refused(f, "R13")
class Conffiles(unittest.TestCase):
# dpkg's two shapes, measured live 2026-10-04: an unchanged file is UPDATED; a locally changed one is KEPT.
def test_updated_vs_kept(self):
f = Fake()
f.install_out = ("Installing new version of config file /etc/debian_version ...\n"
"Configuration file '/etc/ssh/sshd_config'\n ==> Modified (by you or by a script) since installation.\n"
" ==> Keeping old config file as default.\n")
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertIn("os-apply: CONFFILE updated /etc/debian_version (it was not changed locally)", f.logs)
self.assertTrue(any(l.startswith("os-apply: CONFFILE kept /etc/ssh/sshd_config") for l in f.logs), f.logs)
self.assertEqual(rep["conffiles_kept"], ["/etc/ssh/sshd_config"])
self.assertFalse(any("kept /etc/debian_version" in l for l in f.logs), "an updated file must not be reported as kept")
class Failure(unittest.TestCase):
def test_install_failure_is_rc3_with_dpkg_state(self):
f = Fake()
f.install_rc = 100
rc, rep = run(f)
self.assertEqual(rc, 3)
self.assertEqual(rep["failed"]["rc"], 100)
self.assertTrue(any(l.startswith("os-apply: FAILED rc=100 step=install") for l in f.logs))
if __name__ == "__main__":
unittest.main()
+10 -1
View File
@@ -2,6 +2,7 @@ package backup
import ( import (
"context" "context"
"fmt"
"encoding/json" "encoding/json"
"errors" "errors"
"io" "io"
@@ -21,6 +22,8 @@ type fakeBackupAPI struct {
vzdumpErr error vzdumpErr error
waitErr error waitErr error
cfg proxmox.GuestConfig cfg proxmox.GuestConfig
goneGuests map[int]bool // R-689: vmids whose config lookup answers "does not exist"
aclGuests map[int]bool // R-689: vmids outside the token's ACL — PVE answers 403 "permission denied"
cfgErr error cfgErr error
content []proxmox.StorageContent content []proxmox.StorageContent
contentErr error contentErr error
@@ -43,7 +46,13 @@ func (f *fakeBackupAPI) WaitTask(_ context.Context, _ string, _ proxmox.WaitOpti
} }
return proxmox.TaskStatus{Status: "stopped", ExitStatus: "OK"}, f.waitErr return proxmox.TaskStatus{Status: "stopped", ExitStatus: "OK"}, f.waitErr
} }
func (f *fakeBackupAPI) GuestConfig(_ context.Context, _ int) (proxmox.GuestConfig, error) { func (f *fakeBackupAPI) GuestConfig(_ context.Context, vmid int) (proxmox.GuestConfig, error) {
if f.aclGuests[vmid] {
return proxmox.GuestConfig{}, fmt.Errorf("proxmox: GET /nodes/n/lxc/%d/config -> HTTP 403: permission denied at /vms/%d (missing privilege VM.Audit)", vmid, vmid)
}
if f.goneGuests[vmid] { // R-689: PVE's answer for a deleted guest
return proxmox.GuestConfig{}, fmt.Errorf("proxmox: GET /nodes/n/lxc/%d/config -> HTTP 500: Configuration file 'nodes/n/lxc/%d.conf' does not exist", vmid, vmid)
}
return f.cfg, f.cfgErr return f.cfg, f.cfgErr
} }
func (f *fakeBackupAPI) StorageContent(_ context.Context, _ string) ([]proxmox.StorageContent, error) { func (f *fakeBackupAPI) StorageContent(_ context.Context, _ string) ([]proxmox.StorageContent, error) {
+56
View File
@@ -2,6 +2,7 @@ package backup
import ( import (
"context" "context"
"fmt"
"testing" "testing"
"time" "time"
@@ -54,3 +55,58 @@ func TestR689_GuestBackupArchiveShapes(t *testing.T) {
} }
} }
} }
// R-689 (v0.136.0) — the measured demo-hp shape right after v0.135.0: the golden (skipped), a leftover archive
// of guest 9100 deleted in August (settled), and today's archive of 9201 (not settled yet). The pick must be
// NOTHING — never the deleted guest's archive. With 9201's archive settled, that one.
//
// COMPANION RED-PROOF (REPORT.md): drop the known-guest check — the pick is the 9100 leftover.
func TestR689_AnArchiveOfADeletedGuestIsNeverPicked(t *testing.T) {
const day = int64(86400)
now := int64(1790476000)
api := &fakeBackupAPI{goneGuests: map[int]bool{9100: true}, content: []proxmox.StorageContent{
{VolID: "local:backup/felhom-golden-0.236.0.tar.zst", Content: "backup", Size: 654115664, CTime: now - 14*day},
{VolID: "local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst", Content: "backup", VMID: 9100, Size: 656970239, CTime: now - 37*day},
{VolID: "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 7*3600},
}}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now-day, 0).UTC())
if err != nil || got != "" {
t.Fatalf("picked %q err=%v — a deleted guest's archive proves nothing about this box", got, err)
}
got, _, _ = r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now, 0).UTC())
if got != "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst" {
t.Fatalf("with 9201's archive settled the pick is %q", got)
}
}
// Any OTHER lookup failure is not "the guest is gone": the tier must read UNKNOWN (an error), never
// "nothing to prove".
func TestR689_AGuestLookupFailureIsUnknownNotEmpty(t *testing.T) {
api := &fakeBackupAPI{cfgErr: fmt.Errorf("proxmox: connection refused"), content: []proxmox.StorageContent{
{VolID: "local:backup/vzdump-lxc-9201-x.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: 10},
}}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Time{}); err == nil {
t.Fatal("a failed guest lookup read as a clean answer")
}
}
// v0.137.0 — THE MEASURED ANSWER: the agent's token sees only its pool, so for the deleted guest PVE says 403
// "permission denied at /vms/9100", not "does not exist" (demo-hp, right after v0.136.0 — the local tier read
// UNKNOWN). Such a guest is not one this agent manages: its archive is skipped, the tier is not an error.
//
// COMPANION RED-PROOF (REPORT.md): drop the "permission denied" case — the pick errors.
func TestR689_AGuestOutsideTheAgentsACLIsNotAKnownGuest(t *testing.T) {
const day = int64(86400)
now := int64(1790476000)
api := &fakeBackupAPI{aclGuests: map[int]bool{9100: true}, content: []proxmox.StorageContent{
{VolID: "local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst", Content: "backup", VMID: 9100, Size: 656970239, CTime: now - 37*day},
{VolID: "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 7*3600},
}}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now-day, 0).UTC())
if err != nil || got != "" {
t.Fatalf("picked %q err=%v — want nothing and no error (the only settled archive is not ours)", got, err)
}
}
+74
View File
@@ -0,0 +1,74 @@
package backup
import (
"context"
"errors"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
const (
thisBoxKey = "de:51:7a:18:cb:39:22:30:2c:84:f5:8b:d1:91:4b:7e:81:bb:69:b8:89:0f:57:ac:d3:59:e1:1a:62:25:11:2c"
earlierBox1 = "6b:ca:5f:3f:ca:0f:e2:3f:fb:24:62:89:bf:e7:64:59:9a:41:c5:e6:e3:9f:3f:5f:e1:71:7b:a1:9d:24:67:82"
earlierBox2 = "fe:3d:db:95:d4:df:ab:e1:7d:4a:89:fa:2b:07:53:6a:e4:d2:85:95:d1:90:27:4b:d9:c6:92:20:95:04:e5:d4"
)
// R-727 (v0.138.0) — the 2026-09-30 shape, measured on a fresh box for a returning customer: the PBS
// namespace held two archives of earlier boxes (same guest 9201, same token) and this box's own, which was not
// settled yet. The old picker chose the earlier box's newest settled archive and failed `wrong key`.
// The CONSEQUENCE asserted: no archive of another box is ever picked; with this box's archive settled it is picked.
// COMPANION RED-PROOF: remove the `ownKey != "" && !EqualFold(...)` skip → the first case picks 2026-09-16T21:59:54Z.
func TestR727_TheRestoreTestTakesOnlyThisBoxsArchives(t *testing.T) {
day := int64(86400)
now := int64(1790740000) // 2026-09-30 ~04:00Z
own := proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-09-29T19:37:07Z", Content: "backup", VMID: 9201, Size: 3490689830, CTime: 1790710627, Encrypted: thisBoxKey}
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "felhom-pbs", Type: "pbs", EncryptionKey: thisBoxKey}},
content: []proxmox.StorageContent{
{VolID: "felhom-pbs:backup/ct/9201/2026-09-16T17:27:32Z", Content: "backup", VMID: 9201, Size: 4774114206, CTime: 1789579652, Encrypted: earlierBox2},
{VolID: "felhom-pbs:backup/ct/9201/2026-09-16T21:59:54Z", Content: "backup", VMID: 9201, Size: 20811501236, CTime: 1789595994, Encrypted: earlierBox1},
own,
},
}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
// 1. The night of 2026-09-30: this box's own archive is ~6 h old, not settled (cutoff 24 h) — nothing to prove.
got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Unix(now-day, 0).UTC())
if err != nil {
t.Fatal(err)
}
if got != "" {
t.Fatalf("picked %q — an archive of ANOTHER box is never this box's proof (R-727)", got)
}
// 2. A day later this box's own archive is settled — it is the one picked.
got, _, err = r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Unix(now+day, 0).UTC())
if err != nil {
t.Fatal(err)
}
if got != own.VolID {
t.Fatalf("picked %q, want this box's own %q", got, own.VolID)
}
}
// An unencrypted storage (a local dir) holds only this box's vzdumps — no key filter applies.
func TestR727_UnencryptedStorageIsNotFiltered(t *testing.T) {
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "local", Type: "dir"}},
content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-2026_09_29-21_27_05.tar.zst", Content: "backup", VMID: 9201, Size: 955425507, CTime: 1790710025}},
}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
if got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Time{}); err != nil || got == "" {
t.Fatalf("got %q err %v", got, err)
}
}
// A storage-list failure makes the tier UNKNOWN (an error), never "nothing to prove".
func TestR727_KeyLookupFailureIsUnknown(t *testing.T) {
api := &fakeBackupAPI{storageErr: errors.New("proxmox: GET /storage -> HTTP 500"), content: []proxmox.StorageContent{{VolID: "felhom-pbs:backup/ct/9201/x", Content: "backup", VMID: 9201}}}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Time{}); err == nil {
t.Fatal("a failed key lookup must surface as an error (tier UNKNOWN)")
}
}
+64
View File
@@ -357,8 +357,19 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
if err != nil { if err != nil {
return "", time.Time{}, err return "", time.Time{}, err
} }
// R-727 (v0.138.0): on an ENCRYPTED storage, only archives written with THIS storage's key are this box's.
// Measured 2026-09-30 on a fresh box for a returning customer: the PBS namespace still held two archives
// of earlier boxes (same guest id 9201, same token), the newest settled one was an earlier box's, and the
// test failed `wrong key` every evaluation. The archive carries no host id; its key fingerprint is the
// discriminator (PVE's content `encrypted`, the storage's `encryption-key`). A lookup failure returns
// the error — the tier reads UNKNOWN, never "nothing to prove".
ownKey, err := r.storageKeyFingerprint(ctx, target)
if err != nil {
return "", time.Time{}, fmt.Errorf("reading the key fingerprint of storage %s: %w", target, err)
}
var best string var best string
var bestCTime int64 = -1 var bestCTime int64 = -1
known := map[int]bool{} // vmid → the guest exists on this node (asked once per vmid per pick)
for _, e := range contents { for _, e := range contents {
if e.Content != "backup" { if e.Content != "backup" {
continue continue
@@ -371,6 +382,33 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
r.noteNotAGuestBackupOnce(e, why) r.noteNotAGuestBackupOnce(e, why)
continue continue
} }
if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) {
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this box's proof", shortFP(e.Encrypted), shortFP(ownKey)))
continue
}
// R-689 (v0.136.0): … OF A GUEST THAT STILL EXISTS here. Measured on demo-hp 2026-09-27 right after
// v0.135.0: with the golden skipped, the pick fell to `vzdump-lxc-9100-2026_08_21…`, a leftover of a
// guest deleted in August — proving nothing about any guest this box runs. "Does not exist" skips the
// archive; any OTHER lookup failure is returned, so the tier reads UNKNOWN, never "nothing to prove".
if _, seen := known[e.VMID]; !seen {
_, err := r.api.GuestConfig(ctx, e.VMID)
switch {
case err == nil:
known[e.VMID] = true
case strings.Contains(err.Error(), "does not exist"), strings.Contains(err.Error(), "permission denied"):
// v0.137.0: PVE answers 403 "permission denied at /vms/<id>" — not "does not exist" — for a guest
// outside the agent's ACL (the `felhom` pool). Measured on demo-hp after v0.136.0: the deleted
// guest 9100's archive made the local tier UNKNOWN every evaluation. A guest the agent cannot
// read is not one it manages; its archive is not a candidate.
known[e.VMID] = false
default:
return "", time.Time{}, fmt.Errorf("checking whether guest %d still exists: %w", e.VMID, err)
}
}
if !known[e.VMID] {
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("guest %d does not exist on this node or is not one this agent manages", e.VMID))
continue
}
if !notAfter.IsZero() && e.CTime > notAfter.Unix() { if !notAfter.IsZero() && e.CTime > notAfter.Unix() {
continue // not settled yet — a newer archive is not a reason to re-prove an older one continue // not settled yet — a newer archive is not a reason to re-prove an older one
} }
@@ -643,3 +681,29 @@ func (r *BackupRunner) noteNotAGuestBackupOnce(e proxmox.StorageContent, why str
"target", r.target, "volid", e.VolID, "size_bytes", e.Size, "reason", why) "target", r.target, "volid", e.VolID, "size_bytes", e.Size, "reason", why)
} }
} }
// storageKeyFingerprint returns the named storage's client-side encryption key fingerprint ("" when the
// storage is not encrypted — a local dir holds only this box's own vzdumps).
func (r *BackupRunner) storageKeyFingerprint(ctx context.Context, target string) (string, error) {
sts, err := r.api.ListStorage(ctx)
if err != nil {
return "", err
}
for _, st := range sts {
if st.Storage == target {
return strings.TrimSpace(st.EncryptionKey), nil
}
}
return "", nil
}
// shortFP is the first 8 bytes of a key fingerprint, for a log line.
func shortFP(fp string) string {
if fp == "" {
return "none"
}
if len(fp) > 23 {
return fp[:23] + "…"
}
return fp
}
+3
View File
@@ -123,6 +123,9 @@ var manifest = []Capability{
{"dnsmasq-install", "dnsmasq package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "dnsmasq"}, false, ""}, {"dnsmasq-install", "dnsmasq package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "dnsmasq"}, false, ""},
{"dnsmasq-write", "dnsmasq drop-in write", "/usr/bin/install", []string{"-m", "0644", "/tmp/felhom-resolver-x.conf", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""}, {"dnsmasq-write", "dnsmasq drop-in write", "/usr/bin/install", []string{"-m", "0644", "/tmp/felhom-resolver-x.conf", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""},
{"dnsmasq-enable", "dnsmasq enable", "/usr/bin/systemctl", []string{"enable", "--now", "dnsmasq"}, false, ""}, {"dnsmasq-enable", "dnsmasq enable", "/usr/bin/systemctl", []string{"enable", "--now", "dnsmasq"}, false, ""},
// ---- OS updates, guest fast lane (`11` §5.4.1; the wrapper holds every rule) ----
{"osapply-run", "OS update wrapper (guest fast lane)", "/usr/local/sbin/felhom-os-apply", []string{"--plan", "/var/lib/felhom-agent/os/plan-x.json"}, false, ""},
{"dnsmasq-reload", "dnsmasq reload", "/usr/bin/systemctl", []string{"reload", "dnsmasq"}, false, ""}, {"dnsmasq-reload", "dnsmasq reload", "/usr/bin/systemctl", []string{"reload", "dnsmasq"}, false, ""},
{"dnsmasq-restart", "dnsmasq restart (LAN-DNS self-heal)", "/usr/bin/systemctl", []string{"restart", "dnsmasq"}, false, ""}, {"dnsmasq-restart", "dnsmasq restart (LAN-DNS self-heal)", "/usr/bin/systemctl", []string{"restart", "dnsmasq"}, false, ""},
{"dnsmasq-rm", "dnsmasq drop-in remove (decommission)", "/usr/bin/rm", []string{"-f", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""}, {"dnsmasq-rm", "dnsmasq drop-in remove (decommission)", "/usr/bin/rm", []string{"-f", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""},
+24
View File
@@ -425,3 +425,27 @@ func (c *Client) FetchRetainedIdentityEscrow(ctx context.Context) (*RetainedEscr
} }
return &out, nil return &out, nil
} }
// PostOSReport sends the OS-update leg's report after every run (hub v0.130.0): POST /api/v1/hosts/{id}/os-report.
// Per-host key, self-scoped on the hub. Errors are typed like RegisterWG's and never include the bearer.
func (c *Client) PostOSReport(ctx context.Context, body []byte) error {
if c.hostID == "" {
return fmt.Errorf("hub: PostOSReport requires a configured host_id")
}
req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+"/api/v1/hosts/"+c.hostID+"/os-report", bytes.NewReader(body))
if err != nil {
return fmt.Errorf("hub: building os-report request: %w", err)
}
req.Header.Set("Authorization", "Bearer "+c.apiKey)
req.Header.Set("Content-Type", "application/json")
resp, err := c.hc.Do(req)
if err != nil {
return &TransportError{Err: err}
}
defer resp.Body.Close()
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return &HTTPError{StatusCode: resp.StatusCode, BodyTail: tail(raw, 256)}
}
return nil
}
+30
View File
@@ -0,0 +1,30 @@
package hub
import (
"encoding/json"
"os"
"testing"
)
// The os_update block is a cross-repo contract: testdata/desired-state-osupdate.golden.json is byte-identical with
// felhom.eu/hub/internal/api/testdata (the hub's TestOSUpdate_DesiredBlockMatchesTheGolden proves the hub SERVES
// it). Here: the agent DECODES every field. A renamed json tag on either side fails one of the two tests.
func TestOSUpdateGolden_Decodes(t *testing.T) {
raw, err := os.ReadFile("testdata/desired-state-osupdate.golden.json")
if err != nil {
t.Fatal(err)
}
var resp DesiredStateResponse
if err := json.Unmarshal(raw, &resp); err != nil {
t.Fatal(err)
}
o := resp.DesiredState.OSUpdate
if o == nil || o.Ring != 1 || !o.Enabled || o.Release == nil {
t.Fatalf("os_update = %+v", o)
}
r := o.Release
if r.ID != "os-20261004-120000" || r.Snapshot != "20261004T120000Z" || len(r.Packages) != 2 ||
r.Packages[1].Name != "openssl" || r.Packages[1].Version != "3.5.7-1~deb13u3" || r.Packages[1].Origin != "Debian-Security" {
t.Fatalf("release = %+v", r)
}
}
+26
View File
@@ -548,6 +548,32 @@ type WireDesiredState struct {
RestoreDirective *WireRestoreDirective `json:"restore_directive,omitempty"` // slice 10D (forward-compat) RestoreDirective *WireRestoreDirective `json:"restore_directive,omitempty"` // slice 10D (forward-compat)
Wireguard *WireWireguard `json:"wireguard,omitempty"` // S3 (doc 06 §3.2; golden-pinned) Wireguard *WireWireguard `json:"wireguard,omitempty"` // S3 (doc 06 §3.2; golden-pinned)
PBSDR *WirePBSDR `json:"pbs_dr,omitempty"` // PBS DR tier (slice 2 consumer) PBSDR *WirePBSDR `json:"pbs_dr,omitempty"` // PBS DR tier (slice 2 consumer)
OSUpdate *WireOSUpdate `json:"os_update,omitempty"` // OS updates, guest fast lane (agent v0.140.0)
}
// WireOSUpdate is the hub-OWNED OS-update block (hub v0.130.0, `11-os-updates.md` §5.3), merged into the served
// document at read time. Ring 0 installs every pending Debian / Debian-Security fix; ring 1 installs exactly the
// newest approved release. Absent (older hub) → the agent treats the box as ring 1, ON, no release: it reports
// and installs nothing. Golden: testdata/desired-state-osupdate.golden.json (byte-identical with the hub's).
type WireOSUpdate struct {
Ring int `json:"ring"`
Enabled bool `json:"enabled"`
Release *WireOSRelease `json:"release,omitempty"`
}
// WireOSRelease is an approved version set; Snapshot is the approval time (YYYYMMDDTHHMMSSZ) the wrapper uses
// for snapshot.debian.org when Debian has already replaced a version (decision 79).
type WireOSRelease struct {
ID string `json:"id"`
Snapshot string `json:"snapshot"`
Packages []WireOSPackage `json:"packages"`
}
// WireOSPackage is one approved name=version and its origin ("Debian" | "Debian-Security").
type WireOSPackage struct {
Name string `json:"name"`
Version string `json:"version"`
Origin string `json:"origin"`
} }
// WirePBSDR is the hub's PBS-DR-tier descriptor (PBS DR slice 1, hub/internal/web/pbsdr.go // WirePBSDR is the hub's PBS-DR-tier descriptor (PBS DR slice 1, hub/internal/web/pbsdr.go
@@ -0,0 +1,17 @@
{
"generation": 1,
"desired_state": {
"os_update": {
"ring": 1,
"enabled": true,
"release": {
"id": "os-20261004-120000",
"snapshot": "20261004T120000Z",
"packages": [
{"name": "libc6", "version": "2.41-12+deb13u4", "origin": "Debian"},
{"name": "openssl", "version": "3.5.7-1~deb13u3", "origin": "Debian-Security"}
]
}
}
}
}
+63
View File
@@ -0,0 +1,63 @@
package localapi
import (
"context"
"net/http"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
)
// The OS leg (agent v0.140.0) runs after a SUCCESSFUL primary backup, and only then; and it runs BEFORE the
// host-wide heavy-op gate is released, so a restore-test cannot start in the middle of it (`11` C10).
// Red-proof: drop the `b.Success &&` guard and the failed-backup sub-case fails; move the call after release()
// and the gate sub-case fails.
func TestAfterPrimaryBackup(t *testing.T) {
run := func(t *testing.T, failErr string) (calls []int, gateHeld bool) {
gate := &backup.InFlight{}
b := &fakeBackups{failErr: failErr}
srv := newTestServerS(t, &fakeGuests{}, b, &fakeStore{}, nil)
srv.inFlight = gate
var mu sync.Mutex
done := make(chan struct{}, 1)
srv.SetAfterPrimaryBackup(func(_ context.Context, vmid int) {
rel, _, ok := gate.TryAcquire("probe")
mu.Lock()
calls = append(calls, vmid)
gateHeld = !ok
mu.Unlock()
if ok {
rel()
}
done <- struct{}{}
})
h := srv.Handler()
if do(t, h, "POST", "/backup", "A", "").Code != http.StatusAccepted {
t.Fatal("POST /backup not accepted")
}
select {
case <-done:
case <-time.After(500 * time.Millisecond):
}
time.Sleep(20 * time.Millisecond)
mu.Lock()
defer mu.Unlock()
return calls, gateHeld
}
t.Run("success runs the leg under the gate", func(t *testing.T) {
calls, held := run(t, "")
if len(calls) != 1 {
t.Fatalf("the leg ran %d time(s), want 1", len(calls))
}
if !held {
t.Fatal("the heavy-op gate was free while the leg ran — a restore-test could overlap it")
}
})
t.Run("a failed backup runs nothing", func(t *testing.T) {
if calls, _ := run(t, "vzdump exploded"); len(calls) != 0 {
t.Fatalf("the leg ran after a FAILED backup: %v", calls)
}
})
}
+13
View File
@@ -160,6 +160,10 @@ type Options struct {
// NetStorage is the privileged network-mount (NAS) surface (Part A1). OPTIONAL — when nil, the // NetStorage is the privileged network-mount (NAS) surface (Part A1). OPTIONAL — when nil, the
// /netstorage endpoints report "not configured". Satisfied by *storage.SudoHostOps. // /netstorage endpoints report "not configured". Satisfied by *storage.SudoHostOps.
NetStorage NetworkStorageOps NetStorage NetworkStorageOps
// AfterPrimaryBackup (agent v0.140.0, `11-os-updates.md` §8 step 2) runs right after a SUCCESSFUL backup on the
// PRIMARY tier, inside the backup goroutine and BEFORE the host-wide heavy-op gate is released — so the OS leg
// that it starts can never overlap another backup or a restore-test (`11` C10). OPTIONAL — nil → nothing runs.
AfterPrimaryBackup func(ctx context.Context, vmid int)
// Privileged runs the fenced root wrappers (E-2a: felhom-backup-target-apply). OPTIONAL — when // Privileged runs the fenced root wrappers (E-2a: felhom-backup-target-apply). OPTIONAL — when
// nil, POST /backup/target reports "not configured". Satisfied by *proxmox.ExecRunner. // nil, POST /backup/target reports "not configured". Satisfied by *proxmox.ExecRunner.
Privileged PrivilegedRunner Privileged PrivilegedRunner
@@ -276,6 +280,7 @@ type Server struct {
tiers []BackupTier tiers []BackupTier
// inFlight (R-85) is shared with the restore-test scheduler so the two never run together. // inFlight (R-85) is shared with the restore-test scheduler so the two never run together.
inFlight *backup.InFlight inFlight *backup.InFlight
afterPrimaryBackup func(ctx context.Context, vmid int) // the OS leg (agent v0.140.0); nil = none
logger *slog.Logger logger *slog.Logger
now func() time.Time now func() time.Time
@@ -469,6 +474,7 @@ func NewServer(o Options) (*Server, error) {
// the primary is always first, because that is what the untargeted endpoints act on. // the primary is always first, because that is what the untargeted endpoints act on.
s.tiers = normalizeBackupTiers(o.BackupTiers, o.Backups, cadence) s.tiers = normalizeBackupTiers(o.BackupTiers, o.Backups, cadence)
s.inFlight = o.InFlight s.inFlight = o.InFlight
s.afterPrimaryBackup = o.AfterPrimaryBackup
if s.backups == nil && len(s.tiers) > 0 { if s.backups == nil && len(s.tiers) > 0 {
s.backups = s.tiers[0].Service s.backups = s.tiers[0].Service
} }
@@ -894,6 +900,10 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
} }
s.store.RecordBackup(b) s.store.RecordBackup(b)
s.finishJob(key, jobID, b) s.finishJob(key, jobID, b)
// OS leg (agent v0.140.0): after the night's whole-guest copy exists, still holding the heavy-op gate.
if b.Success && tier.Primary && s.afterPrimaryBackup != nil {
s.afterPrimaryBackup(base, vmid)
}
}() }()
writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: jobID, Phase: PhaseRunning}, "") writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: jobID, Phase: PhaseRunning}, "")
} }
@@ -1465,3 +1475,6 @@ func writeStatus(w http.ResponseWriter, code int, ok bool, data any, errMsg stri
w.WriteHeader(code) w.WriteHeader(code)
_ = json.NewEncoder(w).Encode(apiResponse{OK: ok, Data: data, Error: errMsg}) _ = json.NewEncoder(w).Encode(apiResponse{OK: ok, Data: data, Error: errMsg})
} }
// SetAfterPrimaryBackup wires the hook that runs after a successful primary-tier backup (the OS leg, agent v0.140.0).
func (s *Server) SetAfterPrimaryBackup(fn func(ctx context.Context, vmid int)) { s.afterPrimaryBackup = fn }
+459
View File
@@ -0,0 +1,459 @@
// Package osupdate is the agent's OS-update leg for the customer GUEST (`11-os-updates.md` §8 step 2, agent v0.140.0).
//
// It runs right after the night's successful whole-guest backup, while the backup goroutine still holds the host-wide
// heavy-op gate (so it never overlaps a backup or a restore-test, `11` C10), at most once per night. All root work is
// the wrapper `felhom-os-apply` (configs/, its own tests); this package only builds plans, calls the wrapper through
// sudo, judges health and reports to the hub.
//
// NO AUTOMATIC UNDO in this release (R-837, measured 2026-10-04): a customer guest cannot be snapshotted — PVE
// refuses any snapshot not named `vzdump` when the guest has host-path binds (mp8, mp9). A failed health check
// therefore stops, reports `health_failed` and the hub mails the operator; the whole-guest backup taken minutes
// earlier is the undo, by hand.
package osupdate
import (
"context"
"encoding/json"
"fmt"
"log/slog"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// WrapperPath is the pinned sudoers vector (configs/felhom-agent.sudoers FELHOM_OSAPPLY).
const WrapperPath = "/usr/local/sbin/felhom-os-apply"
// DefaultPlanDir is where plans are written (the sudoers glob names it).
const DefaultPlanDir = "/var/lib/felhom-agent/os"
// Package is one name=version with its origin.
type Package struct {
Name string `json:"name"`
Version string `json:"version"`
Origin string `json:"origin"`
}
// Pending is one update the guest's sources offer (origin as apt names it, possibly several).
type Pending struct {
Name string `json:"name"`
From string `json:"from"`
To string `json:"to"`
Origin []string `json:"origin"`
}
// Container is one container's state as the wrapper saw it.
type Container struct {
State string `json:"state"`
Health string `json:"health"` // healthy | unhealthy | starting | none
}
// Health is the guest's health snapshot (the wrapper's `health` object).
type Health struct {
DockerOK bool `json:"docker_ok"`
NetworkOK bool `json:"network_ok"`
Controller string `json:"controller"`
Containers map[string]Container `json:"containers"`
}
// WrapperReport is the wrapper's OSAPPLY-REPORT object.
type WrapperReport struct {
Mode string `json:"mode"`
Refused json.RawMessage `json:"refused"`
Failed json.RawMessage `json:"failed"`
Upgraded []Package `json:"upgraded"`
Installed []Package `json:"installed"`
Pending []Pending `json:"pending"`
RestartNeeded []string `json:"restart_needed"`
DockerRestartNeeded bool `json:"docker_restart_needed"`
RebootNeeded bool `json:"reboot_needed"`
HealthBefore *Health `json:"health_before"`
HealthAfter *Health `json:"health_after"`
Health *Health `json:"health"`
}
func (w WrapperReport) refused() bool { return len(w.Refused) > 0 && string(w.Refused) != "null" }
func (w WrapperReport) failed() bool { return len(w.Failed) > 0 && string(w.Failed) != "null" }
// Report is what the hub receives (hub osupdates.Report — field-exact).
type Report struct {
RunID string `json:"run_id"`
Trigger string `json:"trigger"`
Mode string `json:"mode"`
Ring int `json:"ring"`
ReleaseID string `json:"release_id"`
Outcome string `json:"outcome"`
Healthy bool `json:"healthy"`
HealthReason string `json:"health_reason,omitempty"`
VMID int `json:"vmid"`
Upgraded []Package `json:"upgraded,omitempty"`
Installed []Package `json:"installed,omitempty"`
Pending []Pending `json:"pending,omitempty"`
NotCovered []string `json:"not_covered,omitempty"`
RestartNeeded []string `json:"restart_needed,omitempty"`
DockerRestartNeeded bool `json:"docker_restart_needed,omitempty"`
RebootNeeded bool `json:"reboot_needed,omitempty"`
Refused json.RawMessage `json:"refused,omitempty"`
}
// Reporter posts a report to the hub (*hub.Client).
type Reporter interface {
PostOSReport(ctx context.Context, body []byte) error
}
// Leg runs one OS-update pass for the customer guest.
type Leg struct {
Runner proxmox.Runner
Hub Reporter
Logger *slog.Logger
PlanDir string
StatePath string // last night run (once per night)
HealthWait time.Duration // how long health may take to come back (default 5 min)
HealthPoll time.Duration // default 15 s
MinGap time.Duration // between night runs (default 20 h)
Now func() time.Time
Sleep func(context.Context, time.Duration)
mu sync.Mutex
block *hub.WireOSUpdate
have bool
}
// OnDesiredState stores the hub's os_update block (desired.RawConsumer — store only, never block).
func (l *Leg) OnDesiredState(_ context.Context, resp *hub.DesiredStateResponse) {
if resp == nil {
return
}
l.mu.Lock()
defer l.mu.Unlock()
l.block, l.have = resp.DesiredState.OSUpdate, true
}
// Block returns the newest os_update block. No block (an older hub, or nothing fetched yet) = ring 1, ON, no
// release: the box reports and installs nothing.
func (l *Leg) Block() hub.WireOSUpdate {
l.mu.Lock()
defer l.mu.Unlock()
if l.block == nil {
return hub.WireOSUpdate{Ring: 1, Enabled: true}
}
return *l.block
}
// SetBlock sets the block directly (the selftest fetches the desired state itself).
func (l *Leg) SetBlock(b *hub.WireOSUpdate) {
l.mu.Lock()
defer l.mu.Unlock()
l.block, l.have = b, true
}
func (l *Leg) now() time.Time {
if l.Now != nil {
return l.Now()
}
return time.Now()
}
func (l *Leg) log() *slog.Logger {
if l.Logger != nil {
return l.Logger
}
return slog.Default()
}
func (l *Leg) sleep(ctx context.Context, d time.Duration) {
if l.Sleep != nil {
l.Sleep(ctx, d)
return
}
select {
case <-ctx.Done():
case <-time.After(d):
}
}
// IsFast reports whether every origin apt names is Debian / Debian-Security (the fast lane, `11` C3).
func IsFast(origins []string) bool {
if len(origins) == 0 {
return false
}
for _, o := range origins {
if o != "Debian" && o != "Debian-Security" {
return false
}
}
return true
}
func fastOrigin(origins []string) string {
for _, o := range origins {
if o == "Debian-Security" {
return o
}
}
return "Debian"
}
// HealthVerdict is THE health rule (`11` §5.4.1; pinned by TestHealthVerdict_*): after the run, docker answers, the
// guest's network resolves, the controller's own health check is `healthy`, and every container that was running
// before is running again — and healthy again if it was healthy before. "starting" is not yet healthy.
func HealthVerdict(before, after *Health) (bool, string) {
if after == nil {
return false, "no health reading"
}
if !after.DockerOK {
return false, "docker does not answer"
}
if !after.NetworkOK {
return false, "the guest cannot resolve deb.debian.org"
}
if after.Controller != "healthy" {
return false, "the controller is " + after.Controller
}
if before == nil {
return true, ""
}
names := make([]string, 0, len(before.Containers))
for n := range before.Containers {
names = append(names, n)
}
sort.Strings(names)
for _, n := range names {
b := before.Containers[n]
if b.State != "running" {
continue
}
a, ok := after.Containers[n]
if !ok || a.State != "running" {
return false, n + " was running and is not"
}
if b.Health == "healthy" && a.Health != "healthy" {
return false, n + " was healthy and is " + a.Health
}
}
return true, ""
}
// MergeBaseline is the health baseline from two readings: a container counts as running (and healthy) if EITHER
// reading saw it so. nil-safe. Pinned by TestHealth_BaselineIsTheStartOfTheLeg.
func MergeBaseline(a, b *Health) *Health {
if a == nil {
return b
}
if b == nil {
return a
}
out := &Health{DockerOK: a.DockerOK && b.DockerOK, NetworkOK: a.NetworkOK && b.NetworkOK, Controller: b.Controller,
Containers: map[string]Container{}}
for _, h := range []*Health{a, b} {
for n, c := range h.Containers {
cur, seen := out.Containers[n]
if !seen || (cur.State != "running" && c.State == "running") {
out.Containers[n] = c
continue
}
if c.State == "running" && c.Health == "healthy" {
out.Containers[n] = c
}
}
}
return out
}
// call writes the plan and runs the wrapper once.
func (l *Leg) call(ctx context.Context, runID, mode string, vmid int, rel hub.WireOSRelease, pkgs []Package) (WrapperReport, error) {
dir := l.PlanDir
if dir == "" {
dir = DefaultPlanDir
}
if err := os.MkdirAll(dir, 0o700); err != nil {
return WrapperReport{}, fmt.Errorf("osupdate: plan dir: %w", err)
}
plan := map[string]any{"release_id": rel.ID, "layer": "guest", "lane": "fast", "vmid": vmid, "mode": mode,
"snapshot": rel.Snapshot, "packages": pkgs}
if pkgs == nil {
plan["packages"] = []Package{}
}
b, _ := json.Marshal(plan)
path := filepath.Join(dir, "plan-"+runID+"-"+mode+".json")
if err := os.WriteFile(path, b, 0o600); err != nil {
return WrapperReport{}, fmt.Errorf("osupdate: write plan: %w", err)
}
defer os.Remove(path)
stdout, stderr, err := l.Runner.Run(ctx, WrapperPath, "--plan", path)
for _, line := range strings.Split(strings.TrimSpace(string(stderr)), "\n") {
if strings.HasPrefix(line, "os-apply: ") {
l.log().Info("osupdate: wrapper", "line", line)
}
}
var rep WrapperReport
found := false
for _, line := range strings.Split(string(stdout), "\n") {
if strings.HasPrefix(line, "OSAPPLY-REPORT ") {
if jerr := json.Unmarshal([]byte(strings.TrimPrefix(line, "OSAPPLY-REPORT ")), &rep); jerr == nil {
found = true
}
}
}
if !found {
return rep, fmt.Errorf("osupdate: wrapper gave no report (err %v): %s", err, strings.TrimSpace(string(stderr)))
}
return rep, nil // a refusal / failure is IN the report (exit 2 / 3), not an error here
}
// Run is one pass: inventory → (switch, ring, plan) → apply → health → report. trigger is "night" or "debug".
func (l *Leg) Run(ctx context.Context, vmid int, trigger string) Report {
runID := l.now().UTC().Format("20060102T150405Z")
blk := l.Block()
rel := hub.WireOSRelease{ID: "ring0-" + runID}
if blk.Ring == 1 {
rel = hub.WireOSRelease{}
if blk.Release != nil {
rel = *blk.Release
}
}
rep := Report{RunID: runID, Trigger: trigger, Ring: blk.Ring, VMID: vmid, ReleaseID: rel.ID, Mode: "inventory"}
lg := l.log().With("run", runID, "vmid", vmid, "ring", blk.Ring, "trigger", trigger)
if trigger == "night" && l.StatePath != "" {
gap := l.MinGap
if gap == 0 {
gap = 20 * time.Hour
}
if b, err := os.ReadFile(l.StatePath); err == nil {
if last, perr := time.Parse(time.RFC3339, strings.TrimSpace(string(b))); perr == nil && l.now().Sub(last) < gap {
lg.Info("osupdate: skipped — already ran tonight", "last", last.UTC().Format(time.RFC3339))
rep.Outcome = "skipped"
return rep
}
}
}
lg.Info("osupdate: START", "enabled", blk.Enabled, "release", rel.ID)
inv, err := l.call(ctx, runID, "inventory", vmid, rel, nil)
if err != nil || inv.refused() || inv.failed() {
rep.Outcome, rep.Refused = "refused", inv.Refused
if err != nil {
rep.Outcome, rep.HealthReason = "failed", err.Error()
}
return l.finish(ctx, lg, rep)
}
var plan []Package
switch {
case !blk.Enabled:
rep.Outcome = "inventory"
lg.Info("osupdate: switched OFF for this box — reporting only")
case blk.Ring == 0:
for _, p := range inv.Pending {
if IsFast(p.Origin) {
plan = append(plan, Package{Name: p.Name, Version: p.To, Origin: fastOrigin(p.Origin)})
}
}
default:
for _, p := range rel.Packages {
plan = append(plan, Package{Name: p.Name, Version: p.Version, Origin: p.Origin})
}
}
pkgNames := map[string]bool{}
for _, p := range plan {
pkgNames[p.Name] = true
}
if blk.Enabled && len(plan) == 0 {
rep.Outcome = "nothing"
}
final := inv
if blk.Enabled && len(plan) > 0 {
rep.Mode = "apply"
ap, err := l.call(ctx, runID, "apply", vmid, rel, plan)
switch {
case err != nil:
rep.Outcome, rep.HealthReason = "failed", err.Error()
return l.finish(ctx, lg, rep)
case ap.refused():
rep.Outcome, rep.Refused = "refused", ap.Refused
return l.finish(ctx, lg, rep)
case ap.failed():
rep.Outcome, rep.Refused = "failed", ap.Failed
}
final = ap
rep.Upgraded = ap.Upgraded
if rep.Outcome == "" {
if len(ap.Upgraded) == 0 {
rep.Outcome = "nothing"
} else {
rep.Outcome = "applied"
}
}
// Health: compare with what the guest looked like BEFORE the run; give restarted services time.
wait, poll := l.HealthWait, l.HealthPoll
if wait == 0 {
wait = 5 * time.Minute
}
if poll == 0 {
poll = 15 * time.Second
}
deadline := l.now().Add(wait)
cur := ap.HealthAfter
// The baseline is the guest as it was at the START of the leg (the inventory's reading) merged with the
// apply's own "before": an app that stops at any point during the run counts. Found live 2026-10-04: an app
// stopped between the inventory and the apply's own reading was taken as "stopped before" and ignored.
base := MergeBaseline(inv.HealthAfter, ap.HealthBefore)
for {
ok, why := HealthVerdict(base, cur)
rep.Healthy, rep.HealthReason = ok, why
if ok || !l.now().Before(deadline) || ctx.Err() != nil {
break
}
l.sleep(ctx, poll)
hr, herr := l.call(ctx, runID, "health", vmid, rel, nil)
if herr == nil && hr.Health != nil {
cur = hr.Health
}
}
if !rep.Healthy && rep.Outcome == "applied" {
rep.Outcome = "health_failed"
}
} else {
ok, why := HealthVerdict(nil, inv.HealthAfter)
rep.Healthy, rep.HealthReason = ok, why
}
rep.Installed, rep.Pending = final.Installed, final.Pending
rep.RestartNeeded, rep.DockerRestartNeeded, rep.RebootNeeded = final.RestartNeeded, final.DockerRestartNeeded, final.RebootNeeded
rep.NotCovered = notCovered(final.Pending, blk.Ring, pkgNames)
if trigger == "night" && l.StatePath != "" {
_ = os.WriteFile(l.StatePath, []byte(l.now().UTC().Format(time.RFC3339)), 0o600)
}
return l.finish(ctx, lg, rep)
}
// notCovered lists pending updates no approved release covers: in ring 0 everything outside the fast lane; in
// ring 1 also every fast-lane update the release did not name.
func notCovered(pending []Pending, ring int, planned map[string]bool) []string {
var out []string
for _, p := range pending {
if !IsFast(p.Origin) || (ring == 1 && !planned[p.Name]) {
out = append(out, p.Name)
}
}
return out
}
func (l *Leg) finish(ctx context.Context, lg *slog.Logger, rep Report) Report {
lg.Info("osupdate: DONE", "outcome", rep.Outcome, "healthy", rep.Healthy, "reason", rep.HealthReason,
"upgraded", len(rep.Upgraded), "pending", len(rep.Pending), "not_covered", len(rep.NotCovered), "restart_needed", len(rep.RestartNeeded))
if l.Hub != nil {
body, _ := json.Marshal(rep)
rctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), time.Minute)
defer cancel()
if err := l.Hub.PostOSReport(rctx, body); err != nil {
lg.Warn("osupdate: reporting to the hub failed (the run itself is done)", "err", err)
}
}
return rep
}
+281
View File
@@ -0,0 +1,281 @@
package osupdate
import (
"context"
"encoding/json"
"io"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// fakeWrapper plays /usr/local/sbin/felhom-os-apply: it reads the plan the leg wrote and answers per mode.
type fakeWrapper struct {
t *testing.T
pending []Pending
applyRep WrapperReport
healthSeq []*Health // answers to successive "health" calls
plans []map[string]any
}
func (f *fakeWrapper) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
if name != WrapperPath || len(args) != 2 || args[0] != "--plan" {
f.t.Fatalf("unexpected command %s %v", name, args)
}
b, err := os.ReadFile(args[1])
if err != nil {
f.t.Fatal(err)
}
var plan map[string]any
json.Unmarshal(b, &plan)
f.plans = append(f.plans, plan)
var rep WrapperReport
healthy := &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{
"felhom-controller": {State: "running", Health: "healthy"}, "app": {State: "running", Health: "healthy"}}}
switch plan["mode"] {
case "inventory":
rep = WrapperReport{Mode: "inventory", Pending: f.pending, HealthAfter: healthy,
Installed: []Package{{Name: "libc6", Version: "u3", Origin: "Debian"}}}
case "apply":
rep = f.applyRep
rep.Mode = "apply"
if rep.HealthBefore == nil {
rep.HealthBefore = healthy
}
if rep.HealthAfter == nil {
rep.HealthAfter = healthy
}
case "health":
if len(f.healthSeq) > 0 {
rep.Health, f.healthSeq = f.healthSeq[0], f.healthSeq[1:]
} else {
rep.Health = healthy
}
}
out, _ := json.Marshal(rep)
return []byte("OSAPPLY-REPORT " + string(out) + "\n"), []byte("os-apply: DONE rc=0\n"), nil
}
func (f *fakeWrapper) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return f.Run(ctx, name, args...)
}
type fakeHub struct{ reports []Report }
func (h *fakeHub) PostOSReport(_ context.Context, body []byte) error {
var r Report
json.Unmarshal(body, &r)
h.reports = append(h.reports, r)
return nil
}
func newLeg(t *testing.T, w *fakeWrapper, blk *hub.WireOSUpdate) (*Leg, *fakeHub) {
h := &fakeHub{}
now := time.Date(2026, 10, 4, 4, 0, 0, 0, time.UTC)
l := &Leg{Runner: w, Hub: h, PlanDir: t.TempDir(), StatePath: filepath.Join(t.TempDir(), "last"),
HealthWait: time.Minute, HealthPoll: 10 * time.Second,
Now: func() time.Time { return now },
Sleep: func(_ context.Context, d time.Duration) { now = now.Add(d) }}
if blk != nil {
l.SetBlock(blk)
}
return l, h
}
var pend = []Pending{
{Name: "libc6", From: "u3", To: "u4", Origin: []string{"Debian"}},
{Name: "openssl", From: "u1", To: "u3", Origin: []string{"Debian-Security", "Debian"}},
{Name: "docker-ce", From: "29.7", To: "29.8", Origin: []string{"Docker CE"}},
}
func modes(w *fakeWrapper) string {
var m []string
for _, p := range w.plans {
m = append(m, p["mode"].(string))
}
return strings.Join(m, ",")
}
// Ring 0 plans every pending FAST-LANE update (Docker excluded, `11` C3) and reports what it now runs.
func TestRing0_PlansTheFastLaneOnly(t *testing.T) {
w := &fakeWrapper{t: t, pending: pend, applyRep: WrapperReport{Upgraded: []Package{{Name: "libc6", Version: "u4"}, {Name: "openssl", Version: "u3"}}, Pending: pend[2:]}}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
rep := l.Run(context.Background(), 9201, "night")
if rep.Outcome != "applied" || !rep.Healthy {
t.Fatalf("rep = %+v", rep)
}
pk := w.plans[1]["packages"].([]any)
if len(pk) != 2 {
t.Fatalf("ring-0 plan = %v, want libc6 + openssl only", pk)
}
if o := pk[1].(map[string]any)["origin"]; o != "Debian-Security" {
t.Fatalf("openssl origin = %v", o)
}
if w.plans[1]["snapshot"] != "" {
t.Fatalf("ring 0 installs from live sources, snapshot = %v", w.plans[1]["snapshot"])
}
if len(rep.NotCovered) != 1 || rep.NotCovered[0] != "docker-ce" {
t.Fatalf("not covered = %v", rep.NotCovered)
}
if len(h.reports) != 1 || h.reports[0].Outcome != "applied" {
t.Fatalf("hub got %+v", h.reports)
}
}
// Ring 1 installs EXACTLY the approved release (its versions, its snapshot), nothing it computed itself.
func TestRing1_InstallsExactlyTheRelease(t *testing.T) {
w := &fakeWrapper{t: t, pending: pend, applyRep: WrapperReport{Upgraded: []Package{{Name: "libc6", Version: "u4-approved"}}, Pending: pend[1:]}}
rel := &hub.WireOSRelease{ID: "os-1", Snapshot: "20261004T080000Z", Packages: []hub.WireOSPackage{{Name: "libc6", Version: "u4-approved", Origin: "Debian"}}}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true, Release: rel})
rep := l.Run(context.Background(), 9201, "night")
if rep.Outcome != "applied" || rep.ReleaseID != "os-1" {
t.Fatalf("rep = %+v", rep)
}
ap := w.plans[1]
pk := ap["packages"].([]any)
if len(pk) != 1 || pk[0].(map[string]any)["version"] != "u4-approved" || ap["snapshot"] != "20261004T080000Z" || ap["release_id"] != "os-1" {
t.Fatalf("ring-1 plan = %v", ap)
}
// openssl is pending and fast-lane but NOT in the release → not covered; docker-ce is never covered.
if strings.Join(rep.NotCovered, ",") != "openssl,docker-ce" {
t.Fatalf("not covered = %v", rep.NotCovered)
}
}
func TestRing1_NoReleaseInstallsNothing(t *testing.T) {
w := &fakeWrapper{t: t, pending: pend}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
rep := l.Run(context.Background(), 9201, "night")
if rep.Outcome != "nothing" || modes(w) != "inventory" {
t.Fatalf("rep=%+v modes=%s", rep, modes(w))
}
}
// No block from the hub (an older hub): ring 1, ON, no release → reports, installs nothing.
func TestNoBlock_IsRing1Nothing(t *testing.T) {
w := &fakeWrapper{t: t, pending: pend}
l, _ := newLeg(t, w, nil)
if rep := l.Run(context.Background(), 9201, "night"); rep.Outcome != "nothing" || rep.Ring != 1 || modes(w) != "inventory" {
t.Fatalf("rep=%+v modes=%s", rep, modes(w))
}
}
// Switched OFF: the box reports but installs nothing (the brief, Part D 2).
func TestSwitchOff_ReportsOnly(t *testing.T) {
w := &fakeWrapper{t: t, pending: pend}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: false})
rep := l.Run(context.Background(), 9201, "night")
if rep.Outcome != "inventory" || modes(w) != "inventory" || len(h.reports) != 1 || len(rep.Pending) != 3 {
t.Fatalf("rep=%+v modes=%s", rep, modes(w))
}
}
// Unhealthy after the run, and still unhealthy at the end of the wait → health_failed (the hub mails the operator).
func TestHealth_FailsAfterTheWait(t *testing.T) {
bad := &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{
"felhom-controller": {State: "running", Health: "healthy"}, "app": {State: "exited"}}}
w := &fakeWrapper{t: t, pending: pend, applyRep: WrapperReport{Upgraded: []Package{{Name: "libc6"}}, HealthAfter: bad},
healthSeq: []*Health{bad, bad, bad, bad, bad, bad, bad, bad}}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
rep := l.Run(context.Background(), 9201, "night")
if rep.Outcome != "health_failed" || rep.Healthy || !strings.Contains(rep.HealthReason, "app was running") {
t.Fatalf("rep = %+v", rep)
}
if h.reports[0].Outcome != "health_failed" {
t.Fatal("the hub was not told")
}
}
// A service that takes a moment to come back is not a failure: the poll sees it recover inside the wait.
func TestHealth_RecoversInsideTheWait(t *testing.T) {
starting := &Health{DockerOK: true, NetworkOK: true, Controller: "starting"}
w := &fakeWrapper{t: t, pending: pend, applyRep: WrapperReport{Upgraded: []Package{{Name: "libc6"}}, HealthAfter: starting},
healthSeq: []*Health{starting}}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
if rep := l.Run(context.Background(), 9201, "night"); rep.Outcome != "applied" || !rep.Healthy {
t.Fatalf("rep = %+v", rep)
}
}
func TestHealthVerdict(t *testing.T) {
ok := &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{"a": {State: "running", Health: "healthy"}}}
cases := []struct {
name string
before *Health
after *Health
want bool
}{
{"all good", ok, ok, true},
{"no reading", ok, nil, false},
{"docker down", ok, &Health{NetworkOK: true, Controller: "healthy"}, false},
{"no network", ok, &Health{DockerOK: true, Controller: "healthy"}, false},
{"controller starting", ok, &Health{DockerOK: true, NetworkOK: true, Controller: "starting"}, false},
{"only the controller differs", ok, &Health{DockerOK: true, NetworkOK: true, Controller: "unhealthy", Containers: map[string]Container{"a": {State: "running", Health: "healthy"}}}, false},
{"app gone", ok, &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{}}, false},
{"app unhealthy", ok, &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{"a": {State: "running", Health: "unhealthy"}}}, false},
{"stopped before stays stopped", &Health{Containers: map[string]Container{"x": {State: "exited"}}}, &Health{DockerOK: true, NetworkOK: true, Controller: "healthy"}, true},
}
for _, c := range cases {
if got, why := HealthVerdict(c.before, c.after); got != c.want {
t.Errorf("%s: got %v (%s), want %v", c.name, got, why, c.want)
}
}
}
func TestOncePerNight(t *testing.T) {
w := &fakeWrapper{t: t, pending: pend, applyRep: WrapperReport{Upgraded: []Package{{Name: "libc6"}}}}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
l.Run(context.Background(), 9201, "night")
n := len(w.plans)
if rep := l.Run(context.Background(), 9201, "night"); rep.Outcome != "skipped" || len(w.plans) != n {
t.Fatalf("a second night run in the same night ran: %+v", rep)
}
if rep := l.Run(context.Background(), 9201, "debug"); rep.Outcome == "skipped" {
t.Fatal("the debug action must not be throttled")
}
}
func TestRefusedIsReported(t *testing.T) {
w := &fakeWrapper{t: t, pending: pend, applyRep: WrapperReport{Refused: json.RawMessage(`{"code":"R6","reason":"x"}`)}}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
if rep := l.Run(context.Background(), 9201, "night"); rep.Outcome != "refused" || h.reports[0].Outcome != "refused" {
t.Fatalf("rep = %+v", rep)
}
}
// The root wrapper's own suite (configs/test_felhom_os_apply.py) runs with `go test ./...` so CI covers it.
func TestWrapperSuite(t *testing.T) {
py, err := exec.LookPath("python3")
if err != nil {
t.Skip("python3 not available")
}
cmd := exec.Command(py, "../../configs/test_felhom_os_apply.py")
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("wrapper suite failed: %v\n%s", err, out)
}
if !strings.Contains(string(out), "OK") {
t.Fatalf("wrapper suite did not report OK:\n%s", out)
}
}
// An app that stops BETWEEN the start of the leg and the apply's own "before" reading still fails the run.
// Measured live 2026-10-04 on demo-hp (privatebin stopped 1 s after the apply plan was written: the old rule
// passed). Red-proof: use ap.HealthBefore alone as the baseline and this fails.
func TestHealth_BaselineIsTheStartOfTheLeg(t *testing.T) {
stoppedEarly := &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{
"felhom-controller": {State: "running", Health: "healthy"}, "app": {State: "exited"}}}
w := &fakeWrapper{t: t, pending: pend, applyRep: WrapperReport{Upgraded: []Package{{Name: "libc6"}},
HealthBefore: stoppedEarly, HealthAfter: stoppedEarly},
healthSeq: []*Health{stoppedEarly, stoppedEarly, stoppedEarly, stoppedEarly, stoppedEarly, stoppedEarly, stoppedEarly}}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
rep := l.Run(context.Background(), 9201, "night")
if rep.Outcome != "health_failed" || !strings.Contains(rep.HealthReason, "app was running") {
t.Fatalf("an app that stopped during the run passed: %+v", rep)
}
}
+15 -9
View File
@@ -219,15 +219,18 @@ type Storage struct {
UsedFraction float64 `json:"used_fraction,omitempty"` UsedFraction float64 `json:"used_fraction,omitempty"`
// Type-specific config (durable_id sources). // Type-specific config (durable_id sources).
Server string `json:"server,omitempty"` // nfs/cifs/pbs server host Server string `json:"server,omitempty"` // nfs/cifs/pbs server host
Export string `json:"export,omitempty"` // nfs export path // EncryptionKey is the storage's own client-side key FINGERPRINT (pbs; the key itself stays in
Share string `json:"share,omitempty"` // cifs share name // /etc/pve/priv). R-727: an archive encrypted with any other key was written by another box.
Datastore string `json:"datastore,omitempty"` // pbs datastore name EncryptionKey string `json:"encryption-key,omitempty"`
Fingerprint string `json:"fingerprint,omitempty"` // pbs server cert fingerprint Export string `json:"export,omitempty"` // nfs export path
Username string `json:"username,omitempty"` // pbs auth id, e.g. "felhom@pbs!n100" Share string `json:"share,omitempty"` // cifs share name
Namespace string `json:"namespace,omitempty"` // pbs namespace ("" = root; per-customer tenancy = S4) Datastore string `json:"datastore,omitempty"` // pbs datastore name
VGName string `json:"vgname,omitempty"` // lvm/lvmthin volume group Fingerprint string `json:"fingerprint,omitempty"` // pbs server cert fingerprint
ThinPool string `json:"thinpool,omitempty"` // lvmthin pool LV name Username string `json:"username,omitempty"` // pbs auth id, e.g. "felhom@pbs!n100"
Namespace string `json:"namespace,omitempty"` // pbs namespace ("" = root; per-customer tenancy = S4)
VGName string `json:"vgname,omitempty"` // lvm/lvmthin volume group
ThinPool string `json:"thinpool,omitempty"` // lvmthin pool LV name
} }
// StorageContent is one entry of GET /nodes/{node}/storage/{store}/content // StorageContent is one entry of GET /nodes/{node}/storage/{store}/content
@@ -239,4 +242,7 @@ type StorageContent struct {
Size int64 `json:"size"` Size int64 `json:"size"`
CTime int64 `json:"ctime"` CTime int64 `json:"ctime"`
VMID int `json:"vmid,omitempty"` VMID int `json:"vmid,omitempty"`
// Encrypted is the fingerprint of the key a PBS archive was encrypted with ("" = not encrypted). R-727:
// the restore test reads it to tell THIS box's archives from an earlier box's in the same namespace.
Encrypted string `json:"encrypted,omitempty"`
} }
+57
View File
@@ -302,6 +302,19 @@ func (e *Engine) runBringUp(ctx context.Context, spec BringUpSpec, res *BringUpR
} }
} }
// R-834: DR keeps the archive's `onboot: 1`, binds the host's REAL drives (4d) and STARTS the guest
// — right on a replaced host, where the original is gone. Beside a LIVE original it would be a
// second controller for the same household on the same drives. So DR refuses when this host
// still carries the original (the archive's source VMID) or any guest that binds the drives.
// A copy beside the original is the restore-test's job (onboot=0, throwaway stand-ins, torn
// down) or the runbook's beside-restore. Pinned by TestRunBringUp_DRRefusesBesideALiveOriginal.
if spec.Mode == ModeDRGuestLoss {
if why := e.liveOriginalBeside(ctx, lxc, spec.Archive); why != "" {
res.Err = fmt.Errorf("reconcile: dr bring-up refused: %s — a DR restore beside a live original would run two boxes on the same drives (R-834)", why)
return
}
}
base := JournalEntry{OpID: e.bringUpOpID(spec.VMID), VMID: spec.VMID, Kind: bringUpKind, Rollback: true} base := JournalEntry{OpID: e.bringUpOpID(spec.VMID), VMID: spec.VMID, Kind: bringUpKind, Rollback: true}
// OWN the rollback BEFORE any mutation. From here a crash leaves an in-flight Rollback // OWN the rollback BEFORE any mutation. From here a crash leaves an in-flight Rollback
@@ -734,3 +747,47 @@ func net0MAC(cfg proxmox.GuestConfig) string {
} }
return "" return ""
} }
// archiveSourceVMID reads the source guest's VMID from a backup volid: a vzdump file
// (`…/vzdump-lxc-<vmid>-<date>.tar.zst`) or a PBS snapshot (`…:backup/ct/<vmid>/<time>`). 0 = unknown.
func archiveSourceVMID(archive string) int {
if i := strings.Index(archive, "vzdump-lxc-"); i >= 0 {
rest := archive[i+len("vzdump-lxc-"):]
if j := strings.Index(rest, "-"); j > 0 {
if n, err := strconv.Atoi(rest[:j]); err == nil {
return n
}
}
}
if i := strings.Index(archive, "ct/"); i >= 0 {
rest := archive[i+len("ct/"):]
if j := strings.Index(rest, "/"); j > 0 {
if n, err := strconv.Atoi(rest[:j]); err == nil {
return n
}
}
}
return 0
}
// liveOriginalBeside says why a DR bring-up would land beside a live original ("" = it would not):
// the archive's source guest still exists here, or a guest binds the drives parent. Fails CLOSED: a
// guest whose config cannot be read cannot be ruled out.
func (e *Engine) liveOriginalBeside(ctx context.Context, lxc []proxmox.Guest, archive string) string {
src := archiveSourceVMID(archive)
for _, g := range lxc {
if src > 0 && g.VMID == src {
return fmt.Sprintf("the archive's source guest %d still exists on this host (status %s)", g.VMID, g.Status)
}
cfg, err := e.api.GuestConfig(ctx, g.VMID)
if err != nil {
return fmt.Sprintf("guest %d's config could not be read to rule out a live original: %v", g.VMID, err)
}
for slot, v := range cfg.MountPoints() {
if source, _, _ := strings.Cut(v, ","); source == structuralParentDir {
return fmt.Sprintf("guest %d binds the household drives (%s %s)", g.VMID, slot, structuralParentDir)
}
}
}
return ""
}
+138
View File
@@ -0,0 +1,138 @@
package reconcile
import (
"context"
"encoding/json"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-834: a whole-guest restore BESIDE a live original must never come up as a second box on the same
// drives. The DR route keeps `onboot: 1`, binds the real drives and starts the guest, so it refuses
// when the original (or any guest binding the drives) is still on this host; on a replaced host it
// proceeds and keeps its binds. Red-proof: make liveOriginalBeside return "" and the refusals pass
// the restore through (the "refused" sub-tests fail).
func TestRunBringUp_DRRefusesBesideALiveOriginal(t *testing.T) {
const target = 9299
drivesBind := proxmox.GuestConfig{Extra: map[string]json.RawMessage{
"mp8": json.RawMessage(`"/mnt/felhom-drives,mp=/mnt/felhom-drives"`),
}}
cases := []struct {
name string
archive string
lxc []proxmox.Guest
cfg map[int]proxmox.GuestConfig
refuse string // substring of the refusal; "" = must proceed
}{
{"the source guest still exists", "local:backup/vzdump-lxc-9201-2026_10_04-04_34_55.tar.zst",
[]proxmox.Guest{{VMID: 9201, Status: "running"}}, map[int]proxmox.GuestConfig{9201: scratchCfg()}, "source guest 9201"},
{"another guest binds the drives (PBS archive)", "felhom-pbs:backup/ct/9201/2026-10-04T02:34:55Z",
[]proxmox.Guest{{VMID: 9300, Status: "stopped"}}, map[int]proxmox.GuestConfig{9300: drivesBind}, "binds the household drives"},
{"a guest whose config cannot be read", "local:backup/vzdump-lxc-9201-x.tar.zst",
[]proxmox.Guest{{VMID: 9400, Status: "running"}}, map[int]proxmox.GuestConfig{}, "could not be read"},
{"replaced host: only an unrelated scratch guest", "local:backup/vzdump-lxc-9201-x.tar.zst",
[]proxmox.Guest{{VMID: 9202, Status: "running"}}, map[int]proxmox.GuestConfig{9202: scratchCfg()}, ""},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
cfg := c.cfg
cfg[target] = scratchCfg()
api := &fakeAPI{lxc: c.lxc, cfg: cfg}
e, fr, _, q := newDREngine(t, api)
defer q.Close()
res := e.RunBringUp(context.Background(), BringUpSpec{
Mode: ModeDRGuestLoss, Archive: c.archive, VMID: target, RestoreStorage: "local-lvm", KeepMAC: true,
})
if c.refuse != "" {
if res.Err == nil || !strings.Contains(res.Err.Error(), c.refuse) || !strings.Contains(res.Err.Error(), "R-834") {
t.Fatalf("want a refusal naming %q, got %+v", c.refuse, res)
}
if len(api.restores) != 0 || len(api.starts) != 0 || len(fr.cmds) != 0 {
t.Fatalf("a refused DR touched the host: restores=%d starts=%v cmds=%v", len(api.restores), api.starts, fr.cmds)
}
return
}
if res.Err != nil || !res.Pass {
t.Fatalf("a DR on a replaced host must proceed, got %+v", res)
}
// … and there it keeps the REAL drives bind (the right binds on a replaced host).
joined := strings.Join(fr.cmds, "\n")
if !strings.Contains(joined, "-mp8 /mnt/felhom-drives,mp=/mnt/felhom-drives") {
t.Fatalf("the DR guest lost its drives bind: %v", fr.cmds)
}
})
}
}
// Provisioning restores the GOLDEN (no drives, onboot set by the back-half on purpose): a drives-
// binding guest on the host does not block it — the rule is DR's alone.
func TestRunBringUp_ProvisionNotBlockedByADrivesBind(t *testing.T) {
api := &fakeAPI{
lxc: []proxmox.Guest{{VMID: 9201, Status: "running"}},
cfg: map[int]proxmox.GuestConfig{
9201: {Extra: map[string]json.RawMessage{"mp8": json.RawMessage(`"/mnt/felhom-drives,mp=/mnt/felhom-drives"`)}},
9203: scratchCfg(),
},
}
e, _, q := newEngine(t, api, EmptyProvider{})
defer q.Close()
res := e.RunBringUp(context.Background(), BringUpSpec{Mode: ModeProvision, Archive: "local:vztmpl/felhom-golden.tar.zst", VMID: 9203, RestoreStorage: "local-lvm"})
if res.Err != nil || !res.Pass {
t.Fatalf("provision must proceed, got %+v", res)
}
}
func TestArchiveSourceVMID(t *testing.T) {
for in, want := range map[string]int{
"local:backup/vzdump-lxc-9201-2026_10_04-04_34_55.tar.zst": 9201,
"felhom-pbs:backup/ct/9201/2026-10-04T02:34:55Z": 9201,
"tmp-dooplex-copy:backup/ct/9201/2026-10-03T19:00:00Z": 9201,
"local:vztmpl/felhom-golden.tar.zst": 0,
"vol": 0,
} {
if got := archiveSourceVMID(in); got != want {
t.Errorf("archiveSourceVMID(%q) = %d, want %d", in, got, want)
}
}
}
// R-834, the restore-test route: its scratch guest sits BESIDE the live original by design, so it must
// carry no host-path bind — the archive's mp8 (the household's drives) and mp9 (the original's
// bootstrap) are replaced by throwaway volumes AT restore time. Measured live 2026-10-04 on demo-hp
// (`audits/backup-close-2026-10-04/partA/`): onboot 0 and no host bind on every poll. Red-proof:
// make drRestoreOverrides return the archive's own mp8 value and this fails.
func TestRestoreTest_NoHostPathBindBesideTheOriginal(t *testing.T) {
api := &fakeAPI{
cfg: map[int]proxmox.GuestConfig{990000: scratchCfg()},
extractCfg: "hostname: demo-hp\nonboot: 1\nrootfs: local-lvm:vm-9201-disk-0,size=16G\n" +
"mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G\n" +
"mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives\n" +
"mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1\n",
}
e, _, q := newEngine(t, api, EmptyProvider{})
defer q.Close()
_ = e.RunRestoreTest(context.Background(), RestoreTestSpec{
Archive: "local:backup/vzdump-lxc-9201-x.tar.zst", RestoreStorage: "local-lvm",
ScratchMin: 990000, ScratchMax: 990009, SourceTier: "local",
})
if len(api.restores) != 1 {
t.Fatalf("want one restore, got %+v", api.restores)
}
r := api.restores[0]
for _, slot := range []string{"mp8", "mp9"} {
v, ok := r.MountOverrides[slot]
if !ok || strings.HasPrefix(v, "/") {
t.Fatalf("%s = %q (present=%v): the scratch beside the original must get a throwaway volume, never the host path", slot, v, ok)
}
}
for slot, v := range r.MountOverrides {
if strings.HasPrefix(v, "/") {
t.Fatalf("%s carries a host path %q into the scratch guest", slot, v)
}
}
if r.ConfigOverrides["onboot"] != "0" {
t.Fatalf("onboot = %q, want 0", r.ConfigOverrides["onboot"])
}
}