Compare commits
18 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 9cac3462bb | |||
| 1bb8608e88 | |||
| b84e0dd1bd | |||
| 23a8ef3de4 | |||
| 596238cc2e | |||
| 475bdce7e4 | |||
| d766666ff8 | |||
| a4c09a7c11 | |||
| 904dc20466 | |||
| e1b8269be0 | |||
| 5c68c869b6 | |||
| 728d12b1a0 | |||
| dd81866b16 | |||
| 3ef095fb71 | |||
| 7c986915ca | |||
| 16dbc83221 | |||
| 9555a7f93b | |||
| 9ff937d8fb |
@@ -9,3 +9,6 @@
|
|||||||
|
|
||||||
# go
|
# go
|
||||||
/vendor/
|
/vendor/
|
||||||
|
|
||||||
|
# Python bytecode written by configs/test_felhom_os_apply.py
|
||||||
|
configs/__pycache__/
|
||||||
|
|||||||
@@ -1,3 +1,92 @@
|
|||||||
|
## v0.139.0 — a DR restore never lands beside a live original (2026-10-04, R-834)
|
||||||
|
|
||||||
|
> **RELEASED 2026-10-04** by `scripts/release-agent.sh` — tag `v0.139.0` (`475bdce`), sha256
|
||||||
|
> `8534a9be368a6d24d8065db77436e86900443c5c6554c71f6fe91c2bdb9d0b9c`, verified by download. **Not vouched.**
|
||||||
|
|
||||||
|
**MinAgent impact:** none required by any controller.
|
||||||
|
|
||||||
|
- The DR bring-up (`--selftest=bring-up -mode dr`) keeps the archive's `onboot: 1`, binds the host's REAL drives
|
||||||
|
(`mp8 /mnt/felhom-drives`) and STARTS the guest — right on a replaced host, wrong beside a live original (a second
|
||||||
|
controller for the same household on the same drives). It now REFUSES, before any restore, when the archive's
|
||||||
|
source guest still exists on the host, when any guest binds the drives parent, or when a guest's config cannot be
|
||||||
|
read (fail closed). On a replaced host it proceeds and keeps its binds, unchanged.
|
||||||
|
- The restore-test was MEASURED safe live on demo-hp (onboot 0 and throwaway stand-ins for mp8/mp9 from the first
|
||||||
|
config read to teardown); a test now pins its "no host path" half beside the existing onboot test.
|
||||||
|
- No sudoers change: the restore-test sets onboot 0 through the API create call, and DR refuses rather than degrade,
|
||||||
|
so no `-onboot 0` line is needed.
|
||||||
|
- Tests: `TestRunBringUp_DRRefusesBesideALiveOriginal` (source guest present / drives bind on another guest / an
|
||||||
|
unreadable config refuse; a replaced host proceeds and keeps the drives bind), `TestRunBringUp_ProvisionNotBlockedByADrivesBind`,
|
||||||
|
`TestArchiveSourceVMID`, `TestRestoreTest_NoHostPathBindBesideTheOriginal`. Red-proofs: the DR check returning ""
|
||||||
|
→ three refusal cases restore and START; the restore-test's mp8 override set to the host path → fails.
|
||||||
|
|
||||||
|
## v0.138.0 — the restore test takes only THIS box's archives (2026-09-30, R-727, `09` §3 decision 51)
|
||||||
|
|
||||||
|
> **RELEASED 2026-09-30** by `scripts/release-agent.sh` — tag `v0.138.0` (`e1b8269`), sha256
|
||||||
|
> `55916026001790a79ebf97d32c032610cfde8e09d02979b9b9d8c2cbc5d88195`, verified by download. **Not vouched** (the golden
|
||||||
|
> keeps 0.137.0 until the next bake); delivered to the demo boxes by signed `agent_update` jobs.
|
||||||
|
>
|
||||||
|
> **VOUCHED 2026-09-30** with golden 0.283.1 (`min_agent` 0.131.0), on the operator's word: the hub logged
|
||||||
|
> `Artifact manifest set: agent=0.138.0 golden=0.283.1 min_agent="0.131.0"`. Evidence:
|
||||||
|
> `felhom.eu/documentation/audits/evidence-golden-0283-2026-09-30/`.
|
||||||
|
|
||||||
|
**MinAgent impact:** none required by any controller.
|
||||||
|
|
||||||
|
- A returning customer's PBS namespace can hold archives of EARLIER boxes: same guest id (9201), same token, written
|
||||||
|
with a different key. Measured 2026-09-30: the newest SETTLED archive was an earlier box's, and the test failed
|
||||||
|
`wrong key … manifest's key 6b:ca:5f:3f… does not match provided key de:51:7a:18…` every evaluation. The archive
|
||||||
|
carries no host id; it carries its key fingerprint (PVE content `encrypted`), and the storage carries its own
|
||||||
|
(`GET /storage` → `encryption-key`). `PickSettledRestoreCandidateOn` now skips — and logs by name, once — an archive
|
||||||
|
whose fingerprint is not the storage's own; an unencrypted storage is not filtered; a failed storage read is an
|
||||||
|
error (tier UNKNOWN), never "nothing to prove".
|
||||||
|
- Tests: `TestR727_TheRestoreTestTakesOnlyThisBoxsArchives` (the 2026-09-30 shape: nothing picked while this box's
|
||||||
|
archive settles, then exactly it), `TestR727_UnencryptedStorageIsNotFiltered`, `TestR727_KeyLookupFailureIsUnknown`.
|
||||||
|
Red-proof RP39: the skip removed → the earlier box's `2026-09-16T21:59:54Z` is picked.
|
||||||
|
|
||||||
|
## v0.137.0 — a guest outside the agent's ACL is not a known guest (2026-09-27, R-689, v0.136.0 regression)
|
||||||
|
|
||||||
|
> **RELEASED 2026-09-27** by `scripts/release-agent.sh` — tag `v0.137.0` (`3ef095f`), sha256 `766c9166916a1bd3674b0dc69081f8a7619e770f1402d8ad7705b395937e7627`, verified by download. **NOT vouched** (the operator's act).
|
||||||
|
>
|
||||||
|
> **VOUCHED 2026-09-28** with golden 0.276.0 (`min_agent` 0.131.0), on the operator's word of 2026-09-27: the hub logged
|
||||||
|
> `Artifact manifest set: agent=0.137.0 golden=0.276.0 min_agent="0.131.0"`, and a Day-0 test install fetched this binary
|
||||||
|
> through the manifest and sha-verified it. Evidence: `felhom.eu/documentation/audits/evidence-golden-0276-2026-09-28/`.
|
||||||
|
|
||||||
|
**MinAgent impact:** none required by any controller.
|
||||||
|
|
||||||
|
- v0.136.0 asked `GuestConfig` whether an archive's guest exists and treated anything but "does not exist" as a lookup
|
||||||
|
failure. PVE answers **403 "permission denied at /vms/<id>"** for a vmid outside the token's pool — so on demo-hp the
|
||||||
|
deleted guest 9100's archive made the local tier UNKNOWN every evaluation. A guest the agent cannot read is not one it
|
||||||
|
manages; its archive is skipped. `TestR689_AGuestOutsideTheAgentsACLIsNotAKnownGuest`, red-proofed. Verified read-only
|
||||||
|
on demo-hp with the pre-release binary before the release.
|
||||||
|
|
||||||
|
## v0.136.0 — … of a guest that still EXISTS (2026-09-27, R-689 second half)
|
||||||
|
|
||||||
|
> **RELEASED 2026-09-27** by `scripts/release-agent.sh` — tag `v0.136.0` (`16dbc83`), sha256 `2eb0b5ebe253defd68b322312bbac12418c051b0a7a0d2d1831310d97fa6d755`, verified by download. **NOT vouched** (the operator's act).
|
||||||
|
|
||||||
|
**MinAgent impact:** none required by any controller.
|
||||||
|
|
||||||
|
- **R-689, second half.** Read on demo-hp right after v0.135.0 with the read-only `-selftest=restore-test-due`: the
|
||||||
|
golden was skipped, and the pick fell to `vzdump-lxc-9100-2026_08_21…` — a leftover of a guest deleted in August. A
|
||||||
|
candidate's guest must now exist on this node (`GuestConfig`): "does not exist" skips the archive (INFO once per
|
||||||
|
volid); any other lookup failure is returned, so the tier reads UNKNOWN, never "nothing to prove". Tests
|
||||||
|
`TestR689_AnArchiveOfADeletedGuestIsNeverPicked` (red-proofed: without the check it picks the 9100 leftover),
|
||||||
|
`TestR689_AGuestLookupFailureIsUnknownNotEmpty`. A second agent release in one session — the first half was found
|
||||||
|
incomplete on the box.
|
||||||
|
|
||||||
|
## v0.135.0 — the restore test proves only backups OF A GUEST (2026-09-27, R-689)
|
||||||
|
|
||||||
|
> **RELEASED 2026-09-27** by `scripts/release-agent.sh` — tag `v0.135.0` (`d4be12c`), sha256 `ad4e75f16d338552f4588d3fe64c51cbf9651220b9d223b5386b85b6c37fd4c3`, verified by download. **NOT vouched** (the operator's act).
|
||||||
|
|
||||||
|
**MinAgent impact:** none required by any controller.
|
||||||
|
|
||||||
|
- **R-689** (`backup/runner.go` `guestBackupArchive`). demo-hp keeps its golden template in `local:backup/` — content
|
||||||
|
"backup", 654 MB, plausibly complete, the newest settled entry — and the scheduled restore test picked it every 6 h and
|
||||||
|
failed `extractconfig` with a 403, while the guest's real archive went untested. A restore-test candidate is now a
|
||||||
|
`vzdump-{lxc,qemu}-<vmid>-…` file or a PBS `backup/{ct,vm}/<vmid>/…` snapshot whose vmid the storage reports; anything
|
||||||
|
else is skipped with one INFO line per volid (not the "INCOMPLETE archive" WARN). Tests
|
||||||
|
`TestR689_TheRestoreTestNeverPicksTheGolden` (red-proof: without the check it picks the golden) and
|
||||||
|
`TestR689_GuestBackupArchiveShapes`; two older picker tests' fixtures moved to real archive names.
|
||||||
|
Evidence: `felhom.eu/documentation/audits/version-travel-2026-09-26/D1/`.
|
||||||
|
|
||||||
## v0.134.0 — a whole-box backup that cannot fit is skipped with a reason, before anything starts (2026-09-25 night, R-685)
|
## v0.134.0 — a whole-box backup that cannot fit is skipped with a reason, before anything starts (2026-09-25 night, R-685)
|
||||||
|
|
||||||
> **RELEASED 2026-09-24 night** by `scripts/release-agent.sh` — tag `v0.134.0` (`0722b2c`), sha256 `7593bebe03234c7d22f3ade384e7ed7787dc659aa8c8594b3ee19af2ce81c72d`, verified by download. **NOT vouched** (Day-0 stays on the previous version).
|
> **RELEASED 2026-09-24 night** by `scripts/release-agent.sh` — tag `v0.134.0` (`0722b2c`), sha256 `7593bebe03234c7d22f3ade384e7ed7787dc659aa8c8594b3ee19af2ce81c72d`, verified by download. **NOT vouched** (Day-0 stays on the previous version).
|
||||||
|
|||||||
@@ -1,17 +1,10 @@
|
|||||||
# REPORT — agents v0.133.0 + v0.134.0 delivered; v0.134.0 released (night 2026-09-25)
|
# REPORT — 2026-10-04: v0.139.0 (R-834)
|
||||||
|
|
||||||
**Delivered** to demo-felhom and demo-hp by CC-signed `agent_update` jobs (ruling 1, 2026-09-16), each verified
|
Full session report: `felhom.eu/REPORT-backup-close-os-spike-2026-10-04.md`.
|
||||||
against the published package's sha256 first: v0.133.0 (committed 21:35 / 21:50 CEST), then v0.134.0 (22:52 /
|
|
||||||
22:58). Hub reads 0.134.0 on both. Peti's box: nothing. Restore test back ON on both (A3: demo-felhom PASS 85 s;
|
|
||||||
demo-hp refused for space, correctly). demo-hp `local_backup_retention: 1` (operator option A).
|
|
||||||
|
|
||||||
**v0.134.0 = R-685 (agent half):** a vzdump to a LOCAL target needs free ≥ newest archive × 1.25 + 1 GiB; a
|
- **Measured** on demo-hp: the scheduled restore-test's scratch guest has `onboot: 0` and throwaway stand-ins for
|
||||||
shortfall is a named skip before anything starts; fail-open on PBS / first backup / unknown usage. Tag `v0.134.0`
|
mp8/mp9 on every config read until teardown — it was already safe. Evidence `felhom.eu/documentation/audits/backup-close-2026-10-04/partA/`.
|
||||||
(`0722b2c`), sha256 `7593bebe03234c7d22f3ade384e7ed7787dc659aa8c8594b3ee19af2ce81c72d`, verified by download; CI jobs
|
- **Fixed:** the DR bring-up refuses beside a live original (source guest present, a drives bind on another guest,
|
||||||
975–977 success. **Not vouched** for Day-0.
|
or an unreadable config). On a replaced host it is unchanged.
|
||||||
|
- Tests `TestRunBringUp_DRRefusesBesideALiveOriginal`, `TestRestoreTest_NoHostPathBindBesideTheOriginal` and two
|
||||||
**Found live before the release:** the first build read free space from `GET /storage` (no usage) and failed open —
|
more; both rules red-proved. No sudoers change (said why in the CHANGELOG).
|
||||||
its own "must refuse" test started a real vzdump of demo-hp 9201, aborted by CC after 5 min 16 s, no archive left.
|
|
||||||
Fixed (`NodeStorage`), test fake made honest (`ListStorage` strips usage), two red-proofs, re-proven live with safe
|
|
||||||
builds (a hard stop before vzdump): ×10 refused with real numbers, ×1.25 passed. Evidence:
|
|
||||||
`felhom.eu/documentation/audits/night-2026-09-25/F/`.
|
|
||||||
|
|||||||
@@ -48,6 +48,7 @@ import (
|
|||||||
"gitea.dooplex.hu/admin/felhom-agent/internal/pbsdr"
|
"gitea.dooplex.hu/admin/felhom-agent/internal/pbsdr"
|
||||||
"gitea.dooplex.hu/admin/felhom-agent/internal/poke"
|
"gitea.dooplex.hu/admin/felhom-agent/internal/poke"
|
||||||
"gitea.dooplex.hu/admin/felhom-agent/internal/provision"
|
"gitea.dooplex.hu/admin/felhom-agent/internal/provision"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/osupdate"
|
||||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||||
"gitea.dooplex.hu/admin/felhom-agent/internal/restorespace"
|
"gitea.dooplex.hu/admin/felhom-agent/internal/restorespace"
|
||||||
@@ -242,6 +243,8 @@ func main() {
|
|||||||
os.Exit(runSelftestRestoreTest(context.Background(), cfg, logger, archive))
|
os.Exit(runSelftestRestoreTest(context.Background(), cfg, logger, archive))
|
||||||
case "restore-test-due":
|
case "restore-test-due":
|
||||||
os.Exit(runSelftestRestoreTestDue(context.Background(), cfg, logger))
|
os.Exit(runSelftestRestoreTestDue(context.Background(), cfg, logger))
|
||||||
|
case "os-update":
|
||||||
|
os.Exit(runSelftestOSUpdate(context.Background(), cfg, logger, vmid))
|
||||||
case "pbs-verify":
|
case "pbs-verify":
|
||||||
os.Exit(runSelftestPBSVerify(context.Background(), cfg, logger))
|
os.Exit(runSelftestPBSVerify(context.Background(), cfg, logger))
|
||||||
case "lanresolver":
|
case "lanresolver":
|
||||||
@@ -839,6 +842,10 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
|||||||
// The "Down" channel sync hook: on each heartbeat, fetch desired-state when the generation
|
// The "Down" channel sync hook: on each heartbeat, fetch desired-state when the generation
|
||||||
// advances. The loop calls it via the EnvelopeObserver seam (hub does not import desired).
|
// advances. The loop calls it via the EnvelopeObserver seam (hub does not import desired).
|
||||||
desiredSyncer := desired.NewSyncer(client, desiredProvider, logger)
|
desiredSyncer := desired.NewSyncer(client, desiredProvider, logger)
|
||||||
|
// OS updates, guest fast lane (agent v0.140.0, `11-os-updates.md` §8 step 2): the leg consumes the hub's
|
||||||
|
// os_update block and runs after each successful primary whole-guest backup (wired on the local API below).
|
||||||
|
osLeg := newOSLeg(cfg, client, logger)
|
||||||
|
desiredSyncer.AddConsumer(osLeg)
|
||||||
// S5: consume a host_loss restore_directive into an inspectable restore PLAN (derive + surface,
|
// S5: consume a host_loss restore_directive into an inspectable restore PLAN (derive + surface,
|
||||||
// execute nothing). The recipe is fetched on-demand (rare directive) via a fresh Collect.
|
// execute nothing). The recipe is fetched on-demand (rare directive) via a fresh Collect.
|
||||||
desiredSyncer.AddConsumer(dr.NewConsumer(func(ctx context.Context) *hub.DRRecipeHostHalf {
|
desiredSyncer.AddConsumer(dr.NewConsumer(func(ctx context.Context) *hub.DRRecipeHostHalf {
|
||||||
@@ -1112,6 +1119,17 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
|
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
|
||||||
|
if localSrv != nil {
|
||||||
|
localSrv.SetAfterPrimaryBackup(func(ctx context.Context, vmid int) {
|
||||||
|
// Let the controller finish bringing its apps back after the backup, then run (still under the gate).
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case <-time.After(90 * time.Second):
|
||||||
|
}
|
||||||
|
osLeg.Run(ctx, vmid, "night")
|
||||||
|
})
|
||||||
|
}
|
||||||
if localTokens != nil {
|
if localTokens != nil {
|
||||||
defer localTokens.Close()
|
defer localTokens.Close()
|
||||||
}
|
}
|
||||||
@@ -3478,8 +3496,66 @@ func (f *selftestFlag) Set(v string) error {
|
|||||||
f.mode = "identity-consume"
|
f.mode = "identity-consume"
|
||||||
case "controller-swap":
|
case "controller-swap":
|
||||||
f.mode = "controller-swap"
|
f.mode = "controller-swap"
|
||||||
|
case "os-update":
|
||||||
|
f.mode = "os-update"
|
||||||
|
case "wgtunnel": // dispatched since S3 but refused here until 2026-10-04 (TestSelftestFlag_AcceptsEveryDispatchedMode)
|
||||||
|
f.mode = "wgtunnel"
|
||||||
default:
|
default:
|
||||||
return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test|restore-test-due|pbs-verify|bring-up|provision|escrow-create|escrow-consume|identity-consume|controller-swap)", v)
|
return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test|restore-test-due|pbs-verify|lanresolver|wgtunnel|bring-up|provision|escrow-create|escrow-consume|identity-consume|controller-swap|os-update)", v)
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// newOSLeg builds the OS-update leg (agent v0.140.0). The wrapper runs through sudo (FELHOM_OSAPPLY); the plan and
|
||||||
|
// the once-per-night marker live in the agent's own os/ dir.
|
||||||
|
func newOSLeg(cfg config.Config, client *hub.Client, logger *slog.Logger) *osupdate.Leg {
|
||||||
|
mode := proxmox.RunnerMode(cfg.Privileged.Mode)
|
||||||
|
if mode == "" {
|
||||||
|
mode = proxmox.RunnerSudo
|
||||||
|
}
|
||||||
|
l := &osupdate.Leg{
|
||||||
|
Runner: &proxmox.ExecRunner{Mode: mode, SudoPath: cfg.Privileged.SudoPath},
|
||||||
|
Logger: logger,
|
||||||
|
PlanDir: osupdate.DefaultPlanDir,
|
||||||
|
StatePath: filepath.Join(osupdate.DefaultPlanDir, "last-night-run"),
|
||||||
|
}
|
||||||
|
if client != nil {
|
||||||
|
l.Hub = client
|
||||||
|
}
|
||||||
|
return l
|
||||||
|
}
|
||||||
|
|
||||||
|
// runSelftestOSUpdate is the OS leg's DEBUG ACTION (agent v0.140.0): one pass for -vmid, now, exactly as the night
|
||||||
|
// runs it after a backup — the hub's os_update block (fetched fresh), the wrapper via sudo, the health wait, the
|
||||||
|
// report to the hub — with trigger "debug" (never throttled, and it does NOT count as a night run for approval).
|
||||||
|
// Run it as the agent user: sudo -u felhom-agent felhom-agent --config … --selftest=os-update -vmid 9201
|
||||||
|
func runSelftestOSUpdate(ctx context.Context, cfg config.Config, logger *slog.Logger, vmid int) int {
|
||||||
|
if vmid <= 0 {
|
||||||
|
fmt.Fprintln(os.Stderr, "selftest=os-update: -vmid is required")
|
||||||
|
return 2
|
||||||
|
}
|
||||||
|
client, err := hub.NewClient(cfg.Hub, logger)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(os.Stderr, "selftest=os-update: hub client:", err)
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
leg := newOSLeg(cfg, client, logger)
|
||||||
|
resp, err := client.FetchDesiredState(ctx)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(os.Stderr, "selftest=os-update: desired state:", err)
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
leg.SetBlock(resp.DesiredState.OSUpdate)
|
||||||
|
b := leg.Block()
|
||||||
|
fmt.Printf("=== felhom-agent %s selftest=os-update vmid=%d ring=%d enabled=%v release=%v ===\n", version, vmid, b.Ring, b.Enabled, b.Release != nil)
|
||||||
|
rep := leg.Run(ctx, vmid, "debug")
|
||||||
|
printJSON("os-update report", map[string]any{"run_id": rep.RunID, "ring": rep.Ring, "release_id": rep.ReleaseID,
|
||||||
|
"mode": rep.Mode, "outcome": rep.Outcome, "healthy": rep.Healthy, "health_reason": rep.HealthReason,
|
||||||
|
"upgraded": rep.Upgraded, "pending": len(rep.Pending), "not_covered": rep.NotCovered,
|
||||||
|
"restart_needed": rep.RestartNeeded, "docker_restart_needed": rep.DockerRestartNeeded, "refused": rep.Refused})
|
||||||
|
switch rep.Outcome {
|
||||||
|
case "applied", "nothing", "inventory", "skipped":
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,39 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Every mode the dispatcher (`switch selftest.mode`) runs must be ACCEPTED by the --selftest flag. Found live
|
||||||
|
// 2026-10-04: --selftest=os-update had a dispatch case and a function but the flag's allow-list refused it, so the
|
||||||
|
// debug action could not run. Red-proof: drop the "os-update" case from selftestFlag.Set and this fails.
|
||||||
|
func TestSelftestFlag_AcceptsEveryDispatchedMode(t *testing.T) {
|
||||||
|
src, err := os.ReadFile("main.go")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
s := string(src)
|
||||||
|
i := strings.Index(s, "switch selftest.mode {")
|
||||||
|
if i < 0 {
|
||||||
|
t.Fatal("dispatch switch not found")
|
||||||
|
}
|
||||||
|
block := s[i:]
|
||||||
|
block = block[:strings.Index(block, "\n\t}\n")]
|
||||||
|
modes := regexp.MustCompile(`(?m)^\tcase "([a-z-]+)":`).FindAllStringSubmatch(block, -1)
|
||||||
|
if len(modes) < 5 {
|
||||||
|
t.Fatalf("parsed only %d dispatch cases — the parser is wrong", len(modes))
|
||||||
|
}
|
||||||
|
var bad []string
|
||||||
|
for _, m := range modes {
|
||||||
|
var f selftestFlag
|
||||||
|
if err := f.Set(m[1]); err != nil {
|
||||||
|
bad = append(bad, m[1])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(bad) > 0 {
|
||||||
|
t.Fatalf("dispatched but refused by --selftest: %v", bad)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -299,10 +299,17 @@ Cmnd_Alias FELHOM_SELFHEAL = \
|
|||||||
# argument after the numeric vmid is a literal, so the grant cannot be widened by anything the guest or
|
# argument after the numeric vmid is a literal, so the grant cannot be widened by anything the guest or
|
||||||
# the hub says. The address read is deliberately NOT duplicated here — it is already FELHOM_DNSMASQ's,
|
# the hub says. The address read is deliberately NOT duplicated here — it is already FELHOM_DNSMASQ's,
|
||||||
# and the same command must not be granted twice under two names.
|
# and the same command must not be granted twice under two names.
|
||||||
|
# OS updates, guest fast lane (`11-os-updates.md` §5.4.1, agent v0.140.0). The ONLY entry: the root wrapper with
|
||||||
|
# one plan file in the agent's own os/ dir. Every safety rule (no removal, no downgrade, no new or unlisted package,
|
||||||
|
# Debian origin only, the box's own customer guest only) lives in the wrapper, red-proved per rule
|
||||||
|
# (configs/test_felhom_os_apply.py). The agent gets NO apt grant of its own.
|
||||||
|
Cmnd_Alias FELHOM_OSAPPLY = \
|
||||||
|
/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json
|
||||||
|
|
||||||
Cmnd_Alias FELHOM_GUESTNET = \
|
Cmnd_Alias FELHOM_GUESTNET = \
|
||||||
/usr/sbin/pct exec [0-9]* -- ip route show default, \
|
/usr/sbin/pct exec [0-9]* -- ip route show default, \
|
||||||
/usr/sbin/pct exec [0-9]* -- cat /etc/network/interfaces, \
|
/usr/sbin/pct exec [0-9]* -- cat /etc/network/interfaces, \
|
||||||
/usr/sbin/pct exec [0-9]* -- pgrep -x dhclient, \
|
/usr/sbin/pct exec [0-9]* -- pgrep -x dhclient, \
|
||||||
/usr/sbin/pct exec [0-9]* -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
|
/usr/sbin/pct exec [0-9]* -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
|
||||||
|
|
||||||
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN
|
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN, FELHOM_OSAPPLY
|
||||||
|
|||||||
Executable
+486
@@ -0,0 +1,486 @@
|
|||||||
|
#!/usr/bin/python3
|
||||||
|
# felhom-os-apply — the ROOT half of the agent's operating-system update leg (`11-os-updates.md` §5.4.1).
|
||||||
|
#
|
||||||
|
# Install as /usr/local/sbin/felhom-os-apply (0755 root:root). The non-root agent invokes it via `sudo -n`
|
||||||
|
# (FELHOM_OSAPPLY alias) with EXACTLY: felhom-os-apply --plan /var/lib/felhom-agent/os/plan-<id>.json
|
||||||
|
# Nothing else on the command line is accepted. Python 3, standard library only (a JSON plan cannot be parsed
|
||||||
|
# safely in sh). Tests: configs/test_felhom_os_apply.py (a fake runner; nothing real is executed).
|
||||||
|
#
|
||||||
|
# THE TRUST MODEL. The plan is written by the agent, so a broken-into agent writes whatever plan it likes. The
|
||||||
|
# protection is therefore what this file REFUSES, not where the plan came from: no removal, no downgrade, no new
|
||||||
|
# package, no package outside the plan, only Debian origin in the fast lane, only the box's own customer guest.
|
||||||
|
# Package signatures stay Debian's: apt checks every Release file against the guest's keyring, including the
|
||||||
|
# snapshot.debian.org fallback (decision 79). Nothing here is overridable from the environment.
|
||||||
|
#
|
||||||
|
# THIS RELEASE: layer "guest", lane "fast" only. The host layer and the slow lane exist in the interface and are
|
||||||
|
# REFUSED (R3, R12) until `11` §8 steps 3 and 5 enable them.
|
||||||
|
#
|
||||||
|
# Modes (plan field "mode"):
|
||||||
|
# inventory read-only for packages: `apt-get update` in the guest, then report what is installed (with origin),
|
||||||
|
# what is pending, restart-needed and health. Installs nothing.
|
||||||
|
# apply repair first, check every refusal on an `apt-get -s` simulation of EXACTLY name=version, then
|
||||||
|
# install, clean, and report the same as inventory.
|
||||||
|
# health report health only (the agent polls it after a run).
|
||||||
|
# Output: log lines on stderr and the journal (tag felhom-os-apply); the LAST stdout line is
|
||||||
|
# OSAPPLY-REPORT <one JSON object>
|
||||||
|
# which is what the agent parses. Exit 0 = done; 2 = refused (nothing changed); 3 = failed during install.
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import stat
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
|
||||||
|
PLAN_DIR = "/var/lib/felhom-agent/os"
|
||||||
|
PLAN_RE = re.compile(r"^plan-[A-Za-z0-9._-]{1,80}\.json$")
|
||||||
|
AGENT_USER = "felhom-agent"
|
||||||
|
FAST_ORIGINS = ("Debian", "Debian-Security")
|
||||||
|
# Debian package name and version grammar (Debian policy §5.6.1, §5.6.12).
|
||||||
|
NAME_RE = re.compile(r"^[a-z0-9][a-z0-9+.-]+$")
|
||||||
|
VERSION_RE = re.compile(r"^(?:[0-9]+:)?[0-9][A-Za-z0-9.+~-]*$")
|
||||||
|
SNAP_RE = re.compile(r"^[0-9]{8}T[0-9]{6}Z$")
|
||||||
|
RESERVED_VMIDS = set(range(990000, 990010)) | {9999}
|
||||||
|
DRIVES_PARENT = "/mnt/felhom-drives"
|
||||||
|
SNAPSHOT_LIST = "/etc/apt/sources.list.d/felhom-os-snapshot.list"
|
||||||
|
APT_ENV = ["env", "DEBIAN_FRONTEND=noninteractive", "APT_LISTCHANGES_FRONTEND=none", "NEEDRESTART_MODE=l", "LC_ALL=C"]
|
||||||
|
DPKG_OPTS = ["-o", "Dpkg::Options::=--force-confold", "-o", "Dpkg::Options::=--force-confdef"]
|
||||||
|
MIN_FREE = 500 * 1024 * 1024
|
||||||
|
|
||||||
|
|
||||||
|
class Refused(Exception):
|
||||||
|
def __init__(self, code, reason):
|
||||||
|
super().__init__(f"{code} {reason}")
|
||||||
|
self.code, self.reason = code, reason
|
||||||
|
|
||||||
|
|
||||||
|
class Runner:
|
||||||
|
"""Runs commands for real. Tests replace it with a fake. `guest` runs inside the container via pct exec."""
|
||||||
|
|
||||||
|
def host(self, argv, timeout=600):
|
||||||
|
p = subprocess.run(argv, capture_output=True, text=True, timeout=timeout)
|
||||||
|
return p.returncode, p.stdout, p.stderr
|
||||||
|
|
||||||
|
def guest(self, vmid, argv, timeout=1800):
|
||||||
|
return self.host(["/usr/sbin/pct", "exec", str(vmid), "--"] + argv, timeout)
|
||||||
|
|
||||||
|
def read_file(self, path):
|
||||||
|
with open(path) as f:
|
||||||
|
return f.read()
|
||||||
|
|
||||||
|
def stat(self, path):
|
||||||
|
return os.lstat(path)
|
||||||
|
|
||||||
|
def agent_uid(self):
|
||||||
|
import pwd
|
||||||
|
return pwd.getpwnam(AGENT_USER).pw_uid
|
||||||
|
|
||||||
|
def log(self, line):
|
||||||
|
print(line, file=sys.stderr, flush=True)
|
||||||
|
try:
|
||||||
|
subprocess.run(["logger", "-t", "felhom-os-apply", line], timeout=10)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class Apply:
|
||||||
|
def __init__(self, runner, plan_path):
|
||||||
|
self.r = runner
|
||||||
|
self.plan_path = plan_path
|
||||||
|
self.report = {"refused": None, "mode": None}
|
||||||
|
|
||||||
|
# ---------- checks ----------
|
||||||
|
def load_plan(self):
|
||||||
|
p = self.plan_path
|
||||||
|
d, base = os.path.dirname(p), os.path.basename(p)
|
||||||
|
if d != PLAN_DIR or not PLAN_RE.match(base) or ".." in p:
|
||||||
|
raise Refused("R1", f"the plan must be {PLAN_DIR}/plan-<id>.json, got {p!r}")
|
||||||
|
try:
|
||||||
|
st = self.r.stat(p)
|
||||||
|
except OSError as e:
|
||||||
|
raise Refused("R1", f"cannot stat the plan: {e}")
|
||||||
|
if not stat.S_ISREG(st.st_mode):
|
||||||
|
raise Refused("R1", "the plan is not a regular file (a symlink or a device is refused)")
|
||||||
|
if st.st_uid != self.r.agent_uid():
|
||||||
|
raise Refused("R1", f"the plan is not owned by {AGENT_USER}")
|
||||||
|
if st.st_size > 2 * 1024 * 1024:
|
||||||
|
raise Refused("R1", "the plan is larger than 2 MB")
|
||||||
|
try:
|
||||||
|
plan = json.loads(self.r.read_file(p))
|
||||||
|
except (OSError, ValueError) as e:
|
||||||
|
raise Refused("R1", f"the plan is not valid JSON: {e}")
|
||||||
|
if not isinstance(plan, dict):
|
||||||
|
raise Refused("R1", "the plan is not a JSON object")
|
||||||
|
return plan
|
||||||
|
|
||||||
|
def check_plan(self, plan):
|
||||||
|
mode = plan.get("mode", "apply")
|
||||||
|
if mode not in ("apply", "inventory", "health"):
|
||||||
|
raise Refused("R11", f"unknown mode {mode!r}")
|
||||||
|
if plan.get("layer") != "guest":
|
||||||
|
raise Refused("R12", f"layer {plan.get('layer')!r} is refused in this release (guest only)")
|
||||||
|
if plan.get("lane", "fast") != "fast":
|
||||||
|
raise Refused("R3", "the slow lane is refused in this release")
|
||||||
|
vmid = plan.get("vmid")
|
||||||
|
if not isinstance(vmid, int) or isinstance(vmid, bool) or vmid <= 0:
|
||||||
|
raise Refused("R11", f"vmid must be a positive integer, got {vmid!r}")
|
||||||
|
rid = plan.get("release_id", "")
|
||||||
|
if not isinstance(rid, str) or not re.match(r"^[A-Za-z0-9._:-]{1,80}$", rid):
|
||||||
|
raise Refused("R11", f"release_id {rid!r} is not a plain id")
|
||||||
|
if plan.get("allow_new"):
|
||||||
|
raise Refused("R6", "allow_new is a slow-lane field; the fast lane never adds a package")
|
||||||
|
pk = plan.get("packages", [])
|
||||||
|
if not isinstance(pk, list) or (mode == "apply" and not pk):
|
||||||
|
raise Refused("R11", "packages must be a non-empty list in apply mode")
|
||||||
|
seen = set()
|
||||||
|
for e in pk:
|
||||||
|
if not isinstance(e, dict):
|
||||||
|
raise Refused("R11", "every package entry must be an object")
|
||||||
|
n, v, o = e.get("name"), e.get("version"), e.get("origin")
|
||||||
|
if not isinstance(n, str) or not NAME_RE.match(n):
|
||||||
|
raise Refused("R11", f"package name {n!r} is not a Debian package name")
|
||||||
|
if not isinstance(v, str) or not VERSION_RE.match(v):
|
||||||
|
raise Refused("R11", f"version {v!r} of {n} is not a Debian version string")
|
||||||
|
if n in seen:
|
||||||
|
raise Refused("R11", f"package {n} is named twice")
|
||||||
|
seen.add(n)
|
||||||
|
if o not in FAST_ORIGINS:
|
||||||
|
raise Refused("R2", f"{n}: origin {o!r} is not Debian / Debian-Security (the fast lane, `11` C3)")
|
||||||
|
snap = plan.get("snapshot", "")
|
||||||
|
if snap and not SNAP_RE.match(snap):
|
||||||
|
raise Refused("R11", f"snapshot {snap!r} is not YYYYMMDDTHHMMSSZ")
|
||||||
|
return mode, vmid
|
||||||
|
|
||||||
|
def check_guest(self, vmid):
|
||||||
|
if vmid in RESERVED_VMIDS:
|
||||||
|
raise Refused("R10", f"vmid {vmid} is a reserved scratch vmid")
|
||||||
|
try:
|
||||||
|
conf = self.r.read_file(f"/etc/pve/lxc/{vmid}.conf")
|
||||||
|
except OSError:
|
||||||
|
raise Refused("R10", f"vmid {vmid} is not a container on this host")
|
||||||
|
cur = conf.split("\n[", 1)[0] # the current config, not a snapshot section
|
||||||
|
binds = [l for l in cur.splitlines() if re.match(r"^mp[0-9]+: " + re.escape(DRIVES_PARENT) + r",", l)]
|
||||||
|
if not binds:
|
||||||
|
raise Refused("R10", f"vmid {vmid} does not bind {DRIVES_PARENT} — it is not this box's customer guest")
|
||||||
|
lock = [l for l in cur.splitlines() if l.startswith("lock:")]
|
||||||
|
if lock:
|
||||||
|
raise Refused("R9", f"vmid {vmid} is locked ({lock[0].split(':', 1)[1].strip()}) — a backup or restore is running")
|
||||||
|
rc, out, _ = self.r.host(["/usr/sbin/pct", "status", str(vmid)])
|
||||||
|
if rc != 0 or "running" not in out:
|
||||||
|
raise Refused("R10", f"vmid {vmid} is not running")
|
||||||
|
|
||||||
|
# ---------- guest helpers ----------
|
||||||
|
def g(self, argv, timeout=1800):
|
||||||
|
return self.r.guest(self.vmid, argv, timeout)
|
||||||
|
|
||||||
|
def installed(self):
|
||||||
|
rc, out, _ = self.g(["dpkg-query", "-W", "-f", "${Package}\t${Version}\t${db:Status-Abbrev}\n"])
|
||||||
|
res = {}
|
||||||
|
for l in out.splitlines():
|
||||||
|
parts = l.split("\t")
|
||||||
|
if len(parts) == 3 and parts[2].startswith("ii"):
|
||||||
|
res[parts[0]] = parts[1]
|
||||||
|
return res
|
||||||
|
|
||||||
|
def dpkg_cmp(self, a, op, b):
|
||||||
|
rc, _, _ = self.g(["dpkg", "--compare-versions", a, op, b])
|
||||||
|
return rc == 0
|
||||||
|
|
||||||
|
def madison(self, name):
|
||||||
|
rc, out, _ = self.g(["apt-cache", "madison", name])
|
||||||
|
vs = set()
|
||||||
|
for l in out.splitlines():
|
||||||
|
f = [x.strip() for x in l.split("|")]
|
||||||
|
if len(f) >= 3 and f[0] == name:
|
||||||
|
vs.add(f[1])
|
||||||
|
return vs
|
||||||
|
|
||||||
|
def simulate(self, args):
|
||||||
|
rc, out, err = self.g(APT_ENV + ["apt-get", "-s", "-q"] + args)
|
||||||
|
inst, remv = [], []
|
||||||
|
for l in out.splitlines():
|
||||||
|
m = re.match(r"^Inst (\S+) (?:\[([^]]*)\] )?\((\S+) (.*?) \[[a-z0-9]+\]\)", l)
|
||||||
|
if m:
|
||||||
|
inst.append({"name": m.group(1), "from": m.group(2), "to": m.group(3), "origin": m.group(4)})
|
||||||
|
m = re.match(r"^Remv (\S+)", l)
|
||||||
|
if m:
|
||||||
|
remv.append(m.group(1))
|
||||||
|
return rc, inst, remv, out + err
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def origin_name(origin):
|
||||||
|
# "Debian:13.7/stable, Debian-Security:13/stable-security" -> {"Debian", "Debian-Security"}
|
||||||
|
return {o.strip().split(":")[0] for o in origin.split(",") if o.strip()}
|
||||||
|
|
||||||
|
def free_bytes(self):
|
||||||
|
rc, out, _ = self.g(["df", "-B1", "--output=avail", "/"])
|
||||||
|
try:
|
||||||
|
return int(out.strip().splitlines()[-1])
|
||||||
|
except (ValueError, IndexError):
|
||||||
|
return -1
|
||||||
|
|
||||||
|
def apt_lock_held(self):
|
||||||
|
rc, out, _ = self.g(["fuser", "/var/lib/dpkg/lock-frontend", "/var/lib/dpkg/lock"])
|
||||||
|
return rc == 0 and out.strip() != ""
|
||||||
|
|
||||||
|
def health(self):
|
||||||
|
"""The guest's signals: every container's state + health, the controller's own health, the network."""
|
||||||
|
rc, out, _ = self.g(["docker", "ps", "-a", "--format", "{{.Names}}\t{{.State}}\t{{.Status}}"], timeout=60)
|
||||||
|
cont = {}
|
||||||
|
for l in out.splitlines():
|
||||||
|
p = l.split("\t")
|
||||||
|
if len(p) == 3:
|
||||||
|
h = "healthy" if "(healthy)" in p[2] else "unhealthy" if "(unhealthy)" in p[2] else \
|
||||||
|
"starting" if "(health: starting)" in p[2] else "none"
|
||||||
|
cont[p[0]] = {"state": p[1], "health": h}
|
||||||
|
nrc, _, _ = self.g(["getent", "hosts", "deb.debian.org"], timeout=30)
|
||||||
|
return {"docker_ok": rc == 0, "containers": cont,
|
||||||
|
"controller": cont.get("felhom-controller", {}).get("health", "absent"),
|
||||||
|
"network_ok": nrc == 0}
|
||||||
|
|
||||||
|
def restart_needed(self):
|
||||||
|
"""Processes still mapping deleted files, OUTSIDE docker containers (C11)."""
|
||||||
|
script = ('for p in /proc/[0-9]*; do grep -q "(deleted)" $p/maps 2>/dev/null || continue; '
|
||||||
|
'grep -q "docker" $p/cgroup 2>/dev/null && continue; echo "${p#/proc/} $(cat $p/comm 2>/dev/null)"; done')
|
||||||
|
rc, out, _ = self.g(["sh", "-c", script], timeout=120)
|
||||||
|
procs = sorted({l.split(" ", 1)[1] for l in out.splitlines() if " " in l})
|
||||||
|
pid1 = any(l.split(" ", 1)[0] == "1" for l in out.splitlines())
|
||||||
|
return procs, pid1
|
||||||
|
|
||||||
|
def inventory(self):
|
||||||
|
inst = self.installed()
|
||||||
|
names = sorted(inst)
|
||||||
|
origins = {}
|
||||||
|
for i in range(0, len(names), 200):
|
||||||
|
rc, out, _ = self.g(["apt-cache", "policy"] + names[i:i + 200])
|
||||||
|
cur, star = None, False
|
||||||
|
for l in out.splitlines():
|
||||||
|
if not l.startswith(" "):
|
||||||
|
cur, star = l.rstrip(":"), False
|
||||||
|
continue
|
||||||
|
s = l.strip()
|
||||||
|
if s.startswith("*** "):
|
||||||
|
star = True
|
||||||
|
continue
|
||||||
|
if star and cur and re.match(r"^[0-9-]+ ", s):
|
||||||
|
if "/var/lib/dpkg/status" in s:
|
||||||
|
origins.setdefault(cur, "local")
|
||||||
|
else:
|
||||||
|
origins[cur] = s
|
||||||
|
continue
|
||||||
|
if star and not re.match(r"^[0-9-]+ ", s):
|
||||||
|
star = False
|
||||||
|
# Map an index URL to an origin name the hub understands.
|
||||||
|
def oname(src):
|
||||||
|
if src in (None, "local"):
|
||||||
|
return "unknown"
|
||||||
|
if "security" in src and "debian" in src:
|
||||||
|
return "Debian-Security"
|
||||||
|
if "docker.com" in src:
|
||||||
|
return "Docker"
|
||||||
|
if "debian" in src:
|
||||||
|
return "Debian"
|
||||||
|
return "other"
|
||||||
|
rc, pend, remv, _ = self.simulate(["dist-upgrade"])
|
||||||
|
return {
|
||||||
|
"installed": [{"name": n, "version": inst[n], "origin": oname(origins.get(n))} for n in names],
|
||||||
|
"pending": [{"name": p["name"], "from": p["from"], "to": p["to"],
|
||||||
|
"origin": sorted(self.origin_name(p["origin"]))} for p in pend],
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------- the run ----------
|
||||||
|
def run(self):
|
||||||
|
plan = self.load_plan()
|
||||||
|
self.mode, self.vmid = self.check_plan(plan)
|
||||||
|
self.report.update(mode=self.mode, release_id=plan.get("release_id"), vmid=self.vmid)
|
||||||
|
self.check_guest(self.vmid)
|
||||||
|
log = self.r.log
|
||||||
|
if self.mode == "health":
|
||||||
|
self.report["health"] = self.health()
|
||||||
|
return 0
|
||||||
|
log(f"os-apply: START release={plan.get('release_id')} layer=guest:{self.vmid} lane=fast mode={self.mode} packages={len(plan.get('packages', []))}")
|
||||||
|
if self.apt_lock_held():
|
||||||
|
raise Refused("R9", "another apt/dpkg holds the lock in the guest")
|
||||||
|
self.report["health_before"] = self.health()
|
||||||
|
if self.mode == "apply":
|
||||||
|
self.repair()
|
||||||
|
rc, out, err = self.g(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
|
||||||
|
if rc != 0:
|
||||||
|
raise Refused("R7", f"apt-get update failed in the guest: {(out + err).strip().splitlines()[-1:]}")
|
||||||
|
if self.mode == "apply":
|
||||||
|
rc = self.apply(plan)
|
||||||
|
if rc:
|
||||||
|
return rc
|
||||||
|
procs, pid1 = self.restart_needed()
|
||||||
|
self.report.update(self.inventory())
|
||||||
|
self.report["restart_needed"] = procs
|
||||||
|
self.report["docker_restart_needed"] = any(p in ("dockerd", "containerd") for p in procs)
|
||||||
|
self.report["reboot_needed"] = pid1
|
||||||
|
self.report["health_after"] = self.health()
|
||||||
|
return 0
|
||||||
|
|
||||||
|
def repair(self):
|
||||||
|
rc, before, _ = self.g(["dpkg", "--audit"])
|
||||||
|
self.g(APT_ENV + ["dpkg", "--configure", "-a", "--force-confold"])
|
||||||
|
rc2, out, err = self.g(APT_ENV + ["apt-get", "-f", "install", "-y", "-q"] + DPKG_OPTS)
|
||||||
|
_, after, _ = self.g(["dpkg", "--audit"])
|
||||||
|
configured = len([l for l in before.splitlines() if l.startswith(" ")])
|
||||||
|
fixed = len(re.findall(r"^Setting up ", out, re.M))
|
||||||
|
self.report["repair"] = {"half_configured_before": configured, "fixed": fixed, "clean_after": after.strip() == ""}
|
||||||
|
self.r.log(f"os-apply: REPAIR configured={configured} fixed={fixed}")
|
||||||
|
if after.strip():
|
||||||
|
raise Refused("R13", "dpkg is still broken after the repair: " + after.strip().splitlines()[0])
|
||||||
|
|
||||||
|
def apply(self, plan):
|
||||||
|
log = self.r.log
|
||||||
|
inst = self.installed()
|
||||||
|
upgrade, already, notinst = [], 0, 0
|
||||||
|
for e in plan["packages"]:
|
||||||
|
n, v = e["name"], e["version"]
|
||||||
|
if n not in inst:
|
||||||
|
notinst += 1
|
||||||
|
continue
|
||||||
|
if not self.dpkg_cmp(v, "gt", inst[n]):
|
||||||
|
already += 1
|
||||||
|
continue
|
||||||
|
upgrade.append((n, v))
|
||||||
|
from_snap = 0
|
||||||
|
missing = [(n, v) for n, v in upgrade if v not in self.madison(n)]
|
||||||
|
if missing:
|
||||||
|
snap = plan.get("snapshot", "")
|
||||||
|
if not snap:
|
||||||
|
raise Refused("R7", f"{missing[0][0]}={missing[0][1]} is not downloadable and the plan names no snapshot")
|
||||||
|
self.add_snapshot_sources(snap)
|
||||||
|
still = [(n, v) for n, v in missing if v not in self.madison(n)]
|
||||||
|
if still:
|
||||||
|
self.remove_snapshot_sources()
|
||||||
|
raise Refused("R7", f"{still[0][0]}={still[0][1]} is not downloadable, not even from snapshot {snap}")
|
||||||
|
from_snap = len(missing)
|
||||||
|
try:
|
||||||
|
log(f"os-apply: PLAN upgrade={len(upgrade)} already={already} not-installed={notinst} from-snapshot={from_snap}")
|
||||||
|
self.report["plan"] = {"upgrade": len(upgrade), "already": already, "not_installed": notinst, "from_snapshot": from_snap}
|
||||||
|
if not upgrade:
|
||||||
|
self.report["upgraded"] = []
|
||||||
|
log("os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)")
|
||||||
|
return 0
|
||||||
|
args = ["install", "--only-upgrade", "--no-install-recommends"] + [f"{n}={v}" for n, v in upgrade]
|
||||||
|
rc, sim, remv, text = self.simulate(args)
|
||||||
|
if rc != 0:
|
||||||
|
tail = text.strip().splitlines()[-1] if text.strip() else ""
|
||||||
|
raise Refused("R7", "the simulation failed: " + tail)
|
||||||
|
if remv:
|
||||||
|
raise Refused("R4", f"the plan would remove {', '.join(remv[:5])}")
|
||||||
|
want = dict(upgrade)
|
||||||
|
for p in sim:
|
||||||
|
if p["from"] is None:
|
||||||
|
raise Refused("R6", f"the plan would add a package that is not installed: {p['name']}")
|
||||||
|
if p["name"] not in want:
|
||||||
|
raise Refused("R6", f"the plan would touch {p['name']}, which is not in the plan")
|
||||||
|
if p["to"] != want[p["name"]]:
|
||||||
|
raise Refused("R6", f"{p['name']} would go to {p['to']}, not the approved {want[p['name']]}")
|
||||||
|
if not self.dpkg_cmp(p["to"], "gt", p["from"]):
|
||||||
|
raise Refused("R5", f"{p['name']} would be downgraded {p['from']} -> {p['to']}")
|
||||||
|
if not self.origin_name(p["origin"]) & set(FAST_ORIGINS):
|
||||||
|
raise Refused("R2", f"{p['name']} would come from {p['origin']}, not Debian")
|
||||||
|
need = self.download_bytes(args)
|
||||||
|
free = self.free_bytes()
|
||||||
|
if free >= 0 and free < max(MIN_FREE, 3 * need):
|
||||||
|
raise Refused("R8", f"free space {free} B is below max(500 MB, 3 x download {need} B)")
|
||||||
|
t0 = time.time()
|
||||||
|
rc, out, err = self.g(APT_ENV + ["apt-get", "-y", "-q"] + DPKG_OPTS + args)
|
||||||
|
secs = time.time() - t0
|
||||||
|
# dpkg says "Installing new version of config file X" when X was NOT changed locally (the package's new
|
||||||
|
# version is taken), and "Configuration file 'X'" + "Keeping old config file" when it was (--force-confold
|
||||||
|
# keeps the local one; the package's version lands as X.dpkg-dist). Measured live 2026-10-04 (debian_version).
|
||||||
|
conflict = None
|
||||||
|
for l in (out + err).splitlines():
|
||||||
|
m = re.search(r"Installing new version of config file (\S+?)\s*\.\.\.", l)
|
||||||
|
if m:
|
||||||
|
log(f"os-apply: CONFFILE updated {m.group(1)} (it was not changed locally)")
|
||||||
|
m = re.search(r"Configuration file '([^']+)'", l)
|
||||||
|
if m:
|
||||||
|
conflict = m.group(1)
|
||||||
|
if conflict and "Keeping old config file" in l:
|
||||||
|
log(f"os-apply: CONFFILE kept {conflict} (changed locally; the package's version is {conflict}.dpkg-dist)")
|
||||||
|
self.report.setdefault("conffiles_kept", []).append(conflict)
|
||||||
|
conflict = None
|
||||||
|
self.g(["apt-get", "clean"])
|
||||||
|
if rc != 0:
|
||||||
|
_, aud, _ = self.g(["dpkg", "--audit"])
|
||||||
|
first = aud.strip().splitlines()[0] if aud.strip() else "clean"
|
||||||
|
log(f"os-apply: FAILED rc={rc} step=install — dpkg state: {first}")
|
||||||
|
self.report["failed"] = {"rc": rc, "dpkg_audit": first, "tail": (out + err).strip().splitlines()[-3:]}
|
||||||
|
return 3
|
||||||
|
self.report["upgraded"] = [{"name": n, "version": v} for n, v in upgrade]
|
||||||
|
self.report["seconds"] = round(secs, 1)
|
||||||
|
log(f"os-apply: DONE rc=0 seconds={secs:.1f} upgraded={len(upgrade)}")
|
||||||
|
return 0
|
||||||
|
finally:
|
||||||
|
if from_snap:
|
||||||
|
self.remove_snapshot_sources()
|
||||||
|
|
||||||
|
def download_bytes(self, args):
|
||||||
|
rc, out, _ = self.g(APT_ENV + ["apt-get", "-s", "-o", "Debug::NoLocking=1", "--print-uris", "-q"] + args)
|
||||||
|
total = 0
|
||||||
|
for l in out.splitlines():
|
||||||
|
m = re.match(r"^'[^']+' \S+ ([0-9]+) ", l)
|
||||||
|
if m:
|
||||||
|
total += int(m.group(1))
|
||||||
|
return total
|
||||||
|
|
||||||
|
def add_snapshot_sources(self, snap):
|
||||||
|
rc, out, _ = self.g(["sh", "-c", ". /etc/os-release && echo $VERSION_CODENAME"])
|
||||||
|
code = out.strip()
|
||||||
|
if not re.match(r"^[a-z]+$", code):
|
||||||
|
raise Refused("R7", f"cannot read the guest's Debian codename ({code!r})")
|
||||||
|
body = (f"deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/{snap} {code} main\n"
|
||||||
|
f"deb [check-valid-until=no] http://snapshot.debian.org/archive/debian-security/{snap} {code}-security main\n")
|
||||||
|
self.r.guest_write(self.vmid, SNAPSHOT_LIST, body)
|
||||||
|
self.r.log(f"os-apply: SNAPSHOT using snapshot.debian.org/{snap} for versions no longer published (decision 79)")
|
||||||
|
rc, out, err = self.g(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
|
||||||
|
if rc != 0:
|
||||||
|
self.remove_snapshot_sources()
|
||||||
|
raise Refused("R7", "apt-get update against snapshot.debian.org failed")
|
||||||
|
|
||||||
|
def remove_snapshot_sources(self):
|
||||||
|
self.g(["rm", "-f", SNAPSHOT_LIST])
|
||||||
|
self.g(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
|
||||||
|
|
||||||
|
|
||||||
|
def guest_write(self, vmid, path, body):
|
||||||
|
"""Write a small text file inside the guest via `pct exec … tee` (stdin), never via a shell string."""
|
||||||
|
p = subprocess.run(["/usr/sbin/pct", "exec", str(vmid), "--", "tee", path], input=body,
|
||||||
|
capture_output=True, text=True, timeout=60)
|
||||||
|
if p.returncode != 0:
|
||||||
|
raise Refused("R7", f"could not write {path} in the guest")
|
||||||
|
|
||||||
|
|
||||||
|
Runner.guest_write = guest_write
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv, runner=None):
|
||||||
|
r = runner or Runner()
|
||||||
|
if len(argv) != 3 or argv[1] != "--plan":
|
||||||
|
r.log("os-apply: REFUSED: R1 usage: felhom-os-apply --plan /var/lib/felhom-agent/os/plan-<id>.json")
|
||||||
|
print("OSAPPLY-REPORT " + json.dumps({"refused": {"code": "R1", "reason": "usage"}}))
|
||||||
|
return 2
|
||||||
|
a = Apply(r, argv[2])
|
||||||
|
try:
|
||||||
|
rc = a.run()
|
||||||
|
except Refused as e:
|
||||||
|
r.log(f"os-apply: REFUSED: {e.code} {e.reason}")
|
||||||
|
a.report["refused"] = {"code": e.code, "reason": e.reason}
|
||||||
|
rc = 2
|
||||||
|
except subprocess.TimeoutExpired as e:
|
||||||
|
r.log(f"os-apply: FAILED rc=124 step=timeout — {e.cmd}")
|
||||||
|
a.report["failed"] = {"rc": 124, "timeout": str(e.cmd)[:200]}
|
||||||
|
rc = 3
|
||||||
|
print("OSAPPLY-REPORT " + json.dumps(a.report, sort_keys=True))
|
||||||
|
return rc
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
if os.geteuid() != 0:
|
||||||
|
print("felhom-os-apply: must run as root (via sudo)", file=sys.stderr)
|
||||||
|
sys.exit(2)
|
||||||
|
sys.exit(main(sys.argv))
|
||||||
@@ -0,0 +1,467 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Tests for configs/felhom-os-apply (`11` §5.4.1). A fake runner plays the host and the guest: nothing is executed
|
||||||
|
for real except the local `dpkg --compare-versions` (pure, no network). Every refusal R1–R13 has a test; the red-proof
|
||||||
|
(each test fails when its rule is removed) is `audits/os-guest-lane-2026-10-04/partB/redproof.txt`.
|
||||||
|
|
||||||
|
Run: python3 configs/test_felhom_os_apply.py (also run by internal/osupdate's Go test)
|
||||||
|
"""
|
||||||
|
import importlib.machinery
|
||||||
|
import importlib.util
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import pathlib
|
||||||
|
import re
|
||||||
|
import stat as statmod
|
||||||
|
import subprocess
|
||||||
|
import unittest
|
||||||
|
|
||||||
|
HERE = pathlib.Path(__file__).resolve().parent
|
||||||
|
_loader = importlib.machinery.SourceFileLoader("osapply", os.environ.get("OSAPPLY_UNDER_TEST", str(HERE / "felhom-os-apply"))) # red-proof seam
|
||||||
|
_spec = importlib.util.spec_from_loader("osapply", _loader)
|
||||||
|
osapply = importlib.util.module_from_spec(_spec)
|
||||||
|
_loader.exec_module(osapply)
|
||||||
|
|
||||||
|
PLAN = "/var/lib/felhom-agent/os/plan-t1.json"
|
||||||
|
CONF_OK = ("arch: amd64\nmp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G\n"
|
||||||
|
"mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives\nrootfs: local-lvm:vm-9201-disk-0,size=32G\n")
|
||||||
|
DEB = "Debian:13.7/stable"
|
||||||
|
SEC = "Debian-Security:13/stable-security"
|
||||||
|
|
||||||
|
|
||||||
|
def dpkg_cmp(a, op, b):
|
||||||
|
return subprocess.run(["dpkg", "--compare-versions", a, op, b]).returncode == 0
|
||||||
|
|
||||||
|
|
||||||
|
class St:
|
||||||
|
def __init__(self, mode=statmod.S_IFREG | 0o600, uid=999, size=100):
|
||||||
|
self.st_mode, self.st_uid, self.st_size = mode, uid, size
|
||||||
|
|
||||||
|
|
||||||
|
class Fake:
|
||||||
|
"""The host + one guest. `installed` / `live` (name -> versions in the live archive) / `snapshot` (versions
|
||||||
|
the snapshot archive adds) / `extra_sim` (lines the simulation adds) / `dpkg_audit` / `free`."""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self.plan = {"release_id": "os-t1", "layer": "guest", "lane": "fast", "vmid": 9201, "mode": "apply",
|
||||||
|
"snapshot": "20261004T080000Z",
|
||||||
|
"packages": [{"name": "libc6", "version": "2.41-12+deb13u4", "origin": "Debian"},
|
||||||
|
{"name": "openssl", "version": "3.5.7-1~deb13u3", "origin": "Debian-Security"}]}
|
||||||
|
self.files = {"/etc/pve/lxc/9201.conf": CONF_OK}
|
||||||
|
self.stats = {PLAN: St()}
|
||||||
|
self.installed = {"libc6": "2.41-12+deb13u3", "openssl": "3.5.6-1~deb13u1", "bash": "5.2.37-2+b9"}
|
||||||
|
self.live = {"libc6": {"2.41-12+deb13u4"}, "openssl": {"3.5.7-1~deb13u3"}}
|
||||||
|
self.snapshot = {}
|
||||||
|
self.snap_active = False
|
||||||
|
self.extra_sim = []
|
||||||
|
self.dpkg_audit = ""
|
||||||
|
self.free = 10 * 1024 ** 3
|
||||||
|
self.install_rc = 0
|
||||||
|
self.calls = []
|
||||||
|
self.logs = []
|
||||||
|
self.written = {}
|
||||||
|
self.status = "status: running"
|
||||||
|
self.lock_held = False
|
||||||
|
|
||||||
|
# Runner interface
|
||||||
|
def read_file(self, p):
|
||||||
|
if p == PLAN:
|
||||||
|
return json.dumps(self.plan)
|
||||||
|
if p not in self.files:
|
||||||
|
raise OSError("no such file")
|
||||||
|
return self.files[p]
|
||||||
|
|
||||||
|
def stat(self, p):
|
||||||
|
if p not in self.stats:
|
||||||
|
raise OSError("no such file")
|
||||||
|
return self.stats[p]
|
||||||
|
|
||||||
|
def agent_uid(self):
|
||||||
|
return 999
|
||||||
|
|
||||||
|
def log(self, line):
|
||||||
|
self.logs.append(line)
|
||||||
|
|
||||||
|
def host(self, argv, timeout=600):
|
||||||
|
self.calls.append(("host", argv))
|
||||||
|
if argv[1] == "status":
|
||||||
|
return 0, self.status + "\n", ""
|
||||||
|
return 1, "", "unexpected host call"
|
||||||
|
|
||||||
|
def guest_write(self, vmid, path, body):
|
||||||
|
self.written[path] = body
|
||||||
|
if path == osapply.SNAPSHOT_LIST:
|
||||||
|
self.snap_active = True
|
||||||
|
|
||||||
|
def avail(self, n):
|
||||||
|
v = set(self.live.get(n, set()))
|
||||||
|
if self.snap_active:
|
||||||
|
v |= self.snapshot.get(n, set())
|
||||||
|
return v
|
||||||
|
|
||||||
|
def guest(self, vmid, argv, timeout=1800):
|
||||||
|
self.calls.append(("guest", vmid, argv))
|
||||||
|
a = [x for x in argv if not re.match(r"^[A-Z_]+=", x) and x != "env"]
|
||||||
|
cmd = a[0]
|
||||||
|
if cmd == "dpkg-query":
|
||||||
|
return 0, "".join(f"{n}\t{v}\tii \n" for n, v in self.installed.items()), ""
|
||||||
|
if cmd == "dpkg" and a[1] == "--compare-versions":
|
||||||
|
return (0 if dpkg_cmp(a[2], a[3], a[4]) else 1), "", ""
|
||||||
|
if cmd == "dpkg" and a[1] == "--audit":
|
||||||
|
return 0, self.dpkg_audit, ""
|
||||||
|
if cmd == "dpkg" and a[1] == "--configure":
|
||||||
|
return 0, "", ""
|
||||||
|
if cmd == "fuser":
|
||||||
|
return (0, " 123", "") if self.lock_held else (1, "", "")
|
||||||
|
if cmd == "apt-cache" and a[1] == "madison":
|
||||||
|
return 0, "".join(f" {a[2]} | {v} | http://deb.debian.org trixie/main amd64 Packages\n" for v in self.avail(a[2])), ""
|
||||||
|
if cmd == "apt-cache" and a[1] == "policy":
|
||||||
|
out = ""
|
||||||
|
for n in a[2:]:
|
||||||
|
out += f"{n}:\n Installed: {self.installed.get(n)}\n Version table:\n *** {self.installed.get(n)} 500\n 500 http://deb.debian.org/debian trixie/main amd64 Packages\n"
|
||||||
|
return 0, out, ""
|
||||||
|
if cmd == "apt-get":
|
||||||
|
if "update" in a:
|
||||||
|
return 0, "", ""
|
||||||
|
if "clean" in a:
|
||||||
|
return 0, "", ""
|
||||||
|
if "-f" in a:
|
||||||
|
self.dpkg_audit = ""
|
||||||
|
return 0, "Setting up x (1) ...\n" if getattr(self, "repaired", False) else "", ""
|
||||||
|
if "-s" in a:
|
||||||
|
return self.sim(a)
|
||||||
|
if "install" in a:
|
||||||
|
if self.install_rc:
|
||||||
|
return self.install_rc, "", "E: boom"
|
||||||
|
for x in a:
|
||||||
|
if "=" in x and not x.startswith("-") and "::" not in x:
|
||||||
|
n, v = x.split("=", 1)
|
||||||
|
self.installed[n] = v
|
||||||
|
return 0, getattr(self, "install_out", "Setting up libc6 ...\n"), ""
|
||||||
|
if cmd == "df":
|
||||||
|
return 0, f"Avail\n{self.free}\n", ""
|
||||||
|
if cmd == "docker":
|
||||||
|
return 0, "felhom-controller\trunning\tUp 1 hour (healthy)\napp\trunning\tUp 1 hour (healthy)\n", ""
|
||||||
|
if cmd == "getent":
|
||||||
|
return 0, "1.2.3.4 deb.debian.org\n", ""
|
||||||
|
if cmd == "sh":
|
||||||
|
if "os-release" in a[2]:
|
||||||
|
return 0, "trixie\n", ""
|
||||||
|
return 0, "", ""
|
||||||
|
if cmd == "rm":
|
||||||
|
self.snap_active = False
|
||||||
|
return 0, "", ""
|
||||||
|
return 1, "", f"unexpected guest call {a}"
|
||||||
|
|
||||||
|
def sim(self, a):
|
||||||
|
if "--print-uris" in a:
|
||||||
|
return 0, "'http://x/libc6.deb' libc6.deb 4000000 SHA256:x\n", ""
|
||||||
|
if "dist-upgrade" in a:
|
||||||
|
return 0, "Inst bash [5.2.37-2+b9] (5.2.37-2+b10 Debian:13.7/stable [amd64])\n", ""
|
||||||
|
out = ""
|
||||||
|
for x in a:
|
||||||
|
if "=" in x and not x.startswith("-") and "::" not in x:
|
||||||
|
n, v = x.split("=", 1)
|
||||||
|
if v not in self.avail(n):
|
||||||
|
return 100, "", f"E: Version '{v}' for '{n}' was not found"
|
||||||
|
origin = SEC if n == "openssl" else DEB
|
||||||
|
out += f"Inst {n} [{self.installed[n]}] ({v} {origin} [amd64])\n"
|
||||||
|
out += "".join(l + "\n" for l in self.extra_sim)
|
||||||
|
return 0, out, ""
|
||||||
|
|
||||||
|
|
||||||
|
def run(f):
|
||||||
|
import io
|
||||||
|
import contextlib
|
||||||
|
buf = io.StringIO()
|
||||||
|
with contextlib.redirect_stdout(buf):
|
||||||
|
rc = osapply.main(["felhom-os-apply", "--plan", PLAN], runner=f)
|
||||||
|
line = [l for l in buf.getvalue().splitlines() if l.startswith("OSAPPLY-REPORT ")][-1]
|
||||||
|
return rc, json.loads(line[len("OSAPPLY-REPORT "):])
|
||||||
|
|
||||||
|
|
||||||
|
class Happy(unittest.TestCase):
|
||||||
|
def test_apply_installs_exactly_the_plan(self):
|
||||||
|
f = Fake()
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(f.installed["libc6"], "2.41-12+deb13u4")
|
||||||
|
self.assertEqual(f.installed["openssl"], "3.5.7-1~deb13u3")
|
||||||
|
self.assertEqual(f.installed["bash"], "5.2.37-2+b9", "a package outside the plan was changed")
|
||||||
|
self.assertEqual(rep["plan"]["upgrade"], 2)
|
||||||
|
self.assertIn("installed", rep)
|
||||||
|
self.assertEqual(rep["pending"][0]["name"], "bash")
|
||||||
|
self.assertTrue(any(l.startswith("os-apply: REPAIR ") for l in f.logs), "the repair line must always print")
|
||||||
|
self.assertTrue(any(l.startswith("os-apply: DONE rc=0") for l in f.logs))
|
||||||
|
inst = [c for c in f.calls if c[0] == "guest" and "install" in c[2] and "-s" not in c[2] and "-f" not in c[2]]
|
||||||
|
self.assertTrue(inst and "Dpkg::Options::=--force-confold" in inst[0][2], "must keep existing config files")
|
||||||
|
|
||||||
|
def test_already_current_is_a_no_op(self):
|
||||||
|
f = Fake()
|
||||||
|
f.installed.update(libc6="2.41-12+deb13u4", openssl="3.5.7-1~deb13u3")
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0)
|
||||||
|
self.assertEqual(rep["plan"]["upgrade"], 0)
|
||||||
|
|
||||||
|
def test_inventory_installs_nothing(self):
|
||||||
|
f = Fake()
|
||||||
|
f.plan["mode"] = "inventory"
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(f.installed["libc6"], "2.41-12+deb13u3")
|
||||||
|
self.assertIn("installed", rep)
|
||||||
|
self.assertIn("restart_needed", rep)
|
||||||
|
|
||||||
|
def test_health_mode(self):
|
||||||
|
f = Fake()
|
||||||
|
f.plan["mode"] = "health"
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0)
|
||||||
|
self.assertEqual(rep["health"]["controller"], "healthy")
|
||||||
|
|
||||||
|
|
||||||
|
class Repair(unittest.TestCase):
|
||||||
|
def test_repair_runs_first_and_is_reported(self):
|
||||||
|
f = Fake()
|
||||||
|
f.dpkg_audit = "The following packages have been unpacked but not yet configured.\n perl Larry Wall's\n"
|
||||||
|
f.repaired = True
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(rep["repair"]["half_configured_before"], 1)
|
||||||
|
self.assertEqual(rep["repair"]["fixed"], 1)
|
||||||
|
order = [i for i, c in enumerate(f.calls) if c[0] == "guest" and c[2][-1:] != ["update"]]
|
||||||
|
first_cfg = next(i for i, c in enumerate(f.calls) if c[0] == "guest" and "--configure" in c[2])
|
||||||
|
first_upd = next(i for i, c in enumerate(f.calls) if c[0] == "guest" and "update" in c[2])
|
||||||
|
self.assertLess(first_cfg, first_upd, "the repair must run before anything else touches apt")
|
||||||
|
self.assertTrue(order)
|
||||||
|
|
||||||
|
|
||||||
|
class Snapshot(unittest.TestCase):
|
||||||
|
def test_a_replaced_version_comes_from_the_snapshot(self):
|
||||||
|
f = Fake()
|
||||||
|
f.live["openssl"] = {"3.5.7-1~deb13u4"} # Debian moved on
|
||||||
|
f.snapshot["openssl"] = {"3.5.7-1~deb13u3"}
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(rep["plan"]["from_snapshot"], 1)
|
||||||
|
self.assertEqual(f.installed["openssl"], "3.5.7-1~deb13u3", "must install the APPROVED version, not the newer one")
|
||||||
|
body = f.written[osapply.SNAPSHOT_LIST]
|
||||||
|
self.assertIn("snapshot.debian.org/archive/debian/20261004T080000Z trixie main", body)
|
||||||
|
self.assertIn("debian-security/20261004T080000Z trixie-security main", body)
|
||||||
|
self.assertFalse(f.snap_active, "the temporary snapshot sources must be removed after the run")
|
||||||
|
|
||||||
|
def test_snapshot_does_not_have_it_either(self):
|
||||||
|
f = Fake()
|
||||||
|
f.live["openssl"] = set()
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual((rc, rep["refused"]["code"]), (2, "R7"))
|
||||||
|
self.assertFalse(f.snap_active)
|
||||||
|
|
||||||
|
|
||||||
|
class Refusals(unittest.TestCase):
|
||||||
|
def refused(self, f, code):
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 2, rep)
|
||||||
|
self.assertEqual(rep["refused"]["code"], code, rep)
|
||||||
|
self.assertTrue(any(l.startswith(f"os-apply: REFUSED: {code} ") for l in f.logs), f.logs)
|
||||||
|
inst = [c for c in f.calls if c[0] == "guest" and "install" in c[2] and "-s" not in c[2] and "-f" not in c[2]]
|
||||||
|
self.assertEqual(inst, [], "a refusal must install nothing")
|
||||||
|
return rep
|
||||||
|
|
||||||
|
def test_R1_usage(self):
|
||||||
|
import io
|
||||||
|
import contextlib
|
||||||
|
f = Fake()
|
||||||
|
with contextlib.redirect_stdout(io.StringIO()):
|
||||||
|
self.assertEqual(osapply.main(["felhom-os-apply", "--plan", PLAN, "--extra"], runner=f), 2)
|
||||||
|
self.assertEqual(osapply.main(["felhom-os-apply", "--plan"], runner=f), 2)
|
||||||
|
|
||||||
|
def test_R1_path_outside_the_plan_dir(self):
|
||||||
|
import io
|
||||||
|
import contextlib
|
||||||
|
f = Fake()
|
||||||
|
with contextlib.redirect_stdout(io.StringIO()):
|
||||||
|
rc = osapply.main(["felhom-os-apply", "--plan", "/tmp/plan-x.json"], runner=f)
|
||||||
|
self.assertEqual(rc, 2)
|
||||||
|
self.assertTrue(any("R1" in l for l in f.logs))
|
||||||
|
|
||||||
|
def test_R1_symlink(self):
|
||||||
|
f = Fake()
|
||||||
|
f.stats[PLAN] = St(mode=statmod.S_IFLNK | 0o777)
|
||||||
|
self.refused(f, "R1")
|
||||||
|
|
||||||
|
def test_R1_not_owned_by_the_agent(self):
|
||||||
|
f = Fake()
|
||||||
|
f.stats[PLAN] = St(uid=0)
|
||||||
|
self.refused(f, "R1")
|
||||||
|
|
||||||
|
def test_R2_non_debian_origin_in_the_plan(self):
|
||||||
|
f = Fake()
|
||||||
|
f.plan["packages"][0]["origin"] = "Proxmox"
|
||||||
|
self.refused(f, "R2")
|
||||||
|
|
||||||
|
def test_R2_non_debian_origin_in_the_simulation(self):
|
||||||
|
f = Fake()
|
||||||
|
f.installed["libc6"] = "2.41-12+deb13u3"
|
||||||
|
orig = f.sim
|
||||||
|
|
||||||
|
def sim(a):
|
||||||
|
rc, out, err = orig(a)
|
||||||
|
return rc, out.replace("Debian:13.7/stable", "Proxmox Debian Repository:stable"), err
|
||||||
|
f.sim = sim
|
||||||
|
self.refused(f, "R2")
|
||||||
|
|
||||||
|
def test_R3_slow_lane(self):
|
||||||
|
f = Fake()
|
||||||
|
f.plan["lane"] = "slow"
|
||||||
|
self.refused(f, "R3")
|
||||||
|
|
||||||
|
def test_R4_removal(self):
|
||||||
|
f = Fake()
|
||||||
|
f.extra_sim = ["Remv bash [5.2.37-2+b9]"]
|
||||||
|
self.refused(f, "R4")
|
||||||
|
|
||||||
|
def test_R5_downgrade(self):
|
||||||
|
f = Fake()
|
||||||
|
f.installed["libc6"] = "2.41-12+deb13u4"
|
||||||
|
f.plan["packages"] = [{"name": "openssl", "version": "3.5.7-1~deb13u3", "origin": "Debian-Security"}]
|
||||||
|
f.extra_sim = ["Inst openssl [3.5.6-1~deb13u1] (3.5.5-1 Debian:13.7/stable [amd64])"]
|
||||||
|
orig = f.sim
|
||||||
|
|
||||||
|
def sim(a): # the simulation answers with a LOWER version than installed
|
||||||
|
rc, out, err = orig(a)
|
||||||
|
return rc, "\n".join(l for l in out.splitlines() if not l.startswith("Inst openssl [3.5.6-1~deb13u1] (3.5.7")) + "\n", err
|
||||||
|
f.sim = sim
|
||||||
|
f.plan["packages"][0]["version"] = "3.5.7-1~deb13u3"
|
||||||
|
rep = run(f)[1]
|
||||||
|
# The plan asks 3.5.7; the simulation goes to 3.5.5: that is BOTH a wrong version (R6) and a downgrade.
|
||||||
|
self.assertIn(rep["refused"]["code"], ("R5", "R6"))
|
||||||
|
|
||||||
|
def test_R5_downgrade_exact(self):
|
||||||
|
f = Fake()
|
||||||
|
f.plan["packages"] = [{"name": "openssl", "version": "3.5.7-1~deb13u3", "origin": "Debian-Security"}]
|
||||||
|
f.installed["openssl"] = "3.5.6-1~deb13u1"
|
||||||
|
orig = f.sim
|
||||||
|
|
||||||
|
def sim(a):
|
||||||
|
rc, out, err = orig(a)
|
||||||
|
return rc, out.replace("[3.5.6-1~deb13u1]", "[3.5.8-1]"), err
|
||||||
|
f.sim = sim
|
||||||
|
self.refused(f, "R5")
|
||||||
|
|
||||||
|
def test_R6_new_package(self):
|
||||||
|
f = Fake()
|
||||||
|
f.extra_sim = ["Inst newthing (1.0 Debian:13.7/stable [amd64])"]
|
||||||
|
self.refused(f, "R6")
|
||||||
|
|
||||||
|
def test_R6_unlisted_package(self):
|
||||||
|
f = Fake()
|
||||||
|
f.extra_sim = ["Inst bash [5.2.37-2+b9] (5.2.37-2+b10 Debian:13.7/stable [amd64])"]
|
||||||
|
self.refused(f, "R6")
|
||||||
|
|
||||||
|
def test_R6_allow_new_is_slow_lane(self):
|
||||||
|
f = Fake()
|
||||||
|
f.plan["allow_new"] = ["proxmox-kernel-x"]
|
||||||
|
self.refused(f, "R6")
|
||||||
|
|
||||||
|
def test_R7_not_downloadable_and_no_snapshot(self):
|
||||||
|
f = Fake()
|
||||||
|
f.live["openssl"] = set()
|
||||||
|
f.plan["snapshot"] = ""
|
||||||
|
self.refused(f, "R7")
|
||||||
|
|
||||||
|
def test_R8_free_space(self):
|
||||||
|
f = Fake()
|
||||||
|
f.free = 100 * 1024 * 1024
|
||||||
|
self.refused(f, "R8")
|
||||||
|
|
||||||
|
def test_R9_guest_locked_by_a_backup(self):
|
||||||
|
f = Fake()
|
||||||
|
f.files["/etc/pve/lxc/9201.conf"] = CONF_OK + "lock: backup\n"
|
||||||
|
self.refused(f, "R9")
|
||||||
|
|
||||||
|
def test_R9_apt_lock_held(self):
|
||||||
|
f = Fake()
|
||||||
|
f.lock_held = True
|
||||||
|
self.refused(f, "R9")
|
||||||
|
|
||||||
|
def test_R10_not_the_boxs_own_guest(self):
|
||||||
|
f = Fake()
|
||||||
|
f.files["/etc/pve/lxc/9201.conf"] = CONF_OK.replace("mp8: /mnt/felhom-drives,", "mp8: /mnt/hdd_1/scratch,")
|
||||||
|
self.refused(f, "R10")
|
||||||
|
|
||||||
|
def test_R10_reserved_vmid(self):
|
||||||
|
f = Fake()
|
||||||
|
f.plan["vmid"] = 990003
|
||||||
|
self.refused(f, "R10")
|
||||||
|
|
||||||
|
def test_R10_bind_only_in_a_snapshot_section(self):
|
||||||
|
f = Fake()
|
||||||
|
f.files["/etc/pve/lxc/9201.conf"] = "rootfs: x\n[snap1]\nmp8: /mnt/felhom-drives,mp=/mnt/felhom-drives\n"
|
||||||
|
self.refused(f, "R10")
|
||||||
|
|
||||||
|
def test_R10_not_running(self):
|
||||||
|
f = Fake()
|
||||||
|
f.status = "status: stopped"
|
||||||
|
self.refused(f, "R10")
|
||||||
|
|
||||||
|
def test_R11_duplicate(self):
|
||||||
|
f = Fake()
|
||||||
|
f.plan["packages"].append(dict(f.plan["packages"][0]))
|
||||||
|
self.refused(f, "R11")
|
||||||
|
|
||||||
|
def test_R11_bad_version_string(self):
|
||||||
|
f = Fake()
|
||||||
|
f.plan["packages"][0]["version"] = "1.0; rm -rf /"
|
||||||
|
self.refused(f, "R11")
|
||||||
|
|
||||||
|
def test_R11_bad_name(self):
|
||||||
|
f = Fake()
|
||||||
|
f.plan["packages"][0]["name"] = "--purge"
|
||||||
|
self.refused(f, "R11")
|
||||||
|
|
||||||
|
def test_R12_host_layer(self):
|
||||||
|
f = Fake()
|
||||||
|
f.plan["layer"] = "host"
|
||||||
|
self.refused(f, "R12")
|
||||||
|
|
||||||
|
def test_R13_repair_does_not_fix_it(self):
|
||||||
|
f = Fake()
|
||||||
|
f.dpkg_audit = "The following packages are broken\n perl\n"
|
||||||
|
orig = f.guest
|
||||||
|
|
||||||
|
def guest(vmid, argv, timeout=1800):
|
||||||
|
rc, out, err = orig(vmid, argv, timeout)
|
||||||
|
if "-f" in argv:
|
||||||
|
f.dpkg_audit = "The following packages are broken\n perl\n"
|
||||||
|
return rc, out, err
|
||||||
|
f.guest = guest
|
||||||
|
self.refused(f, "R13")
|
||||||
|
|
||||||
|
|
||||||
|
class Conffiles(unittest.TestCase):
|
||||||
|
# dpkg's two shapes, measured live 2026-10-04: an unchanged file is UPDATED; a locally changed one is KEPT.
|
||||||
|
def test_updated_vs_kept(self):
|
||||||
|
f = Fake()
|
||||||
|
f.install_out = ("Installing new version of config file /etc/debian_version ...\n"
|
||||||
|
"Configuration file '/etc/ssh/sshd_config'\n ==> Modified (by you or by a script) since installation.\n"
|
||||||
|
" ==> Keeping old config file as default.\n")
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertIn("os-apply: CONFFILE updated /etc/debian_version (it was not changed locally)", f.logs)
|
||||||
|
self.assertTrue(any(l.startswith("os-apply: CONFFILE kept /etc/ssh/sshd_config") for l in f.logs), f.logs)
|
||||||
|
self.assertEqual(rep["conffiles_kept"], ["/etc/ssh/sshd_config"])
|
||||||
|
self.assertFalse(any("kept /etc/debian_version" in l for l in f.logs), "an updated file must not be reported as kept")
|
||||||
|
|
||||||
|
|
||||||
|
class Failure(unittest.TestCase):
|
||||||
|
def test_install_failure_is_rc3_with_dpkg_state(self):
|
||||||
|
f = Fake()
|
||||||
|
f.install_rc = 100
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 3)
|
||||||
|
self.assertEqual(rep["failed"]["rc"], 100)
|
||||||
|
self.assertTrue(any(l.startswith("os-apply: FAILED rc=100 step=install") for l in f.logs))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -2,6 +2,7 @@ package backup
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"fmt"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
"io"
|
"io"
|
||||||
@@ -21,6 +22,8 @@ type fakeBackupAPI struct {
|
|||||||
vzdumpErr error
|
vzdumpErr error
|
||||||
waitErr error
|
waitErr error
|
||||||
cfg proxmox.GuestConfig
|
cfg proxmox.GuestConfig
|
||||||
|
goneGuests map[int]bool // R-689: vmids whose config lookup answers "does not exist"
|
||||||
|
aclGuests map[int]bool // R-689: vmids outside the token's ACL — PVE answers 403 "permission denied"
|
||||||
cfgErr error
|
cfgErr error
|
||||||
content []proxmox.StorageContent
|
content []proxmox.StorageContent
|
||||||
contentErr error
|
contentErr error
|
||||||
@@ -43,7 +46,13 @@ func (f *fakeBackupAPI) WaitTask(_ context.Context, _ string, _ proxmox.WaitOpti
|
|||||||
}
|
}
|
||||||
return proxmox.TaskStatus{Status: "stopped", ExitStatus: "OK"}, f.waitErr
|
return proxmox.TaskStatus{Status: "stopped", ExitStatus: "OK"}, f.waitErr
|
||||||
}
|
}
|
||||||
func (f *fakeBackupAPI) GuestConfig(_ context.Context, _ int) (proxmox.GuestConfig, error) {
|
func (f *fakeBackupAPI) GuestConfig(_ context.Context, vmid int) (proxmox.GuestConfig, error) {
|
||||||
|
if f.aclGuests[vmid] {
|
||||||
|
return proxmox.GuestConfig{}, fmt.Errorf("proxmox: GET /nodes/n/lxc/%d/config -> HTTP 403: permission denied at /vms/%d (missing privilege VM.Audit)", vmid, vmid)
|
||||||
|
}
|
||||||
|
if f.goneGuests[vmid] { // R-689: PVE's answer for a deleted guest
|
||||||
|
return proxmox.GuestConfig{}, fmt.Errorf("proxmox: GET /nodes/n/lxc/%d/config -> HTTP 500: Configuration file 'nodes/n/lxc/%d.conf' does not exist", vmid, vmid)
|
||||||
|
}
|
||||||
return f.cfg, f.cfgErr
|
return f.cfg, f.cfgErr
|
||||||
}
|
}
|
||||||
func (f *fakeBackupAPI) StorageContent(_ context.Context, _ string) ([]proxmox.StorageContent, error) {
|
func (f *fakeBackupAPI) StorageContent(_ context.Context, _ string) ([]proxmox.StorageContent, error) {
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package backup
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"fmt"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -54,3 +55,58 @@ func TestR689_GuestBackupArchiveShapes(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// R-689 (v0.136.0) — the measured demo-hp shape right after v0.135.0: the golden (skipped), a leftover archive
|
||||||
|
// of guest 9100 deleted in August (settled), and today's archive of 9201 (not settled yet). The pick must be
|
||||||
|
// NOTHING — never the deleted guest's archive. With 9201's archive settled, that one.
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (REPORT.md): drop the known-guest check — the pick is the 9100 leftover.
|
||||||
|
func TestR689_AnArchiveOfADeletedGuestIsNeverPicked(t *testing.T) {
|
||||||
|
const day = int64(86400)
|
||||||
|
now := int64(1790476000)
|
||||||
|
api := &fakeBackupAPI{goneGuests: map[int]bool{9100: true}, content: []proxmox.StorageContent{
|
||||||
|
{VolID: "local:backup/felhom-golden-0.236.0.tar.zst", Content: "backup", Size: 654115664, CTime: now - 14*day},
|
||||||
|
{VolID: "local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst", Content: "backup", VMID: 9100, Size: 656970239, CTime: now - 37*day},
|
||||||
|
{VolID: "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 7*3600},
|
||||||
|
}}
|
||||||
|
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||||
|
got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now-day, 0).UTC())
|
||||||
|
if err != nil || got != "" {
|
||||||
|
t.Fatalf("picked %q err=%v — a deleted guest's archive proves nothing about this box", got, err)
|
||||||
|
}
|
||||||
|
got, _, _ = r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now, 0).UTC())
|
||||||
|
if got != "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst" {
|
||||||
|
t.Fatalf("with 9201's archive settled the pick is %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Any OTHER lookup failure is not "the guest is gone": the tier must read UNKNOWN (an error), never
|
||||||
|
// "nothing to prove".
|
||||||
|
func TestR689_AGuestLookupFailureIsUnknownNotEmpty(t *testing.T) {
|
||||||
|
api := &fakeBackupAPI{cfgErr: fmt.Errorf("proxmox: connection refused"), content: []proxmox.StorageContent{
|
||||||
|
{VolID: "local:backup/vzdump-lxc-9201-x.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: 10},
|
||||||
|
}}
|
||||||
|
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||||
|
if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Time{}); err == nil {
|
||||||
|
t.Fatal("a failed guest lookup read as a clean answer")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// v0.137.0 — THE MEASURED ANSWER: the agent's token sees only its pool, so for the deleted guest PVE says 403
|
||||||
|
// "permission denied at /vms/9100", not "does not exist" (demo-hp, right after v0.136.0 — the local tier read
|
||||||
|
// UNKNOWN). Such a guest is not one this agent manages: its archive is skipped, the tier is not an error.
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (REPORT.md): drop the "permission denied" case — the pick errors.
|
||||||
|
func TestR689_AGuestOutsideTheAgentsACLIsNotAKnownGuest(t *testing.T) {
|
||||||
|
const day = int64(86400)
|
||||||
|
now := int64(1790476000)
|
||||||
|
api := &fakeBackupAPI{aclGuests: map[int]bool{9100: true}, content: []proxmox.StorageContent{
|
||||||
|
{VolID: "local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst", Content: "backup", VMID: 9100, Size: 656970239, CTime: now - 37*day},
|
||||||
|
{VolID: "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 7*3600},
|
||||||
|
}}
|
||||||
|
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||||
|
got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now-day, 0).UTC())
|
||||||
|
if err != nil || got != "" {
|
||||||
|
t.Fatalf("picked %q err=%v — want nothing and no error (the only settled archive is not ours)", got, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,74 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
thisBoxKey = "de:51:7a:18:cb:39:22:30:2c:84:f5:8b:d1:91:4b:7e:81:bb:69:b8:89:0f:57:ac:d3:59:e1:1a:62:25:11:2c"
|
||||||
|
earlierBox1 = "6b:ca:5f:3f:ca:0f:e2:3f:fb:24:62:89:bf:e7:64:59:9a:41:c5:e6:e3:9f:3f:5f:e1:71:7b:a1:9d:24:67:82"
|
||||||
|
earlierBox2 = "fe:3d:db:95:d4:df:ab:e1:7d:4a:89:fa:2b:07:53:6a:e4:d2:85:95:d1:90:27:4b:d9:c6:92:20:95:04:e5:d4"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-727 (v0.138.0) — the 2026-09-30 shape, measured on a fresh box for a returning customer: the PBS
|
||||||
|
// namespace held two archives of earlier boxes (same guest 9201, same token) and this box's own, which was not
|
||||||
|
// settled yet. The old picker chose the earlier box's newest settled archive and failed `wrong key`.
|
||||||
|
// The CONSEQUENCE asserted: no archive of another box is ever picked; with this box's archive settled it is picked.
|
||||||
|
// COMPANION RED-PROOF: remove the `ownKey != "" && !EqualFold(...)` skip → the first case picks 2026-09-16T21:59:54Z.
|
||||||
|
func TestR727_TheRestoreTestTakesOnlyThisBoxsArchives(t *testing.T) {
|
||||||
|
day := int64(86400)
|
||||||
|
now := int64(1790740000) // 2026-09-30 ~04:00Z
|
||||||
|
own := proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-09-29T19:37:07Z", Content: "backup", VMID: 9201, Size: 3490689830, CTime: 1790710627, Encrypted: thisBoxKey}
|
||||||
|
api := &fakeBackupAPI{
|
||||||
|
storages: []proxmox.Storage{{Storage: "felhom-pbs", Type: "pbs", EncryptionKey: thisBoxKey}},
|
||||||
|
content: []proxmox.StorageContent{
|
||||||
|
{VolID: "felhom-pbs:backup/ct/9201/2026-09-16T17:27:32Z", Content: "backup", VMID: 9201, Size: 4774114206, CTime: 1789579652, Encrypted: earlierBox2},
|
||||||
|
{VolID: "felhom-pbs:backup/ct/9201/2026-09-16T21:59:54Z", Content: "backup", VMID: 9201, Size: 20811501236, CTime: 1789595994, Encrypted: earlierBox1},
|
||||||
|
own,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||||
|
|
||||||
|
// 1. The night of 2026-09-30: this box's own archive is ~6 h old, not settled (cutoff 24 h) — nothing to prove.
|
||||||
|
got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Unix(now-day, 0).UTC())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if got != "" {
|
||||||
|
t.Fatalf("picked %q — an archive of ANOTHER box is never this box's proof (R-727)", got)
|
||||||
|
}
|
||||||
|
// 2. A day later this box's own archive is settled — it is the one picked.
|
||||||
|
got, _, err = r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Unix(now+day, 0).UTC())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if got != own.VolID {
|
||||||
|
t.Fatalf("picked %q, want this box's own %q", got, own.VolID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// An unencrypted storage (a local dir) holds only this box's vzdumps — no key filter applies.
|
||||||
|
func TestR727_UnencryptedStorageIsNotFiltered(t *testing.T) {
|
||||||
|
api := &fakeBackupAPI{
|
||||||
|
storages: []proxmox.Storage{{Storage: "local", Type: "dir"}},
|
||||||
|
content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-2026_09_29-21_27_05.tar.zst", Content: "backup", VMID: 9201, Size: 955425507, CTime: 1790710025}},
|
||||||
|
}
|
||||||
|
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||||
|
if got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Time{}); err != nil || got == "" {
|
||||||
|
t.Fatalf("got %q err %v", got, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A storage-list failure makes the tier UNKNOWN (an error), never "nothing to prove".
|
||||||
|
func TestR727_KeyLookupFailureIsUnknown(t *testing.T) {
|
||||||
|
api := &fakeBackupAPI{storageErr: errors.New("proxmox: GET /storage -> HTTP 500"), content: []proxmox.StorageContent{{VolID: "felhom-pbs:backup/ct/9201/x", Content: "backup", VMID: 9201}}}
|
||||||
|
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||||
|
if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Time{}); err == nil {
|
||||||
|
t.Fatal("a failed key lookup must surface as an error (tier UNKNOWN)")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -357,8 +357,19 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return "", time.Time{}, err
|
return "", time.Time{}, err
|
||||||
}
|
}
|
||||||
|
// R-727 (v0.138.0): on an ENCRYPTED storage, only archives written with THIS storage's key are this box's.
|
||||||
|
// Measured 2026-09-30 on a fresh box for a returning customer: the PBS namespace still held two archives
|
||||||
|
// of earlier boxes (same guest id 9201, same token), the newest settled one was an earlier box's, and the
|
||||||
|
// test failed `wrong key` every evaluation. The archive carries no host id; its key fingerprint is the
|
||||||
|
// discriminator (PVE's content `encrypted`, the storage's `encryption-key`). A lookup failure returns
|
||||||
|
// the error — the tier reads UNKNOWN, never "nothing to prove".
|
||||||
|
ownKey, err := r.storageKeyFingerprint(ctx, target)
|
||||||
|
if err != nil {
|
||||||
|
return "", time.Time{}, fmt.Errorf("reading the key fingerprint of storage %s: %w", target, err)
|
||||||
|
}
|
||||||
var best string
|
var best string
|
||||||
var bestCTime int64 = -1
|
var bestCTime int64 = -1
|
||||||
|
known := map[int]bool{} // vmid → the guest exists on this node (asked once per vmid per pick)
|
||||||
for _, e := range contents {
|
for _, e := range contents {
|
||||||
if e.Content != "backup" {
|
if e.Content != "backup" {
|
||||||
continue
|
continue
|
||||||
@@ -371,6 +382,33 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
|
|||||||
r.noteNotAGuestBackupOnce(e, why)
|
r.noteNotAGuestBackupOnce(e, why)
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) {
|
||||||
|
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this box's proof", shortFP(e.Encrypted), shortFP(ownKey)))
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
// R-689 (v0.136.0): … OF A GUEST THAT STILL EXISTS here. Measured on demo-hp 2026-09-27 right after
|
||||||
|
// v0.135.0: with the golden skipped, the pick fell to `vzdump-lxc-9100-2026_08_21…`, a leftover of a
|
||||||
|
// guest deleted in August — proving nothing about any guest this box runs. "Does not exist" skips the
|
||||||
|
// archive; any OTHER lookup failure is returned, so the tier reads UNKNOWN, never "nothing to prove".
|
||||||
|
if _, seen := known[e.VMID]; !seen {
|
||||||
|
_, err := r.api.GuestConfig(ctx, e.VMID)
|
||||||
|
switch {
|
||||||
|
case err == nil:
|
||||||
|
known[e.VMID] = true
|
||||||
|
case strings.Contains(err.Error(), "does not exist"), strings.Contains(err.Error(), "permission denied"):
|
||||||
|
// v0.137.0: PVE answers 403 "permission denied at /vms/<id>" — not "does not exist" — for a guest
|
||||||
|
// outside the agent's ACL (the `felhom` pool). Measured on demo-hp after v0.136.0: the deleted
|
||||||
|
// guest 9100's archive made the local tier UNKNOWN every evaluation. A guest the agent cannot
|
||||||
|
// read is not one it manages; its archive is not a candidate.
|
||||||
|
known[e.VMID] = false
|
||||||
|
default:
|
||||||
|
return "", time.Time{}, fmt.Errorf("checking whether guest %d still exists: %w", e.VMID, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !known[e.VMID] {
|
||||||
|
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("guest %d does not exist on this node or is not one this agent manages", e.VMID))
|
||||||
|
continue
|
||||||
|
}
|
||||||
if !notAfter.IsZero() && e.CTime > notAfter.Unix() {
|
if !notAfter.IsZero() && e.CTime > notAfter.Unix() {
|
||||||
continue // not settled yet — a newer archive is not a reason to re-prove an older one
|
continue // not settled yet — a newer archive is not a reason to re-prove an older one
|
||||||
}
|
}
|
||||||
@@ -643,3 +681,29 @@ func (r *BackupRunner) noteNotAGuestBackupOnce(e proxmox.StorageContent, why str
|
|||||||
"target", r.target, "volid", e.VolID, "size_bytes", e.Size, "reason", why)
|
"target", r.target, "volid", e.VolID, "size_bytes", e.Size, "reason", why)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// storageKeyFingerprint returns the named storage's client-side encryption key fingerprint ("" when the
|
||||||
|
// storage is not encrypted — a local dir holds only this box's own vzdumps).
|
||||||
|
func (r *BackupRunner) storageKeyFingerprint(ctx context.Context, target string) (string, error) {
|
||||||
|
sts, err := r.api.ListStorage(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
for _, st := range sts {
|
||||||
|
if st.Storage == target {
|
||||||
|
return strings.TrimSpace(st.EncryptionKey), nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return "", nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// shortFP is the first 8 bytes of a key fingerprint, for a log line.
|
||||||
|
func shortFP(fp string) string {
|
||||||
|
if fp == "" {
|
||||||
|
return "none"
|
||||||
|
}
|
||||||
|
if len(fp) > 23 {
|
||||||
|
return fp[:23] + "…"
|
||||||
|
}
|
||||||
|
return fp
|
||||||
|
}
|
||||||
|
|||||||
@@ -123,6 +123,9 @@ var manifest = []Capability{
|
|||||||
{"dnsmasq-install", "dnsmasq package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "dnsmasq"}, false, ""},
|
{"dnsmasq-install", "dnsmasq package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "dnsmasq"}, false, ""},
|
||||||
{"dnsmasq-write", "dnsmasq drop-in write", "/usr/bin/install", []string{"-m", "0644", "/tmp/felhom-resolver-x.conf", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""},
|
{"dnsmasq-write", "dnsmasq drop-in write", "/usr/bin/install", []string{"-m", "0644", "/tmp/felhom-resolver-x.conf", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""},
|
||||||
{"dnsmasq-enable", "dnsmasq enable", "/usr/bin/systemctl", []string{"enable", "--now", "dnsmasq"}, false, ""},
|
{"dnsmasq-enable", "dnsmasq enable", "/usr/bin/systemctl", []string{"enable", "--now", "dnsmasq"}, false, ""},
|
||||||
|
|
||||||
|
// ---- OS updates, guest fast lane (`11` §5.4.1; the wrapper holds every rule) ----
|
||||||
|
{"osapply-run", "OS update wrapper (guest fast lane)", "/usr/local/sbin/felhom-os-apply", []string{"--plan", "/var/lib/felhom-agent/os/plan-x.json"}, false, ""},
|
||||||
{"dnsmasq-reload", "dnsmasq reload", "/usr/bin/systemctl", []string{"reload", "dnsmasq"}, false, ""},
|
{"dnsmasq-reload", "dnsmasq reload", "/usr/bin/systemctl", []string{"reload", "dnsmasq"}, false, ""},
|
||||||
{"dnsmasq-restart", "dnsmasq restart (LAN-DNS self-heal)", "/usr/bin/systemctl", []string{"restart", "dnsmasq"}, false, ""},
|
{"dnsmasq-restart", "dnsmasq restart (LAN-DNS self-heal)", "/usr/bin/systemctl", []string{"restart", "dnsmasq"}, false, ""},
|
||||||
{"dnsmasq-rm", "dnsmasq drop-in remove (decommission)", "/usr/bin/rm", []string{"-f", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""},
|
{"dnsmasq-rm", "dnsmasq drop-in remove (decommission)", "/usr/bin/rm", []string{"-f", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""},
|
||||||
|
|||||||
@@ -425,3 +425,27 @@ func (c *Client) FetchRetainedIdentityEscrow(ctx context.Context) (*RetainedEscr
|
|||||||
}
|
}
|
||||||
return &out, nil
|
return &out, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// PostOSReport sends the OS-update leg's report after every run (hub v0.130.0): POST /api/v1/hosts/{id}/os-report.
|
||||||
|
// Per-host key, self-scoped on the hub. Errors are typed like RegisterWG's and never include the bearer.
|
||||||
|
func (c *Client) PostOSReport(ctx context.Context, body []byte) error {
|
||||||
|
if c.hostID == "" {
|
||||||
|
return fmt.Errorf("hub: PostOSReport requires a configured host_id")
|
||||||
|
}
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+"/api/v1/hosts/"+c.hostID+"/os-report", bytes.NewReader(body))
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("hub: building os-report request: %w", err)
|
||||||
|
}
|
||||||
|
req.Header.Set("Authorization", "Bearer "+c.apiKey)
|
||||||
|
req.Header.Set("Content-Type", "application/json")
|
||||||
|
resp, err := c.hc.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return &TransportError{Err: err}
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
|
||||||
|
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||||
|
return &HTTPError{StatusCode: resp.StatusCode, BodyTail: tail(raw, 256)}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,30 @@
|
|||||||
|
package hub
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"os"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The os_update block is a cross-repo contract: testdata/desired-state-osupdate.golden.json is byte-identical with
|
||||||
|
// felhom.eu/hub/internal/api/testdata (the hub's TestOSUpdate_DesiredBlockMatchesTheGolden proves the hub SERVES
|
||||||
|
// it). Here: the agent DECODES every field. A renamed json tag on either side fails one of the two tests.
|
||||||
|
func TestOSUpdateGolden_Decodes(t *testing.T) {
|
||||||
|
raw, err := os.ReadFile("testdata/desired-state-osupdate.golden.json")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var resp DesiredStateResponse
|
||||||
|
if err := json.Unmarshal(raw, &resp); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
o := resp.DesiredState.OSUpdate
|
||||||
|
if o == nil || o.Ring != 1 || !o.Enabled || o.Release == nil {
|
||||||
|
t.Fatalf("os_update = %+v", o)
|
||||||
|
}
|
||||||
|
r := o.Release
|
||||||
|
if r.ID != "os-20261004-120000" || r.Snapshot != "20261004T120000Z" || len(r.Packages) != 2 ||
|
||||||
|
r.Packages[1].Name != "openssl" || r.Packages[1].Version != "3.5.7-1~deb13u3" || r.Packages[1].Origin != "Debian-Security" {
|
||||||
|
t.Fatalf("release = %+v", r)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -548,6 +548,32 @@ type WireDesiredState struct {
|
|||||||
RestoreDirective *WireRestoreDirective `json:"restore_directive,omitempty"` // slice 10D (forward-compat)
|
RestoreDirective *WireRestoreDirective `json:"restore_directive,omitempty"` // slice 10D (forward-compat)
|
||||||
Wireguard *WireWireguard `json:"wireguard,omitempty"` // S3 (doc 06 §3.2; golden-pinned)
|
Wireguard *WireWireguard `json:"wireguard,omitempty"` // S3 (doc 06 §3.2; golden-pinned)
|
||||||
PBSDR *WirePBSDR `json:"pbs_dr,omitempty"` // PBS DR tier (slice 2 consumer)
|
PBSDR *WirePBSDR `json:"pbs_dr,omitempty"` // PBS DR tier (slice 2 consumer)
|
||||||
|
OSUpdate *WireOSUpdate `json:"os_update,omitempty"` // OS updates, guest fast lane (agent v0.140.0)
|
||||||
|
}
|
||||||
|
|
||||||
|
// WireOSUpdate is the hub-OWNED OS-update block (hub v0.130.0, `11-os-updates.md` §5.3), merged into the served
|
||||||
|
// document at read time. Ring 0 installs every pending Debian / Debian-Security fix; ring 1 installs exactly the
|
||||||
|
// newest approved release. Absent (older hub) → the agent treats the box as ring 1, ON, no release: it reports
|
||||||
|
// and installs nothing. Golden: testdata/desired-state-osupdate.golden.json (byte-identical with the hub's).
|
||||||
|
type WireOSUpdate struct {
|
||||||
|
Ring int `json:"ring"`
|
||||||
|
Enabled bool `json:"enabled"`
|
||||||
|
Release *WireOSRelease `json:"release,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// WireOSRelease is an approved version set; Snapshot is the approval time (YYYYMMDDTHHMMSSZ) the wrapper uses
|
||||||
|
// for snapshot.debian.org when Debian has already replaced a version (decision 79).
|
||||||
|
type WireOSRelease struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Snapshot string `json:"snapshot"`
|
||||||
|
Packages []WireOSPackage `json:"packages"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// WireOSPackage is one approved name=version and its origin ("Debian" | "Debian-Security").
|
||||||
|
type WireOSPackage struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
Version string `json:"version"`
|
||||||
|
Origin string `json:"origin"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// WirePBSDR is the hub's PBS-DR-tier descriptor (PBS DR slice 1, hub/internal/web/pbsdr.go
|
// WirePBSDR is the hub's PBS-DR-tier descriptor (PBS DR slice 1, hub/internal/web/pbsdr.go
|
||||||
|
|||||||
@@ -0,0 +1,17 @@
|
|||||||
|
{
|
||||||
|
"generation": 1,
|
||||||
|
"desired_state": {
|
||||||
|
"os_update": {
|
||||||
|
"ring": 1,
|
||||||
|
"enabled": true,
|
||||||
|
"release": {
|
||||||
|
"id": "os-20261004-120000",
|
||||||
|
"snapshot": "20261004T120000Z",
|
||||||
|
"packages": [
|
||||||
|
{"name": "libc6", "version": "2.41-12+deb13u4", "origin": "Debian"},
|
||||||
|
{"name": "openssl", "version": "3.5.7-1~deb13u3", "origin": "Debian-Security"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
package localapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"net/http"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The OS leg (agent v0.140.0) runs after a SUCCESSFUL primary backup, and only then; and it runs BEFORE the
|
||||||
|
// host-wide heavy-op gate is released, so a restore-test cannot start in the middle of it (`11` C10).
|
||||||
|
// Red-proof: drop the `b.Success &&` guard and the failed-backup sub-case fails; move the call after release()
|
||||||
|
// and the gate sub-case fails.
|
||||||
|
func TestAfterPrimaryBackup(t *testing.T) {
|
||||||
|
run := func(t *testing.T, failErr string) (calls []int, gateHeld bool) {
|
||||||
|
gate := &backup.InFlight{}
|
||||||
|
b := &fakeBackups{failErr: failErr}
|
||||||
|
srv := newTestServerS(t, &fakeGuests{}, b, &fakeStore{}, nil)
|
||||||
|
srv.inFlight = gate
|
||||||
|
var mu sync.Mutex
|
||||||
|
done := make(chan struct{}, 1)
|
||||||
|
srv.SetAfterPrimaryBackup(func(_ context.Context, vmid int) {
|
||||||
|
rel, _, ok := gate.TryAcquire("probe")
|
||||||
|
mu.Lock()
|
||||||
|
calls = append(calls, vmid)
|
||||||
|
gateHeld = !ok
|
||||||
|
mu.Unlock()
|
||||||
|
if ok {
|
||||||
|
rel()
|
||||||
|
}
|
||||||
|
done <- struct{}{}
|
||||||
|
})
|
||||||
|
h := srv.Handler()
|
||||||
|
if do(t, h, "POST", "/backup", "A", "").Code != http.StatusAccepted {
|
||||||
|
t.Fatal("POST /backup not accepted")
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-done:
|
||||||
|
case <-time.After(500 * time.Millisecond):
|
||||||
|
}
|
||||||
|
time.Sleep(20 * time.Millisecond)
|
||||||
|
mu.Lock()
|
||||||
|
defer mu.Unlock()
|
||||||
|
return calls, gateHeld
|
||||||
|
}
|
||||||
|
t.Run("success runs the leg under the gate", func(t *testing.T) {
|
||||||
|
calls, held := run(t, "")
|
||||||
|
if len(calls) != 1 {
|
||||||
|
t.Fatalf("the leg ran %d time(s), want 1", len(calls))
|
||||||
|
}
|
||||||
|
if !held {
|
||||||
|
t.Fatal("the heavy-op gate was free while the leg ran — a restore-test could overlap it")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("a failed backup runs nothing", func(t *testing.T) {
|
||||||
|
if calls, _ := run(t, "vzdump exploded"); len(calls) != 0 {
|
||||||
|
t.Fatalf("the leg ran after a FAILED backup: %v", calls)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
@@ -160,6 +160,10 @@ type Options struct {
|
|||||||
// NetStorage is the privileged network-mount (NAS) surface (Part A1). OPTIONAL — when nil, the
|
// NetStorage is the privileged network-mount (NAS) surface (Part A1). OPTIONAL — when nil, the
|
||||||
// /netstorage endpoints report "not configured". Satisfied by *storage.SudoHostOps.
|
// /netstorage endpoints report "not configured". Satisfied by *storage.SudoHostOps.
|
||||||
NetStorage NetworkStorageOps
|
NetStorage NetworkStorageOps
|
||||||
|
// AfterPrimaryBackup (agent v0.140.0, `11-os-updates.md` §8 step 2) runs right after a SUCCESSFUL backup on the
|
||||||
|
// PRIMARY tier, inside the backup goroutine and BEFORE the host-wide heavy-op gate is released — so the OS leg
|
||||||
|
// that it starts can never overlap another backup or a restore-test (`11` C10). OPTIONAL — nil → nothing runs.
|
||||||
|
AfterPrimaryBackup func(ctx context.Context, vmid int)
|
||||||
// Privileged runs the fenced root wrappers (E-2a: felhom-backup-target-apply). OPTIONAL — when
|
// Privileged runs the fenced root wrappers (E-2a: felhom-backup-target-apply). OPTIONAL — when
|
||||||
// nil, POST /backup/target reports "not configured". Satisfied by *proxmox.ExecRunner.
|
// nil, POST /backup/target reports "not configured". Satisfied by *proxmox.ExecRunner.
|
||||||
Privileged PrivilegedRunner
|
Privileged PrivilegedRunner
|
||||||
@@ -276,6 +280,7 @@ type Server struct {
|
|||||||
tiers []BackupTier
|
tiers []BackupTier
|
||||||
// inFlight (R-85) is shared with the restore-test scheduler so the two never run together.
|
// inFlight (R-85) is shared with the restore-test scheduler so the two never run together.
|
||||||
inFlight *backup.InFlight
|
inFlight *backup.InFlight
|
||||||
|
afterPrimaryBackup func(ctx context.Context, vmid int) // the OS leg (agent v0.140.0); nil = none
|
||||||
logger *slog.Logger
|
logger *slog.Logger
|
||||||
now func() time.Time
|
now func() time.Time
|
||||||
|
|
||||||
@@ -469,6 +474,7 @@ func NewServer(o Options) (*Server, error) {
|
|||||||
// the primary is always first, because that is what the untargeted endpoints act on.
|
// the primary is always first, because that is what the untargeted endpoints act on.
|
||||||
s.tiers = normalizeBackupTiers(o.BackupTiers, o.Backups, cadence)
|
s.tiers = normalizeBackupTiers(o.BackupTiers, o.Backups, cadence)
|
||||||
s.inFlight = o.InFlight
|
s.inFlight = o.InFlight
|
||||||
|
s.afterPrimaryBackup = o.AfterPrimaryBackup
|
||||||
if s.backups == nil && len(s.tiers) > 0 {
|
if s.backups == nil && len(s.tiers) > 0 {
|
||||||
s.backups = s.tiers[0].Service
|
s.backups = s.tiers[0].Service
|
||||||
}
|
}
|
||||||
@@ -894,6 +900,10 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
|
|||||||
}
|
}
|
||||||
s.store.RecordBackup(b)
|
s.store.RecordBackup(b)
|
||||||
s.finishJob(key, jobID, b)
|
s.finishJob(key, jobID, b)
|
||||||
|
// OS leg (agent v0.140.0): after the night's whole-guest copy exists, still holding the heavy-op gate.
|
||||||
|
if b.Success && tier.Primary && s.afterPrimaryBackup != nil {
|
||||||
|
s.afterPrimaryBackup(base, vmid)
|
||||||
|
}
|
||||||
}()
|
}()
|
||||||
writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: jobID, Phase: PhaseRunning}, "")
|
writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: jobID, Phase: PhaseRunning}, "")
|
||||||
}
|
}
|
||||||
@@ -1465,3 +1475,6 @@ func writeStatus(w http.ResponseWriter, code int, ok bool, data any, errMsg stri
|
|||||||
w.WriteHeader(code)
|
w.WriteHeader(code)
|
||||||
_ = json.NewEncoder(w).Encode(apiResponse{OK: ok, Data: data, Error: errMsg})
|
_ = json.NewEncoder(w).Encode(apiResponse{OK: ok, Data: data, Error: errMsg})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SetAfterPrimaryBackup wires the hook that runs after a successful primary-tier backup (the OS leg, agent v0.140.0).
|
||||||
|
func (s *Server) SetAfterPrimaryBackup(fn func(ctx context.Context, vmid int)) { s.afterPrimaryBackup = fn }
|
||||||
|
|||||||
@@ -0,0 +1,459 @@
|
|||||||
|
// Package osupdate is the agent's OS-update leg for the customer GUEST (`11-os-updates.md` §8 step 2, agent v0.140.0).
|
||||||
|
//
|
||||||
|
// It runs right after the night's successful whole-guest backup, while the backup goroutine still holds the host-wide
|
||||||
|
// heavy-op gate (so it never overlaps a backup or a restore-test, `11` C10), at most once per night. All root work is
|
||||||
|
// the wrapper `felhom-os-apply` (configs/, its own tests); this package only builds plans, calls the wrapper through
|
||||||
|
// sudo, judges health and reports to the hub.
|
||||||
|
//
|
||||||
|
// NO AUTOMATIC UNDO in this release (R-837, measured 2026-10-04): a customer guest cannot be snapshotted — PVE
|
||||||
|
// refuses any snapshot not named `vzdump` when the guest has host-path binds (mp8, mp9). A failed health check
|
||||||
|
// therefore stops, reports `health_failed` and the hub mails the operator; the whole-guest backup taken minutes
|
||||||
|
// earlier is the undo, by hand.
|
||||||
|
package osupdate
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||||
|
)
|
||||||
|
|
||||||
|
// WrapperPath is the pinned sudoers vector (configs/felhom-agent.sudoers FELHOM_OSAPPLY).
|
||||||
|
const WrapperPath = "/usr/local/sbin/felhom-os-apply"
|
||||||
|
|
||||||
|
// DefaultPlanDir is where plans are written (the sudoers glob names it).
|
||||||
|
const DefaultPlanDir = "/var/lib/felhom-agent/os"
|
||||||
|
|
||||||
|
// Package is one name=version with its origin.
|
||||||
|
type Package struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
Version string `json:"version"`
|
||||||
|
Origin string `json:"origin"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Pending is one update the guest's sources offer (origin as apt names it, possibly several).
|
||||||
|
type Pending struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
From string `json:"from"`
|
||||||
|
To string `json:"to"`
|
||||||
|
Origin []string `json:"origin"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Container is one container's state as the wrapper saw it.
|
||||||
|
type Container struct {
|
||||||
|
State string `json:"state"`
|
||||||
|
Health string `json:"health"` // healthy | unhealthy | starting | none
|
||||||
|
}
|
||||||
|
|
||||||
|
// Health is the guest's health snapshot (the wrapper's `health` object).
|
||||||
|
type Health struct {
|
||||||
|
DockerOK bool `json:"docker_ok"`
|
||||||
|
NetworkOK bool `json:"network_ok"`
|
||||||
|
Controller string `json:"controller"`
|
||||||
|
Containers map[string]Container `json:"containers"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// WrapperReport is the wrapper's OSAPPLY-REPORT object.
|
||||||
|
type WrapperReport struct {
|
||||||
|
Mode string `json:"mode"`
|
||||||
|
Refused json.RawMessage `json:"refused"`
|
||||||
|
Failed json.RawMessage `json:"failed"`
|
||||||
|
Upgraded []Package `json:"upgraded"`
|
||||||
|
Installed []Package `json:"installed"`
|
||||||
|
Pending []Pending `json:"pending"`
|
||||||
|
RestartNeeded []string `json:"restart_needed"`
|
||||||
|
DockerRestartNeeded bool `json:"docker_restart_needed"`
|
||||||
|
RebootNeeded bool `json:"reboot_needed"`
|
||||||
|
HealthBefore *Health `json:"health_before"`
|
||||||
|
HealthAfter *Health `json:"health_after"`
|
||||||
|
Health *Health `json:"health"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (w WrapperReport) refused() bool { return len(w.Refused) > 0 && string(w.Refused) != "null" }
|
||||||
|
func (w WrapperReport) failed() bool { return len(w.Failed) > 0 && string(w.Failed) != "null" }
|
||||||
|
|
||||||
|
// Report is what the hub receives (hub osupdates.Report — field-exact).
|
||||||
|
type Report struct {
|
||||||
|
RunID string `json:"run_id"`
|
||||||
|
Trigger string `json:"trigger"`
|
||||||
|
Mode string `json:"mode"`
|
||||||
|
Ring int `json:"ring"`
|
||||||
|
ReleaseID string `json:"release_id"`
|
||||||
|
Outcome string `json:"outcome"`
|
||||||
|
Healthy bool `json:"healthy"`
|
||||||
|
HealthReason string `json:"health_reason,omitempty"`
|
||||||
|
VMID int `json:"vmid"`
|
||||||
|
Upgraded []Package `json:"upgraded,omitempty"`
|
||||||
|
Installed []Package `json:"installed,omitempty"`
|
||||||
|
Pending []Pending `json:"pending,omitempty"`
|
||||||
|
NotCovered []string `json:"not_covered,omitempty"`
|
||||||
|
RestartNeeded []string `json:"restart_needed,omitempty"`
|
||||||
|
DockerRestartNeeded bool `json:"docker_restart_needed,omitempty"`
|
||||||
|
RebootNeeded bool `json:"reboot_needed,omitempty"`
|
||||||
|
Refused json.RawMessage `json:"refused,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Reporter posts a report to the hub (*hub.Client).
|
||||||
|
type Reporter interface {
|
||||||
|
PostOSReport(ctx context.Context, body []byte) error
|
||||||
|
}
|
||||||
|
|
||||||
|
// Leg runs one OS-update pass for the customer guest.
|
||||||
|
type Leg struct {
|
||||||
|
Runner proxmox.Runner
|
||||||
|
Hub Reporter
|
||||||
|
Logger *slog.Logger
|
||||||
|
PlanDir string
|
||||||
|
StatePath string // last night run (once per night)
|
||||||
|
HealthWait time.Duration // how long health may take to come back (default 5 min)
|
||||||
|
HealthPoll time.Duration // default 15 s
|
||||||
|
MinGap time.Duration // between night runs (default 20 h)
|
||||||
|
Now func() time.Time
|
||||||
|
Sleep func(context.Context, time.Duration)
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
block *hub.WireOSUpdate
|
||||||
|
have bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// OnDesiredState stores the hub's os_update block (desired.RawConsumer — store only, never block).
|
||||||
|
func (l *Leg) OnDesiredState(_ context.Context, resp *hub.DesiredStateResponse) {
|
||||||
|
if resp == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
l.block, l.have = resp.DesiredState.OSUpdate, true
|
||||||
|
}
|
||||||
|
|
||||||
|
// Block returns the newest os_update block. No block (an older hub, or nothing fetched yet) = ring 1, ON, no
|
||||||
|
// release: the box reports and installs nothing.
|
||||||
|
func (l *Leg) Block() hub.WireOSUpdate {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
if l.block == nil {
|
||||||
|
return hub.WireOSUpdate{Ring: 1, Enabled: true}
|
||||||
|
}
|
||||||
|
return *l.block
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetBlock sets the block directly (the selftest fetches the desired state itself).
|
||||||
|
func (l *Leg) SetBlock(b *hub.WireOSUpdate) {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
l.block, l.have = b, true
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *Leg) now() time.Time {
|
||||||
|
if l.Now != nil {
|
||||||
|
return l.Now()
|
||||||
|
}
|
||||||
|
return time.Now()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *Leg) log() *slog.Logger {
|
||||||
|
if l.Logger != nil {
|
||||||
|
return l.Logger
|
||||||
|
}
|
||||||
|
return slog.Default()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *Leg) sleep(ctx context.Context, d time.Duration) {
|
||||||
|
if l.Sleep != nil {
|
||||||
|
l.Sleep(ctx, d)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
case <-time.After(d):
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// IsFast reports whether every origin apt names is Debian / Debian-Security (the fast lane, `11` C3).
|
||||||
|
func IsFast(origins []string) bool {
|
||||||
|
if len(origins) == 0 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
for _, o := range origins {
|
||||||
|
if o != "Debian" && o != "Debian-Security" {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
func fastOrigin(origins []string) string {
|
||||||
|
for _, o := range origins {
|
||||||
|
if o == "Debian-Security" {
|
||||||
|
return o
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return "Debian"
|
||||||
|
}
|
||||||
|
|
||||||
|
// HealthVerdict is THE health rule (`11` §5.4.1; pinned by TestHealthVerdict_*): after the run, docker answers, the
|
||||||
|
// guest's network resolves, the controller's own health check is `healthy`, and every container that was running
|
||||||
|
// before is running again — and healthy again if it was healthy before. "starting" is not yet healthy.
|
||||||
|
func HealthVerdict(before, after *Health) (bool, string) {
|
||||||
|
if after == nil {
|
||||||
|
return false, "no health reading"
|
||||||
|
}
|
||||||
|
if !after.DockerOK {
|
||||||
|
return false, "docker does not answer"
|
||||||
|
}
|
||||||
|
if !after.NetworkOK {
|
||||||
|
return false, "the guest cannot resolve deb.debian.org"
|
||||||
|
}
|
||||||
|
if after.Controller != "healthy" {
|
||||||
|
return false, "the controller is " + after.Controller
|
||||||
|
}
|
||||||
|
if before == nil {
|
||||||
|
return true, ""
|
||||||
|
}
|
||||||
|
names := make([]string, 0, len(before.Containers))
|
||||||
|
for n := range before.Containers {
|
||||||
|
names = append(names, n)
|
||||||
|
}
|
||||||
|
sort.Strings(names)
|
||||||
|
for _, n := range names {
|
||||||
|
b := before.Containers[n]
|
||||||
|
if b.State != "running" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
a, ok := after.Containers[n]
|
||||||
|
if !ok || a.State != "running" {
|
||||||
|
return false, n + " was running and is not"
|
||||||
|
}
|
||||||
|
if b.Health == "healthy" && a.Health != "healthy" {
|
||||||
|
return false, n + " was healthy and is " + a.Health
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true, ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// MergeBaseline is the health baseline from two readings: a container counts as running (and healthy) if EITHER
|
||||||
|
// reading saw it so. nil-safe. Pinned by TestHealth_BaselineIsTheStartOfTheLeg.
|
||||||
|
func MergeBaseline(a, b *Health) *Health {
|
||||||
|
if a == nil {
|
||||||
|
return b
|
||||||
|
}
|
||||||
|
if b == nil {
|
||||||
|
return a
|
||||||
|
}
|
||||||
|
out := &Health{DockerOK: a.DockerOK && b.DockerOK, NetworkOK: a.NetworkOK && b.NetworkOK, Controller: b.Controller,
|
||||||
|
Containers: map[string]Container{}}
|
||||||
|
for _, h := range []*Health{a, b} {
|
||||||
|
for n, c := range h.Containers {
|
||||||
|
cur, seen := out.Containers[n]
|
||||||
|
if !seen || (cur.State != "running" && c.State == "running") {
|
||||||
|
out.Containers[n] = c
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if c.State == "running" && c.Health == "healthy" {
|
||||||
|
out.Containers[n] = c
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// call writes the plan and runs the wrapper once.
|
||||||
|
func (l *Leg) call(ctx context.Context, runID, mode string, vmid int, rel hub.WireOSRelease, pkgs []Package) (WrapperReport, error) {
|
||||||
|
dir := l.PlanDir
|
||||||
|
if dir == "" {
|
||||||
|
dir = DefaultPlanDir
|
||||||
|
}
|
||||||
|
if err := os.MkdirAll(dir, 0o700); err != nil {
|
||||||
|
return WrapperReport{}, fmt.Errorf("osupdate: plan dir: %w", err)
|
||||||
|
}
|
||||||
|
plan := map[string]any{"release_id": rel.ID, "layer": "guest", "lane": "fast", "vmid": vmid, "mode": mode,
|
||||||
|
"snapshot": rel.Snapshot, "packages": pkgs}
|
||||||
|
if pkgs == nil {
|
||||||
|
plan["packages"] = []Package{}
|
||||||
|
}
|
||||||
|
b, _ := json.Marshal(plan)
|
||||||
|
path := filepath.Join(dir, "plan-"+runID+"-"+mode+".json")
|
||||||
|
if err := os.WriteFile(path, b, 0o600); err != nil {
|
||||||
|
return WrapperReport{}, fmt.Errorf("osupdate: write plan: %w", err)
|
||||||
|
}
|
||||||
|
defer os.Remove(path)
|
||||||
|
stdout, stderr, err := l.Runner.Run(ctx, WrapperPath, "--plan", path)
|
||||||
|
for _, line := range strings.Split(strings.TrimSpace(string(stderr)), "\n") {
|
||||||
|
if strings.HasPrefix(line, "os-apply: ") {
|
||||||
|
l.log().Info("osupdate: wrapper", "line", line)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
var rep WrapperReport
|
||||||
|
found := false
|
||||||
|
for _, line := range strings.Split(string(stdout), "\n") {
|
||||||
|
if strings.HasPrefix(line, "OSAPPLY-REPORT ") {
|
||||||
|
if jerr := json.Unmarshal([]byte(strings.TrimPrefix(line, "OSAPPLY-REPORT ")), &rep); jerr == nil {
|
||||||
|
found = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
return rep, fmt.Errorf("osupdate: wrapper gave no report (err %v): %s", err, strings.TrimSpace(string(stderr)))
|
||||||
|
}
|
||||||
|
return rep, nil // a refusal / failure is IN the report (exit 2 / 3), not an error here
|
||||||
|
}
|
||||||
|
|
||||||
|
// Run is one pass: inventory → (switch, ring, plan) → apply → health → report. trigger is "night" or "debug".
|
||||||
|
func (l *Leg) Run(ctx context.Context, vmid int, trigger string) Report {
|
||||||
|
runID := l.now().UTC().Format("20060102T150405Z")
|
||||||
|
blk := l.Block()
|
||||||
|
rel := hub.WireOSRelease{ID: "ring0-" + runID}
|
||||||
|
if blk.Ring == 1 {
|
||||||
|
rel = hub.WireOSRelease{}
|
||||||
|
if blk.Release != nil {
|
||||||
|
rel = *blk.Release
|
||||||
|
}
|
||||||
|
}
|
||||||
|
rep := Report{RunID: runID, Trigger: trigger, Ring: blk.Ring, VMID: vmid, ReleaseID: rel.ID, Mode: "inventory"}
|
||||||
|
lg := l.log().With("run", runID, "vmid", vmid, "ring", blk.Ring, "trigger", trigger)
|
||||||
|
|
||||||
|
if trigger == "night" && l.StatePath != "" {
|
||||||
|
gap := l.MinGap
|
||||||
|
if gap == 0 {
|
||||||
|
gap = 20 * time.Hour
|
||||||
|
}
|
||||||
|
if b, err := os.ReadFile(l.StatePath); err == nil {
|
||||||
|
if last, perr := time.Parse(time.RFC3339, strings.TrimSpace(string(b))); perr == nil && l.now().Sub(last) < gap {
|
||||||
|
lg.Info("osupdate: skipped — already ran tonight", "last", last.UTC().Format(time.RFC3339))
|
||||||
|
rep.Outcome = "skipped"
|
||||||
|
return rep
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
lg.Info("osupdate: START", "enabled", blk.Enabled, "release", rel.ID)
|
||||||
|
|
||||||
|
inv, err := l.call(ctx, runID, "inventory", vmid, rel, nil)
|
||||||
|
if err != nil || inv.refused() || inv.failed() {
|
||||||
|
rep.Outcome, rep.Refused = "refused", inv.Refused
|
||||||
|
if err != nil {
|
||||||
|
rep.Outcome, rep.HealthReason = "failed", err.Error()
|
||||||
|
}
|
||||||
|
return l.finish(ctx, lg, rep)
|
||||||
|
}
|
||||||
|
var plan []Package
|
||||||
|
switch {
|
||||||
|
case !blk.Enabled:
|
||||||
|
rep.Outcome = "inventory"
|
||||||
|
lg.Info("osupdate: switched OFF for this box — reporting only")
|
||||||
|
case blk.Ring == 0:
|
||||||
|
for _, p := range inv.Pending {
|
||||||
|
if IsFast(p.Origin) {
|
||||||
|
plan = append(plan, Package{Name: p.Name, Version: p.To, Origin: fastOrigin(p.Origin)})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
for _, p := range rel.Packages {
|
||||||
|
plan = append(plan, Package{Name: p.Name, Version: p.Version, Origin: p.Origin})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
pkgNames := map[string]bool{}
|
||||||
|
for _, p := range plan {
|
||||||
|
pkgNames[p.Name] = true
|
||||||
|
}
|
||||||
|
if blk.Enabled && len(plan) == 0 {
|
||||||
|
rep.Outcome = "nothing"
|
||||||
|
}
|
||||||
|
final := inv
|
||||||
|
if blk.Enabled && len(plan) > 0 {
|
||||||
|
rep.Mode = "apply"
|
||||||
|
ap, err := l.call(ctx, runID, "apply", vmid, rel, plan)
|
||||||
|
switch {
|
||||||
|
case err != nil:
|
||||||
|
rep.Outcome, rep.HealthReason = "failed", err.Error()
|
||||||
|
return l.finish(ctx, lg, rep)
|
||||||
|
case ap.refused():
|
||||||
|
rep.Outcome, rep.Refused = "refused", ap.Refused
|
||||||
|
return l.finish(ctx, lg, rep)
|
||||||
|
case ap.failed():
|
||||||
|
rep.Outcome, rep.Refused = "failed", ap.Failed
|
||||||
|
}
|
||||||
|
final = ap
|
||||||
|
rep.Upgraded = ap.Upgraded
|
||||||
|
if rep.Outcome == "" {
|
||||||
|
if len(ap.Upgraded) == 0 {
|
||||||
|
rep.Outcome = "nothing"
|
||||||
|
} else {
|
||||||
|
rep.Outcome = "applied"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Health: compare with what the guest looked like BEFORE the run; give restarted services time.
|
||||||
|
wait, poll := l.HealthWait, l.HealthPoll
|
||||||
|
if wait == 0 {
|
||||||
|
wait = 5 * time.Minute
|
||||||
|
}
|
||||||
|
if poll == 0 {
|
||||||
|
poll = 15 * time.Second
|
||||||
|
}
|
||||||
|
deadline := l.now().Add(wait)
|
||||||
|
cur := ap.HealthAfter
|
||||||
|
// The baseline is the guest as it was at the START of the leg (the inventory's reading) merged with the
|
||||||
|
// apply's own "before": an app that stops at any point during the run counts. Found live 2026-10-04: an app
|
||||||
|
// stopped between the inventory and the apply's own reading was taken as "stopped before" and ignored.
|
||||||
|
base := MergeBaseline(inv.HealthAfter, ap.HealthBefore)
|
||||||
|
for {
|
||||||
|
ok, why := HealthVerdict(base, cur)
|
||||||
|
rep.Healthy, rep.HealthReason = ok, why
|
||||||
|
if ok || !l.now().Before(deadline) || ctx.Err() != nil {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
l.sleep(ctx, poll)
|
||||||
|
hr, herr := l.call(ctx, runID, "health", vmid, rel, nil)
|
||||||
|
if herr == nil && hr.Health != nil {
|
||||||
|
cur = hr.Health
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !rep.Healthy && rep.Outcome == "applied" {
|
||||||
|
rep.Outcome = "health_failed"
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
ok, why := HealthVerdict(nil, inv.HealthAfter)
|
||||||
|
rep.Healthy, rep.HealthReason = ok, why
|
||||||
|
}
|
||||||
|
rep.Installed, rep.Pending = final.Installed, final.Pending
|
||||||
|
rep.RestartNeeded, rep.DockerRestartNeeded, rep.RebootNeeded = final.RestartNeeded, final.DockerRestartNeeded, final.RebootNeeded
|
||||||
|
rep.NotCovered = notCovered(final.Pending, blk.Ring, pkgNames)
|
||||||
|
if trigger == "night" && l.StatePath != "" {
|
||||||
|
_ = os.WriteFile(l.StatePath, []byte(l.now().UTC().Format(time.RFC3339)), 0o600)
|
||||||
|
}
|
||||||
|
return l.finish(ctx, lg, rep)
|
||||||
|
}
|
||||||
|
|
||||||
|
// notCovered lists pending updates no approved release covers: in ring 0 everything outside the fast lane; in
|
||||||
|
// ring 1 also every fast-lane update the release did not name.
|
||||||
|
func notCovered(pending []Pending, ring int, planned map[string]bool) []string {
|
||||||
|
var out []string
|
||||||
|
for _, p := range pending {
|
||||||
|
if !IsFast(p.Origin) || (ring == 1 && !planned[p.Name]) {
|
||||||
|
out = append(out, p.Name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *Leg) finish(ctx context.Context, lg *slog.Logger, rep Report) Report {
|
||||||
|
lg.Info("osupdate: DONE", "outcome", rep.Outcome, "healthy", rep.Healthy, "reason", rep.HealthReason,
|
||||||
|
"upgraded", len(rep.Upgraded), "pending", len(rep.Pending), "not_covered", len(rep.NotCovered), "restart_needed", len(rep.RestartNeeded))
|
||||||
|
if l.Hub != nil {
|
||||||
|
body, _ := json.Marshal(rep)
|
||||||
|
rctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), time.Minute)
|
||||||
|
defer cancel()
|
||||||
|
if err := l.Hub.PostOSReport(rctx, body); err != nil {
|
||||||
|
lg.Warn("osupdate: reporting to the hub failed (the run itself is done)", "err", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return rep
|
||||||
|
}
|
||||||
@@ -0,0 +1,281 @@
|
|||||||
|
package osupdate
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
|
)
|
||||||
|
|
||||||
|
// fakeWrapper plays /usr/local/sbin/felhom-os-apply: it reads the plan the leg wrote and answers per mode.
|
||||||
|
type fakeWrapper struct {
|
||||||
|
t *testing.T
|
||||||
|
pending []Pending
|
||||||
|
applyRep WrapperReport
|
||||||
|
healthSeq []*Health // answers to successive "health" calls
|
||||||
|
plans []map[string]any
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeWrapper) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||||
|
if name != WrapperPath || len(args) != 2 || args[0] != "--plan" {
|
||||||
|
f.t.Fatalf("unexpected command %s %v", name, args)
|
||||||
|
}
|
||||||
|
b, err := os.ReadFile(args[1])
|
||||||
|
if err != nil {
|
||||||
|
f.t.Fatal(err)
|
||||||
|
}
|
||||||
|
var plan map[string]any
|
||||||
|
json.Unmarshal(b, &plan)
|
||||||
|
f.plans = append(f.plans, plan)
|
||||||
|
var rep WrapperReport
|
||||||
|
healthy := &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{
|
||||||
|
"felhom-controller": {State: "running", Health: "healthy"}, "app": {State: "running", Health: "healthy"}}}
|
||||||
|
switch plan["mode"] {
|
||||||
|
case "inventory":
|
||||||
|
rep = WrapperReport{Mode: "inventory", Pending: f.pending, HealthAfter: healthy,
|
||||||
|
Installed: []Package{{Name: "libc6", Version: "u3", Origin: "Debian"}}}
|
||||||
|
case "apply":
|
||||||
|
rep = f.applyRep
|
||||||
|
rep.Mode = "apply"
|
||||||
|
if rep.HealthBefore == nil {
|
||||||
|
rep.HealthBefore = healthy
|
||||||
|
}
|
||||||
|
if rep.HealthAfter == nil {
|
||||||
|
rep.HealthAfter = healthy
|
||||||
|
}
|
||||||
|
case "health":
|
||||||
|
if len(f.healthSeq) > 0 {
|
||||||
|
rep.Health, f.healthSeq = f.healthSeq[0], f.healthSeq[1:]
|
||||||
|
} else {
|
||||||
|
rep.Health = healthy
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out, _ := json.Marshal(rep)
|
||||||
|
return []byte("OSAPPLY-REPORT " + string(out) + "\n"), []byte("os-apply: DONE rc=0\n"), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeWrapper) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||||
|
return f.Run(ctx, name, args...)
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeHub struct{ reports []Report }
|
||||||
|
|
||||||
|
func (h *fakeHub) PostOSReport(_ context.Context, body []byte) error {
|
||||||
|
var r Report
|
||||||
|
json.Unmarshal(body, &r)
|
||||||
|
h.reports = append(h.reports, r)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func newLeg(t *testing.T, w *fakeWrapper, blk *hub.WireOSUpdate) (*Leg, *fakeHub) {
|
||||||
|
h := &fakeHub{}
|
||||||
|
now := time.Date(2026, 10, 4, 4, 0, 0, 0, time.UTC)
|
||||||
|
l := &Leg{Runner: w, Hub: h, PlanDir: t.TempDir(), StatePath: filepath.Join(t.TempDir(), "last"),
|
||||||
|
HealthWait: time.Minute, HealthPoll: 10 * time.Second,
|
||||||
|
Now: func() time.Time { return now },
|
||||||
|
Sleep: func(_ context.Context, d time.Duration) { now = now.Add(d) }}
|
||||||
|
if blk != nil {
|
||||||
|
l.SetBlock(blk)
|
||||||
|
}
|
||||||
|
return l, h
|
||||||
|
}
|
||||||
|
|
||||||
|
var pend = []Pending{
|
||||||
|
{Name: "libc6", From: "u3", To: "u4", Origin: []string{"Debian"}},
|
||||||
|
{Name: "openssl", From: "u1", To: "u3", Origin: []string{"Debian-Security", "Debian"}},
|
||||||
|
{Name: "docker-ce", From: "29.7", To: "29.8", Origin: []string{"Docker CE"}},
|
||||||
|
}
|
||||||
|
|
||||||
|
func modes(w *fakeWrapper) string {
|
||||||
|
var m []string
|
||||||
|
for _, p := range w.plans {
|
||||||
|
m = append(m, p["mode"].(string))
|
||||||
|
}
|
||||||
|
return strings.Join(m, ",")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ring 0 plans every pending FAST-LANE update (Docker excluded, `11` C3) and reports what it now runs.
|
||||||
|
func TestRing0_PlansTheFastLaneOnly(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, pending: pend, applyRep: WrapperReport{Upgraded: []Package{{Name: "libc6", Version: "u4"}, {Name: "openssl", Version: "u3"}}, Pending: pend[2:]}}
|
||||||
|
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||||
|
rep := l.Run(context.Background(), 9201, "night")
|
||||||
|
if rep.Outcome != "applied" || !rep.Healthy {
|
||||||
|
t.Fatalf("rep = %+v", rep)
|
||||||
|
}
|
||||||
|
pk := w.plans[1]["packages"].([]any)
|
||||||
|
if len(pk) != 2 {
|
||||||
|
t.Fatalf("ring-0 plan = %v, want libc6 + openssl only", pk)
|
||||||
|
}
|
||||||
|
if o := pk[1].(map[string]any)["origin"]; o != "Debian-Security" {
|
||||||
|
t.Fatalf("openssl origin = %v", o)
|
||||||
|
}
|
||||||
|
if w.plans[1]["snapshot"] != "" {
|
||||||
|
t.Fatalf("ring 0 installs from live sources, snapshot = %v", w.plans[1]["snapshot"])
|
||||||
|
}
|
||||||
|
if len(rep.NotCovered) != 1 || rep.NotCovered[0] != "docker-ce" {
|
||||||
|
t.Fatalf("not covered = %v", rep.NotCovered)
|
||||||
|
}
|
||||||
|
if len(h.reports) != 1 || h.reports[0].Outcome != "applied" {
|
||||||
|
t.Fatalf("hub got %+v", h.reports)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ring 1 installs EXACTLY the approved release (its versions, its snapshot), nothing it computed itself.
|
||||||
|
func TestRing1_InstallsExactlyTheRelease(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, pending: pend, applyRep: WrapperReport{Upgraded: []Package{{Name: "libc6", Version: "u4-approved"}}, Pending: pend[1:]}}
|
||||||
|
rel := &hub.WireOSRelease{ID: "os-1", Snapshot: "20261004T080000Z", Packages: []hub.WireOSPackage{{Name: "libc6", Version: "u4-approved", Origin: "Debian"}}}
|
||||||
|
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true, Release: rel})
|
||||||
|
rep := l.Run(context.Background(), 9201, "night")
|
||||||
|
if rep.Outcome != "applied" || rep.ReleaseID != "os-1" {
|
||||||
|
t.Fatalf("rep = %+v", rep)
|
||||||
|
}
|
||||||
|
ap := w.plans[1]
|
||||||
|
pk := ap["packages"].([]any)
|
||||||
|
if len(pk) != 1 || pk[0].(map[string]any)["version"] != "u4-approved" || ap["snapshot"] != "20261004T080000Z" || ap["release_id"] != "os-1" {
|
||||||
|
t.Fatalf("ring-1 plan = %v", ap)
|
||||||
|
}
|
||||||
|
// openssl is pending and fast-lane but NOT in the release → not covered; docker-ce is never covered.
|
||||||
|
if strings.Join(rep.NotCovered, ",") != "openssl,docker-ce" {
|
||||||
|
t.Fatalf("not covered = %v", rep.NotCovered)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRing1_NoReleaseInstallsNothing(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, pending: pend}
|
||||||
|
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||||
|
rep := l.Run(context.Background(), 9201, "night")
|
||||||
|
if rep.Outcome != "nothing" || modes(w) != "inventory" {
|
||||||
|
t.Fatalf("rep=%+v modes=%s", rep, modes(w))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// No block from the hub (an older hub): ring 1, ON, no release → reports, installs nothing.
|
||||||
|
func TestNoBlock_IsRing1Nothing(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, pending: pend}
|
||||||
|
l, _ := newLeg(t, w, nil)
|
||||||
|
if rep := l.Run(context.Background(), 9201, "night"); rep.Outcome != "nothing" || rep.Ring != 1 || modes(w) != "inventory" {
|
||||||
|
t.Fatalf("rep=%+v modes=%s", rep, modes(w))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Switched OFF: the box reports but installs nothing (the brief, Part D 2).
|
||||||
|
func TestSwitchOff_ReportsOnly(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, pending: pend}
|
||||||
|
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: false})
|
||||||
|
rep := l.Run(context.Background(), 9201, "night")
|
||||||
|
if rep.Outcome != "inventory" || modes(w) != "inventory" || len(h.reports) != 1 || len(rep.Pending) != 3 {
|
||||||
|
t.Fatalf("rep=%+v modes=%s", rep, modes(w))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Unhealthy after the run, and still unhealthy at the end of the wait → health_failed (the hub mails the operator).
|
||||||
|
func TestHealth_FailsAfterTheWait(t *testing.T) {
|
||||||
|
bad := &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{
|
||||||
|
"felhom-controller": {State: "running", Health: "healthy"}, "app": {State: "exited"}}}
|
||||||
|
w := &fakeWrapper{t: t, pending: pend, applyRep: WrapperReport{Upgraded: []Package{{Name: "libc6"}}, HealthAfter: bad},
|
||||||
|
healthSeq: []*Health{bad, bad, bad, bad, bad, bad, bad, bad}}
|
||||||
|
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||||
|
rep := l.Run(context.Background(), 9201, "night")
|
||||||
|
if rep.Outcome != "health_failed" || rep.Healthy || !strings.Contains(rep.HealthReason, "app was running") {
|
||||||
|
t.Fatalf("rep = %+v", rep)
|
||||||
|
}
|
||||||
|
if h.reports[0].Outcome != "health_failed" {
|
||||||
|
t.Fatal("the hub was not told")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A service that takes a moment to come back is not a failure: the poll sees it recover inside the wait.
|
||||||
|
func TestHealth_RecoversInsideTheWait(t *testing.T) {
|
||||||
|
starting := &Health{DockerOK: true, NetworkOK: true, Controller: "starting"}
|
||||||
|
w := &fakeWrapper{t: t, pending: pend, applyRep: WrapperReport{Upgraded: []Package{{Name: "libc6"}}, HealthAfter: starting},
|
||||||
|
healthSeq: []*Health{starting}}
|
||||||
|
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||||
|
if rep := l.Run(context.Background(), 9201, "night"); rep.Outcome != "applied" || !rep.Healthy {
|
||||||
|
t.Fatalf("rep = %+v", rep)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHealthVerdict(t *testing.T) {
|
||||||
|
ok := &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{"a": {State: "running", Health: "healthy"}}}
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
before *Health
|
||||||
|
after *Health
|
||||||
|
want bool
|
||||||
|
}{
|
||||||
|
{"all good", ok, ok, true},
|
||||||
|
{"no reading", ok, nil, false},
|
||||||
|
{"docker down", ok, &Health{NetworkOK: true, Controller: "healthy"}, false},
|
||||||
|
{"no network", ok, &Health{DockerOK: true, Controller: "healthy"}, false},
|
||||||
|
{"controller starting", ok, &Health{DockerOK: true, NetworkOK: true, Controller: "starting"}, false},
|
||||||
|
{"only the controller differs", ok, &Health{DockerOK: true, NetworkOK: true, Controller: "unhealthy", Containers: map[string]Container{"a": {State: "running", Health: "healthy"}}}, false},
|
||||||
|
{"app gone", ok, &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{}}, false},
|
||||||
|
{"app unhealthy", ok, &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{"a": {State: "running", Health: "unhealthy"}}}, false},
|
||||||
|
{"stopped before stays stopped", &Health{Containers: map[string]Container{"x": {State: "exited"}}}, &Health{DockerOK: true, NetworkOK: true, Controller: "healthy"}, true},
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
if got, why := HealthVerdict(c.before, c.after); got != c.want {
|
||||||
|
t.Errorf("%s: got %v (%s), want %v", c.name, got, why, c.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOncePerNight(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, pending: pend, applyRep: WrapperReport{Upgraded: []Package{{Name: "libc6"}}}}
|
||||||
|
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||||
|
l.Run(context.Background(), 9201, "night")
|
||||||
|
n := len(w.plans)
|
||||||
|
if rep := l.Run(context.Background(), 9201, "night"); rep.Outcome != "skipped" || len(w.plans) != n {
|
||||||
|
t.Fatalf("a second night run in the same night ran: %+v", rep)
|
||||||
|
}
|
||||||
|
if rep := l.Run(context.Background(), 9201, "debug"); rep.Outcome == "skipped" {
|
||||||
|
t.Fatal("the debug action must not be throttled")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRefusedIsReported(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, pending: pend, applyRep: WrapperReport{Refused: json.RawMessage(`{"code":"R6","reason":"x"}`)}}
|
||||||
|
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||||
|
if rep := l.Run(context.Background(), 9201, "night"); rep.Outcome != "refused" || h.reports[0].Outcome != "refused" {
|
||||||
|
t.Fatalf("rep = %+v", rep)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The root wrapper's own suite (configs/test_felhom_os_apply.py) runs with `go test ./...` so CI covers it.
|
||||||
|
func TestWrapperSuite(t *testing.T) {
|
||||||
|
py, err := exec.LookPath("python3")
|
||||||
|
if err != nil {
|
||||||
|
t.Skip("python3 not available")
|
||||||
|
}
|
||||||
|
cmd := exec.Command(py, "../../configs/test_felhom_os_apply.py")
|
||||||
|
out, err := cmd.CombinedOutput()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("wrapper suite failed: %v\n%s", err, out)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(out), "OK") {
|
||||||
|
t.Fatalf("wrapper suite did not report OK:\n%s", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// An app that stops BETWEEN the start of the leg and the apply's own "before" reading still fails the run.
|
||||||
|
// Measured live 2026-10-04 on demo-hp (privatebin stopped 1 s after the apply plan was written: the old rule
|
||||||
|
// passed). Red-proof: use ap.HealthBefore alone as the baseline and this fails.
|
||||||
|
func TestHealth_BaselineIsTheStartOfTheLeg(t *testing.T) {
|
||||||
|
stoppedEarly := &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{
|
||||||
|
"felhom-controller": {State: "running", Health: "healthy"}, "app": {State: "exited"}}}
|
||||||
|
w := &fakeWrapper{t: t, pending: pend, applyRep: WrapperReport{Upgraded: []Package{{Name: "libc6"}},
|
||||||
|
HealthBefore: stoppedEarly, HealthAfter: stoppedEarly},
|
||||||
|
healthSeq: []*Health{stoppedEarly, stoppedEarly, stoppedEarly, stoppedEarly, stoppedEarly, stoppedEarly, stoppedEarly}}
|
||||||
|
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||||
|
rep := l.Run(context.Background(), 9201, "night")
|
||||||
|
if rep.Outcome != "health_failed" || !strings.Contains(rep.HealthReason, "app was running") {
|
||||||
|
t.Fatalf("an app that stopped during the run passed: %+v", rep)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -219,15 +219,18 @@ type Storage struct {
|
|||||||
UsedFraction float64 `json:"used_fraction,omitempty"`
|
UsedFraction float64 `json:"used_fraction,omitempty"`
|
||||||
|
|
||||||
// Type-specific config (durable_id sources).
|
// Type-specific config (durable_id sources).
|
||||||
Server string `json:"server,omitempty"` // nfs/cifs/pbs server host
|
Server string `json:"server,omitempty"` // nfs/cifs/pbs server host
|
||||||
Export string `json:"export,omitempty"` // nfs export path
|
// EncryptionKey is the storage's own client-side key FINGERPRINT (pbs; the key itself stays in
|
||||||
Share string `json:"share,omitempty"` // cifs share name
|
// /etc/pve/priv). R-727: an archive encrypted with any other key was written by another box.
|
||||||
Datastore string `json:"datastore,omitempty"` // pbs datastore name
|
EncryptionKey string `json:"encryption-key,omitempty"`
|
||||||
Fingerprint string `json:"fingerprint,omitempty"` // pbs server cert fingerprint
|
Export string `json:"export,omitempty"` // nfs export path
|
||||||
Username string `json:"username,omitempty"` // pbs auth id, e.g. "felhom@pbs!n100"
|
Share string `json:"share,omitempty"` // cifs share name
|
||||||
Namespace string `json:"namespace,omitempty"` // pbs namespace ("" = root; per-customer tenancy = S4)
|
Datastore string `json:"datastore,omitempty"` // pbs datastore name
|
||||||
VGName string `json:"vgname,omitempty"` // lvm/lvmthin volume group
|
Fingerprint string `json:"fingerprint,omitempty"` // pbs server cert fingerprint
|
||||||
ThinPool string `json:"thinpool,omitempty"` // lvmthin pool LV name
|
Username string `json:"username,omitempty"` // pbs auth id, e.g. "felhom@pbs!n100"
|
||||||
|
Namespace string `json:"namespace,omitempty"` // pbs namespace ("" = root; per-customer tenancy = S4)
|
||||||
|
VGName string `json:"vgname,omitempty"` // lvm/lvmthin volume group
|
||||||
|
ThinPool string `json:"thinpool,omitempty"` // lvmthin pool LV name
|
||||||
}
|
}
|
||||||
|
|
||||||
// StorageContent is one entry of GET /nodes/{node}/storage/{store}/content
|
// StorageContent is one entry of GET /nodes/{node}/storage/{store}/content
|
||||||
@@ -239,4 +242,7 @@ type StorageContent struct {
|
|||||||
Size int64 `json:"size"`
|
Size int64 `json:"size"`
|
||||||
CTime int64 `json:"ctime"`
|
CTime int64 `json:"ctime"`
|
||||||
VMID int `json:"vmid,omitempty"`
|
VMID int `json:"vmid,omitempty"`
|
||||||
|
// Encrypted is the fingerprint of the key a PBS archive was encrypted with ("" = not encrypted). R-727:
|
||||||
|
// the restore test reads it to tell THIS box's archives from an earlier box's in the same namespace.
|
||||||
|
Encrypted string `json:"encrypted,omitempty"`
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -302,6 +302,19 @@ func (e *Engine) runBringUp(ctx context.Context, spec BringUpSpec, res *BringUpR
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// R-834: DR keeps the archive's `onboot: 1`, binds the host's REAL drives (4d) and STARTS the guest
|
||||||
|
// — right on a replaced host, where the original is gone. Beside a LIVE original it would be a
|
||||||
|
// second controller for the same household on the same drives. So DR refuses when this host
|
||||||
|
// still carries the original (the archive's source VMID) or any guest that binds the drives.
|
||||||
|
// A copy beside the original is the restore-test's job (onboot=0, throwaway stand-ins, torn
|
||||||
|
// down) or the runbook's beside-restore. Pinned by TestRunBringUp_DRRefusesBesideALiveOriginal.
|
||||||
|
if spec.Mode == ModeDRGuestLoss {
|
||||||
|
if why := e.liveOriginalBeside(ctx, lxc, spec.Archive); why != "" {
|
||||||
|
res.Err = fmt.Errorf("reconcile: dr bring-up refused: %s — a DR restore beside a live original would run two boxes on the same drives (R-834)", why)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
base := JournalEntry{OpID: e.bringUpOpID(spec.VMID), VMID: spec.VMID, Kind: bringUpKind, Rollback: true}
|
base := JournalEntry{OpID: e.bringUpOpID(spec.VMID), VMID: spec.VMID, Kind: bringUpKind, Rollback: true}
|
||||||
|
|
||||||
// OWN the rollback BEFORE any mutation. From here a crash leaves an in-flight Rollback
|
// OWN the rollback BEFORE any mutation. From here a crash leaves an in-flight Rollback
|
||||||
@@ -734,3 +747,47 @@ func net0MAC(cfg proxmox.GuestConfig) string {
|
|||||||
}
|
}
|
||||||
return ""
|
return ""
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// archiveSourceVMID reads the source guest's VMID from a backup volid: a vzdump file
|
||||||
|
// (`…/vzdump-lxc-<vmid>-<date>.tar.zst`) or a PBS snapshot (`…:backup/ct/<vmid>/<time>`). 0 = unknown.
|
||||||
|
func archiveSourceVMID(archive string) int {
|
||||||
|
if i := strings.Index(archive, "vzdump-lxc-"); i >= 0 {
|
||||||
|
rest := archive[i+len("vzdump-lxc-"):]
|
||||||
|
if j := strings.Index(rest, "-"); j > 0 {
|
||||||
|
if n, err := strconv.Atoi(rest[:j]); err == nil {
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if i := strings.Index(archive, "ct/"); i >= 0 {
|
||||||
|
rest := archive[i+len("ct/"):]
|
||||||
|
if j := strings.Index(rest, "/"); j > 0 {
|
||||||
|
if n, err := strconv.Atoi(rest[:j]); err == nil {
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
// liveOriginalBeside says why a DR bring-up would land beside a live original ("" = it would not):
|
||||||
|
// the archive's source guest still exists here, or a guest binds the drives parent. Fails CLOSED: a
|
||||||
|
// guest whose config cannot be read cannot be ruled out.
|
||||||
|
func (e *Engine) liveOriginalBeside(ctx context.Context, lxc []proxmox.Guest, archive string) string {
|
||||||
|
src := archiveSourceVMID(archive)
|
||||||
|
for _, g := range lxc {
|
||||||
|
if src > 0 && g.VMID == src {
|
||||||
|
return fmt.Sprintf("the archive's source guest %d still exists on this host (status %s)", g.VMID, g.Status)
|
||||||
|
}
|
||||||
|
cfg, err := e.api.GuestConfig(ctx, g.VMID)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Sprintf("guest %d's config could not be read to rule out a live original: %v", g.VMID, err)
|
||||||
|
}
|
||||||
|
for slot, v := range cfg.MountPoints() {
|
||||||
|
if source, _, _ := strings.Cut(v, ","); source == structuralParentDir {
|
||||||
|
return fmt.Sprintf("guest %d binds the household drives (%s %s)", g.VMID, slot, structuralParentDir)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,138 @@
|
|||||||
|
package reconcile
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-834: a whole-guest restore BESIDE a live original must never come up as a second box on the same
|
||||||
|
// drives. The DR route keeps `onboot: 1`, binds the real drives and starts the guest, so it refuses
|
||||||
|
// when the original (or any guest binding the drives) is still on this host; on a replaced host it
|
||||||
|
// proceeds and keeps its binds. Red-proof: make liveOriginalBeside return "" and the refusals pass
|
||||||
|
// the restore through (the "refused" sub-tests fail).
|
||||||
|
func TestRunBringUp_DRRefusesBesideALiveOriginal(t *testing.T) {
|
||||||
|
const target = 9299
|
||||||
|
drivesBind := proxmox.GuestConfig{Extra: map[string]json.RawMessage{
|
||||||
|
"mp8": json.RawMessage(`"/mnt/felhom-drives,mp=/mnt/felhom-drives"`),
|
||||||
|
}}
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
archive string
|
||||||
|
lxc []proxmox.Guest
|
||||||
|
cfg map[int]proxmox.GuestConfig
|
||||||
|
refuse string // substring of the refusal; "" = must proceed
|
||||||
|
}{
|
||||||
|
{"the source guest still exists", "local:backup/vzdump-lxc-9201-2026_10_04-04_34_55.tar.zst",
|
||||||
|
[]proxmox.Guest{{VMID: 9201, Status: "running"}}, map[int]proxmox.GuestConfig{9201: scratchCfg()}, "source guest 9201"},
|
||||||
|
{"another guest binds the drives (PBS archive)", "felhom-pbs:backup/ct/9201/2026-10-04T02:34:55Z",
|
||||||
|
[]proxmox.Guest{{VMID: 9300, Status: "stopped"}}, map[int]proxmox.GuestConfig{9300: drivesBind}, "binds the household drives"},
|
||||||
|
{"a guest whose config cannot be read", "local:backup/vzdump-lxc-9201-x.tar.zst",
|
||||||
|
[]proxmox.Guest{{VMID: 9400, Status: "running"}}, map[int]proxmox.GuestConfig{}, "could not be read"},
|
||||||
|
{"replaced host: only an unrelated scratch guest", "local:backup/vzdump-lxc-9201-x.tar.zst",
|
||||||
|
[]proxmox.Guest{{VMID: 9202, Status: "running"}}, map[int]proxmox.GuestConfig{9202: scratchCfg()}, ""},
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
t.Run(c.name, func(t *testing.T) {
|
||||||
|
cfg := c.cfg
|
||||||
|
cfg[target] = scratchCfg()
|
||||||
|
api := &fakeAPI{lxc: c.lxc, cfg: cfg}
|
||||||
|
e, fr, _, q := newDREngine(t, api)
|
||||||
|
defer q.Close()
|
||||||
|
res := e.RunBringUp(context.Background(), BringUpSpec{
|
||||||
|
Mode: ModeDRGuestLoss, Archive: c.archive, VMID: target, RestoreStorage: "local-lvm", KeepMAC: true,
|
||||||
|
})
|
||||||
|
if c.refuse != "" {
|
||||||
|
if res.Err == nil || !strings.Contains(res.Err.Error(), c.refuse) || !strings.Contains(res.Err.Error(), "R-834") {
|
||||||
|
t.Fatalf("want a refusal naming %q, got %+v", c.refuse, res)
|
||||||
|
}
|
||||||
|
if len(api.restores) != 0 || len(api.starts) != 0 || len(fr.cmds) != 0 {
|
||||||
|
t.Fatalf("a refused DR touched the host: restores=%d starts=%v cmds=%v", len(api.restores), api.starts, fr.cmds)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if res.Err != nil || !res.Pass {
|
||||||
|
t.Fatalf("a DR on a replaced host must proceed, got %+v", res)
|
||||||
|
}
|
||||||
|
// … and there it keeps the REAL drives bind (the right binds on a replaced host).
|
||||||
|
joined := strings.Join(fr.cmds, "\n")
|
||||||
|
if !strings.Contains(joined, "-mp8 /mnt/felhom-drives,mp=/mnt/felhom-drives") {
|
||||||
|
t.Fatalf("the DR guest lost its drives bind: %v", fr.cmds)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Provisioning restores the GOLDEN (no drives, onboot set by the back-half on purpose): a drives-
|
||||||
|
// binding guest on the host does not block it — the rule is DR's alone.
|
||||||
|
func TestRunBringUp_ProvisionNotBlockedByADrivesBind(t *testing.T) {
|
||||||
|
api := &fakeAPI{
|
||||||
|
lxc: []proxmox.Guest{{VMID: 9201, Status: "running"}},
|
||||||
|
cfg: map[int]proxmox.GuestConfig{
|
||||||
|
9201: {Extra: map[string]json.RawMessage{"mp8": json.RawMessage(`"/mnt/felhom-drives,mp=/mnt/felhom-drives"`)}},
|
||||||
|
9203: scratchCfg(),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
e, _, q := newEngine(t, api, EmptyProvider{})
|
||||||
|
defer q.Close()
|
||||||
|
res := e.RunBringUp(context.Background(), BringUpSpec{Mode: ModeProvision, Archive: "local:vztmpl/felhom-golden.tar.zst", VMID: 9203, RestoreStorage: "local-lvm"})
|
||||||
|
if res.Err != nil || !res.Pass {
|
||||||
|
t.Fatalf("provision must proceed, got %+v", res)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestArchiveSourceVMID(t *testing.T) {
|
||||||
|
for in, want := range map[string]int{
|
||||||
|
"local:backup/vzdump-lxc-9201-2026_10_04-04_34_55.tar.zst": 9201,
|
||||||
|
"felhom-pbs:backup/ct/9201/2026-10-04T02:34:55Z": 9201,
|
||||||
|
"tmp-dooplex-copy:backup/ct/9201/2026-10-03T19:00:00Z": 9201,
|
||||||
|
"local:vztmpl/felhom-golden.tar.zst": 0,
|
||||||
|
"vol": 0,
|
||||||
|
} {
|
||||||
|
if got := archiveSourceVMID(in); got != want {
|
||||||
|
t.Errorf("archiveSourceVMID(%q) = %d, want %d", in, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// R-834, the restore-test route: its scratch guest sits BESIDE the live original by design, so it must
|
||||||
|
// carry no host-path bind — the archive's mp8 (the household's drives) and mp9 (the original's
|
||||||
|
// bootstrap) are replaced by throwaway volumes AT restore time. Measured live 2026-10-04 on demo-hp
|
||||||
|
// (`audits/backup-close-2026-10-04/partA/`): onboot 0 and no host bind on every poll. Red-proof:
|
||||||
|
// make drRestoreOverrides return the archive's own mp8 value and this fails.
|
||||||
|
func TestRestoreTest_NoHostPathBindBesideTheOriginal(t *testing.T) {
|
||||||
|
api := &fakeAPI{
|
||||||
|
cfg: map[int]proxmox.GuestConfig{990000: scratchCfg()},
|
||||||
|
extractCfg: "hostname: demo-hp\nonboot: 1\nrootfs: local-lvm:vm-9201-disk-0,size=16G\n" +
|
||||||
|
"mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G\n" +
|
||||||
|
"mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives\n" +
|
||||||
|
"mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1\n",
|
||||||
|
}
|
||||||
|
e, _, q := newEngine(t, api, EmptyProvider{})
|
||||||
|
defer q.Close()
|
||||||
|
_ = e.RunRestoreTest(context.Background(), RestoreTestSpec{
|
||||||
|
Archive: "local:backup/vzdump-lxc-9201-x.tar.zst", RestoreStorage: "local-lvm",
|
||||||
|
ScratchMin: 990000, ScratchMax: 990009, SourceTier: "local",
|
||||||
|
})
|
||||||
|
if len(api.restores) != 1 {
|
||||||
|
t.Fatalf("want one restore, got %+v", api.restores)
|
||||||
|
}
|
||||||
|
r := api.restores[0]
|
||||||
|
for _, slot := range []string{"mp8", "mp9"} {
|
||||||
|
v, ok := r.MountOverrides[slot]
|
||||||
|
if !ok || strings.HasPrefix(v, "/") {
|
||||||
|
t.Fatalf("%s = %q (present=%v): the scratch beside the original must get a throwaway volume, never the host path", slot, v, ok)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for slot, v := range r.MountOverrides {
|
||||||
|
if strings.HasPrefix(v, "/") {
|
||||||
|
t.Fatalf("%s carries a host path %q into the scratch guest", slot, v)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if r.ConfigOverrides["onboot"] != "0" {
|
||||||
|
t.Fatalf("onboot = %q, want 0", r.ConfigOverrides["onboot"])
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user