hub v0.131.0: the tunnel status is true (R-841: three states, tunnel_down after two not_running reports, unknown never alarms); OS updates per layer (guest/host separate approved sets, host candidate leaves out kernel/boot/firmware, host_release in the box block), the fleet view and four hourly operator alarms (11 §8 steps 3+4); red-proofs in audits/os-host-lane-2026-10-04
gates / gates (push) Successful in 30s
gates / gates (push) Successful in 30s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -0,0 +1,9 @@
|
||||
# agent v0.141.0 tunnel-probe red-proofs (each mutation compiles; result line printed)
|
||||
== mutation: container-state-only
|
||||
cloudflared_test.go:32: running but not connected: got "running" (container running but the tunnel is NOT connected (cloudflared /ready fails)), want "not_running" (…NOT connected…)
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/hub 0.009s
|
||||
== mutation: unknown-as-down
|
||||
cloudflared_test.go:32: guest not running: got "not_running" (could not ask the guest: CT 9201 not running), want "unknown" (…could not ask…)
|
||||
cloudflared_test.go:32: sudo refused: got "not_running" (could not ask the guest: sudo: a password is required), want "unknown" (…could not ask…)
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/hub 0.009s
|
||||
ok gitea.dooplex.hu/admin/felhom-agent/internal/hub (cached)
|
||||
@@ -0,0 +1,3 @@
|
||||
== RP (controller): the healthcheck block removed from the cloudflared template
|
||||
infra_tunnel_test.go:300: tunnel=false: compose lacks "test: [\"CMD\", \"cloudflared\", \"tunnel\", \"--metrics\", \"localhost:20241\", \"ready\"]":
|
||||
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.007s
|
||||
@@ -0,0 +1,17 @@
|
||||
# hub v0.131.0 tunnel-alarm red-proofs, 2026-10-04 (each mutation compiles; the package result line is printed)
|
||||
== mutation: one-report
|
||||
--- FAIL: TestTunnelAlarm_Sequences/a_single_bad_report_does_not_alarm (0.02s)
|
||||
tunnel_test.go:47: tunnel_down=1 tunnel_recovered=0, want 0 / 0
|
||||
--- FAIL: TestTunnelAlarm_Sequences/unknown_never_alarms_and_breaks_the_run (0.03s)
|
||||
tunnel_test.go:47: tunnel_down=1 tunnel_recovered=0, want 0 / 0
|
||||
== mutation: unknown-counts-as-down
|
||||
--- FAIL: TestTunnelAlarm_Sequences/unknown_never_alarms_and_breaks_the_run (0.03s)
|
||||
tunnel_test.go:47: tunnel_down=1 tunnel_recovered=0, want 0 / 0
|
||||
FAIL gitea.dooplex.hu/admin/felhom-hub/internal/api 0.164s
|
||||
== mutation: no-alarm
|
||||
--- FAIL: TestTunnelAlarm_Sequences/two_in_a_row_alarm_once (0.03s)
|
||||
tunnel_test.go:47: tunnel_down=0 tunnel_recovered=0, want 1 / 0
|
||||
--- FAIL: TestTunnelAlarm_Sequences/recovery_after_an_alarm (0.04s)
|
||||
tunnel_test.go:47: tunnel_down=0 tunnel_recovered=1, want 1 / 1
|
||||
== reverted
|
||||
ok gitea.dooplex.hu/admin/felhom-hub/internal/api 0.168s
|
||||
@@ -0,0 +1,22 @@
|
||||
# agent v0.141.0 leg red-proofs (each mutation compiles; result line printed)
|
||||
== mutation: byo-gets-host
|
||||
--- FAIL: TestBYO_NoHostPlan (0.00s)
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.440s
|
||||
== mutation: host-after-failed-guest
|
||||
--- FAIL: TestGuestFailure_SkipsTheHost (0.00s)
|
||||
--- FAIL: TestHealth_FailsAfterTheWait (0.00s)
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.420s
|
||||
== mutation: host-ignores-services
|
||||
--- FAIL: TestHostHealthVerdict (0.00s)
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.441s
|
||||
== mutation: host-ignores-guest
|
||||
--- FAIL: TestHostHealthVerdict (0.00s)
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.410s
|
||||
== mutation: host-ignores-tunnel
|
||||
--- FAIL: TestHost_TunnelDownFailsTheHostStep (0.00s)
|
||||
--- FAIL: TestHostHealthVerdict (0.00s)
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.427s
|
||||
== mutation: host-uses-guest-release
|
||||
--- FAIL: TestRing1_EachLayerItsOwnRelease (0.00s)
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.461s
|
||||
ok gitea.dooplex.hu/admin/felhom-agent/internal/osupdate (cached)
|
||||
@@ -0,0 +1,20 @@
|
||||
# felhom-os-apply (agent v0.141.0) red-proof, 2026-10-04: each mutation is applied to a COPY, syntax-checked, and the suite run against it.
|
||||
R1 (7 raise(s) removed): compiles=True suite rc=1; failing: test_R1_not_owned_by_the_agent, test_R1_path_outside_the_plan_dir, test_R1_symlink; expected test failed: YES
|
||||
R2 (2 raise(s) removed): compiles=True suite rc=1; failing: test_R2_non_debian_origin_in_the_plan, test_R2_non_debian_origin_in_the_simulation; expected test failed: YES
|
||||
R3 (1 raise(s) removed): compiles=True suite rc=1; failing: test_R3_slow_lane; expected test failed: YES
|
||||
R4 (1 raise(s) removed): compiles=True suite rc=1; failing: test_R4_removal; expected test failed: YES
|
||||
R5 (1 raise(s) removed): compiles=True suite rc=1; failing: test_R5_downgrade_exact; expected test failed: YES
|
||||
R6 (4 raise(s) removed): compiles=True suite rc=1; failing: test_R6_allow_new_is_slow_lane, test_R6_new_package, test_R6_unlisted_package; expected test failed: YES
|
||||
R7 (7 raise(s) removed): compiles=True suite rc=1; failing: test_R7_not_downloadable_and_no_snapshot, test_snapshot_does_not_have_it_either; expected test failed: YES
|
||||
R8 (1 raise(s) removed): compiles=True suite rc=1; failing: test_R8_free_space; expected test failed: YES
|
||||
R9 (2 raise(s) removed): compiles=True suite rc=1; failing: test_R9_apt_lock_held, test_R9_guest_locked_by_a_backup; expected test failed: YES
|
||||
R10 (4 raise(s) removed): compiles=True suite rc=1; failing: test_R10_bind_only_in_a_snapshot_section, test_R10_not_running, test_R10_not_the_boxs_own_guest, test_R10_reserved_vmid; expected test failed: YES
|
||||
R11 (12 raise(s) removed): compiles=True suite rc=1; failing: test_R11_bad_name, test_R11_bad_version_string, test_R11_duplicate; expected test failed: YES
|
||||
R12 (5 raise(s) removed): compiles=True suite rc=1; failing: test_R12_host_on_a_byo_box, test_R12_host_record_not_root_owned, test_R12_host_without_an_install_record, test_R12_unknown_layer; expected test failed: YES
|
||||
R13 (1 raise(s) removed): compiles=True suite rc=1; failing: test_R13_repair_does_not_fix_it; expected test failed: YES
|
||||
R14 (2 raise(s) removed): compiles=True suite rc=1; failing: test_R14_kernel_package_in_a_host_plan, test_R14_kernel_package_pulled_by_the_simulation; expected test failed: YES
|
||||
host layer runs in the guest instead: compiles=True suite rc=1; failing: test_host_runs_on_the_host_not_in_the_guest; expected test failed: YES
|
||||
pending-fast takes every pending upgrade: compiles=True suite rc=1; failing: test_pending_fast_skips_proxmox_docker_and_kernel; expected test failed: YES
|
||||
version checks in the target (pct exec per package): compiles=True suite rc=1; failing: test_no_per_package_guest_calls; expected test failed: YES
|
||||
lxc-start does not mean reboot: compiles=True suite rc=0; failing: NONE; expected test failed: NO
|
||||
lxc-start does not mean reboot (after adding test_reboot_needed_for_lxc_start_alone): suite rc=1; failing: test_reboot_needed_for_lxc_start_alone
|
||||
@@ -0,0 +1,8 @@
|
||||
# Agent red-proof: the host_release wire tag (contract with the hub golden), 2026-10-04T11:04:25Z
|
||||
# mutation: report.go json:"host_release" -> json:"hostrelease"; restored after
|
||||
=== RUN TestOSUpdateGolden_Decodes
|
||||
osupdate_contract_test.go:34: host_release = <nil>
|
||||
--- FAIL: TestOSUpdateGolden_Decodes (0.00s)
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/hub 0.007s
|
||||
FAIL
|
||||
@@ -0,0 +1,30 @@
|
||||
# Hub red-proofs, Parts B+C (hub v0.131.0), 2026-10-04T11:03:52Z
|
||||
# script: each mutation applied, the named test run with -v, file restored.
|
||||
[CAUGHT] host candidate leaves out kernel/boot/firmware: mutated internal/osupdates/service.go; TestCandidate_HostLeavesOutKernelBootFirmware ran=True failed=True rc=1
|
||||
alarms_test.go:27: host candidate = map[firmware-realtek:{Name:firmware-realtek Version:20250410-2 Origin:Debian} grub-efi-amd64:{Name:grub-efi-amd64 Version:2.12-9 Origin:Debian} intel-microcode:{Name:intel-microcode Ve
|
||||
FAIL gitea.dooplex.hu/admin/felhom-hub/internal/osupdates 0.034s
|
||||
[CAUGHT] host nights counted per layer: mutated internal/osupdates/service.go; TestLayers_SeparateSets ran=True failed=True rc=1
|
||||
alarms_test.go:60: host approved on a GUEST night: &{ID:os-host-20261005-130000 Layer:host Fingerprint:535006df2f78491d ApprovedAt:2026-10-05 13:00:00 +0000 UTC ApprovedBy:auto PackagesJSON:[{"name":"libssl3t64","version
|
||||
FAIL gitea.dooplex.hu/admin/felhom-hub/internal/osupdates 0.042s
|
||||
[CAUGHT] alarm 1 stale threshold: mutated internal/osupdates/service.go; TestAlarm_StaleLeg ran=True failed=True rc=1
|
||||
alarms_test.go:151: no alarm at 8 days: []
|
||||
FAIL gitea.dooplex.hu/admin/felhom-hub/internal/osupdates 0.041s
|
||||
[CAUGHT] alarm 1 switch OFF not watched: mutated internal/osupdates/service.go; TestAlarm_StaleLeg ran=True failed=True rc=1
|
||||
alarms_test.go:168: alarmed with the switch OFF
|
||||
FAIL gitea.dooplex.hu/admin/felhom-hub/internal/osupdates 0.054s
|
||||
[CAUGHT] alarm 2 reboot-needed counts from the FIRST report: mutated internal/osupdates/service.go; TestAlarm_RebootNeeded ran=True failed=True rc=1
|
||||
alarms_test.go:211: reboot-needed since = 2026-10-17 12:00:00 +0000 UTC
|
||||
FAIL gitea.dooplex.hu/admin/felhom-hub/internal/osupdates 0.045s
|
||||
[CAUGHT] alarm 3 counts only fast-lane pending: mutated internal/osupdates/service.go; TestAlarm_Ring0Stalled ran=True failed=True rc=1
|
||||
alarms_test.go:235: a pending KERNEL counted as a stalled fast lane
|
||||
FAIL gitea.dooplex.hu/admin/felhom-hub/internal/osupdates 0.040s
|
||||
[CAUGHT] alarm 4 ring 0 never alarms: mutated internal/osupdates/service.go; TestAlarm_NotCovered ran=True failed=True rc=1
|
||||
alarms_test.go:278: not-covered alarms = ["OS updates (guest): cust1 has had 1 fast-lane package(s) no approved release covers since 2026-10-04: libsomething-hw. Ring 0 does not run them (other hardware?) — add matching
|
||||
FAIL gitea.dooplex.hu/admin/felhom-hub/internal/osupdates 0.060s
|
||||
[CAUGHT] alarm events are operator-only: mutated internal/notify/dispatcher.go; TestOSUpdateEvents_OperatorOnlyExceptApplied ran=True failed=True rc=1
|
||||
os_alarms_operator_test.go:17: os_ring0_stalled is not operator-only
|
||||
FAIL gitea.dooplex.hu/admin/felhom-hub/internal/notify 0.009s
|
||||
[CAUGHT] host_release served to ring 1: mutated internal/osupdates/service.go; TestOSUpdate_DesiredBlockMatchesTheGolden ran=True failed=True rc=1
|
||||
os_updates_test.go:59: served os_update diverged from the golden:
|
||||
FAIL gitea.dooplex.hu/admin/felhom-hub/internal/api 0.055s
|
||||
after restore: ok gitea.dooplex.hu/admin/felhom-hub/internal/osupdates 0.544s | ok gitea.dooplex.hu/admin/felhom-hub/internal/notify 1.169s | ok gitea.dooplex.hu/admin/felhom-hub/internal/api 4.690s rc= 0
|
||||
@@ -0,0 +1,44 @@
|
||||
import subprocess, sys, shutil
|
||||
M = [
|
||||
("host candidate leaves out kernel/boot/firmware", "internal/osupdates/service.go",
|
||||
"if layer == LayerHost && hostSlowRE.MatchString(p.Name) {\n\t\t\t\tcontinue", "if false && hostSlowRE.MatchString(p.Name) {\n\t\t\t\tcontinue",
|
||||
"./internal/osupdates/", "TestCandidate_HostLeavesOutKernelBootFirmware"),
|
||||
("host nights counted per layer", "internal/osupdates/service.go",
|
||||
"reps, err := s.Store.OSReportsSince(h, layer, first)", "reps, err := s.Store.OSReportsSince(h, LayerGuest, first)",
|
||||
"./internal/osupdates/", "TestLayers_SeparateSets"),
|
||||
("alarm 1 stale threshold", "internal/osupdates/service.go",
|
||||
"now.Sub(ref) >= stale", "now.Sub(ref) >= 10*stale", "./internal/osupdates/", "TestAlarm_StaleLeg"),
|
||||
("alarm 1 switch OFF not watched", "internal/osupdates/service.go",
|
||||
"holds := st.Enabled && canRun", "holds := canRun", "./internal/osupdates/", "TestAlarm_StaleLeg"),
|
||||
("alarm 2 reboot-needed counts from the FIRST report", "internal/osupdates/service.go",
|
||||
"\t\tsince = rep.ReceivedAt\n", "\t\tif since.IsZero() {\n\t\t\tsince = rep.ReceivedAt\n\t\t}\n", "./internal/osupdates/", "TestAlarm_RebootNeeded"),
|
||||
("alarm 3 counts only fast-lane pending", "internal/osupdates/service.go",
|
||||
"pendingAny += len(fastPending(layer, r.Pending))", "pendingAny += len(r.Pending)", "./internal/osupdates/", "TestAlarm_Ring0Stalled"),
|
||||
("alarm 4 ring 0 never alarms", "internal/osupdates/service.go",
|
||||
"\tif ring == 0 {\n\t\treturn nil\n\t}\n", "", "./internal/osupdates/", "TestAlarm_NotCovered"),
|
||||
("alarm events are operator-only", "internal/notify/dispatcher.go",
|
||||
'\t"os_ring0_stalled": true,\n', "", "./internal/notify/", "TestOSUpdateEvents_OperatorOnlyExceptApplied"),
|
||||
("host_release served to ring 1", "internal/osupdates/service.go",
|
||||
"b.Release, b.HostRelease = s.releaseBlock(LayerGuest), s.releaseBlock(LayerHost)", "b.Release = s.releaseBlock(LayerGuest)",
|
||||
"./internal/api/", "TestOSUpdate_DesiredBlockMatchesTheGolden"),
|
||||
]
|
||||
ok = True
|
||||
for name, f, old, new, pkg, test in M:
|
||||
src = open(f).read()
|
||||
assert src.count(old) == 1, (name, src.count(old))
|
||||
shutil.copy(f, f + ".bak")
|
||||
open(f, "w").write(src.replace(old, new))
|
||||
r = subprocess.run(["go", "test", pkg, "-run", "^" + test + "$", "-v", "-count=1"], capture_output=True, text=True)
|
||||
shutil.move(f + ".bak", f)
|
||||
out = r.stdout + r.stderr
|
||||
ran = ("=== RUN " + test) in out
|
||||
failed = ("--- FAIL: " + test) in out
|
||||
verdict = "CAUGHT" if (ran and failed and r.returncode != 0) else "NOT CAUGHT"
|
||||
if verdict != "CAUGHT": ok = False
|
||||
print(f"[{verdict}] {name}: mutated {f}; {test} ran={ran} failed={failed} rc={r.returncode}")
|
||||
for l in out.splitlines():
|
||||
if "_test.go:" in l or l.startswith("FAIL\t") or "build failed" in l or l.startswith("#"):
|
||||
print(" " + l.strip()[:220])
|
||||
r = subprocess.run(["go", "test", "./internal/osupdates/", "./internal/notify/", "./internal/api/", "-count=1"], capture_output=True, text=True)
|
||||
print("after restore:", r.stdout.strip().replace("\n", " | "), "rc=", r.returncode)
|
||||
sys.exit(0 if ok and r.returncode == 0 else 1)
|
||||
Reference in New Issue
Block a user