admin
51871a7ea6
gates / gates (push) Successful in 8s
Every weekly off-site run uploaded successfully and then failed the job on a prune the box's token is deliberately refused — R-89 moved off-site pruning server-side to ep0 and box tokens stay write-only. The 2026-07-26 'two weeks' ruling was not reversed; where it is enforced moved, and keep_last: 2 did not follow. Now 0, which the agent's existing guard already reads as 'never prune from the box'. Verified read-only on ep0 before changing it: both namespaces have a prune job at 03:30 keep-last 2 that has run every day since 2026-07-27 — 18 tasks, all OK, the newest keeping exactly two. Without that check this would have traded a weekly false alarm for unbounded growth. A gate asserts the offsite tier carries no client-side prune. The local tier is untouched.
267 lines
15 KiB
Python
267 lines
15 KiB
Python
# -*- coding: utf-8 -*-
|
|
"""DR-tier-by-default installer gates — mechanical grep-assertions against
|
|
felhom-host-install.sh's known failure mode: a Day-0 that silently regresses one of the
|
|
drill-swept findings (DRILL-day0-vm-2026-07-12 F-1/F-7/F-9/F-10 + the ACL-narrowing 403).
|
|
Run from the repo root: python scripts/hostinstall_gates.py
|
|
|
|
Gates (all must pass; non-zero exit on any failure):
|
|
1. version — exactly ONE version source: SCRIPT_VERSION exists, the header line carries no
|
|
version literal, and **the hub carries no host-install version literal at all**.
|
|
The third assertion inverted on 2026-08-02 (R-94): it used to require the hub's
|
|
`hostInstallVersion` const to EQUAL SCRIPT_VERSION, which is unachievable
|
|
honestly — the Option-1 install command downloads felhom-host-install.sh from
|
|
the website at RUN TIME and the website git-syncs `main` every 30 seconds
|
|
(R-110), so the hub cannot know which version a given box will run. A
|
|
build-time literal there is a guess with a version number's authority, and the
|
|
real one drifted to 1.19.0-vs-1.22.0 and stayed wrong for 19 days. The label was
|
|
deleted rather than derived; this gate now pins its absence (F-1 structural fix,
|
|
second form).
|
|
2. age — the `age` package is installed by the agent-install step (F-10)
|
|
3. pbs-apply — configs/felhom-pbs-apply is fetched + installed to
|
|
/usr/local/sbin/felhom-pbs-apply (F-7), and the uninstall removes it
|
|
4. wg — the rendered agent.json defaults wg_tunnel enabled=true (F-9 / decision 5),
|
|
and the byo assert no longer forbids it
|
|
5. acl — the default PVE_STORAGES set still contains felhom-pbs (narrowing it is the
|
|
drill's apply-bridge 403)
|
|
"""
|
|
import io, os, re, sys
|
|
|
|
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
|
SCRIPT = os.path.join(ROOT, "scripts", "felhom-host-install.sh")
|
|
HUB_CONFIGS = os.path.join(ROOT, "hub", "internal", "web", "configs.go")
|
|
|
|
fails = []
|
|
|
|
|
|
def fail(msg):
|
|
fails.append(msg)
|
|
print("FAIL:", msg)
|
|
|
|
|
|
def ok(msg):
|
|
print(" ok:", msg)
|
|
|
|
|
|
with io.open(SCRIPT, "r", encoding="utf-8") as f:
|
|
src = f.read()
|
|
lines = src.splitlines()
|
|
|
|
# ── 1. version single-source (F-1) ──────────────────────────────────────────────
|
|
m = re.search(r'^SCRIPT_VERSION="(\d+\.\d+\.\d+)"', src, re.M)
|
|
if not m:
|
|
fail("SCRIPT_VERSION=\"x.y.z\" not found — the single version source is gone")
|
|
script_ver = None
|
|
else:
|
|
script_ver = m.group(1)
|
|
ok("SCRIPT_VERSION=%s" % script_ver)
|
|
|
|
# header (first 10 lines) must NOT carry its own version literal — that is the F-1 drift.
|
|
header = "\n".join(lines[:10])
|
|
if re.search(r'felhom-host-install\.sh\s+v\d+\.\d+\.\d+', header):
|
|
fail("header line carries a hardcoded version — SCRIPT_VERSION is the only source (F-1)")
|
|
else:
|
|
ok("header has no version literal")
|
|
|
|
# The hub must carry NO host-install version literal at all (R-94, 2026-08-02). It cannot know
|
|
# which version a box will run — the Option-1 command fetches the script from the website at run
|
|
# time and the website git-syncs `main` every 30s. The const this replaced said 1.19.0 while the
|
|
# served script was 1.22.0, and had been wrong since 2026-07-14.
|
|
#
|
|
# Matched in CODE SHAPES, never as bare prose: the deleted declarations, the struct field, the
|
|
# assignment and the template action, plus a rename-proof generic form of each. Comments are
|
|
# deliberately NOT stripped (a `//` inside a URL string literal would truncate the scan and turn
|
|
# this gate blind); a comment that merely NAMES the identifier is allowed, and configs.go carries
|
|
# exactly such a note explaining the absence.
|
|
BANNED = [
|
|
(r'\bconst\s+hostInstallVersion\b', "const hostInstallVersion"),
|
|
(r'\bhostInstallVersion\s*=', "hostInstallVersion assignment"),
|
|
(r'(?i)\bconst\s+\w*hostinstall\w*version\b', "a renamed host-install version const"),
|
|
(r'\bScriptVersion\s+string\b', "ScriptVersion struct field"),
|
|
(r'\bScriptVersion\s*:', "ScriptVersion struct assignment"),
|
|
(r'\{\{\s*\.ScriptVersion\s*\}\}', "{{.ScriptVersion}} template action"),
|
|
]
|
|
HUB_DIR = os.path.join(ROOT, "hub")
|
|
if not os.path.isdir(HUB_DIR):
|
|
fail("hub/ not found at %s — cannot assert the absence of a host-install version literal" % HUB_DIR)
|
|
else:
|
|
scanned, hits = 0, 0
|
|
for dirpath, dirs, files in os.walk(HUB_DIR):
|
|
dirs[:] = [d for d in dirs if d not in (".git", "vendor", "node_modules")]
|
|
for fn in files:
|
|
if not (fn.endswith(".go") or fn.endswith(".html")):
|
|
continue
|
|
fp = os.path.join(dirpath, fn)
|
|
scanned += 1
|
|
with io.open(fp, "r", encoding="utf-8") as f:
|
|
for lineno, line in enumerate(f, 1):
|
|
for pat, what in BANNED:
|
|
if re.search(pat, line):
|
|
hits += 1
|
|
fail("%s:%d carries %s — the hub must render NO host-install version "
|
|
"(R-94: the served script is fetched at run time, so no build-time "
|
|
"value can be true). Single source: scripts/felhom-host-install.sh "
|
|
"SCRIPT_VERSION. Line: %s"
|
|
% (os.path.relpath(fp, ROOT), lineno, what, line.strip()[:120]))
|
|
if not hits:
|
|
ok("hub carries no host-install version literal (%d .go/.html files scanned, %d shapes checked)"
|
|
% (scanned, len(BANNED)))
|
|
|
|
# ── 2. age package (F-10) ───────────────────────────────────────────────────────
|
|
# must match the REAL install invocation, not the log_dry echo (red-proof-hardened twice:
|
|
# a prefix regex matched "agekit", then a loose one matched the dry-run print line).
|
|
if re.search(r'DEBIAN_FRONTEND=noninteractive apt-get install -y -q age\b', src):
|
|
ok("age is in the installed package set")
|
|
else:
|
|
fail("`age` install not found (F-10 — the fresh-box escrow ceremony dies without it)")
|
|
|
|
# ── 3. pbs-apply wrapper shipped + removed (F-7) ────────────────────────────────
|
|
if 'fetch_raw "configs/felhom-pbs-apply"' in src:
|
|
ok("felhom-pbs-apply is fetched from the agent repo")
|
|
else:
|
|
fail("configs/felhom-pbs-apply fetch not found (F-7 — pbsdr capabilities born DEGRADED)")
|
|
if re.search(r'install -m 0755 -o root -g root "\$patmp" /usr/local/sbin/felhom-pbs-apply', src):
|
|
ok("felhom-pbs-apply installed 0755 to /usr/local/sbin")
|
|
else:
|
|
fail("felhom-pbs-apply install line not found (F-7)")
|
|
if re.search(r'rm -f /usr/local/sbin/felhom-pbs-apply', src):
|
|
ok("uninstall removes felhom-pbs-apply")
|
|
else:
|
|
fail("uninstall does not remove /usr/local/sbin/felhom-pbs-apply")
|
|
|
|
# ── 4. wg_tunnel default-on (F-9 / decision 5) ──────────────────────────────────
|
|
if re.search(r"base\.setdefault\('wg_tunnel',\s*\{\"enabled\":\s*True\}\)", src):
|
|
ok("rendered agent.json defaults wg_tunnel.enabled=true")
|
|
else:
|
|
fail("wg_tunnel enabled-by-default missing from the agent.json render (F-9)")
|
|
# the byo assert must NOT forbid wg_tunnel any more (decision 5: WG is base infrastructure).
|
|
byo_assert = re.search(r"byo-forbidden config keys.*?sys\.exit\(1\)", src, re.S)
|
|
if byo_assert and "wg_tunnel" in byo_assert.group(0):
|
|
fail("the byo config assert still forbids wg_tunnel.enabled (decision 5 retired that)")
|
|
else:
|
|
ok("byo assert no longer forbids wg_tunnel")
|
|
|
|
# ── 5. default ACL keeps felhom-pbs (the drill 403) ─────────────────────────────
|
|
if re.search(r'^PVE_STORAGES=\([^)]*felhom-pbs[^)]*\)', src, re.M):
|
|
ok("PVE_STORAGES default contains felhom-pbs")
|
|
else:
|
|
fail("felhom-pbs missing from the default PVE_STORAGES — narrowing it 403s the PBS-DR apply-bridge")
|
|
|
|
# ── 6. the publish channel is pinned, not floating (R-110 / R-183) ──────────────
|
|
#
|
|
# WHAT THIS ASSERTS, AND WHAT IT DELIBERATELY DOES NOT.
|
|
#
|
|
# It does NOT assert "a tag exists for the current SCRIPT_VERSION". That gate would fail the very
|
|
# push that bumps SCRIPT_VERSION, before publishing has happened — and publishing being a SEPARATE
|
|
# deliberate act is the whole point of R-110's ruling. A gate that goes red on the normal path is a
|
|
# gate people learn to ignore, which is the reasoning the task's own §8.4 applies to the agent-side
|
|
# gate; it applies here identically. "Is the vouched version actually downloadable" is a real
|
|
# invariant and it lives where a missing artifact genuinely breaks day-0 — `felhom-agent`'s
|
|
# `agent_gates.py`, which has the network access to answer it.
|
|
#
|
|
# What it asserts instead are the two STRUCTURAL regressions that would silently return the
|
|
# installer to a floating channel, both answerable by reading files (no network, so this stays in
|
|
# `--fast` and therefore runs in CI on every push):
|
|
#
|
|
# 6a. no `raw/branch/` ref anywhere in the installer — one of the sixteen agent-config fetches
|
|
# slipping back to `main` is exactly how a channel stays floating unnoticed, and it is
|
|
# invisible in a diff that touches one line.
|
|
# 6b. `fetch_raw` still pins to the resolved agent version — the positive form, so the mechanism
|
|
# cannot be quietly deleted rather than regressed.
|
|
# 6c. the website manifest still syncs `/scripts/` from a TAG ref and the website from `main` —
|
|
# the split is the deploy-side half of the same channel, and reverting it is one word.
|
|
branch_refs = [l for l in lines if "raw/branch/" in l and not l.lstrip().startswith("#")]
|
|
if branch_refs:
|
|
fail("installer still fetches from a BRANCH ref — the run-time channel is floating again "
|
|
"(R-110/R-183). Offending line(s): %s" % "; ".join(l.strip()[:90] for l in branch_refs))
|
|
else:
|
|
ok("no raw/branch/ ref in the installer — every run-time fetch is pinned")
|
|
|
|
if re.search(r'raw/tag/v\$ART_AGENT_VER/', src):
|
|
ok("fetch_raw pins the agent configs to the vouched agent version")
|
|
else:
|
|
fail("fetch_raw no longer pins to $ART_AGENT_VER — the agent's configs and its binary can "
|
|
"again come from different refs in one install (R-183)")
|
|
|
|
WEBPAGE = os.path.join(ROOT, "manifests", "webpage.yaml")
|
|
try:
|
|
with io.open(WEBPAGE, "r", encoding="utf-8") as f:
|
|
wp = f.read()
|
|
except IOError as e:
|
|
fail("cannot read manifests/webpage.yaml to check the publish channel: %s" % e)
|
|
wp = None
|
|
if wp is not None:
|
|
# The scripts sync must name a tag ref; the website sync must still track main.
|
|
if re.search(r'--ref=installer-v', wp):
|
|
ok("manifest: /scripts/ syncs from an installer tag")
|
|
else:
|
|
fail("manifests/webpage.yaml has no `--ref=installer-v…` sync — /scripts/ is not served "
|
|
"from a tag, so pushing the installer publishes it again (R-110)")
|
|
if re.search(r'--(branch|ref)=main', wp):
|
|
ok("manifest: the website still tracks main (a copy edit must not need a release)")
|
|
else:
|
|
fail("manifests/webpage.yaml no longer tracks main for the website — pinning the SITE to "
|
|
"the installer tag turns every copy edit into a release")
|
|
|
|
# ── R-185: every path that RESOLVES the backup target must also grant on it ──────────────────
|
|
#
|
|
# THE DEFECT THIS WOULD HAVE CAUGHT, measured on both demo boxes 2026-08-03. `configure_backup_target`
|
|
# has two arms. The CASE A arm creates the storage and grants in the same breath. The Scenario-F arm —
|
|
# "the target already exists, leave it alone" — returned WITHOUT granting, so a box whose target
|
|
# pre-dated the install pointed `local_backup_target` at a storage its own token could not read. The
|
|
# API answered `{"data":[]}` while root saw three archives, and nothing said so, because an empty
|
|
# listing is also what a brand-new tier returns.
|
|
#
|
|
# The assertion is deliberately about the FUNCTION, not about PVE_STORAGES: the target's grant belongs
|
|
# with the target's resolution (PVE_STORAGES is granted a step earlier, before the target exists), so
|
|
# what must hold is that no arm of that function can resolve a target and skip the grant.
|
|
fn = re.search(r'^configure_backup_target\(\)\s*\{(.*?)^\}', src, re.S | re.M)
|
|
if not fn:
|
|
fail("cannot find configure_backup_target() — the backup-target ACL assertion cannot run, and a "
|
|
"check that cannot run must never report OK (R-185)")
|
|
else:
|
|
body = fn.group(1)
|
|
resolutions = len(re.findall(r'BACKUP_TARGET_RESOLVED="\$BACKUP_TARGET_ID"', body))
|
|
grants = len(re.findall(r'felhom-backup-target-apply grant', body))
|
|
if resolutions == 0:
|
|
fail("configure_backup_target no longer resolves BACKUP_TARGET_ID anywhere — re-read it")
|
|
elif grants >= resolutions:
|
|
ok("every arm that resolves the backup target also grants on it (%d resolution(s), %d grant(s))"
|
|
% (resolutions, grants))
|
|
else:
|
|
fail("configure_backup_target resolves the backup target in %d place(s) but grants in only %d "
|
|
"— an arm resolves a target the agent may not READ. That is R-185: the tier's archives are "
|
|
"invisible to the agent, it is never restore-tested, and an empty listing looks exactly "
|
|
"like a brand-new tier." % (resolutions, grants))
|
|
|
|
# ── R-191: the OFFSITE tier must not arm a client-side prune ─────────────────────────────────
|
|
#
|
|
# R-89 moved offsite pruning SERVER-SIDE — ep0 runs a per-namespace prune job and box tokens stay
|
|
# write-only, so the box is REFUSED if it asks. When this default was `keep_last: 2` the effect was a
|
|
# weekly lie: vzdump uploaded the snapshot, then failed the whole job on the prune, and the operator
|
|
# was told the offsite backup had failed when it had succeeded.
|
|
#
|
|
# The assertion is on the OFFSITE entry only. The local tier's `local_backup_retention` is untouched
|
|
# and must stay untouched — it prunes correctly and is allowed to.
|
|
m = re.search(r'"backup_targets":\s*\[(.*?)\]', src, re.S)
|
|
if not m:
|
|
fail("cannot find backup_targets in the rendered agent.json defaults — the offsite-retention "
|
|
"assertion cannot run, and a check that cannot run must never report OK (R-191)")
|
|
else:
|
|
targets = m.group(1)
|
|
kl = re.search(r'"keep_last"\s*:\s*(\d+)', targets)
|
|
if not kl:
|
|
fail("the offsite backup_target carries no keep_last at all — expected an explicit 0 "
|
|
"(R-191: 0 means 'never prune from the box'; absent is not the same statement)")
|
|
elif kl.group(1) != "0":
|
|
fail("the offsite backup_target arms a CLIENT-SIDE prune (keep_last=%s). R-89 moved offsite "
|
|
"pruning server-side to ep0 and box tokens are write-only, so every weekly run will "
|
|
"upload successfully and then FAIL the job on a refused prune (R-191)." % kl.group(1))
|
|
else:
|
|
ok("the offsite tier arms no client-side prune (keep_last=0; retention is ep0's prune jobs)")
|
|
|
|
print()
|
|
if fails:
|
|
print("hostinstall gates: %d FAILURE(S)" % len(fails))
|
|
sys.exit(1)
|
|
print("hostinstall gates: ALL PASS")
|