Campaign 10 Phase B: 27 cycles, 586 invariant checks, 0 violations
Ran the soak on the Phase A rig. Ended on its own deadline — no watchdog halt, no atom exception, no I11 breach. I1 28+28 pairs, I2 28+28 pairs, I3 56, I4 56, I5/I6 28 each, I7 28, I10 135, I11 28. Zero violations. The row counts are themselves the no-silent-skip check: I3/I4 twice per cycle (both drives), I10 = 5 secret-class fields x 27, REBOOT on cycles 7/14/21 only. I7 is the headline: 28 restores, 28 correct discriminators — never stale, never empty. RTO (Tier 1, rallly, 66 MB): min 38.8s, median 42.0s, p90 42.5s, max 44.3s. That is the S band's lower end ONLY; the 5.5s spread over 28 runs says fixed work dominates, so nothing extrapolates to M or L. RPO not measured. Every atom and invariant was proven BY HAND before automation — the runner asserts nothing that was not first observed live. Caught a Phase A gap before starting: no app had HDD_PATH, so all data sat on the system disk and I3 could never have fired. Deployed calibre-web onto adatok first; otherwise the run would have produced 27 green cycles that tested nothing cross-drive. Investigated and DISPROVED a suspected defect (audit 5.2): /api/disks reports state=attached for a physically absent drive, and intermediary.go:230 really does compute presence from State=="attached". It is inert — planDriveGates only gates paths under /mnt/felhom-drives/ and uses BoundUnderParent there, which was correctly false. The gate fired; the storage page showed "Meghajtó leválasztva". No R-n minted. Honest gaps: 6 of ~12 atom families ran. Not run — Tier 3 (structurally un-isolatable), abort-fs-in-place, kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency atoms, fill-drive-near-full. I8 not checked, I9 not automated (cited from the tester-gate run, not re-claimed). kill_controller is NOT mid-backup and reboot_guest never interleaved with a detach. 27 cycles does not answer the brief's question about drift at the thirty-eighth. Teardown still OWED, including hub customer c10-soak (disposition: DELETE).
This commit is contained in:
@@ -0,0 +1,141 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Campaign 10 — shared driver library. Runs on DooPlex, NOT inside the VM:
|
||||
the soak's atoms kill the controller, the agent and the VM itself, so anything
|
||||
running inside would die with the thing it is testing.
|
||||
|
||||
Canary model (brief A4): every app carries a marker naming the backup GENERATION
|
||||
its data came from. Without it, a restore that returns stale or empty data is
|
||||
indistinguishable from a good one — this arc produced six checks that passed for
|
||||
the wrong reason.
|
||||
|
||||
rallly -> a row in a cc_proof TABLE in postgres, read over the app's real
|
||||
network path (docker run --network container:rallly ... psql -h
|
||||
rallly-postgres). NOT 127.0.0.1 inside the postgres container,
|
||||
which postgres trusts and which produced D5's false pass.
|
||||
others -> a marker file inside the app's own docker volume (what gets dumped).
|
||||
"""
|
||||
import json, os, subprocess, time
|
||||
|
||||
HOME = os.path.expanduser("~/.config/campaign10")
|
||||
VM_IP = "192.168.0.105"
|
||||
JUMP = "demo-hp"
|
||||
VMID = "311"
|
||||
GUEST = "9201"
|
||||
|
||||
HOSTPW = json.load(open(os.path.join(HOME, "host-recovery.json")))["password"]
|
||||
SECRETS = json.load(open(os.path.join(HOME, "app-secrets.json")))
|
||||
|
||||
# app -> (container, volume mount dir) for the file-marker apps
|
||||
FILE_APPS = {
|
||||
"grafana": ("grafana", "/var/lib/grafana"),
|
||||
"homebox": ("homebox", "/data"),
|
||||
"papra": ("papra", "/app/data"),
|
||||
}
|
||||
ALL_APPS = ["rallly", "homebox", "grafana", "papra"]
|
||||
|
||||
|
||||
def _run(args, timeout=300):
|
||||
env = dict(os.environ, SSHPASS=HOSTPW, LC_ALL="C")
|
||||
try:
|
||||
p = subprocess.run(args, capture_output=True, text=True, timeout=timeout, env=env)
|
||||
return p.returncode, p.stdout, p.stderr
|
||||
except subprocess.TimeoutExpired:
|
||||
return 124, "", "TIMEOUT"
|
||||
|
||||
|
||||
def box(cmd, timeout=300):
|
||||
"""Run a command on the campaign PVE host (the VM), via the demo-hp jump."""
|
||||
return _run(["sshpass", "-e", "ssh", "-o", "StrictHostKeyChecking=no",
|
||||
"-o", "ConnectTimeout=20", "-J", JUMP, f"root@{VM_IP}",
|
||||
"LC_ALL=C " + cmd], timeout)
|
||||
|
||||
|
||||
def guest(cmd, timeout=300):
|
||||
"""Run a command inside guest 9201."""
|
||||
return box(f"pct exec {GUEST} -- bash -c {shq(cmd)}", timeout)
|
||||
|
||||
|
||||
def hp(cmd, timeout=180):
|
||||
"""Run a command on demo-hp itself (for qm: reset, stop, disk detach...)."""
|
||||
return _run(["ssh", "-o", "ConnectTimeout=20", JUMP, "LC_ALL=C " + cmd], timeout)
|
||||
|
||||
|
||||
def shq(s):
|
||||
return "'" + s.replace("'", "'\\''") + "'"
|
||||
|
||||
|
||||
def api(method, path, body=None, timeout=300):
|
||||
"""Authenticated controller API call through the on-guest helper."""
|
||||
if body is None:
|
||||
rc, out, _ = guest(f"/root/c10api.sh {method} {path} 2>/dev/null", timeout)
|
||||
return out
|
||||
import base64
|
||||
b64 = base64.b64encode(json.dumps(body).encode()).decode()
|
||||
rc, out, _ = guest(
|
||||
f"echo {b64} | base64 -d > /tmp/.b && /root/c10api.sh {method} {path} "
|
||||
f'"$(cat /tmp/.b)" --json; shred -u /tmp/.b 2>/dev/null', timeout)
|
||||
return out
|
||||
|
||||
|
||||
def jparse(raw):
|
||||
try:
|
||||
return json.loads(raw[raw.index("{"):raw.rindex("}") + 1])
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- canaries
|
||||
|
||||
def _psql(sql, timeout=180):
|
||||
"""Read/write rallly's DB over the path DATABASE_URL actually names."""
|
||||
pw = SECRETS["rallly"]["values"]["DB_PASSWORD"]
|
||||
url = f"postgresql://rallly:{pw}@rallly-postgres:5432/rallly"
|
||||
cmd = ("docker run --rm --network container:rallly postgres:16-alpine "
|
||||
f"psql {shq(url)} -tAc {shq(sql)}")
|
||||
rc, out, err = guest(cmd, timeout)
|
||||
return rc, out.strip(), err.strip()
|
||||
|
||||
|
||||
def seed(gen):
|
||||
"""Write the generation marker into every app. Returns {app: ok}."""
|
||||
res = {}
|
||||
_psql("CREATE TABLE IF NOT EXISTS cc_proof (id serial primary key, gen text, at timestamptz default now())")
|
||||
rc, out, err = _psql(f"INSERT INTO cc_proof (gen) VALUES ('{gen}') RETURNING gen")
|
||||
res["rallly"] = (gen in out)
|
||||
for app, (cont, d) in FILE_APPS.items():
|
||||
# -u 0: papra's volume is root-owned while the container runs as nonroot (R-156),
|
||||
# so the app itself cannot write there. Seeding as root keeps the canary in the
|
||||
# volume the BACKUP dumps, which is what I7 needs to observe.
|
||||
rc, out, err = guest(
|
||||
f"docker exec -u 0 {cont} sh -c {shq(f'echo {gen} > {d}/cc_proof.txt && cat {d}/cc_proof.txt')}", 180)
|
||||
res[app] = (gen in out)
|
||||
return res
|
||||
|
||||
|
||||
def read_canaries():
|
||||
"""Read each app's marker back. Returns {app: value-or-None}."""
|
||||
res = {}
|
||||
rc, out, err = _psql("SELECT gen FROM cc_proof ORDER BY id DESC LIMIT 1")
|
||||
res["rallly"] = out.strip() or None
|
||||
for app, (cont, d) in FILE_APPS.items():
|
||||
rc, out, err = guest(f"docker exec {cont} sh -c {shq(f'cat {d}/cc_proof.txt 2>/dev/null')}", 180)
|
||||
v = out.strip()
|
||||
res[app] = v or None
|
||||
return res
|
||||
|
||||
|
||||
def containers():
|
||||
rc, out, _ = guest("docker ps --format '{{.Names}}|{{.Status}}'", 120)
|
||||
d = {}
|
||||
for line in out.strip().splitlines():
|
||||
if "|" in line:
|
||||
n, s = line.split("|", 1)
|
||||
d[n.strip()] = s.strip()
|
||||
return d
|
||||
|
||||
|
||||
def guest_init_pid():
|
||||
"""I4: a stale-device bind must repair WITHOUT the guest restarting."""
|
||||
rc, out, _ = box(f"pct exec {GUEST} -- cat /proc/1/stat 2>/dev/null", 120)
|
||||
rc2, boot, _ = box(f"pct exec {GUEST} -- stat -c %Y /proc/1 2>/dev/null", 120)
|
||||
return boot.strip()
|
||||
@@ -0,0 +1,333 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Campaign 10 — Phase B soak runner. Runs on DooPlex (outside the VM it abuses).
|
||||
|
||||
Shape (brief B1): a set of atomic operations composed into cycles in varying order,
|
||||
with EVERY invariant checked EVERY cycle — drift at iteration 38 is the point.
|
||||
|
||||
Every atom and every invariant here was proven BY HAND before automation
|
||||
(see the audit §5); nothing in this file asserts a behaviour that was not first
|
||||
observed live.
|
||||
|
||||
Writes: journal.tsv (one row per invariant check), runner.log, status.txt (heartbeat).
|
||||
Halts cleanly on: disk floor breach, VM gone, I11 breach (secret leak).
|
||||
"""
|
||||
import json, os, random, subprocess, sys, time, traceback
|
||||
import c10lib as L
|
||||
|
||||
EV = "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/tests/campaign10-evidence-2026-07-31"
|
||||
STATE = os.path.join(EV, "state")
|
||||
os.makedirs(STATE, exist_ok=True)
|
||||
JOURNAL = os.path.join(STATE, "journal.tsv")
|
||||
RUNLOG = os.path.join(STATE, "runner.log")
|
||||
STATUS = os.path.join(STATE, "status.txt")
|
||||
|
||||
APPS = ["rallly", "homebox", "grafana", "papra"] # docker-volume apps
|
||||
DRIVE_APP = "calibre-web" # binds /mnt/felhom-drives/adatok
|
||||
TARGET_DISK, TARGET_VOL = "scsi2", "c10-scratch:311/vm-311-disk-3.qcow2" # mentes = backup target
|
||||
NONTGT_DISK, NONTGT_VOL = "scsi1", "c10-scratch:311/vm-311-disk-2.qcow2" # adatok
|
||||
FLOOR_GB = 120 # halt if /mnt/nvme-1tb free drops below this
|
||||
MAX_CYCLES = int(os.environ.get("C10_CYCLES", "40"))
|
||||
DEADLINE = time.time() + float(os.environ.get("C10_HOURS", "6")) * 3600
|
||||
|
||||
CYCLE = 0
|
||||
VIOL = []
|
||||
|
||||
|
||||
def log(msg):
|
||||
line = "%s %s" % (time.strftime("%H:%M:%S"), msg)
|
||||
print(line, flush=True)
|
||||
with open(RUNLOG, "a") as fh:
|
||||
fh.write(line + "\n")
|
||||
|
||||
|
||||
def rec(inv, ok, detail=""):
|
||||
"""One journal row per invariant check. Violations do NOT stop the run (brief B2)."""
|
||||
with open(JOURNAL, "a") as fh:
|
||||
fh.write("%d\t%s\t%s\t%s\t%s\n" % (CYCLE, time.strftime("%FT%TZ", time.gmtime()),
|
||||
inv, "PASS" if ok else "VIOLATION", detail[:400]))
|
||||
if not ok:
|
||||
VIOL.append((CYCLE, inv, detail[:300]))
|
||||
log(" !! %s VIOLATION: %s" % (inv, detail[:220]))
|
||||
|
||||
|
||||
def heartbeat(phase):
|
||||
with open(STATUS, "w") as fh:
|
||||
fh.write("cycle=%d\nphase=%s\nutc=%s\nviolations=%d\n"
|
||||
% (CYCLE, phase, time.strftime("%FT%TZ", time.gmtime()), len(VIOL)))
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ helpers
|
||||
|
||||
def hub_events(since="3m"):
|
||||
"""Hub-side event stream: type + severity, the observable I1/I2 assert on."""
|
||||
p = subprocess.run(["sudo", "kubectl", "-n", "felhom-system", "logs", "deploy/hub",
|
||||
"--since=" + since], capture_output=True, text=True, timeout=120)
|
||||
out = []
|
||||
for line in p.stdout.splitlines():
|
||||
if "Event from c10-soak:" in line:
|
||||
seg = line.split("Event from c10-soak:", 1)[1].strip()
|
||||
typ = seg.split()[0]
|
||||
sev = seg.split("(", 1)[1].split(")", 1)[0] if "(" in seg else "?"
|
||||
out.append((typ, sev, seg))
|
||||
return out
|
||||
|
||||
|
||||
def target_state():
|
||||
return (L.jparse(L.api("GET", "/api/storage/backup-target")) or {}).get("data", {}) or {}
|
||||
|
||||
|
||||
def disks():
|
||||
return ((L.jparse(L.api("GET", "/api/disks")) or {}).get("data", {}) or {}).get("disks", [])
|
||||
|
||||
|
||||
def wait_backup(timeout=900):
|
||||
t0 = time.time()
|
||||
while time.time() - t0 < timeout:
|
||||
d = ((L.jparse(L.api("GET", "/api/backup/status")) or {}).get("data") or {})
|
||||
if not d.get("running"):
|
||||
return d
|
||||
time.sleep(10)
|
||||
return {"timeout": True}
|
||||
|
||||
|
||||
def wait_restore(timeout=900):
|
||||
t0 = time.time()
|
||||
while time.time() - t0 < timeout:
|
||||
d = ((L.jparse(L.api("GET", "/api/backup/restore-status")) or {}).get("data") or {})
|
||||
if d and not d.get("running"):
|
||||
return d
|
||||
time.sleep(10)
|
||||
return {"timeout": True}
|
||||
|
||||
|
||||
def settle(seconds=130):
|
||||
"""Agent tick + host-report + hub event propagation. Measured at ~25-60s; 130 gives margin."""
|
||||
time.sleep(seconds)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ atoms
|
||||
|
||||
def atom_backup():
|
||||
L.api("POST", "/api/backup/run", {})
|
||||
d = wait_backup()
|
||||
ok = bool(d.get("db_dump", {}).get("success", True)) and not d.get("timeout")
|
||||
rec("BACKUP", ok, json.dumps(d)[:300])
|
||||
return ok
|
||||
|
||||
|
||||
def atom_restore_verify():
|
||||
"""I7 — the whole point: a restore must return the discriminator asked for."""
|
||||
app = "rallly" # the DB app; its canary is a real DB row
|
||||
genA = "C10-C%03d-A-%s" % (CYCLE, time.strftime("%H%M%S"))
|
||||
L.seed(genA)
|
||||
if not atom_backup():
|
||||
return
|
||||
genB = "C10-C%03d-B-%s" % (CYCLE, time.strftime("%H%M%S"))
|
||||
L.seed(genB) # must NOT survive the restore
|
||||
snaps = L.jparse(L.api("GET", "/api/backup/snapshots?stack=%s" % app)) or {}
|
||||
pts = snaps.get("data") or []
|
||||
if not pts:
|
||||
rec("I7", False, "no restore points for %s" % app)
|
||||
return
|
||||
sid = pts[0].get("short_id")
|
||||
t0 = time.time()
|
||||
L.guest('/root/c10api.sh POST /backup/restore "stack_name=%s&snapshot_id=%s" 2>/dev/null' % (app, sid))
|
||||
d = wait_restore()
|
||||
rto = time.time() - t0
|
||||
got = L.read_canaries().get(app)
|
||||
rec("I7", got == genA,
|
||||
"app=%s want=%s got=%s snap=%s restore_ok=%s rto=%.1fs"
|
||||
% (app, genA, got, sid, (d.get("last") or {}).get("ok"), rto))
|
||||
# RTO/RPO byproduct (brief B3): to app-serving-correct-data, not to "restore returned"
|
||||
with open(os.path.join(STATE, "rto.tsv"), "a") as fh:
|
||||
fh.write("%d\t%s\t%s\ttier1\t%.1f\t%s\n" % (CYCLE, time.strftime("%FT%TZ", time.gmtime()),
|
||||
app, rto, got == genA))
|
||||
|
||||
|
||||
def _detach(disk, vol, label, expect_type, expect_recover):
|
||||
"""I1/I2 — the absent event must be the RIGHT one, and the pair must match."""
|
||||
before_boot = L.guest("stat -c %Y /proc/1")[1].strip()
|
||||
L.hp("qm set 311 --delete %s" % disk)
|
||||
settle()
|
||||
evs = hub_events("4m")
|
||||
absent = [e for e in evs if e[0] in ("backup_target_absent", "storage_disconnected")]
|
||||
got = absent[-1][0] if absent else None
|
||||
sev = absent[-1][1] if absent else None
|
||||
rec("I1" if expect_type == "backup_target_absent" else "I2",
|
||||
got == expect_type and sev == "error",
|
||||
"%s absent -> got=%s sev=%s (want %s/error)" % (label, got, sev, expect_type))
|
||||
|
||||
# I3: an unusable namespace must not read as bound
|
||||
for d in disks():
|
||||
if d.get("guest_path", "") and label in str(d.get("guest_path", "")):
|
||||
rec("I3", d.get("bound_under_parent") is False,
|
||||
"%s bound_under_parent=%s while absent" % (label, d.get("bound_under_parent")))
|
||||
|
||||
L.hp("qm set 311 --%s %s" % (disk, vol))
|
||||
settle()
|
||||
evs2 = hub_events("4m")
|
||||
back = [e for e in evs2 if e[0] in ("backup_target_restored", "storage_reconnected")]
|
||||
got2 = back[-1][0] if back else None
|
||||
sev2 = back[-1][1] if back else None
|
||||
rec("I1-pair" if expect_type == "backup_target_absent" else "I2-pair",
|
||||
got2 == expect_recover and sev2 == "info",
|
||||
"%s return -> got=%s sev=%s (want %s/info)" % (label, got2, sev2, expect_recover))
|
||||
|
||||
# I4: the bind repairs WITHOUT the guest restarting
|
||||
after_boot = L.guest("stat -c %Y /proc/1")[1].strip()
|
||||
rec("I4", before_boot == after_boot and before_boot != "",
|
||||
"guest init boot before=%s after=%s (must be equal)" % (before_boot, after_boot))
|
||||
|
||||
|
||||
def atom_detach_target():
|
||||
_detach(TARGET_DISK, TARGET_VOL, "mentes", "backup_target_absent", "backup_target_restored")
|
||||
|
||||
|
||||
def atom_detach_nontarget():
|
||||
_detach(NONTGT_DISK, NONTGT_VOL, "adatok", "storage_disconnected", "storage_reconnected")
|
||||
|
||||
|
||||
def atom_reboot_guest():
|
||||
L.box("pct reboot 9201", timeout=300)
|
||||
for _ in range(40):
|
||||
time.sleep(15)
|
||||
if "felhom-controller" in L.containers():
|
||||
break
|
||||
rec("REBOOT", "felhom-controller" in L.containers(), "controller back after guest reboot")
|
||||
|
||||
|
||||
def atom_kill_controller():
|
||||
L.guest("docker restart felhom-controller >/dev/null 2>&1 &")
|
||||
time.sleep(45)
|
||||
rec("KILL-CTRL", "felhom-controller" in L.containers(), "controller recovered after kill")
|
||||
|
||||
|
||||
def atom_redeploy_app():
|
||||
app = random.choice(["homebox", "grafana"])
|
||||
L.api("POST", "/api/stacks/%s/remove" % app, {})
|
||||
time.sleep(20)
|
||||
sec = L.SECRETS.get(app, {}).get("values", {})
|
||||
if sec:
|
||||
L.api("POST", "/api/stacks/%s/deploy" % app, {"values": sec})
|
||||
for _ in range(20):
|
||||
time.sleep(15)
|
||||
if app in L.containers():
|
||||
break
|
||||
rec("REDEPLOY", app in L.containers(), "%s redeployed" % app)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ per-cycle invariants
|
||||
|
||||
def check_healthy_baseline():
|
||||
"""I5/I6 — the over-correction check: healthy must render nothing."""
|
||||
ts = target_state()
|
||||
degraded = ts.get("degraded")
|
||||
evs = [e for e in hub_events("3m") if e[0] in ("backup_target_absent", "storage_disconnected")]
|
||||
rec("I5", degraded is False and not evs,
|
||||
"degraded=%s absent_events=%d target=%s" % (degraded, len(evs), ts.get("target")))
|
||||
if degraded is False:
|
||||
rec("I6", "message" not in ts or not ts.get("message"),
|
||||
"healthy state must carry no degraded message; got=%s" % str(ts.get("message"))[:120])
|
||||
|
||||
|
||||
UNIT = "/mnt/sys_drive/felhom-data/backups/primary/%s/compose/app.yaml"
|
||||
|
||||
def check_secret_class_travel():
|
||||
"""I10 — `type: secret` travels to the local unit; `type: password` NEVER does.
|
||||
Asserted in BOTH directions: the withheld class staying withheld is the half
|
||||
that silently rots. Key NAMES only; no value is ever read."""
|
||||
for app, meta in L.SECRETS.items():
|
||||
classes = meta.get("secret_classes", {})
|
||||
if not classes:
|
||||
continue
|
||||
rc, keys, _ = L.guest("grep -oE '^ *[A-Z_0-9]+:' " + (UNIT % app) +
|
||||
" 2>/dev/null | tr -d ' :' | tr '\\n' ' '")
|
||||
present = set(keys.split())
|
||||
if not present:
|
||||
rec("I10", False, "%s: recovery unit unreadable/absent" % app)
|
||||
continue
|
||||
for ev, c in classes.items():
|
||||
if c["type"] == "secret":
|
||||
rec("I10", ev in present, "%s/%s type=secret must travel; in_unit=%s"
|
||||
% (app, ev, ev in present))
|
||||
elif c["type"] == "password":
|
||||
rec("I10", ev not in present, "%s/%s type=password must NEVER travel; in_unit=%s"
|
||||
% (app, ev, ev in present))
|
||||
|
||||
|
||||
def check_secrets_absent():
|
||||
"""I11 — no secret value in any log, event or hub payload. Breach STOPS the run."""
|
||||
vals = [v for a in L.SECRETS.values() for k, v in a.get("values", {}).items()
|
||||
if k not in ("DOMAIN", "SUBDOMAIN") and v and len(v) > 8]
|
||||
hay = L.guest("docker logs felhom-controller --since 20m 2>&1 | tail -c 200000")[1]
|
||||
p = subprocess.run(["sudo", "kubectl", "-n", "felhom-system", "logs", "deploy/hub", "--since=20m"],
|
||||
capture_output=True, text=True, timeout=120)
|
||||
hay += p.stdout
|
||||
leaked = [v[:6] + "..." for v in vals if v in hay]
|
||||
rec("I11", not leaked, "leaked=%s" % leaked)
|
||||
return not leaked
|
||||
|
||||
|
||||
def watchdog():
|
||||
rc, out, _ = L.hp("df --output=avail -BG /mnt/nvme-1tb | tail -1")
|
||||
try:
|
||||
free = int(out.strip().rstrip("G"))
|
||||
except Exception:
|
||||
free = 9999
|
||||
rc2, vm, _ = L.hp("qm status 311")
|
||||
alive = "running" in vm
|
||||
if free < FLOOR_GB:
|
||||
log("WATCHDOG: free %dG < floor %dG — halting cleanly" % (free, FLOOR_GB)); return False
|
||||
if not alive:
|
||||
log("WATCHDOG: VM 311 not running (%s) — halting cleanly" % vm.strip()); return False
|
||||
return True
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ main
|
||||
|
||||
ATOMS = [atom_backup, atom_restore_verify, atom_detach_target,
|
||||
atom_detach_nontarget, atom_kill_controller, atom_redeploy_app]
|
||||
|
||||
def main():
|
||||
global CYCLE
|
||||
log("=== Campaign 10 Phase B start — max %d cycles, deadline %s ==="
|
||||
% (MAX_CYCLES, time.strftime("%H:%M", time.localtime(DEADLINE))))
|
||||
if not os.path.exists(JOURNAL):
|
||||
with open(JOURNAL, "w") as fh:
|
||||
fh.write("cycle\tutc\tinvariant\tverdict\tdetail\n")
|
||||
rng = random.Random(20260801)
|
||||
while CYCLE < MAX_CYCLES and time.time() < DEADLINE:
|
||||
CYCLE += 1
|
||||
heartbeat("start")
|
||||
if not watchdog():
|
||||
break
|
||||
order = ATOMS[:]
|
||||
rng.shuffle(order) # permutation, not a fixed catalogue
|
||||
# a reboot is expensive; fold it in every 7th cycle only
|
||||
if CYCLE % 7 == 0:
|
||||
order.append(atom_reboot_guest)
|
||||
log("cycle %d: %s" % (CYCLE, " -> ".join(a.__name__.replace("atom_", "") for a in order)))
|
||||
for a in order:
|
||||
heartbeat(a.__name__)
|
||||
try:
|
||||
a()
|
||||
except Exception as e:
|
||||
rec(a.__name__, False, "EXCEPTION %s" % e)
|
||||
log(" atom %s raised: %s" % (a.__name__, traceback.format_exc().splitlines()[-1]))
|
||||
heartbeat("invariants")
|
||||
try:
|
||||
check_healthy_baseline()
|
||||
check_secret_class_travel()
|
||||
if not check_secrets_absent():
|
||||
log("I11 BREACH — stopping the run immediately per brief B2"); break
|
||||
except Exception as e:
|
||||
log("invariant pass raised: %s" % e)
|
||||
log("cycle %d done — violations so far: %d" % (CYCLE, len(VIOL)))
|
||||
heartbeat("finished")
|
||||
log("=== finished after %d cycles, %d violations ===" % (CYCLE, len(VIOL)))
|
||||
for c, i, d in VIOL:
|
||||
log(" cycle %d %s: %s" % (c, i, d))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,92 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Campaign 10 — deploy apps through the REAL endpoints.
|
||||
|
||||
Mirrors the controller's own generator semantics (internal/stacks/deploy.go:870 generateValue):
|
||||
password:N -> N random alphanumerics
|
||||
hex:N -> N random BYTES, hex-encoded (2N chars)
|
||||
base64key:N-> "base64:" + base64(N random bytes)
|
||||
static:X -> X
|
||||
Runs on DooPlex; talks to the guest via ssh -J demo-hp + pct exec + /root/c10api.sh.
|
||||
Secrets are written to a 0600 manifest, never printed.
|
||||
"""
|
||||
import base64, json, os, secrets, string, subprocess, sys
|
||||
|
||||
APPS = sys.argv[1:] or ["rallly", "homebox", "grafana", "papra"]
|
||||
DOMAIN = "c10.felhom.eu"
|
||||
HOME = os.path.expanduser("~/.config/campaign10")
|
||||
HOSTPW = json.load(open(os.path.join(HOME, "host-recovery.json")))["password"]
|
||||
|
||||
|
||||
def ssh(cmd, timeout=900):
|
||||
full = ["sshpass", "-e", "ssh", "-o", "StrictHostKeyChecking=no",
|
||||
"-o", "ConnectTimeout=25", "-J", "demo-hp", "root@192.168.0.105",
|
||||
"LC_ALL=C " + cmd]
|
||||
env = dict(os.environ, SSHPASS=HOSTPW, LC_ALL="C")
|
||||
return subprocess.run(full, capture_output=True, text=True, timeout=timeout, env=env).stdout
|
||||
|
||||
|
||||
def api(method, path, body=None):
|
||||
if body is None:
|
||||
return ssh(f"pct exec 9201 -- /root/c10api.sh {method} {path} 2>/dev/null")
|
||||
b64 = base64.b64encode(json.dumps(body).encode()).decode()
|
||||
# body travels base64 so no quoting/locale can mangle a secret
|
||||
return ssh("pct exec 9201 -- bash -c \"echo %s | base64 -d > /tmp/.b && "
|
||||
"/root/c10api.sh %s %s \\\"\\$(cat /tmp/.b)\\\" --json; shred -u /tmp/.b\" 2>/dev/null"
|
||||
% (b64, method, path))
|
||||
|
||||
|
||||
def gen(spec):
|
||||
kind, _, param = spec.partition(":")
|
||||
if kind == "password":
|
||||
al = string.ascii_letters + string.digits
|
||||
return "".join(secrets.choice(al) for _ in range(int(param)))
|
||||
if kind == "hex":
|
||||
return secrets.token_bytes(int(param)).hex()
|
||||
if kind == "base64key":
|
||||
return "base64:" + base64.b64encode(secrets.token_bytes(int(param))).decode()
|
||||
if kind == "static":
|
||||
return param
|
||||
raise ValueError("unknown generator %r" % spec)
|
||||
|
||||
|
||||
manifest_path = os.path.join(HOME, "app-secrets.json")
|
||||
manifest = json.load(open(manifest_path)) if os.path.exists(manifest_path) else {}
|
||||
|
||||
for app in APPS:
|
||||
raw = api("GET", f"/api/stacks/{app}/deploy-fields")
|
||||
try:
|
||||
meta = json.loads(raw[raw.index("{"):raw.rindex("}") + 1])["data"]["metadata"]
|
||||
except Exception as e:
|
||||
print(f"{app:12s} FIELDS-FAIL {e} :: {raw[:160]!r}")
|
||||
continue
|
||||
|
||||
values, classes = {}, {}
|
||||
for f in meta.get("deploy_fields", []):
|
||||
ev, typ, spec, dflt = f["env_var"], f.get("type", ""), f.get("generate", ""), f.get("default", "")
|
||||
if typ == "domain":
|
||||
values[ev] = DOMAIN
|
||||
elif spec:
|
||||
values[ev] = gen(spec)
|
||||
elif dflt:
|
||||
values[ev] = dflt
|
||||
elif f.get("required"):
|
||||
print(f"{app:12s} WARN required field {ev} has no default/generator")
|
||||
if typ in ("secret", "password"):
|
||||
classes[ev] = {"type": typ, "data_key": bool(f.get("data_key")), "label": f.get("label", "")}
|
||||
|
||||
manifest[app] = {"values": values, "secret_classes": classes,
|
||||
"subdomain": values.get("SUBDOMAIN", ""), "domain": DOMAIN}
|
||||
|
||||
out = api("POST", f"/api/stacks/{app}/deploy", {"values": values})
|
||||
ok = '"ok":true' in out
|
||||
print(f"{app:12s} deploy -> {'OK' if ok else 'FAIL'} "
|
||||
f"secret={sum(1 for c in classes.values() if c['type']=='secret')} "
|
||||
f"password={sum(1 for c in classes.values() if c['type']=='password')} "
|
||||
f"data_key={sum(1 for c in classes.values() if c['data_key'])}")
|
||||
if not ok:
|
||||
print(" ", out.strip()[:300])
|
||||
|
||||
with open(manifest_path, "w") as fh:
|
||||
json.dump(manifest, fh, indent=1)
|
||||
os.chmod(manifest_path, 0o600)
|
||||
print("\nsecrets + secret-class map written to", manifest_path, "(0600, not printed)")
|
||||
@@ -0,0 +1,62 @@
|
||||
#!/bin/bash
|
||||
# Campaign 10 — blind console driver for VM 311 on demo-hp.
|
||||
# snap <label> : screendump the VM console -> PNG on DooPlex (for visual Read)
|
||||
# key <k> [k...] : sendkey one or more keys (QEMU keynames), 0.25s apart
|
||||
# type <string> : type an ASCII string as sendkey events
|
||||
set -uo pipefail
|
||||
SCRATCH=/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0d68ad3e-5c39-4852-8e38-9c029dff2562/scratchpad
|
||||
VM=311
|
||||
H=demo-hp
|
||||
|
||||
mon() { ssh -o ConnectTimeout=10 "$H" "LC_ALL=C qm monitor $VM" <<<"$1" 2>/dev/null | grep -v '^QEMU\|^(qemu)' ; }
|
||||
|
||||
snap() {
|
||||
local label="${1:-x}"
|
||||
ssh -o ConnectTimeout=10 "$H" "LC_ALL=C qm monitor $VM" <<<"screendump /tmp/c10.ppm" >/dev/null 2>&1
|
||||
scp -q "$H:/tmp/c10.ppm" "$SCRATCH/c10-$label.ppm" 2>/dev/null || { echo "snap: scp failed"; return 1; }
|
||||
convert "$SCRATCH/c10-$label.ppm" "$SCRATCH/c10-$label.png" 2>/dev/null || { echo "snap: convert failed"; return 1; }
|
||||
echo "$SCRATCH/c10-$label.png"
|
||||
}
|
||||
|
||||
key() {
|
||||
local cmds=""
|
||||
for k in "$@"; do cmds+="sendkey $k
|
||||
"; done
|
||||
ssh -o ConnectTimeout=10 "$H" "LC_ALL=C qm monitor $VM" <<<"$cmds" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
# type: map ASCII -> qemu keynames, including shifted chars
|
||||
type_str() {
|
||||
local s="$1" cmds="" c
|
||||
local -A M=( [' ']='spc' ['-']='minus' ['.']='dot' ['/']='slash' ['=']='equal'
|
||||
[',']='comma' [';']='semicolon' ["'"]='apostrophe' ['\']='backslash'
|
||||
['[']='bracket_left' [']']='bracket_right' ['`']='grave_accent' )
|
||||
local -A SH=( ['_']='minus' ['+']='equal' ['?']='slash' [':']='semicolon' ['"']='apostrophe'
|
||||
['!']='1' ['@']='2' ['#']='3' ['$']='4' ['%']='5' ['^']='6' ['&']='7'
|
||||
['*']='8' ['(']='9' [')']='0' ['{']='bracket_left' ['}']='bracket_right'
|
||||
['|']='backslash' ['~']='grave_accent' ['<']='comma' ['>']='dot' )
|
||||
for (( i=0; i<${#s}; i++ )); do
|
||||
c="${s:$i:1}"
|
||||
if [[ -n "${SH[$c]:-}" ]]; then cmds+="sendkey shift-${SH[$c]}
|
||||
"
|
||||
elif [[ -n "${M[$c]:-}" ]]; then cmds+="sendkey ${M[$c]}
|
||||
"
|
||||
elif [[ "$c" =~ [A-Z] ]]; then cmds+="sendkey shift-$(tr '[:upper:]' '[:lower:]' <<<"$c")
|
||||
"
|
||||
elif [[ "$c" =~ [a-z0-9] ]]; then cmds+="sendkey $c
|
||||
"
|
||||
else echo "type_str: unmapped char '$c'" >&2; return 1
|
||||
fi
|
||||
done
|
||||
ssh -o ConnectTimeout=10 "$H" "LC_ALL=C qm monitor $VM" <<<"$cmds" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
case "${1:-}" in
|
||||
snap) shift; snap "$@" ;;
|
||||
key) shift; key "$@" ;;
|
||||
type) shift; type_str "$@" ;;
|
||||
# typefile: read the string from a file so a secret never lands in argv/ps
|
||||
typefile) shift; type_str "$(cat "$1")" ;;
|
||||
mon) shift; mon "$@" ;;
|
||||
*) echo "usage: vm.sh {snap <label>|key <k>...|type <str>|typefile <path>|mon <cmd>}"; exit 2 ;;
|
||||
esac
|
||||
Reference in New Issue
Block a user