Campaign 10 Phase B: 27 cycles, 586 invariant checks, 0 violations
Ran the soak on the Phase A rig. Ended on its own deadline — no watchdog halt, no atom exception, no I11 breach. I1 28+28 pairs, I2 28+28 pairs, I3 56, I4 56, I5/I6 28 each, I7 28, I10 135, I11 28. Zero violations. The row counts are themselves the no-silent-skip check: I3/I4 twice per cycle (both drives), I10 = 5 secret-class fields x 27, REBOOT on cycles 7/14/21 only. I7 is the headline: 28 restores, 28 correct discriminators — never stale, never empty. RTO (Tier 1, rallly, 66 MB): min 38.8s, median 42.0s, p90 42.5s, max 44.3s. That is the S band's lower end ONLY; the 5.5s spread over 28 runs says fixed work dominates, so nothing extrapolates to M or L. RPO not measured. Every atom and invariant was proven BY HAND before automation — the runner asserts nothing that was not first observed live. Caught a Phase A gap before starting: no app had HDD_PATH, so all data sat on the system disk and I3 could never have fired. Deployed calibre-web onto adatok first; otherwise the run would have produced 27 green cycles that tested nothing cross-drive. Investigated and DISPROVED a suspected defect (audit 5.2): /api/disks reports state=attached for a physically absent drive, and intermediary.go:230 really does compute presence from State=="attached". It is inert — planDriveGates only gates paths under /mnt/felhom-drives/ and uses BoundUnderParent there, which was correctly false. The gate fired; the storage page showed "Meghajtó leválasztva". No R-n minted. Honest gaps: 6 of ~12 atom families ran. Not run — Tier 3 (structurally un-isolatable), abort-fs-in-place, kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency atoms, fill-drive-near-full. I8 not checked, I9 not automated (cited from the tester-gate run, not re-claimed). kill_controller is NOT mid-backup and reboot_guest never interleaved with a detach. 27 cycles does not answer the brief's question about drift at the thirty-eighth. Teardown still OWED, including hub customer c10-soak (disposition: DELETE).
This commit is contained in:
@@ -0,0 +1,141 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Campaign 10 — shared driver library. Runs on DooPlex, NOT inside the VM:
|
||||
the soak's atoms kill the controller, the agent and the VM itself, so anything
|
||||
running inside would die with the thing it is testing.
|
||||
|
||||
Canary model (brief A4): every app carries a marker naming the backup GENERATION
|
||||
its data came from. Without it, a restore that returns stale or empty data is
|
||||
indistinguishable from a good one — this arc produced six checks that passed for
|
||||
the wrong reason.
|
||||
|
||||
rallly -> a row in a cc_proof TABLE in postgres, read over the app's real
|
||||
network path (docker run --network container:rallly ... psql -h
|
||||
rallly-postgres). NOT 127.0.0.1 inside the postgres container,
|
||||
which postgres trusts and which produced D5's false pass.
|
||||
others -> a marker file inside the app's own docker volume (what gets dumped).
|
||||
"""
|
||||
import json, os, subprocess, time
|
||||
|
||||
HOME = os.path.expanduser("~/.config/campaign10")
|
||||
VM_IP = "192.168.0.105"
|
||||
JUMP = "demo-hp"
|
||||
VMID = "311"
|
||||
GUEST = "9201"
|
||||
|
||||
HOSTPW = json.load(open(os.path.join(HOME, "host-recovery.json")))["password"]
|
||||
SECRETS = json.load(open(os.path.join(HOME, "app-secrets.json")))
|
||||
|
||||
# app -> (container, volume mount dir) for the file-marker apps
|
||||
FILE_APPS = {
|
||||
"grafana": ("grafana", "/var/lib/grafana"),
|
||||
"homebox": ("homebox", "/data"),
|
||||
"papra": ("papra", "/app/data"),
|
||||
}
|
||||
ALL_APPS = ["rallly", "homebox", "grafana", "papra"]
|
||||
|
||||
|
||||
def _run(args, timeout=300):
|
||||
env = dict(os.environ, SSHPASS=HOSTPW, LC_ALL="C")
|
||||
try:
|
||||
p = subprocess.run(args, capture_output=True, text=True, timeout=timeout, env=env)
|
||||
return p.returncode, p.stdout, p.stderr
|
||||
except subprocess.TimeoutExpired:
|
||||
return 124, "", "TIMEOUT"
|
||||
|
||||
|
||||
def box(cmd, timeout=300):
|
||||
"""Run a command on the campaign PVE host (the VM), via the demo-hp jump."""
|
||||
return _run(["sshpass", "-e", "ssh", "-o", "StrictHostKeyChecking=no",
|
||||
"-o", "ConnectTimeout=20", "-J", JUMP, f"root@{VM_IP}",
|
||||
"LC_ALL=C " + cmd], timeout)
|
||||
|
||||
|
||||
def guest(cmd, timeout=300):
|
||||
"""Run a command inside guest 9201."""
|
||||
return box(f"pct exec {GUEST} -- bash -c {shq(cmd)}", timeout)
|
||||
|
||||
|
||||
def hp(cmd, timeout=180):
|
||||
"""Run a command on demo-hp itself (for qm: reset, stop, disk detach...)."""
|
||||
return _run(["ssh", "-o", "ConnectTimeout=20", JUMP, "LC_ALL=C " + cmd], timeout)
|
||||
|
||||
|
||||
def shq(s):
|
||||
return "'" + s.replace("'", "'\\''") + "'"
|
||||
|
||||
|
||||
def api(method, path, body=None, timeout=300):
|
||||
"""Authenticated controller API call through the on-guest helper."""
|
||||
if body is None:
|
||||
rc, out, _ = guest(f"/root/c10api.sh {method} {path} 2>/dev/null", timeout)
|
||||
return out
|
||||
import base64
|
||||
b64 = base64.b64encode(json.dumps(body).encode()).decode()
|
||||
rc, out, _ = guest(
|
||||
f"echo {b64} | base64 -d > /tmp/.b && /root/c10api.sh {method} {path} "
|
||||
f'"$(cat /tmp/.b)" --json; shred -u /tmp/.b 2>/dev/null', timeout)
|
||||
return out
|
||||
|
||||
|
||||
def jparse(raw):
|
||||
try:
|
||||
return json.loads(raw[raw.index("{"):raw.rindex("}") + 1])
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- canaries
|
||||
|
||||
def _psql(sql, timeout=180):
|
||||
"""Read/write rallly's DB over the path DATABASE_URL actually names."""
|
||||
pw = SECRETS["rallly"]["values"]["DB_PASSWORD"]
|
||||
url = f"postgresql://rallly:{pw}@rallly-postgres:5432/rallly"
|
||||
cmd = ("docker run --rm --network container:rallly postgres:16-alpine "
|
||||
f"psql {shq(url)} -tAc {shq(sql)}")
|
||||
rc, out, err = guest(cmd, timeout)
|
||||
return rc, out.strip(), err.strip()
|
||||
|
||||
|
||||
def seed(gen):
|
||||
"""Write the generation marker into every app. Returns {app: ok}."""
|
||||
res = {}
|
||||
_psql("CREATE TABLE IF NOT EXISTS cc_proof (id serial primary key, gen text, at timestamptz default now())")
|
||||
rc, out, err = _psql(f"INSERT INTO cc_proof (gen) VALUES ('{gen}') RETURNING gen")
|
||||
res["rallly"] = (gen in out)
|
||||
for app, (cont, d) in FILE_APPS.items():
|
||||
# -u 0: papra's volume is root-owned while the container runs as nonroot (R-156),
|
||||
# so the app itself cannot write there. Seeding as root keeps the canary in the
|
||||
# volume the BACKUP dumps, which is what I7 needs to observe.
|
||||
rc, out, err = guest(
|
||||
f"docker exec -u 0 {cont} sh -c {shq(f'echo {gen} > {d}/cc_proof.txt && cat {d}/cc_proof.txt')}", 180)
|
||||
res[app] = (gen in out)
|
||||
return res
|
||||
|
||||
|
||||
def read_canaries():
|
||||
"""Read each app's marker back. Returns {app: value-or-None}."""
|
||||
res = {}
|
||||
rc, out, err = _psql("SELECT gen FROM cc_proof ORDER BY id DESC LIMIT 1")
|
||||
res["rallly"] = out.strip() or None
|
||||
for app, (cont, d) in FILE_APPS.items():
|
||||
rc, out, err = guest(f"docker exec {cont} sh -c {shq(f'cat {d}/cc_proof.txt 2>/dev/null')}", 180)
|
||||
v = out.strip()
|
||||
res[app] = v or None
|
||||
return res
|
||||
|
||||
|
||||
def containers():
|
||||
rc, out, _ = guest("docker ps --format '{{.Names}}|{{.Status}}'", 120)
|
||||
d = {}
|
||||
for line in out.strip().splitlines():
|
||||
if "|" in line:
|
||||
n, s = line.split("|", 1)
|
||||
d[n.strip()] = s.strip()
|
||||
return d
|
||||
|
||||
|
||||
def guest_init_pid():
|
||||
"""I4: a stale-device bind must repair WITHOUT the guest restarting."""
|
||||
rc, out, _ = box(f"pct exec {GUEST} -- cat /proc/1/stat 2>/dev/null", 120)
|
||||
rc2, boot, _ = box(f"pct exec {GUEST} -- stat -c %Y /proc/1 2>/dev/null", 120)
|
||||
return boot.strip()
|
||||
Reference in New Issue
Block a user