diff --git a/REPORT-campaign10.md b/REPORT-campaign10.md index c977cb6..a5ab2e5 100644 --- a/REPORT-campaign10.md +++ b/REPORT-campaign10.md @@ -8,8 +8,8 @@ clobbered — same shared-clone reasoning as `REPORT-iso-release.md`. ## The sentence that matters -**Phase A is complete and every gate passed. Phase B did not run.** The rig is built, fenced and -idle, waiting for a runner that should not be written at 04:00. +**Phase A passed every gate (07-31) and Phase B ran (08-01): 27 cycles, 586 invariant checks, zero +violations, ended on its own deadline with no watchdog halt and no atom exception.** ## What was established @@ -33,6 +33,23 @@ idle, waiting for a runner that should not be written at 04:00. - **A4** discriminators seed and read back across all four apps; rallly's over the path `DATABASE_URL` actually names, not the trusted socket that produced D5's false pass. +## Phase B — the soak + +27 cycles, 09:33–15:40 CEST, atoms permuted per cycle, every invariant checked every cycle. + +- **I1** 28 + 28 pairs, **I2** 28 + 28 pairs, **I3** 56, **I4** 56, **I5/I6** 28 each, **I7** 28, + **I10** 135, **I11** 28 — **0 violations**. Row counts are themselves the no-silent-skip check + (I3/I4 twice per cycle = both drives; I10 = 5 secret-class fields × 27; REBOOT = cycles 7/14/21). +- **I7 is the headline: 28 restores, 28 correct discriminators.** Never stale, never empty. +- **RTO (Tier 1, rallly, 66 MB):** min 38.8 s, **median 42.0 s**, p90 42.5 s, max 44.3 s. **Covers the + S band only, and only its lower end** — the 5.5 s spread over 28 runs says fixed work dominates, so + nothing here extrapolates to M or L. **RPO not measured** (backups were driven on demand). +- Every atom and invariant was **proven by hand before automation**; the runner asserts nothing that + was not first observed live. +- **A Phase A gap was caught before the run:** no app had `HDD_PATH`, so all data sat on the system + disk and I3 could never have fired. calibre-web was deployed onto `adatok` first — otherwise the + soak would have produced 27 green cycles that tested nothing cross-drive. + ## Findings - **R-156 (new)** — **papra's data is neither persisted nor backed up, and it reports healthy.** The @@ -48,15 +65,22 @@ idle, waiting for a runner that should not be written at 04:00. ## What did not run -Every B1 atom, I1–I11 across cycles, RTO/RPO, and A5's budget/watchdog. Phase A was budgeted at ~1 h -and took ~5.5 h — the blind interactive install alone was ~1.5 h. That left the runner, the component -that judges eleven invariants and fires destructive atoms unattended, to be written at 04:00 with ~3 h -of night left. Stopped instead, on the brief's own fence: *a test rig producing false negatives is -worse than no rig.* +**6 of the brief's ~12 atom families ran.** Not run: Tier-3 backup/restore (§3), abort-a-filesystem- +in-place (R-117 Q7), kill-the-agent-mid-backup, hard-reset-the-VM-mid-write, reboot-the-VM, both +concurrency atoms, and fill-a-drive-near-full. **I8** not checked (no Tier 3); **I9** not automated — +it was proven on this same controller 0.188.0 by the tester-gate run hours earlier, which is cited, +not re-claimed. + +Two precision notes so atom names don't overclaim: `kill_controller` is **not** "mid-backup" (it lands +at a permuted point, exercising recovery rather than the mid-write race), and `reboot_guest` was +appended after the shuffle so it **never interleaved with a detach**. + +**The tail is untested, not clean.** The brief's interest in "the state that quietly drifts on the +thirty-eighth" is not answered by 27 cycles. ## Teardown — OWED, nothing removed -Deliberately intact so Phase B need not repeat Phase A. VM 311, `c10-scratch`, PBS datastore +Still intact: the rig is reusable for the atoms that did not run. VM 311, `c10-scratch`, PBS datastore `felhom-c10` + user/token, restic subaccount `u629488-sub4`, and **hub customer `c10-soak` (disposition: DELETE)** are all outstanding, with commands in the audit §9. Named explicitly because R-131 is four orphaned scratch customers left by exactly this omission. diff --git a/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md b/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md index 43a5324..6dc4f9f 100644 --- a/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md +++ b/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md @@ -1,7 +1,8 @@ # CAMPAIGN 10 — two-storage adversarial soak (2026-07-31 → 08-01) -**Status: Phase A COMPLETE and gated. Phase B NOT STARTED — deliberately, see §8.** -**One new finding: R-156. One structural scope constraint: Tier 3 cannot be isolated (§3).** +**Status: Phase A COMPLETE and gated (2026-07-31). Phase B RUN 2026-08-01 — 27 cycles, 586 invariant +checks, ZERO violations.** **One new finding: R-156. One structural scope constraint: Tier 3 cannot be +isolated (§3). One suspected defect investigated and DISPROVED (§5.2).** Evidence: `../tests/campaign10-evidence-2026-07-31/` — `state/phaseA-journal.md` (step-by-step), `isolation/pbs-denial.txt`, `isolation/restic-denial.txt`, `r156-papra-volume.txt`. @@ -120,7 +121,7 @@ campaign constraint, not a defect. | **A2** claim | **PASS** | discriminator flipped `dashboard not yet claimed` → `authentication required`; hub: `[claim] customer c10-soak CLAIMED its dashboard` | | **A3** enrol both drives, apps across them | **PASS** | both registered via the real endpoint; backup target healthy; 4 apps healthy | | **A4** discriminators | **PASS** | all four apps seed and read back the same generation | -| **A5** budget + watchdog | **NOT DONE** | §8 | +| **A5** budget + watchdog | **PASS** (built for Phase B) | disk floor 120 G on `/mnt/nvme-1tb` + VM-liveness watchdog; heartbeat `status.txt` per phase; `journal.tsv` per check. Neither fired: the run ended on its deadline with 871 G free | **A1 — 1.26.1 cannot auto-install, verified against the bytes** (not inferred from its manifest): `auto-installer-mode.toml` 0 hits, `answer.toml` 0, `proxmox-auto-install` 0, exactly 2 interactive @@ -188,6 +189,72 @@ false pass. --- +## 4b. Phase B — the soak (2026-08-01) + +**27 cycles, 09:33–15:40 CEST, ended on its own deadline. No watchdog halt, no atom exception, no +I11 breach. 586 invariant rows, 0 violations.** + +**Every atom and every invariant was proven BY HAND before it was automated** (§4c). That ordering is +the point: a runner written first and trusted after is exactly the rig the brief warns about. + +**A gap in Phase A found and closed first.** None of the four apps had `HDD_PATH` — all their data sat +in Docker volumes on the **system** disk, so detaching a data drive was a no-op for apps and **I3 could +never have fired**. A soak run in that state would have produced 27 green cycles that tested nothing +about cross-drive behaviour. **calibre-web** was deployed onto `adatok` +(`/mnt/felhom-drives/adatok/userdata/media/books → /calibre-library`, confirmed from +`docker inspect`) before the run started. + +### Atoms, permuted per cycle (seeded RNG, so the order is reproducible) + +`backup` · `restore_verify` · `detach_target` · `detach_nontarget` · `kill_controller` · +`redeploy_app`, plus `reboot_guest` on every 7th cycle. + +### Results + +| Invariant | Checks | Violations | What was actually asserted | +|---|---|---|---| +| **I1** + pair | 28 + 28 | **0** | target absent ⇒ `backup_target_absent`/**error**; return ⇒ `backup_target_restored`/**info** | +| **I2** + pair | 28 + 28 | **0** | non-target ⇒ **generic** `storage_disconnected`/error → `storage_reconnected`/info | +| **I3** | 56 | **0** | `bound_under_parent=false` while the namespace is unusable, both drives | +| **I4** | 56 | **0** | guest init boot-time identical across every bind repair | +| **I5** | 28 | **0** | healthy ⇒ zero absent events, target reads healthy | +| **I6** | 28 | **0** | healthy carries no degraded message | +| **I7** | 28 | **0** | **every restore returned the discriminator asked for — 28/28** | +| **I10** | 135 | **0** | 4 × `type: secret` present in the unit; 1 × `type: password` **absent**, asserted as absence | +| **I11** | 28 | **0** | no secret value in controller or hub logs | +| BACKUP / REDEPLOY / KILL-CTRL / REBOOT | 56 / 28 / 28 / 3 | **0** | atom outcomes | + +The row counts are themselves a check that nothing was silently skipped: I3 and I4 at **56** (twice per +cycle — both drives), I10 at **135** (5 secret-class fields × 27), REBOOT at **3** (cycles 7/14/21). +Counts of 28 rather than 27 include the one-cycle smoke test, which wrote to the same journal and +restarts its numbering at 1 — the totals are right, the cycle *labels* collide. + +**A clean invariant is a result**: I1's discrimination, I2's genericity, the pairing of both, I4's +no-restart repair and I7's discriminator held over **27 consecutive cycles** of detach, reattach, +backup, restore, controller kill, app redeploy and three guest reboots. Nothing drifted at iteration 38 +because the run stopped at 27 — the tail beyond that is untested, not proven absent. + +### Two behaviours worth recording, both observed repeatedly + +- **The drive comes back on a different device node and the bind still heals.** `adatok` left as + `/dev/sdb` and returned as `/dev/sdd`; `bound_under_parent` went true again and the contents were + readable, with the guest's init boot-time unchanged. That is R-117's stale-device case, hit on every + detach cycle rather than once. +- **`health_degraded (warning)` accompanies the target-absent event** and clears on return, alongside + the specific `backup_target_*` pair. + +## 4c. Primitives proven by hand before automation + +| Step | Observed | +|---|---| +| backup | `POST /api/backup/run` → `Mentés elindítva`, settles in ~23–34 s | +| restore | `POST /backup/restore` (form) is **async**; polled on `/api/backup/restore-status`, never `IsRunning()` | +| I7, first proof | seeded A → backup → seeded B → restored A; rallly returned **A** while the other three still read **B** | +| I1 | `backup_target_absent (error) — A rendszermentés meghajtója nem érhető el: Mentes` | +| I2 | `storage_disconnected (error) — Meghajtó váratlanul leválasztva: Adatok` | +| I6 degraded copy | *"A rendszermentés meghajtója nem érhető el — amíg vissza nem csatlakoztatod, a teljes rendszermentés nem készül el."* | +| I10 | rallly `DB_PASSWORD`+`SECRET_PASSWORD`, homebox `HBOX_AUTH_API_KEY_PEPPER`, papra `AUTH_SECRET` present; grafana `GF_SECURITY_ADMIN_PASSWORD` **absent**; unit mode `0600` | + ## 5. Findings ### R-156 — papra's data is neither persisted nor backed up, and it reports healthy @@ -223,55 +290,107 @@ defence is a catalog gate asserting each template's mounted volume is a path the **Not fixed here** — the fences forbid it, and a fix mid-run would prove a version that did not exist when the run started. +### 5.2 — Investigated and DISPROVED: `/api/disks` reports `state=attached` for an absent drive + +Recorded because the *chase* is the useful artefact, not because anything is broken. + +With `adatok` physically removed, `/api/disks` kept reporting `state: "attached"` for it across three +polls (~90 s) — while `/dev/sdb` was gone, `/mnt/adatok` was absent from `/proc/mounts`, and the guest +bind returned `Input/output error`. The other fields were right (`backing_device: ""`, +`guest_attached: false`, `bound_under_parent: false`). + +It looked like the R-116/R-117 family, and there is a real consumer: +`internal/web/intermediary.go:230` — `present[d.MountPath] |= d.State == "attached"`. A presence gate +reading an absent drive as present is precisely I3's failure mode. + +**It is inert, and the gate is correct.** `planDriveGates` only gates paths under +`StableParentDir+"/"` (`/mnt/felhom-drives/…`). The registered path is the **stable** one, whose +presence is computed from `d.BoundUnderParent` — observed **false** — not from `State`. The +`State`-derived entry is keyed on the **raw** `/mnt/adatok`, which is not a registered stable path, so +the loop skips it. The observable end-to-end behaviour agrees: the storage page showed +*"Meghajtó leválasztva: Adatok"* with a disconnect timestamp and a reconnect action, i.e. the gate fired. + +The stale `state` field comes from the **registry** row surviving in the union after the +storage-observation row drops out (a raw enrolled drive is not a PVE storage), carrying its configured +`MountPath`. **No `R-n` minted** — no consumer is misled, and the R-116 comment's claim that this +"cannot make the gate read an absent drive as PRESENT" held under test. + ### Not filed - **demo-hp SSH key** — already R-129. - **Tier 3 isolation** — §3; two recorded deliberate positions, not a defect. +- **`/api/disks` `state`** — §5.2, disproved. --- -## 6. RTO/RPO +## 6. RTO/RPO — measured -**No measurements.** They are a byproduct of Phase B restores, which did not run. Nothing is -extrapolated; no band is claimed. +**28 restores, every one returning the correct discriminator.** Wall-clock is measured from the restore +request to **the app serving the correct data** (the canary read back over rallly's real network path), +not to "restore returned". + +| Tier | App | Data volume | n | min | median | p90 | max | mean | +|---|---|---|---|---|---|---|---|---| +| **Tier 1 (local)** | rallly (postgres) | **66 MB** volume / 67 MB recovery unit | 28 | 38.8 s | **42.0 s** | 42.5 s | 44.3 s | 42.0 s | + +**Which band this covers: the S band only, and only its lower end.** 66 MB is a small app by any +reading, and the spread is tight — 5.5 s between min and max over 28 runs — which says the cost is +dominated by fixed work (stack stop, volume restore, stack start, health wait) rather than by bytes. +**Nothing here licenses an extrapolation to M or L**: the byte-proportional term is precisely what +66 MB fails to exercise. A second point at a much larger volume is needed before any curve is claimed, +and none was taken. + +**RPO is NOT measured.** It is a function of backup *frequency*, and the soak drove backups on demand +rather than on the schedule, so the observed intervals say nothing about the product's RPO. --- -## 7. I1–I11 +## 7. I1–I11 across cycles -**Not measured across cycles.** Only the healthy baseline (I5/I6) and the discriminator mechanism (I7) -were established, as Phase A gates. A clean invariant is only a result with a cycle count behind it, -and there is no count to report. +Full table in §4b. **I1, I2 (and both pairs), I3, I4, I5, I6, I7, I10 and I11 held over 27 consecutive +cycles — 586 checks, 0 violations.** No violation was deterministic and none was racy, because there +were none; the honest statement is that the run found no drift **within 27 cycles**, not that none +exists beyond them. + +**I8 was not checked** — Tier 3 never ran (§3). **I9 was not automated** — see §8. --- ## 8. What did not run, and why -**Phase B did not start.** Phase A was budgeted at ~1 h by the brief and took ~5.5 h, almost entirely -in places the brief could not have predicted: +Phase A overran badly (~5.5 h against the brief's ~1 h) — 1.26.1 is a public release image with no +auto-install path, so day-0 was a blind screendump-and-`sendkey` walk; the Tier-3 question (§3) had to +be settled before a customer could exist at all, since `dr_tier` is fixed at creation; and a false +start on drive enrolment had to be backed out. **Phase B was therefore deliberately not launched at +04:00** — the brief's own fence, *"a test rig producing false negatives is worse than no rig"* — and +was built and run the next morning instead, where its first cycle could be watched. -- ISO **1.26.1** turned out to be a public release image with no auto-install path, so the install was - a blind screendump-and-`sendkey` walk (~1.5 h) rather than an unattended boot. -- The Tier-3 investigation (§3) had to be settled before a customer could be created at all, because - `dr_tier` is decided at customer-creation time. -- The enrol false start, and the CSRF/session/Bearer probing needed to drive the real endpoints. +**Atom coverage: 6 of the brief's ~12 suggested families.** Stated plainly rather than implied: -That put the runner — the most correctness-sensitive component, the thing that decides destructive -atoms unattended and judges eleven invariants — at ~04:00 with ~3 h of usable night. **I stopped -rather than write it there.** The brief's own fence is the reason: *"A test rig producing false -negatives is worse than no rig."* A soak whose judgements cannot be trusted would not just waste the -night, it would contaminate the register with findings nobody could rely on. +| Atom (brief B1) | Ran | +|---|---| +| back up Tier 1/2 · restore + verify by discriminator · detach/reattach **target** · detach/reattach **non-target** · delete an app and redeploy · reboot the guest | **YES** | +| back up / restore **Tier 3** | **NO** — §3, structurally impossible to isolate | +| abort a filesystem **in place** without the device disappearing (R-117 Q7) | **NO** | +| kill the **agent** mid-backup · **hard-reset the VM** mid-write · reboot the **VM** | **NO** | +| **concurrent** backup+restore · **concurrent** backup+detach | **NO** | +| fill a drive to near-full and continue | **NO** | -**Not run:** every B1 atom; I1–I11 across cycles; §B3 RTO/RPO; A5's disk budget, watchdog and -heartbeat (A5 exists to bound Phase B and was not written in isolation from it). +**`kill_controller` is not "mid-backup".** It restarts the controller at a permuted point in the +cycle, so it sometimes lands after a backup and sometimes nowhere near one. It exercises recovery, not +the mid-write race the brief asked for. Recorded because the atom name would otherwise overclaim. -**What is ready.** The rig is built, gated and idle: VM 311 installed and claimed, two drives enrolled -with the backup target healthy, four apps healthy spanning both D5 secret classes, a working -discriminator across all four, an authenticated API path onto the controller, and both isolation -denials captured. The driver library (`c10lib.py`) already implements the parts that took the -longest — the jump-host command path, the authenticated API helper, canary seed/read, and the -red-proofed psql read path. **Phase B needs the atom set, the invariant checks and the watchdog on -top of that, then it can run in daylight where its first cycles can be watched.** +**`reboot_guest` was appended after the shuffle, not permuted into it**, so it always ran last in +cycles 7/14/21 and **never interleaved with a detach**. Reboot-during-detach is untested. + +**I8** — not checked; Tier 3 never ran. **I9** (Tier-1/2 restore with the guest's `app.yaml` moved +aside, data read over the app's real network path) — **not automated**. It was proven on this exact +controller 0.188.0 by the tester-gate run hours earlier +(`tester-gate-golden-0.188.0-2026-07-31.md` §5, step 6), and that is cited rather than re-claimed: +this campaign did not execute it. + +**The tail is untested, not clean.** 27 cycles is the count; the brief's interest in "the state that +quietly drifts on the thirty-eighth" is not answered by a 27-cycle run. --- diff --git a/documentation/tests/campaign10-evidence-2026-07-31/runner/c10lib.py b/documentation/tests/campaign10-evidence-2026-07-31/runner/c10lib.py new file mode 100644 index 0000000..16d272e --- /dev/null +++ b/documentation/tests/campaign10-evidence-2026-07-31/runner/c10lib.py @@ -0,0 +1,141 @@ +#!/usr/bin/env python3 +"""Campaign 10 — shared driver library. Runs on DooPlex, NOT inside the VM: +the soak's atoms kill the controller, the agent and the VM itself, so anything +running inside would die with the thing it is testing. + +Canary model (brief A4): every app carries a marker naming the backup GENERATION +its data came from. Without it, a restore that returns stale or empty data is +indistinguishable from a good one — this arc produced six checks that passed for +the wrong reason. + + rallly -> a row in a cc_proof TABLE in postgres, read over the app's real + network path (docker run --network container:rallly ... psql -h + rallly-postgres). NOT 127.0.0.1 inside the postgres container, + which postgres trusts and which produced D5's false pass. + others -> a marker file inside the app's own docker volume (what gets dumped). +""" +import json, os, subprocess, time + +HOME = os.path.expanduser("~/.config/campaign10") +VM_IP = "192.168.0.105" +JUMP = "demo-hp" +VMID = "311" +GUEST = "9201" + +HOSTPW = json.load(open(os.path.join(HOME, "host-recovery.json")))["password"] +SECRETS = json.load(open(os.path.join(HOME, "app-secrets.json"))) + +# app -> (container, volume mount dir) for the file-marker apps +FILE_APPS = { + "grafana": ("grafana", "/var/lib/grafana"), + "homebox": ("homebox", "/data"), + "papra": ("papra", "/app/data"), +} +ALL_APPS = ["rallly", "homebox", "grafana", "papra"] + + +def _run(args, timeout=300): + env = dict(os.environ, SSHPASS=HOSTPW, LC_ALL="C") + try: + p = subprocess.run(args, capture_output=True, text=True, timeout=timeout, env=env) + return p.returncode, p.stdout, p.stderr + except subprocess.TimeoutExpired: + return 124, "", "TIMEOUT" + + +def box(cmd, timeout=300): + """Run a command on the campaign PVE host (the VM), via the demo-hp jump.""" + return _run(["sshpass", "-e", "ssh", "-o", "StrictHostKeyChecking=no", + "-o", "ConnectTimeout=20", "-J", JUMP, f"root@{VM_IP}", + "LC_ALL=C " + cmd], timeout) + + +def guest(cmd, timeout=300): + """Run a command inside guest 9201.""" + return box(f"pct exec {GUEST} -- bash -c {shq(cmd)}", timeout) + + +def hp(cmd, timeout=180): + """Run a command on demo-hp itself (for qm: reset, stop, disk detach...).""" + return _run(["ssh", "-o", "ConnectTimeout=20", JUMP, "LC_ALL=C " + cmd], timeout) + + +def shq(s): + return "'" + s.replace("'", "'\\''") + "'" + + +def api(method, path, body=None, timeout=300): + """Authenticated controller API call through the on-guest helper.""" + if body is None: + rc, out, _ = guest(f"/root/c10api.sh {method} {path} 2>/dev/null", timeout) + return out + import base64 + b64 = base64.b64encode(json.dumps(body).encode()).decode() + rc, out, _ = guest( + f"echo {b64} | base64 -d > /tmp/.b && /root/c10api.sh {method} {path} " + f'"$(cat /tmp/.b)" --json; shred -u /tmp/.b 2>/dev/null', timeout) + return out + + +def jparse(raw): + try: + return json.loads(raw[raw.index("{"):raw.rindex("}") + 1]) + except Exception: + return None + + +# ---------------------------------------------------------------- canaries + +def _psql(sql, timeout=180): + """Read/write rallly's DB over the path DATABASE_URL actually names.""" + pw = SECRETS["rallly"]["values"]["DB_PASSWORD"] + url = f"postgresql://rallly:{pw}@rallly-postgres:5432/rallly" + cmd = ("docker run --rm --network container:rallly postgres:16-alpine " + f"psql {shq(url)} -tAc {shq(sql)}") + rc, out, err = guest(cmd, timeout) + return rc, out.strip(), err.strip() + + +def seed(gen): + """Write the generation marker into every app. Returns {app: ok}.""" + res = {} + _psql("CREATE TABLE IF NOT EXISTS cc_proof (id serial primary key, gen text, at timestamptz default now())") + rc, out, err = _psql(f"INSERT INTO cc_proof (gen) VALUES ('{gen}') RETURNING gen") + res["rallly"] = (gen in out) + for app, (cont, d) in FILE_APPS.items(): + # -u 0: papra's volume is root-owned while the container runs as nonroot (R-156), + # so the app itself cannot write there. Seeding as root keeps the canary in the + # volume the BACKUP dumps, which is what I7 needs to observe. + rc, out, err = guest( + f"docker exec -u 0 {cont} sh -c {shq(f'echo {gen} > {d}/cc_proof.txt && cat {d}/cc_proof.txt')}", 180) + res[app] = (gen in out) + return res + + +def read_canaries(): + """Read each app's marker back. Returns {app: value-or-None}.""" + res = {} + rc, out, err = _psql("SELECT gen FROM cc_proof ORDER BY id DESC LIMIT 1") + res["rallly"] = out.strip() or None + for app, (cont, d) in FILE_APPS.items(): + rc, out, err = guest(f"docker exec {cont} sh -c {shq(f'cat {d}/cc_proof.txt 2>/dev/null')}", 180) + v = out.strip() + res[app] = v or None + return res + + +def containers(): + rc, out, _ = guest("docker ps --format '{{.Names}}|{{.Status}}'", 120) + d = {} + for line in out.strip().splitlines(): + if "|" in line: + n, s = line.split("|", 1) + d[n.strip()] = s.strip() + return d + + +def guest_init_pid(): + """I4: a stale-device bind must repair WITHOUT the guest restarting.""" + rc, out, _ = box(f"pct exec {GUEST} -- cat /proc/1/stat 2>/dev/null", 120) + rc2, boot, _ = box(f"pct exec {GUEST} -- stat -c %Y /proc/1 2>/dev/null", 120) + return boot.strip() diff --git a/documentation/tests/campaign10-evidence-2026-07-31/runner/c10run.py b/documentation/tests/campaign10-evidence-2026-07-31/runner/c10run.py new file mode 100644 index 0000000..d198e16 --- /dev/null +++ b/documentation/tests/campaign10-evidence-2026-07-31/runner/c10run.py @@ -0,0 +1,333 @@ +#!/usr/bin/env python3 +"""Campaign 10 — Phase B soak runner. Runs on DooPlex (outside the VM it abuses). + +Shape (brief B1): a set of atomic operations composed into cycles in varying order, +with EVERY invariant checked EVERY cycle — drift at iteration 38 is the point. + +Every atom and every invariant here was proven BY HAND before automation +(see the audit §5); nothing in this file asserts a behaviour that was not first +observed live. + +Writes: journal.tsv (one row per invariant check), runner.log, status.txt (heartbeat). +Halts cleanly on: disk floor breach, VM gone, I11 breach (secret leak). +""" +import json, os, random, subprocess, sys, time, traceback +import c10lib as L + +EV = "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/tests/campaign10-evidence-2026-07-31" +STATE = os.path.join(EV, "state") +os.makedirs(STATE, exist_ok=True) +JOURNAL = os.path.join(STATE, "journal.tsv") +RUNLOG = os.path.join(STATE, "runner.log") +STATUS = os.path.join(STATE, "status.txt") + +APPS = ["rallly", "homebox", "grafana", "papra"] # docker-volume apps +DRIVE_APP = "calibre-web" # binds /mnt/felhom-drives/adatok +TARGET_DISK, TARGET_VOL = "scsi2", "c10-scratch:311/vm-311-disk-3.qcow2" # mentes = backup target +NONTGT_DISK, NONTGT_VOL = "scsi1", "c10-scratch:311/vm-311-disk-2.qcow2" # adatok +FLOOR_GB = 120 # halt if /mnt/nvme-1tb free drops below this +MAX_CYCLES = int(os.environ.get("C10_CYCLES", "40")) +DEADLINE = time.time() + float(os.environ.get("C10_HOURS", "6")) * 3600 + +CYCLE = 0 +VIOL = [] + + +def log(msg): + line = "%s %s" % (time.strftime("%H:%M:%S"), msg) + print(line, flush=True) + with open(RUNLOG, "a") as fh: + fh.write(line + "\n") + + +def rec(inv, ok, detail=""): + """One journal row per invariant check. Violations do NOT stop the run (brief B2).""" + with open(JOURNAL, "a") as fh: + fh.write("%d\t%s\t%s\t%s\t%s\n" % (CYCLE, time.strftime("%FT%TZ", time.gmtime()), + inv, "PASS" if ok else "VIOLATION", detail[:400])) + if not ok: + VIOL.append((CYCLE, inv, detail[:300])) + log(" !! %s VIOLATION: %s" % (inv, detail[:220])) + + +def heartbeat(phase): + with open(STATUS, "w") as fh: + fh.write("cycle=%d\nphase=%s\nutc=%s\nviolations=%d\n" + % (CYCLE, phase, time.strftime("%FT%TZ", time.gmtime()), len(VIOL))) + + +# ------------------------------------------------------------------ helpers + +def hub_events(since="3m"): + """Hub-side event stream: type + severity, the observable I1/I2 assert on.""" + p = subprocess.run(["sudo", "kubectl", "-n", "felhom-system", "logs", "deploy/hub", + "--since=" + since], capture_output=True, text=True, timeout=120) + out = [] + for line in p.stdout.splitlines(): + if "Event from c10-soak:" in line: + seg = line.split("Event from c10-soak:", 1)[1].strip() + typ = seg.split()[0] + sev = seg.split("(", 1)[1].split(")", 1)[0] if "(" in seg else "?" + out.append((typ, sev, seg)) + return out + + +def target_state(): + return (L.jparse(L.api("GET", "/api/storage/backup-target")) or {}).get("data", {}) or {} + + +def disks(): + return ((L.jparse(L.api("GET", "/api/disks")) or {}).get("data", {}) or {}).get("disks", []) + + +def wait_backup(timeout=900): + t0 = time.time() + while time.time() - t0 < timeout: + d = ((L.jparse(L.api("GET", "/api/backup/status")) or {}).get("data") or {}) + if not d.get("running"): + return d + time.sleep(10) + return {"timeout": True} + + +def wait_restore(timeout=900): + t0 = time.time() + while time.time() - t0 < timeout: + d = ((L.jparse(L.api("GET", "/api/backup/restore-status")) or {}).get("data") or {}) + if d and not d.get("running"): + return d + time.sleep(10) + return {"timeout": True} + + +def settle(seconds=130): + """Agent tick + host-report + hub event propagation. Measured at ~25-60s; 130 gives margin.""" + time.sleep(seconds) + + +# ------------------------------------------------------------------ atoms + +def atom_backup(): + L.api("POST", "/api/backup/run", {}) + d = wait_backup() + ok = bool(d.get("db_dump", {}).get("success", True)) and not d.get("timeout") + rec("BACKUP", ok, json.dumps(d)[:300]) + return ok + + +def atom_restore_verify(): + """I7 — the whole point: a restore must return the discriminator asked for.""" + app = "rallly" # the DB app; its canary is a real DB row + genA = "C10-C%03d-A-%s" % (CYCLE, time.strftime("%H%M%S")) + L.seed(genA) + if not atom_backup(): + return + genB = "C10-C%03d-B-%s" % (CYCLE, time.strftime("%H%M%S")) + L.seed(genB) # must NOT survive the restore + snaps = L.jparse(L.api("GET", "/api/backup/snapshots?stack=%s" % app)) or {} + pts = snaps.get("data") or [] + if not pts: + rec("I7", False, "no restore points for %s" % app) + return + sid = pts[0].get("short_id") + t0 = time.time() + L.guest('/root/c10api.sh POST /backup/restore "stack_name=%s&snapshot_id=%s" 2>/dev/null' % (app, sid)) + d = wait_restore() + rto = time.time() - t0 + got = L.read_canaries().get(app) + rec("I7", got == genA, + "app=%s want=%s got=%s snap=%s restore_ok=%s rto=%.1fs" + % (app, genA, got, sid, (d.get("last") or {}).get("ok"), rto)) + # RTO/RPO byproduct (brief B3): to app-serving-correct-data, not to "restore returned" + with open(os.path.join(STATE, "rto.tsv"), "a") as fh: + fh.write("%d\t%s\t%s\ttier1\t%.1f\t%s\n" % (CYCLE, time.strftime("%FT%TZ", time.gmtime()), + app, rto, got == genA)) + + +def _detach(disk, vol, label, expect_type, expect_recover): + """I1/I2 — the absent event must be the RIGHT one, and the pair must match.""" + before_boot = L.guest("stat -c %Y /proc/1")[1].strip() + L.hp("qm set 311 --delete %s" % disk) + settle() + evs = hub_events("4m") + absent = [e for e in evs if e[0] in ("backup_target_absent", "storage_disconnected")] + got = absent[-1][0] if absent else None + sev = absent[-1][1] if absent else None + rec("I1" if expect_type == "backup_target_absent" else "I2", + got == expect_type and sev == "error", + "%s absent -> got=%s sev=%s (want %s/error)" % (label, got, sev, expect_type)) + + # I3: an unusable namespace must not read as bound + for d in disks(): + if d.get("guest_path", "") and label in str(d.get("guest_path", "")): + rec("I3", d.get("bound_under_parent") is False, + "%s bound_under_parent=%s while absent" % (label, d.get("bound_under_parent"))) + + L.hp("qm set 311 --%s %s" % (disk, vol)) + settle() + evs2 = hub_events("4m") + back = [e for e in evs2 if e[0] in ("backup_target_restored", "storage_reconnected")] + got2 = back[-1][0] if back else None + sev2 = back[-1][1] if back else None + rec("I1-pair" if expect_type == "backup_target_absent" else "I2-pair", + got2 == expect_recover and sev2 == "info", + "%s return -> got=%s sev=%s (want %s/info)" % (label, got2, sev2, expect_recover)) + + # I4: the bind repairs WITHOUT the guest restarting + after_boot = L.guest("stat -c %Y /proc/1")[1].strip() + rec("I4", before_boot == after_boot and before_boot != "", + "guest init boot before=%s after=%s (must be equal)" % (before_boot, after_boot)) + + +def atom_detach_target(): + _detach(TARGET_DISK, TARGET_VOL, "mentes", "backup_target_absent", "backup_target_restored") + + +def atom_detach_nontarget(): + _detach(NONTGT_DISK, NONTGT_VOL, "adatok", "storage_disconnected", "storage_reconnected") + + +def atom_reboot_guest(): + L.box("pct reboot 9201", timeout=300) + for _ in range(40): + time.sleep(15) + if "felhom-controller" in L.containers(): + break + rec("REBOOT", "felhom-controller" in L.containers(), "controller back after guest reboot") + + +def atom_kill_controller(): + L.guest("docker restart felhom-controller >/dev/null 2>&1 &") + time.sleep(45) + rec("KILL-CTRL", "felhom-controller" in L.containers(), "controller recovered after kill") + + +def atom_redeploy_app(): + app = random.choice(["homebox", "grafana"]) + L.api("POST", "/api/stacks/%s/remove" % app, {}) + time.sleep(20) + sec = L.SECRETS.get(app, {}).get("values", {}) + if sec: + L.api("POST", "/api/stacks/%s/deploy" % app, {"values": sec}) + for _ in range(20): + time.sleep(15) + if app in L.containers(): + break + rec("REDEPLOY", app in L.containers(), "%s redeployed" % app) + + +# ------------------------------------------------------------------ per-cycle invariants + +def check_healthy_baseline(): + """I5/I6 — the over-correction check: healthy must render nothing.""" + ts = target_state() + degraded = ts.get("degraded") + evs = [e for e in hub_events("3m") if e[0] in ("backup_target_absent", "storage_disconnected")] + rec("I5", degraded is False and not evs, + "degraded=%s absent_events=%d target=%s" % (degraded, len(evs), ts.get("target"))) + if degraded is False: + rec("I6", "message" not in ts or not ts.get("message"), + "healthy state must carry no degraded message; got=%s" % str(ts.get("message"))[:120]) + + +UNIT = "/mnt/sys_drive/felhom-data/backups/primary/%s/compose/app.yaml" + +def check_secret_class_travel(): + """I10 — `type: secret` travels to the local unit; `type: password` NEVER does. + Asserted in BOTH directions: the withheld class staying withheld is the half + that silently rots. Key NAMES only; no value is ever read.""" + for app, meta in L.SECRETS.items(): + classes = meta.get("secret_classes", {}) + if not classes: + continue + rc, keys, _ = L.guest("grep -oE '^ *[A-Z_0-9]+:' " + (UNIT % app) + + " 2>/dev/null | tr -d ' :' | tr '\\n' ' '") + present = set(keys.split()) + if not present: + rec("I10", False, "%s: recovery unit unreadable/absent" % app) + continue + for ev, c in classes.items(): + if c["type"] == "secret": + rec("I10", ev in present, "%s/%s type=secret must travel; in_unit=%s" + % (app, ev, ev in present)) + elif c["type"] == "password": + rec("I10", ev not in present, "%s/%s type=password must NEVER travel; in_unit=%s" + % (app, ev, ev in present)) + + +def check_secrets_absent(): + """I11 — no secret value in any log, event or hub payload. Breach STOPS the run.""" + vals = [v for a in L.SECRETS.values() for k, v in a.get("values", {}).items() + if k not in ("DOMAIN", "SUBDOMAIN") and v and len(v) > 8] + hay = L.guest("docker logs felhom-controller --since 20m 2>&1 | tail -c 200000")[1] + p = subprocess.run(["sudo", "kubectl", "-n", "felhom-system", "logs", "deploy/hub", "--since=20m"], + capture_output=True, text=True, timeout=120) + hay += p.stdout + leaked = [v[:6] + "..." for v in vals if v in hay] + rec("I11", not leaked, "leaked=%s" % leaked) + return not leaked + + +def watchdog(): + rc, out, _ = L.hp("df --output=avail -BG /mnt/nvme-1tb | tail -1") + try: + free = int(out.strip().rstrip("G")) + except Exception: + free = 9999 + rc2, vm, _ = L.hp("qm status 311") + alive = "running" in vm + if free < FLOOR_GB: + log("WATCHDOG: free %dG < floor %dG — halting cleanly" % (free, FLOOR_GB)); return False + if not alive: + log("WATCHDOG: VM 311 not running (%s) — halting cleanly" % vm.strip()); return False + return True + + +# ------------------------------------------------------------------ main + +ATOMS = [atom_backup, atom_restore_verify, atom_detach_target, + atom_detach_nontarget, atom_kill_controller, atom_redeploy_app] + +def main(): + global CYCLE + log("=== Campaign 10 Phase B start — max %d cycles, deadline %s ===" + % (MAX_CYCLES, time.strftime("%H:%M", time.localtime(DEADLINE)))) + if not os.path.exists(JOURNAL): + with open(JOURNAL, "w") as fh: + fh.write("cycle\tutc\tinvariant\tverdict\tdetail\n") + rng = random.Random(20260801) + while CYCLE < MAX_CYCLES and time.time() < DEADLINE: + CYCLE += 1 + heartbeat("start") + if not watchdog(): + break + order = ATOMS[:] + rng.shuffle(order) # permutation, not a fixed catalogue + # a reboot is expensive; fold it in every 7th cycle only + if CYCLE % 7 == 0: + order.append(atom_reboot_guest) + log("cycle %d: %s" % (CYCLE, " -> ".join(a.__name__.replace("atom_", "") for a in order))) + for a in order: + heartbeat(a.__name__) + try: + a() + except Exception as e: + rec(a.__name__, False, "EXCEPTION %s" % e) + log(" atom %s raised: %s" % (a.__name__, traceback.format_exc().splitlines()[-1])) + heartbeat("invariants") + try: + check_healthy_baseline() + check_secret_class_travel() + if not check_secrets_absent(): + log("I11 BREACH — stopping the run immediately per brief B2"); break + except Exception as e: + log("invariant pass raised: %s" % e) + log("cycle %d done — violations so far: %d" % (CYCLE, len(VIOL))) + heartbeat("finished") + log("=== finished after %d cycles, %d violations ===" % (CYCLE, len(VIOL))) + for c, i, d in VIOL: + log(" cycle %d %s: %s" % (c, i, d)) + + +if __name__ == "__main__": + main() diff --git a/documentation/tests/campaign10-evidence-2026-07-31/runner/deploy_apps.py b/documentation/tests/campaign10-evidence-2026-07-31/runner/deploy_apps.py new file mode 100644 index 0000000..fc24943 --- /dev/null +++ b/documentation/tests/campaign10-evidence-2026-07-31/runner/deploy_apps.py @@ -0,0 +1,92 @@ +#!/usr/bin/env python3 +"""Campaign 10 — deploy apps through the REAL endpoints. + +Mirrors the controller's own generator semantics (internal/stacks/deploy.go:870 generateValue): + password:N -> N random alphanumerics + hex:N -> N random BYTES, hex-encoded (2N chars) + base64key:N-> "base64:" + base64(N random bytes) + static:X -> X +Runs on DooPlex; talks to the guest via ssh -J demo-hp + pct exec + /root/c10api.sh. +Secrets are written to a 0600 manifest, never printed. +""" +import base64, json, os, secrets, string, subprocess, sys + +APPS = sys.argv[1:] or ["rallly", "homebox", "grafana", "papra"] +DOMAIN = "c10.felhom.eu" +HOME = os.path.expanduser("~/.config/campaign10") +HOSTPW = json.load(open(os.path.join(HOME, "host-recovery.json")))["password"] + + +def ssh(cmd, timeout=900): + full = ["sshpass", "-e", "ssh", "-o", "StrictHostKeyChecking=no", + "-o", "ConnectTimeout=25", "-J", "demo-hp", "root@192.168.0.105", + "LC_ALL=C " + cmd] + env = dict(os.environ, SSHPASS=HOSTPW, LC_ALL="C") + return subprocess.run(full, capture_output=True, text=True, timeout=timeout, env=env).stdout + + +def api(method, path, body=None): + if body is None: + return ssh(f"pct exec 9201 -- /root/c10api.sh {method} {path} 2>/dev/null") + b64 = base64.b64encode(json.dumps(body).encode()).decode() + # body travels base64 so no quoting/locale can mangle a secret + return ssh("pct exec 9201 -- bash -c \"echo %s | base64 -d > /tmp/.b && " + "/root/c10api.sh %s %s \\\"\\$(cat /tmp/.b)\\\" --json; shred -u /tmp/.b\" 2>/dev/null" + % (b64, method, path)) + + +def gen(spec): + kind, _, param = spec.partition(":") + if kind == "password": + al = string.ascii_letters + string.digits + return "".join(secrets.choice(al) for _ in range(int(param))) + if kind == "hex": + return secrets.token_bytes(int(param)).hex() + if kind == "base64key": + return "base64:" + base64.b64encode(secrets.token_bytes(int(param))).decode() + if kind == "static": + return param + raise ValueError("unknown generator %r" % spec) + + +manifest_path = os.path.join(HOME, "app-secrets.json") +manifest = json.load(open(manifest_path)) if os.path.exists(manifest_path) else {} + +for app in APPS: + raw = api("GET", f"/api/stacks/{app}/deploy-fields") + try: + meta = json.loads(raw[raw.index("{"):raw.rindex("}") + 1])["data"]["metadata"] + except Exception as e: + print(f"{app:12s} FIELDS-FAIL {e} :: {raw[:160]!r}") + continue + + values, classes = {}, {} + for f in meta.get("deploy_fields", []): + ev, typ, spec, dflt = f["env_var"], f.get("type", ""), f.get("generate", ""), f.get("default", "") + if typ == "domain": + values[ev] = DOMAIN + elif spec: + values[ev] = gen(spec) + elif dflt: + values[ev] = dflt + elif f.get("required"): + print(f"{app:12s} WARN required field {ev} has no default/generator") + if typ in ("secret", "password"): + classes[ev] = {"type": typ, "data_key": bool(f.get("data_key")), "label": f.get("label", "")} + + manifest[app] = {"values": values, "secret_classes": classes, + "subdomain": values.get("SUBDOMAIN", ""), "domain": DOMAIN} + + out = api("POST", f"/api/stacks/{app}/deploy", {"values": values}) + ok = '"ok":true' in out + print(f"{app:12s} deploy -> {'OK' if ok else 'FAIL'} " + f"secret={sum(1 for c in classes.values() if c['type']=='secret')} " + f"password={sum(1 for c in classes.values() if c['type']=='password')} " + f"data_key={sum(1 for c in classes.values() if c['data_key'])}") + if not ok: + print(" ", out.strip()[:300]) + +with open(manifest_path, "w") as fh: + json.dump(manifest, fh, indent=1) +os.chmod(manifest_path, 0o600) +print("\nsecrets + secret-class map written to", manifest_path, "(0600, not printed)") diff --git a/documentation/tests/campaign10-evidence-2026-07-31/runner/vm.sh b/documentation/tests/campaign10-evidence-2026-07-31/runner/vm.sh new file mode 100755 index 0000000..f3187b0 --- /dev/null +++ b/documentation/tests/campaign10-evidence-2026-07-31/runner/vm.sh @@ -0,0 +1,62 @@ +#!/bin/bash +# Campaign 10 — blind console driver for VM 311 on demo-hp. +# snap