admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Campaign 10 Phase B: 27 cycles, 586 invariant checks, 0 violations

Browse Source 
Ran the soak on the Phase A rig. Ended on its own deadline — no watchdog halt,
no atom exception, no I11 breach.

I1 28+28 pairs, I2 28+28 pairs, I3 56, I4 56, I5/I6 28 each, I7 28, I10 135,
I11 28. Zero violations. The row counts are themselves the no-silent-skip check:
I3/I4 twice per cycle (both drives), I10 = 5 secret-class fields x 27, REBOOT on
cycles 7/14/21 only.

I7 is the headline: 28 restores, 28 correct discriminators — never stale, never
empty. RTO (Tier 1, rallly, 66 MB): min 38.8s, median 42.0s, p90 42.5s, max
44.3s. That is the S band's lower end ONLY; the 5.5s spread over 28 runs says
fixed work dominates, so nothing extrapolates to M or L. RPO not measured.

Every atom and invariant was proven BY HAND before automation — the runner
asserts nothing that was not first observed live.

Caught a Phase A gap before starting: no app had HDD_PATH, so all data sat on the
system disk and I3 could never have fired. Deployed calibre-web onto adatok
first; otherwise the run would have produced 27 green cycles that tested nothing
cross-drive.

Investigated and DISPROVED a suspected defect (audit 5.2): /api/disks reports
state=attached for a physically absent drive, and intermediary.go:230 really does
compute presence from State=="attached". It is inert — planDriveGates only gates
paths under /mnt/felhom-drives/ and uses BoundUnderParent there, which was
correctly false. The gate fired; the storage page showed "Meghajtó leválasztva".
No R-n minted.

Honest gaps: 6 of ~12 atom families ran. Not run — Tier 3 (structurally
un-isolatable), abort-fs-in-place, kill-agent-mid-backup, hard-reset-mid-write,
reboot-VM, both concurrency atoms, fill-drive-near-full. I8 not checked, I9 not
automated (cited from the tester-gate run, not re-claimed). kill_controller is
NOT mid-backup and reboot_guest never interleaved with a detach. 27 cycles does
not answer the brief's question about drift at the thirty-eighth.

Teardown still OWED, including hub customer c10-soak (disposition: DELETE).
This commit is contained in:
authentik Default Admin
2026-08-01 15:45:13 +02:00
parent 4691aa1a35
commit 69f896d3cd
10 changed files with 1489 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md
+150 -31
View File
@@ -1,7 +1,8 @@
# CAMPAIGN 10 — two-storage adversarial soak (2026-07-31 → 08-01)
**Status: Phase A COMPLETE and gated. Phase B NOT STARTED — deliberately, see §8.**
**One new finding: R-156. One structural scope constraint: Tier 3 cannot be isolated (§3).**
**Status: Phase A COMPLETE and gated (2026-07-31). Phase B RUN 2026-08-01 — 27 cycles, 586 invariant
checks, ZERO violations.** **One new finding: R-156. One structural scope constraint: Tier 3 cannot be
isolated (§3). One suspected defect investigated and DISPROVED (§5.2).**
Evidence: `../tests/campaign10-evidence-2026-07-31/` — `state/phaseA-journal.md` (step-by-step),
`isolation/pbs-denial.txt`, `isolation/restic-denial.txt`, `r156-papra-volume.txt`.
@@ -120,7 +121,7 @@ campaign constraint, not a defect.
| **A2** claim | **PASS** | discriminator flipped `dashboard not yet claimed` → `authentication required`; hub: `[claim] customer c10-soak CLAIMED its dashboard` |
| **A3** enrol both drives, apps across them | **PASS** | both registered via the real endpoint; backup target healthy; 4 apps healthy |
| **A4** discriminators | **PASS** | all four apps seed and read back the same generation |
| **A5** budget + watchdog | **NOT DONE** | §8 |
| **A5** budget + watchdog | **PASS** (built for Phase B) | disk floor 120 G on `/mnt/nvme-1tb` + VM-liveness watchdog; heartbeat `status.txt` per phase; `journal.tsv` per check. Neither fired: the run ended on its deadline with 871 G free |
**A1 — 1.26.1 cannot auto-install, verified against the bytes** (not inferred from its manifest):
`auto-installer-mode.toml` 0 hits, `answer.toml` 0, `proxmox-auto-install` 0, exactly 2 interactive
@@ -188,6 +189,72 @@ false pass.
---
## 4b. Phase B — the soak (2026-08-01)
**27 cycles, 09:33–15:40 CEST, ended on its own deadline. No watchdog halt, no atom exception, no
I11 breach. 586 invariant rows, 0 violations.**
**Every atom and every invariant was proven BY HAND before it was automated** (§4c). That ordering is
the point: a runner written first and trusted after is exactly the rig the brief warns about.
**A gap in Phase A found and closed first.** None of the four apps had `HDD_PATH` — all their data sat
in Docker volumes on the **system** disk, so detaching a data drive was a no-op for apps and **I3 could
never have fired**. A soak run in that state would have produced 27 green cycles that tested nothing
about cross-drive behaviour. **calibre-web** was deployed onto `adatok`
(`/mnt/felhom-drives/adatok/userdata/media/books → /calibre-library`, confirmed from
`docker inspect`) before the run started.
### Atoms, permuted per cycle (seeded RNG, so the order is reproducible)
`backup` · `restore_verify` · `detach_target` · `detach_nontarget` · `kill_controller` ·
`redeploy_app`, plus `reboot_guest` on every 7th cycle.
### Results
| Invariant | Checks | Violations | What was actually asserted |
|---|---|---|---|
| **I1** + pair | 28 + 28 | **0** | target absent ⇒ `backup_target_absent`/**error**; return ⇒ `backup_target_restored`/**info** |
| **I2** + pair | 28 + 28 | **0** | non-target ⇒ **generic** `storage_disconnected`/error → `storage_reconnected`/info |
| **I3** | 56 | **0** | `bound_under_parent=false` while the namespace is unusable, both drives |
| **I4** | 56 | **0** | guest init boot-time identical across every bind repair |
| **I5** | 28 | **0** | healthy ⇒ zero absent events, target reads healthy |
| **I6** | 28 | **0** | healthy carries no degraded message |
| **I7** | 28 | **0** | **every restore returned the discriminator asked for — 28/28** |
| **I10** | 135 | **0** | 4 × `type: secret` present in the unit; 1 × `type: password` **absent**, asserted as absence |
| **I11** | 28 | **0** | no secret value in controller or hub logs |
| BACKUP / REDEPLOY / KILL-CTRL / REBOOT | 56 / 28 / 28 / 3 | **0** | atom outcomes |
The row counts are themselves a check that nothing was silently skipped: I3 and I4 at **56** (twice per
cycle — both drives), I10 at **135** (5 secret-class fields × 27), REBOOT at **3** (cycles 7/14/21).
Counts of 28 rather than 27 include the one-cycle smoke test, which wrote to the same journal and
restarts its numbering at 1 — the totals are right, the cycle *labels* collide.
**A clean invariant is a result**: I1's discrimination, I2's genericity, the pairing of both, I4's
no-restart repair and I7's discriminator held over **27 consecutive cycles** of detach, reattach,
backup, restore, controller kill, app redeploy and three guest reboots. Nothing drifted at iteration 38
because the run stopped at 27 — the tail beyond that is untested, not proven absent.
### Two behaviours worth recording, both observed repeatedly
- **The drive comes back on a different device node and the bind still heals.** `adatok` left as
`/dev/sdb` and returned as `/dev/sdd`; `bound_under_parent` went true again and the contents were
readable, with the guest's init boot-time unchanged. That is R-117's stale-device case, hit on every
detach cycle rather than once.
- **`health_degraded (warning)` accompanies the target-absent event** and clears on return, alongside
the specific `backup_target_*` pair.
## 4c. Primitives proven by hand before automation
| Step | Observed |
|---|---|
| backup | `POST /api/backup/run` → `Mentés elindítva`, settles in ~23–34 s |
| restore | `POST /backup/restore` (form) is **async**; polled on `/api/backup/restore-status`, never `IsRunning()` |
| I7, first proof | seeded A → backup → seeded B → restored A; rallly returned **A** while the other three still read **B** |
| I1 | `backup_target_absent (error) — A rendszermentés meghajtója nem érhető el: Mentes` |
| I2 | `storage_disconnected (error) — Meghajtó váratlanul leválasztva: Adatok` |
| I6 degraded copy | *"A rendszermentés meghajtója nem érhető el — amíg vissza nem csatlakoztatod, a teljes rendszermentés nem készül el."* |
| I10 | rallly `DB_PASSWORD`+`SECRET_PASSWORD`, homebox `HBOX_AUTH_API_KEY_PEPPER`, papra `AUTH_SECRET` present; grafana `GF_SECURITY_ADMIN_PASSWORD` **absent**; unit mode `0600` |
## 5. Findings
### R-156 — papra's data is neither persisted nor backed up, and it reports healthy
@@ -223,55 +290,107 @@ defence is a catalog gate asserting each template's mounted volume is a path the
**Not fixed here** — the fences forbid it, and a fix mid-run would prove a version that did not exist
when the run started.
### 5.2 — Investigated and DISPROVED: `/api/disks` reports `state=attached` for an absent drive
Recorded because the *chase* is the useful artefact, not because anything is broken.
With `adatok` physically removed, `/api/disks` kept reporting `state: "attached"` for it across three
polls (~90 s) — while `/dev/sdb` was gone, `/mnt/adatok` was absent from `/proc/mounts`, and the guest
bind returned `Input/output error`. The other fields were right (`backing_device: ""`,
`guest_attached: false`, `bound_under_parent: false`).
It looked like the R-116/R-117 family, and there is a real consumer:
`internal/web/intermediary.go:230` — `present[d.MountPath] |= d.State == "attached"`. A presence gate
reading an absent drive as present is precisely I3's failure mode.
**It is inert, and the gate is correct.** `planDriveGates` only gates paths under
`StableParentDir+"/"` (`/mnt/felhom-drives/…`). The registered path is the **stable** one, whose
presence is computed from `d.BoundUnderParent` — observed **false** — not from `State`. The
`State`-derived entry is keyed on the **raw** `/mnt/adatok`, which is not a registered stable path, so
the loop skips it. The observable end-to-end behaviour agrees: the storage page showed
*"Meghajtó leválasztva: Adatok"* with a disconnect timestamp and a reconnect action, i.e. the gate fired.
The stale `state` field comes from the **registry** row surviving in the union after the
storage-observation row drops out (a raw enrolled drive is not a PVE storage), carrying its configured
`MountPath`. **No `R-n` minted** — no consumer is misled, and the R-116 comment's claim that this
"cannot make the gate read an absent drive as PRESENT" held under test.
### Not filed
- **demo-hp SSH key** — already R-129.
- **Tier 3 isolation** — §3; two recorded deliberate positions, not a defect.
- **`/api/disks` `state`** — §5.2, disproved.
---
## 6. RTO/RPO
## 6. RTO/RPO — measured
**No measurements.** They are a byproduct of Phase B restores, which did not run. Nothing is
extrapolated; no band is claimed.
**28 restores, every one returning the correct discriminator.** Wall-clock is measured from the restore
request to **the app serving the correct data** (the canary read back over rallly's real network path),
not to "restore returned".
| Tier | App | Data volume | n | min | median | p90 | max | mean |
|---|---|---|---|---|---|---|---|---|
| **Tier 1 (local)** | rallly (postgres) | **66 MB** volume / 67 MB recovery unit | 28 | 38.8 s | **42.0 s** | 42.5 s | 44.3 s | 42.0 s |
**Which band this covers: the S band only, and only its lower end.** 66 MB is a small app by any
reading, and the spread is tight — 5.5 s between min and max over 28 runs — which says the cost is
dominated by fixed work (stack stop, volume restore, stack start, health wait) rather than by bytes.
**Nothing here licenses an extrapolation to M or L**: the byte-proportional term is precisely what
66 MB fails to exercise. A second point at a much larger volume is needed before any curve is claimed,
and none was taken.
**RPO is NOT measured.** It is a function of backup *frequency*, and the soak drove backups on demand
rather than on the schedule, so the observed intervals say nothing about the product's RPO.
---
## 7. I1–I11
## 7. I1–I11 across cycles
**Not measured across cycles.** Only the healthy baseline (I5/I6) and the discriminator mechanism (I7)
were established, as Phase A gates. A clean invariant is only a result with a cycle count behind it,
and there is no count to report.
Full table in §4b. **I1, I2 (and both pairs), I3, I4, I5, I6, I7, I10 and I11 held over 27 consecutive
cycles — 586 checks, 0 violations.** No violation was deterministic and none was racy, because there
were none; the honest statement is that the run found no drift **within 27 cycles**, not that none
exists beyond them.
**I8 was not checked** — Tier 3 never ran (§3). **I9 was not automated** — see §8.
---
## 8. What did not run, and why
**Phase B did not start.** Phase A was budgeted at ~1 h by the brief and took ~5.5 h, almost entirely
in places the brief could not have predicted:
Phase A overran badly (~5.5 h against the brief's ~1 h) — 1.26.1 is a public release image with no
auto-install path, so day-0 was a blind screendump-and-`sendkey` walk; the Tier-3 question (§3) had to
be settled before a customer could exist at all, since `dr_tier` is fixed at creation; and a false
start on drive enrolment had to be backed out. **Phase B was therefore deliberately not launched at
04:00** — the brief's own fence, *"a test rig producing false negatives is worse than no rig"* — and
was built and run the next morning instead, where its first cycle could be watched.
- ISO **1.26.1** turned out to be a public release image with no auto-install path, so the install was
a blind screendump-and-`sendkey` walk (~1.5 h) rather than an unattended boot.
- The Tier-3 investigation (§3) had to be settled before a customer could be created at all, because
`dr_tier` is decided at customer-creation time.
- The enrol false start, and the CSRF/session/Bearer probing needed to drive the real endpoints.
**Atom coverage: 6 of the brief's ~12 suggested families.** Stated plainly rather than implied:
That put the runner — the most correctness-sensitive component, the thing that decides destructive
atoms unattended and judges eleven invariants — at ~04:00 with ~3 h of usable night. **I stopped
rather than write it there.** The brief's own fence is the reason: *"A test rig producing false
negatives is worse than no rig."* A soak whose judgements cannot be trusted would not just waste the
night, it would contaminate the register with findings nobody could rely on.
| Atom (brief B1) | Ran |
|---|---|
| back up Tier 1/2 · restore + verify by discriminator · detach/reattach **target** · detach/reattach **non-target** · delete an app and redeploy · reboot the guest | **YES** |
| back up / restore **Tier 3** | **NO** — §3, structurally impossible to isolate |
| abort a filesystem **in place** without the device disappearing (R-117 Q7) | **NO** |
| kill the **agent** mid-backup · **hard-reset the VM** mid-write · reboot the **VM** | **NO** |
| **concurrent** backup+restore · **concurrent** backup+detach | **NO** |
| fill a drive to near-full and continue | **NO** |
**Not run:** every B1 atom; I1–I11 across cycles; §B3 RTO/RPO; A5's disk budget, watchdog and
heartbeat (A5 exists to bound Phase B and was not written in isolation from it).
**`kill_controller` is not "mid-backup".** It restarts the controller at a permuted point in the
cycle, so it sometimes lands after a backup and sometimes nowhere near one. It exercises recovery, not
the mid-write race the brief asked for. Recorded because the atom name would otherwise overclaim.
**What is ready.** The rig is built, gated and idle: VM 311 installed and claimed, two drives enrolled
with the backup target healthy, four apps healthy spanning both D5 secret classes, a working
discriminator across all four, an authenticated API path onto the controller, and both isolation
denials captured. The driver library (`c10lib.py`) already implements the parts that took the
longest — the jump-host command path, the authenticated API helper, canary seed/read, and the
red-proofed psql read path. **Phase B needs the atom set, the invariant checks and the watchdog on
top of that, then it can run in daylight where its first cycles can be watched.**
**`reboot_guest` was appended after the shuffle, not permuted into it**, so it always ran last in
cycles 7/14/21 and **never interleaved with a detach**. Reboot-during-detach is untested.
**I8** — not checked; Tier 3 never ran. **I9** (Tier-1/2 restore with the guest's `app.yaml` moved
aside, data read over the app's real network path) — **not automated**. It was proven on this exact
controller 0.188.0 by the tester-gate run hours earlier
(`tester-gate-golden-0.188.0-2026-07-31.md` §5, step 6), and that is cited rather than re-claimed:
this campaign did not execute it.
**The tail is untested, not clean.** 27 cycles is the count; the brief's interest in "the state that
quietly drifts on the thirty-eighth" is not answered by a 27-cycle run.
---