admin
cee8f70e98
gates / gates (push) Failing after 17s
Overnight soak 22:39->06:10 CEST. demo-hp the victim, demo-felhom the untouched observer.
No production code, no golden, no version bump. Report at
documentation/audits/DRILL-soak-2026-08-31/REPORT.md.
VERDICTS: 1 lock-collision FAIL, 2 guard-interactions PASS-with-one-defect, 3 R-357 PASS,
4 proof-edges PASS, 5 mutated-cycle PASS, 6 observer FAIL, 7 teardown PASS.
R-414 - THE MOST VALUABLE FINDING, AND ONLY AN UNTOUCHED BOX COULD HAVE FOUND IT. On
demo-felhom the nightly proof fired for the first time unattended at 05:30 and REFUSED:
"nowhere to restore to - nincs regisztralt adatmeghajto". Cause established, not inferred:
storage_paths is EMPTY, so there is no path to put a scratch on. It will fail this way every
night forever with only a WARN, and because the error path reaches no verdict,
last_proof_result stays ABSENT - which is also what a pre-0.231.0 controller sends. The hub
cannot tell "never ran" from "not deployed": the StatsKnown trap one level up. The box is
NOT unprotected; its off-site backup ran fine in 46.9s. It is the PROOF that cannot run.
R-411 - measured, not reasoned: restic stats TAKES A LOCK; a customer full-restore runs it
while holding no acquireRunning; the integrity check is therefore not blocked, meets that
lock and escalates to unlock --remove-all. The sampler caught "restore ..." and
"unlock --remove-all" in the SAME sample. Contained: the check was classified unreachable,
not damage, so no false alarm.
R-412 - CORRECTED from HIGH to LOW. I filed it on a mechanism I had not finished measuring.
The off-site run has its own pre-push dump leg, so a hollow unit is REPAIRED before it
ships - proven on two apps and confirmed by pulling the snapshot back out of the store.
What survives is a narrow race, plus a success line over a backup holding no data.
R-413 - the R-87 proof caught a product-produced hollow snapshot unattended, and the
nightly job fired on its own schedule at 05:30 for the first time (bentopdf PASSED on
9d002b38 in 2.315s). Both were listed "not yet live-validated" yesterday.
R-403 mirror guard PROVEN live, with a negative control: it fired when a unit was hollow
("The copy was PRESERVED rather than replaced with an empty one") and skipped 0 legs at
teardown when every unit was sound.
R-357 PASS at last, six days owed: a real full filesystem, refused BEFORE StopStack, app
never stopped, live data byte-identical, and it worked once the space came back.
Phase 4 built the false-alarm control the whole R-87 design rests on: bentopdf is the only
template of 53 with neither a database nor a named volume. It passes silently.
EIGHT of my own instrument errors are named in the report, each caught by its own control -
including a time guard that fired an injection four hours early, and filing R-412 at the
wrong severity.
Teardown clean on all three layers of both boxes; both healthy on 0.231.0.
OWED: a golden for 0.231.0, and a decision on keeping bentopdf.
52 lines
2.4 KiB
Markdown
52 lines
2.4 KiB
Markdown
# Phase 7 — restore the world. **PASS**
|
||
|
||
| layer | `demo-hp` | `demo-felhom` |
|
||
|---|---|---|
|
||
| PVE host `/root` | `.cache .ssh` only | soak files: 0 |
|
||
| guest `/root` | soak files: 0 | soak files: 0 |
|
||
| container `/tmp` | **empty** | n/a (never touched) |
|
||
| scratches (`offsite-restore`, `offsite-proof`) | **both empty** | n/a |
|
||
| ballast | 0 files; 949 GB free on `hdd_1` | n/a |
|
||
| controller | 0.231.0, healthy | 0.231.0, healthy |
|
||
| apps | **18 containers, all healthy**, none exited or restarting | all healthy |
|
||
|
||
Local credential copies `shred -u`'d.
|
||
|
||
## One clean cycle of each tier, checked for the RIGHT reason
|
||
|
||
- **Tier-2**: 3 apps copied, and **0 unit legs skipped** — the R-403 guard's *negative* control. It
|
||
fired earlier tonight when a unit was hollow and correctly does **not** fire now that every unit is
|
||
sound.
|
||
- **Proof**: `bookstack` → `pass`.
|
||
- **Off-site**: the real 04:15 run completed in 2 m 57 s. Not re-run.
|
||
|
||
## Every primary unit, final
|
||
|
||
```
|
||
bentopdf dumps=0 tars=0 <- correct: it legitimately has neither (the R-87 control)
|
||
bookstack dumps=1 tars=2 opengist dumps=0 tars=1
|
||
docmost dumps=1 tars=3 privatebin dumps=0 tars=1
|
||
kimai dumps=1 tars=2 calibre-web dumps=0 tars=1
|
||
paperless-ngx dumps=1 tars=3 romm dumps=1 tars=3
|
||
```
|
||
|
||
All Tier-2 copies present and sized as expected.
|
||
|
||
## Two dispositions stated rather than left silent
|
||
|
||
1. **`bentopdf` — KEEP** (recommendation). It is the only template of 53 with neither a database nor
|
||
a named volume, so it is the only possible live subject for R-87's false-alarm control. Removing it
|
||
removes the control. Viktor's call.
|
||
2. **`/mnt/sys_drive/felhom-data/backups/primary/paperless/`** — a stale, manifest-less directory for
|
||
a stack that is not deployed, dated **21–22 August**. **Not mine**, established by mtime after my
|
||
own baseline grep failed its control. Left in place: I did not create it and do not know why it is
|
||
there. Harmless (no manifest ⇒ treated as hollow ⇒ the R-403 guard protects any copy), but it reads
|
||
as an error in any unit sweep.
|
||
|
||
## The hub layer — the one that is invisible from the box
|
||
|
||
The hub's host page mentions `proof` **zero** times: the controller publishes `last_proof_*` and no
|
||
hub surface reads them. That is R-402's shape, already allowlisted with that reason in
|
||
`wire_contract_gate.py`. No scratch customers, appliances or configs were created tonight, so there is
|
||
nothing to discard there.
|