3 Commits

Author	SHA1	Message	Date
admin	cee8f70e98	soak drill COMPLETE: 7 phases, 4 findings, the observer found the worst one (R-414) ... Overnight soak 22:39->06:10 CEST. demo-hp the victim, demo-felhom the untouched observer. No production code, no golden, no version bump. Report at documentation/audits/DRILL-soak-2026-08-31/REPORT.md. VERDICTS: 1 lock-collision FAIL, 2 guard-interactions PASS-with-one-defect, 3 R-357 PASS, 4 proof-edges PASS, 5 mutated-cycle PASS, 6 observer FAIL, 7 teardown PASS. R-414 - THE MOST VALUABLE FINDING, AND ONLY AN UNTOUCHED BOX COULD HAVE FOUND IT. On demo-felhom the nightly proof fired for the first time unattended at 05:30 and REFUSED: "nowhere to restore to - nincs regisztralt adatmeghajto". Cause established, not inferred: storage_paths is EMPTY, so there is no path to put a scratch on. It will fail this way every night forever with only a WARN, and because the error path reaches no verdict, last_proof_result stays ABSENT - which is also what a pre-0.231.0 controller sends. The hub cannot tell "never ran" from "not deployed": the StatsKnown trap one level up. The box is NOT unprotected; its off-site backup ran fine in 46.9s. It is the PROOF that cannot run. R-411 - measured, not reasoned: restic stats TAKES A LOCK; a customer full-restore runs it while holding no acquireRunning; the integrity check is therefore not blocked, meets that lock and escalates to unlock --remove-all. The sampler caught "restore ..." and "unlock --remove-all" in the SAME sample. Contained: the check was classified unreachable, not damage, so no false alarm. R-412 - CORRECTED from HIGH to LOW. I filed it on a mechanism I had not finished measuring. The off-site run has its own pre-push dump leg, so a hollow unit is REPAIRED before it ships - proven on two apps and confirmed by pulling the snapshot back out of the store. What survives is a narrow race, plus a success line over a backup holding no data. R-413 - the R-87 proof caught a product-produced hollow snapshot unattended, and the nightly job fired on its own schedule at 05:30 for the first time (bentopdf PASSED on 9d002b38 in 2.315s). Both were listed "not yet live-validated" yesterday. R-403 mirror guard PROVEN live, with a negative control: it fired when a unit was hollow ("The copy was PRESERVED rather than replaced with an empty one") and skipped 0 legs at teardown when every unit was sound. R-357 PASS at last, six days owed: a real full filesystem, refused BEFORE StopStack, app never stopped, live data byte-identical, and it worked once the space came back. Phase 4 built the false-alarm control the whole R-87 design rests on: bentopdf is the only template of 53 with neither a database nor a named volume. It passes silently. EIGHT of my own instrument errors are named in the report, each caught by its own control - including a time guard that fired an injection four hours early, and filing R-412 at the wrong severity. Teardown clean on all three layers of both boxes; both healthy on 0.231.0. OWED: a golden for 0.231.0, and a decision on keeping bentopdf.	2026-09-01 06:07:13 +02:00
admin	ab8b884763	soak phase 5: R-403 guard PROVEN live; R-412 CORRECTED down after measuring the mechanism ... R-403 MIRROR GUARD - PASS, forced after the natural test evaporated. privatebin was injected hollow at 23:34 to meet the 03:30 mirror; the 02:30 db-dump re-made its tar, so by 03:30 the primary was complete and the guard had nothing to refuse. Forced instead on calibre-web through the real Tier-2 path: the guard fired and named itself - "unit leg SKIPPED ... The copy was PRESERVED rather than replaced with an empty one (R-403). The other legs continue." Secondary byte-identical, 23 files, tar sha d7e7f422. R-412 CORRECTED, AND I OVERSTATED IT WHEN I FILED IT. The first wording claimed the hollow unit sits in the store for a whole cycle because the volume-dump leg runs only on the backup schedule. That is WRONG. The 04:15 off-site run has its OWN pre-push dump leg - "Stopping calibre-web for safe volume dump", "Volume dump: ... -> 877.5 KB" - so a unit that is hollow when a run starts is REPAIRED before it is pushed. Measured twice: opengist and calibre-web both went in hollow and came out complete, and the snapshot pulled back from the store (6fee3b5a) holds the volume tar and all 17 userdata files. What remains real is narrower: the one hollow snapshot that DID reach the store was created when the unit was destroyed INSIDE a run that had already completed that app's dump leg. The race is real and was observed, and "backed up opengist (... 0 mandatory path(s))" is a success line over a backup holding none of the app's data either way. Severity HIGH -> LOW, with the correction stated in the row rather than quietly rewritten. Phase 6 interim: the observer is clean so far - db-dump 674ms, tier2-backup 3ms (a no-op, cause to be established not assumed), zero ERROR/WARN since 23:00. Also recorded: two of Phase 5's four injections were NOT performed, with the reasons established rather than asserted - there is no endpoint that reaches SetDisconnected and a hand-set flag would be reverted by the live monitor before 04:15; and a corrupted manifest provably never reaches the store because the capture rewrites it first.	2026-09-01 04:21:15 +02:00
admin	585ed654b4	soak drill phases 0-4: R-408's hazard is REAL and reachable; R-357 finally live (R-411..R-413) ... Overnight soak on demo-hp, phases 0-4 of 7. Evidence documentation/audits/DRILL-soak-2026-08-31/. No production code written, no golden, no version bump - findings only. PHASE 1 FAIL - R-411. The R-408 hazard was only ever reasoned about; tonight it was measured through the product's own endpoints. restic stats TAKES A LOCK (clean-room: nothing else running, 4x stats, sampler reads locks=1). A customer full-restore runs stats in its preparation while holding NO acquireRunning, so the integrity check is not blocked, runs, meets that lock, and resticStep escalates to unlock --remove-all - the sampler caught "restore ..." and "unlock --remove-all" in the SAME sample at 20:50:51. The log calls it "a stale exclusive lock left by a previous crash"; there was no crash. THE CUSTOMER-FACING CONSEQUENCE IS CONTAINED and that is R-359 working: the check was classified Unreachable, NOT damage, so no alarm and due-ness held. The opposite direction is FENCED - five restores fired into a running check at 5/15/25/35/40s were all refused by restoreOpBlocked with zero restic invoked. PHASE 2 FAIL - R-412, and it is the natural instance of R-403's shape that yesterday's session had to hand-build. A lost primary unit is rebuilt by the capture WITHOUT its volume dumps (185664 B -> 4382 B, volume_dumps: None), and the off-site backup then ships it and logs "backed up opengist ... 0 mandatory path(s)" - a success line over a backup holding none of the app's data. PHASE 2 also R-413: the R-87 proof CAUGHT that hollow snapshot unattended - verdict "fail", volumes_expected_none_captured: opengist_data, one offsite_proof_empty at severity error, while the four apps ahead of it in the rotation passed. 2.3 stale marker PASS, 2.4 two restores PASS, 2.5 corrected the runbook's premise (RunTier2 has zero acquireRunning and zero restic references - a local mirror cannot collide with the repo, so the proof correctly does not skip for it). PHASE 3 PASS - R-357 live-validated at last, six days owed. A real full filesystem (1900544 B free vs 5878421 B needed) on a separate 1 TB device that does not back Docker. Refused BEFORE StopStack: app "Up 6 minutes (healthy)" unchanged, 0 safety dumps, live userdata tree fingerprint 5a9b2db75dfd4ba4131adb3255670472 unchanged, message names both numbers in Hungarian. Ballast removed, the SAME restore then worked and the fingerprint is still identical. On the same full disk the Tier-2 run, the proof and the integrity check all behaved: the proof refused before any download through the shared unitOnlyHeadroom gate extracted today, reached no verdict and did not alarm. PHASE 4 PASS - the false-alarm control the whole R-87 design rests on now EXISTS. Found by reading all 53 catalogue composes: bentopdf is the only template with neither a database service nor a named volume. Deployed, backed up, proved: PASSED in 2.224s with zero alarms. Four of my own instrument errors were caught by their own controls before any result was believed: a hub log line used as a controller positive control, a grep pattern that missed a registered job, a heredoc that mangled a planted marker, and a catalogue scan that read 0 of 0 apps from the wrong path. Phases 5-7 to follow: the real nightly cycle with injected faults, the untouched observer, and teardown.	2026-08-31 23:33:32 +02:00