The first write-up said R-157 reproduced "at the same cycle in both runs -
deterministic, not a coincidence". Wrong. The cycle numbers matched only because
the runner's RNG is seeded so both runs drew the same permutation. The failure
itself is a coin flip: run 2b's four hard resets went PASS(c2), FAIL(c10),
PASS(c18), FAIL(c26); run 2a went PASS(c2), FAIL(c10). Three failures in six.
The correction matters because it changes what kind of bug this is, and it
strengthens rather than weakens the root cause: intermittency is exactly what a
race against container-state settling predicts, whereas a wrong predicate would
fail every time.
Signature is identical on all three occurrences: rallly Exited 255 with
rallly-postgres healthy, bootrecon reporting "no boot-orphaned apps" about 5s
after controller start, and the container count still churning after the sweep
(third occurrence 01:05: refresh 8, bootrecon 01:05:13, then 8 -> 7 -> 8).
Reproduced twice, two independent runs, same cycle (the runner's RNG is seeded so
both drew the same permutation - deterministic, not coincidence).
A hard reset mid-backup brought everything back except the app half of the
DB-backed stack: rallly left Exited 255, oom=false, restarts=0, its own log ending
"Ready" - it died healthy - while rallly-postgres returned healthy.
20:28:13 Status refresh: 8 containers across 55 stacks <-- docker ps -a shows NINE
20:28:18 [bootrecon] Boot reconciliation: no boot-orphaned apps
20:28:25 Status refresh: 7 ... 8 containers <-- still churning AFTER the sweep
20:39:14 [deadapp] 20 scans, 5 deployed evaluated, 1 currently down
The predicate is sound: once settled the controller reports rallly state=degraded
containers=2, and IsDownState includes StateDegraded, so len>0 && IsDownState
holds. The SNAPSHOT was wrong. bootrecon fires as a goroutine ~5s after start
while docker is still restoring containers, and is start-once by design, so it
never re-checks.
Consequence: the app stays down indefinitely. Detection is perfect and recovery
never happens - R-52's original shape, an alarm with no recovery. Not fixed.
Distinguished from this campaign's two earlier HARNESS defects: both drives bound,
every other app returned incl. the drive-backed one, only the app half of a
two-container stack missing while its DB is healthy, and it surfaced through the
fixed check written for exactly this.
Ran the soak on the Phase A rig. Ended on its own deadline — no watchdog halt,
no atom exception, no I11 breach.
I1 28+28 pairs, I2 28+28 pairs, I3 56, I4 56, I5/I6 28 each, I7 28, I10 135,
I11 28. Zero violations. The row counts are themselves the no-silent-skip check:
I3/I4 twice per cycle (both drives), I10 = 5 secret-class fields x 27, REBOOT on
cycles 7/14/21 only.
I7 is the headline: 28 restores, 28 correct discriminators — never stale, never
empty. RTO (Tier 1, rallly, 66 MB): min 38.8s, median 42.0s, p90 42.5s, max
44.3s. That is the S band's lower end ONLY; the 5.5s spread over 28 runs says
fixed work dominates, so nothing extrapolates to M or L. RPO not measured.
Every atom and invariant was proven BY HAND before automation — the runner
asserts nothing that was not first observed live.
Caught a Phase A gap before starting: no app had HDD_PATH, so all data sat on the
system disk and I3 could never have fired. Deployed calibre-web onto adatok
first; otherwise the run would have produced 27 green cycles that tested nothing
cross-drive.
Investigated and DISPROVED a suspected defect (audit 5.2): /api/disks reports
state=attached for a physically absent drive, and intermediary.go:230 really does
compute presence from State=="attached". It is inert — planDriveGates only gates
paths under /mnt/felhom-drives/ and uses BoundUnderParent there, which was
correctly false. The gate fired; the storage page showed "Meghajtó leválasztva".
No R-n minted.
Honest gaps: 6 of ~12 atom families ran. Not run — Tier 3 (structurally
un-isolatable), abort-fs-in-place, kill-agent-mid-backup, hard-reset-mid-write,
reboot-VM, both concurrency atoms, fill-drive-near-full. I8 not checked, I9 not
automated (cited from the tester-gate run, not re-claimed). kill_controller is
NOT mid-backup and reboot_guest never interleaved with a detach. 27 cycles does
not answer the brief's question about drift at the thirty-eighth.
Teardown still OWED, including hub customer c10-soak (disposition: DELETE).
Phase A passed every gate on a fresh box built from the PUBLISHED ISO 1.26.1:
install, claim, two drives enrolled through the real endpoints with the backup
target healthy, four apps spanning both sides of D5's secret split, and a
working discriminator across all four.
Isolation gate: both denials captured, each with a positive control. The PBS
control FAILED first — four clean-looking 403s were worthless because the token
was denied on its own datastore too (PBS token privilege separation). Fixed and
re-run; the denials stand.
R-156 (new, register grepped): papra's data is neither persisted nor backed up,
and it reports healthy. The template mounts papra_data:/app/data; the app writes
/app/app-data/db/db.sqlite. Volume empty and root-owned against a -rootless
image, real DB in the container writable layer, healthcheck only probes the HTTP
port. Its Tier-1/2 backup is real, verifiable and contains nothing. Not fixed.
Tier 3 could not be isolated so it was not run: offsite hard-requires the DR
tier (configs.go:1300) and the DR tier only provisions on ep0 (per-endpoint
allocation deferred, hub/README.md:260). Both are recorded deliberate positions,
so no R-n minted. The campaign touched neither ep0 nor the Storage Box.
Phase B did not start. Phase A was budgeted at ~1h and took ~5.5h (1.26.1 is a
public release image with no auto-install path, so the install was a blind
screendump+sendkey walk). That left the runner — which judges eleven invariants
and fires destructive atoms unattended — to be written at 04:00 with ~3h of
night left. Stopped on the brief's own fence: a rig producing false negatives is
worse than no rig. The rig is built and idle; teardown is OWED and itemised,
including hub customer c10-soak (disposition: DELETE).
admin