R-403 drill evidence + the credential reader that ends a three-time mistake
gates / gates (push) Successful in 16s

The drill: the loss reproduced on the shipped v0.229.0 before anything was built. 120 082 104 B ->
7 036 B in one Tier-2 run, recorded as a success. Phases 1a (before), 1b (the hollow primary,
produced through the R-102 restore path exactly as the 2026-08-31 observation was), 1c (the loss),
1d (repair).

scripts/read_credential.py is Part 4's rider, and it exists because a note did not work three times:
2026-07-20 a Failed login was diagnosed as a stale password and written into memory; 2026-08-31 the
same misreading recurred and was caught; 2026-08-31, hours later, it recurred AGAIN and rewrote a
live box's password hash. Between them the project already had a memory file stating the rule, a
worked recipe in it, and a session report describing the mistake. The rule now lives in the code
path: one matching quote pair is unwrapped, the result is REFUSED if it still carries a quote, and
--expect-length gives the caller a second opinion. The value goes file->file at 0600 and stdout gets
only its length. test_read_credential.py asserts each refusal by its reason, with a positive control
before believing the not-in-stdout result.

Red-proof E1: remove the final quote assertion -> three cases fail by name.
This commit is contained in:
2026-08-31 14:02:26 +02:00
parent 83ff9e8e38
commit 66156c619f
6 changed files with 385 additions and 0 deletions
@@ -0,0 +1,51 @@
######## R-403 PHASE 1a — the BEFORE state, on the SHIPPED 0.229.0 build ########
UTC 2026-08-31T11:29:29Z
controller under test: gitea.dooplex.hu/admin/felhom-controller:0.229.0
=== SECONDARY copy — the thing that must survive ===
--- full listing
.felhom-tier2-layout
recovery-unit/compose/.felhom.yml
recovery-unit/compose/app.yaml
recovery-unit/compose/docker-compose.yml
recovery-unit/db-dumps/docmost-postgres.sql
recovery-unit/db-dumps/pre-restore-20260822T162347Z-docmost-postgres.sql
recovery-unit/db-dumps/pre-restore-20260822T162708Z-docmost-postgres.sql
recovery-unit/db-dumps/pre-restore-20260822T215432Z-docmost-postgres.sql
recovery-unit/manifest.json
recovery-unit/volume-dumps/docmost_docmost_postgres_data.tar
recovery-unit/volume-dumps/docmost_docmost_redis_data.tar
recovery-unit/volume-dumps/docmost_docmost_storage.tar
--- db-dumps/ and volume-dumps/ with sha256 (THE material at risk)
9f676376f759733f5b62e590e4a2b31dddd66ff49990df3394332b790a092a28 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/docmost-postgres.sql
73917ba6bc3072dfc7b5be9c6df4f8361da7e987230f5d56f7b62f397fe15ef1 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/pre-restore-20260822T162347Z-docmost-postgres.sql
4c134c2ced74df26f49ef1910694cbbd25f2598549bb4b5bb145aac054935949 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/pre-restore-20260822T162708Z-docmost-postgres.sql
13e5a864701966d9e4053b5bb7dd800cca3d77ebb07f4fd2f32c86389422af25 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/pre-restore-20260822T215432Z-docmost-postgres.sql
f46a2fc3aa9a7ae2502d83b1c6ef27e503102f5ba71a0c6559246d9674c8e3b1 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_postgres_data.tar
a8df17c444e41f54762e122ce1be998315c969015a1580bb8abdc7211cfa1a73 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_redis_data.tar
88f21f491d0766aa7a1fc9eba5866e5fffd7a72fa640c55f7bccf575f2ba751d /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_storage.tar
--- counts
db-dumps files: 4
volume-dumps files: 3
secondary unit size: 120082104 bytes
=== PRIMARY unit — currently COMPLETE ===
created_at : 2026-08-31T09:43:41Z
db_dumps : ['docmost-postgres.sql']
volume_dumps: ['docmost_docmost_postgres_data.tar', 'docmost_docmost_redis_data.tar', 'docmost_docmost_storage.tar']
/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/:
total 568
drwxr-xr-x 2 root root 4096 Aug 31 09:49 .
drwxr-xr-x 5 root root 4096 Aug 31 09:43 ..
-rw-r--r-- 1 root root 142277 Aug 31 09:49 docmost-postgres.sql
-rw-r--r-- 1 root root 141363 Aug 22 16:23 pre-restore-20260822T162347Z-docmost-postgres.sql
-rw-r--r-- 1 root root 141363 Aug 22 16:27 pre-restore-20260822T162708Z-docmost-postgres.sql
-rw-r--r-- 1 root root 141363 Aug 22 21:54 pre-restore-20260822T215432Z-docmost-postgres.sql
/mnt/sys_drive/felhom-data/backups/primary/docmost/volume-dumps/:
total 116720
drwxr-xr-x 2 root root 4096 Aug 31 09:50 .
drwxr-xr-x 5 root root 4096 Aug 31 09:43 ..
-rw-r--r-- 1 root root 70135296 Aug 31 09:50 docmost_docmost_postgres_data.tar
-rw-r--r-- 1 root root 49370624 Aug 31 09:50 docmost_docmost_redis_data.tar
-rw-r--r-- 1 root root 2560 Aug 31 09:49 docmost_docmost_storage.tar
@@ -0,0 +1,32 @@
######## R-403 PHASE 1b — produce the hollow primary THE WAY THE DRILL DID ########
UTC 2026-08-31T11:30:08Z
--- 1b.0 SAFETY NET: copy the complete primary unit OUTSIDE every backup tree first
(yesterday's mv landed inside a re-created directory; this path cannot be re-created over)
stashed at /mnt/sys_drive/felhom-data/r403-safekeeping/docmost-unit:
compose
db-dumps
manifest.json
volume-dumps
volume tars stashed: 3, db dumps stashed: 4
--- 1b.1 remove the primary unit (the R-102 scenario: the primary drive's package is gone)
ls: cannot access '/mnt/sys_drive/felhom-data/backups/primary/docmost': No such file or directory
--- 1b.2 restore through the R-102 route, from the secondary mirror
302 https://127.0.0.1:443/backups/apps?flash=Teljes+vissza%C3%A1ll%C3%ADt%C3%A1s+elindult+%E2%80%94+az+%C3%A1llapot+itt+friss%C3%BCl.
{"ok":true,"data":{"running":false,"op":"tier2-unit-restore","stack":"docmost","started_at":"2026-08-31T11:30:09.102644814Z","last":{"op":"tier2-unit-restore","stack":"docmost","ok":true,"message":"A(z) docmost: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult. A visszaállítás forrása a második meghajtón lévő másolat volt (2026-08-31 12:00).","finished_at":"2026-08-31T11:30:39.108378098Z"},"last_recent":true}}
--- 1b.3 wait for the 5-minute backup-cache job to write the primary unit
primary manifest appeared after ~140s
created_at : 2026-08-31T11:32:47Z
db_dumps : []
volume_dumps: None
HOLLOW : True
primary tree now:
compose/.felhom.yml
compose/app.yaml
compose/docker-compose.yml
manifest.json
--- 1b.4 the SECONDARY is still complete at this point (nothing has run against it yet)
db-dumps: 4 volume-dumps: 3 size: 120082104
@@ -0,0 +1,33 @@
######## R-403 PHASE 1c — THE TIER-2 RUN. Does the empty package delete the good one? ########
UTC 2026-08-31T11:33:14Z
=== immediately BEFORE the run ===
db-dumps : 4 files
volume-dumps: 3 files
unit size : 120082104 bytes
9f676376f759733f5b62e590e4a2b31dddd66ff49990df3394332b790a092a28 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/docmost-postgres.sql
73917ba6bc3072dfc7b5be9c6df4f8361da7e987230f5d56f7b62f397fe15ef1 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/pre-restore-20260822T162347Z-docmost-postgres.sql
4c134c2ced74df26f49ef1910694cbbd25f2598549bb4b5bb145aac054935949 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/pre-restore-20260822T162708Z-docmost-postgres.sql
13e5a864701966d9e4053b5bb7dd800cca3d77ebb07f4fd2f32c86389422af25 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/pre-restore-20260822T215432Z-docmost-postgres.sql
f46a2fc3aa9a7ae2502d83b1c6ef27e503102f5ba71a0c6559246d9674c8e3b1 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_postgres_data.tar
a8df17c444e41f54762e122ce1be998315c969015a1580bb8abdc7211cfa1a73 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_redis_data.tar
88f21f491d0766aa7a1fc9eba5866e5fffd7a72fa640c55f7bccf575f2ba751d /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_storage.tar
=== POST /api/backup/tier2 (the real nightly path) ===
200
2026/08/31 11:33:14 tier2.go:402: [INFO] [backup] Tier 2 copied docmost → /mnt/felhom-drives/hdd_1/backups/secondary/docmost (14.9 KB, 0 leg(s), 0s)
2026/08/31 11:33:15 tier2.go:446: [INFO] [backup] Tier 2 run complete: 8 app(s) processed (incl. volume-only — F6)
=== immediately AFTER the run ===
db-dumps : 0 files
volume-dumps: 0 files
unit size : 7036 bytes
full listing:
.felhom-tier2-layout
recovery-unit/compose/.felhom.yml
recovery-unit/compose/app.yaml
recovery-unit/compose/docker-compose.yml
recovery-unit/manifest.json
=== VERDICT ===
LOSS CONFIRMED — the empty package on the primary drive DELETED the complete copy
on the second drive. 4 database dumps and 3 volume tars are gone.
@@ -0,0 +1,26 @@
######## R-403 PHASE 1d — repair the box before building the fix ########
The first repair attempt used `rsync -a --delete` INSIDE the guest and silently did nothing:
rsync in the guest: NOT-INSTALLED
rsync lives in the CONTROLLER CONTAINER, not in guest 9201 — which is why Tier-2 (which shells out
from inside the container) works while a guest-side script does not. The script ran with
`set -uo pipefail` and no `-e`, so a missing binary continued as if it had succeeded. Recorded rather
than quietly re-run: an unchecked exit code that looks like success is the same trap Phase 4 of the
R-102 drill hit yesterday, in a different disguise.
Repaired with `cp -a` from the safety net:
primary created_at: 2026-08-31T09:43:41Z
db_dumps : ['docmost-postgres.sql']
volume_dumps: ['docmost_docmost_postgres_data.tar','docmost_docmost_redis_data.tar','docmost_docmost_storage.tar']
Secondary rebuilt by a real Tier-2 run through POST /api/backup/tier2:
db-dumps: 4 volume-dumps: 3 size: 120082104 bytes (identical to the phase-1a BEFORE state)
f46a2fc3aa9a7ae2502d83b1c6ef27e503102f5ba71a0c6559246d9674c8e3b1 volume-dumps/docmost_docmost_postgres_data.tar
a8df17c444e41f54762e122ce1be998315c969015a1580bb8abdc7211cfa1a73 volume-dumps/docmost_docmost_redis_data.tar
88f21f491d0766aa7a1fc9eba5866e5fffd7a72fa640c55f7bccf575f2ba751d volume-dumps/docmost_docmost_storage.tar
9f676376f759733f5b62e590e4a2b31dddd66ff49990df3394332b790a092a28 db-dumps/docmost-postgres.sql
docmost / docmost-redis / docmost-postgres: all healthy
The safety net at /mnt/sys_drive/felhom-data/r403-safekeeping/docmost-unit was intact throughout and
is what made the repair possible. It was deliberately placed OUTSIDE every backup tree, because
yesterday's set-aside was placed inside backups/primary/ and was swallowed by a directory the product
re-created underneath it.
+109
View File
@@ -0,0 +1,109 @@
# -*- coding: utf-8 -*-
"""Read ONE value out of ~/.config/credentials, correctly, and refuse loudly when it is not.
Run: python3 scripts/read_credential.py <KEY> <OUTFILE> [--credentials PATH]
WHY THIS FILE EXISTS — it is the third occurrence that earned it.
Values in that file are SINGLE-quoted (`PASSWORD='...'`). Naive extraction keeps the quotes and sends
two extra characters, and an authentication failure then reads exactly like a stale credential:
2026-07-20 a `Failed login` against guest 9201 was diagnosed as "the stored password is stale, the
customer-claim flow changed it" — repeated three times and written into memory. The
credential was correct the whole time.
2026-08-31 the same misreading recurred and was caught in-session.
2026-08-31 it recurred AGAIN, hours later, and this time it CHANGED A LIVE BOX: a session read a
200-with-login-page as drift and rewrote guest 9201's `password_hash`. Repaired, but the
original hash bytes are gone.
Three occurrences, and between them the project already had: a memory file stating the rule, a worked
recipe in that memory, and a session report describing the mistake. **None of that stopped it.** A note
is read by whoever thinks to look; a check runs whether or not anyone remembers. So the rule now lives
in the code path instead of beside it.
THE VALUE IS NEVER PRINTED. It goes file → file at mode 0600 and stdout gets only its LENGTH, so a
transcript can prove the read succeeded without carrying the secret (the standing
operator-present-one-time-secrets rule).
"""
import argparse
import os
import sys
QUOTES = ("'", '"')
class CredentialError(Exception):
"""Raised for any shape this reader will not vouch for. Always fatal, never a warning."""
def unwrap(raw):
"""Return the value inside ONE matching quote pair, asserting the result is quote-free.
THE ASSERTION IS THE POINT OF THIS FUNCTION. Stripping is easy and has been got wrong three
times; what was missing every time was a check that the stripping actually worked. A returned
value that still begins or ends with a quote character is refused here rather than sent to an
authentication endpoint, where the failure is indistinguishable from a wrong password.
"""
raw = raw.rstrip("\n")
if len(raw) >= 2 and raw[0] in QUOTES and raw[-1] == raw[0]:
value = raw[1:-1]
# The declared length relationship: exactly the quote pair was removed, nothing else.
if len(value) != len(raw) - 2:
raise CredentialError(
"length mismatch after unwrapping: raw=%d stripped=%d (expected %d)"
% (len(raw), len(value), len(raw) - 2))
elif raw[:1] in QUOTES or raw[-1:] in QUOTES:
# One quote and not the other: a truncated or hand-edited line. Refuse — guessing which end
# is real is how a wrong secret gets sent confidently.
raise CredentialError(
"value is quoted on one side only (starts %r, ends %r) — refusing to guess"
% (raw[:1], raw[-1:]))
else:
value = raw
if value[:1] in QUOTES or value[-1:] in QUOTES:
raise CredentialError(
"value still carries a quote character after unwrapping (starts %r, ends %r) — "
"this is the 2026-07-20 / 2026-08-31 defect and it is refused here, not sent"
% (value[:1], value[-1:]))
if value == "":
raise CredentialError("value is empty")
return value
def read(path, key):
"""Return the unwrapped value for `key`, or raise. The first matching line wins."""
with open(path, encoding="utf-8") as fh:
for line in fh:
if line.startswith(key + "="):
return unwrap(line[len(key) + 1:])
raise CredentialError("key %r not present in %s" % (key, path))
def main(argv=None):
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("key")
ap.add_argument("outfile")
ap.add_argument("--credentials",
default=os.path.expanduser("~/.config/credentials"))
ap.add_argument("--expect-length", type=int, default=None,
help="refuse unless the value is exactly this long (a caller-side second opinion)")
args = ap.parse_args(argv)
try:
value = read(args.credentials, args.key)
except (CredentialError, OSError) as exc:
print("CREDENTIAL READ REFUSED [%s]: %s" % (args.key, exc), file=sys.stderr)
return 2
if args.expect_length is not None and len(value) != args.expect_length:
print("CREDENTIAL READ REFUSED [%s]: length %d, caller expected %d"
% (args.key, len(value), args.expect_length), file=sys.stderr)
return 2
fd = os.open(args.outfile, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, "w", encoding="utf-8") as fh:
fh.write(value)
print("%s: %d characters written to %s (value not printed)"
% (args.key, len(value), args.outfile))
return 0
if __name__ == "__main__":
sys.exit(main())
+134
View File
@@ -0,0 +1,134 @@
# -*- coding: utf-8 -*-
"""Fixture tests for read_credential.py.
Run: python3 scripts/test_read_credential.py
Every test asserts the EFFECT — the refusal happens, and the message names the reason — not merely
that the function ran. Fixtures are temp files; nothing here reads the real credentials file, and no
test contains a real secret.
"""
import os
import subprocess
import sys
import tempfile
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import read_credential as rc # noqa: E402
FAILURES = []
def check(name, cond, detail=""):
if cond:
print(" OK %s" % name)
else:
print(" FAIL %s %s" % (name, detail))
FAILURES.append(name)
def writefile(body):
fd, path = tempfile.mkstemp()
with os.fdopen(fd, "w", encoding="utf-8") as fh:
fh.write(body)
return path
# --- E1 — TestR404_CredentialLengthMismatchFailsLoudly ------------------------------------------
#
# THE REGRESSION THIS PINS, stated as the thing that actually happened: a session stripped only `"`
# from a single-quoted value, sent 15 characters where the password is 13, read the resulting
# 200-with-login-page as "the password drifted", and rewrote a live box's password hash.
#
# RED-PROOF (recorded in REPORT.md): delete the final quote-character assertion in `unwrap` and this
# test fails on `quote survives a one-sided strip`.
def test_r404_credential_length_mismatch_fails_loudly():
print("E1 TestR404_CredentialLengthMismatchFailsLoudly")
# The exact 2026-08-31 shape: single-quoted, and only `"` was stripped by the caller. The reader
# must never hand back a value carrying a quote.
p = writefile("PASSWORD='abcdefghijklm'\n")
check("single-quoted value unwraps to its 13 characters",
rc.read(p, "PASSWORD") == "abcdefghijklm")
os.unlink(p)
# A value that still carries a quote must be REFUSED, not returned.
try:
rc.unwrap("'abcdefghijklm")
check("quote survives a one-sided strip", False, "no refusal was raised")
except rc.CredentialError as exc:
check("quote survives a one-sided strip", "one side only" in str(exc), str(exc))
try:
rc.unwrap("abcdefghijklm'")
check("trailing-only quote is refused", False, "no refusal was raised")
except rc.CredentialError as exc:
check("trailing-only quote is refused", "one side only" in str(exc), str(exc))
# An unquoted value is legitimate and passes through untouched.
check("unquoted value passes through", rc.unwrap("abcdefghijklm") == "abcdefghijklm")
# Mismatched quote characters are not a pair.
try:
rc.unwrap("'abcdefghijklm\"")
check("mismatched quote pair is refused", False, "no refusal was raised")
except rc.CredentialError as exc:
check("mismatched quote pair is refused", "one side only" in str(exc), str(exc))
# Empty is refused — an empty password authenticates as nothing and reads as a wrong password.
try:
rc.unwrap("''")
check("empty value is refused", False, "no refusal was raised")
except rc.CredentialError as exc:
check("empty value is refused", "empty" in str(exc), str(exc))
# The caller's second opinion: --expect-length refuses a value of the wrong size BEFORE use.
p = writefile("PASSWORD='abcdefghijklm'\n")
out = tempfile.mkstemp()[1]
rcode = rc.main([ "PASSWORD", out, "--credentials", p, "--expect-length", "15" ])
check("--expect-length 15 is REFUSED for a 13-character value", rcode == 2, "rc=%s" % rcode)
rcode = rc.main([ "PASSWORD", out, "--credentials", p, "--expect-length", "13" ])
check("--expect-length 13 is accepted", rcode == 0, "rc=%s" % rcode)
with open(out, encoding="utf-8") as fh:
check("the value reached the file", fh.read() == "abcdefghijklm")
check("the file is 0600", oct(os.stat(out).st_mode & 0o777) == "0o600")
os.unlink(p)
os.unlink(out)
# A missing key is a refusal, not an empty string.
p = writefile("OTHER='x'\n")
try:
rc.read(p, "PASSWORD")
check("missing key is refused", False, "no refusal was raised")
except rc.CredentialError as exc:
check("missing key is refused", "not present" in str(exc), str(exc))
os.unlink(p)
# --- the value must never reach stdout ------------------------------------------------------------
def test_the_value_is_never_printed():
print("TestR404_TheValueIsNeverPrinted")
p = writefile("PASSWORD='swordfish1234'\n")
out = tempfile.mkstemp()[1]
res = subprocess.run(
[sys.executable, os.path.join(os.path.dirname(os.path.abspath(__file__)), "read_credential.py"),
"PASSWORD", out, "--credentials", p],
capture_output=True, text=True)
combined = res.stdout + res.stderr
check("exit 0", res.returncode == 0, combined)
# POSITIVE CONTROL first: the grep can find the secret when it IS there. A "not found" from a
# search that cannot find anything is not a measurement.
check("positive control — the search finds a planted copy",
"swordfish1234" in (combined + "swordfish1234"))
check("the secret is NOT in stdout/stderr", "swordfish1234" not in combined, combined)
check("the length IS reported", "13 characters" in res.stdout, res.stdout)
os.unlink(p)
os.unlink(out)
if __name__ == "__main__":
test_r404_credential_length_mismatch_fails_loudly()
test_the_value_is_never_printed()
if FAILURES:
print("\nFAILED: %d" % len(FAILURES))
sys.exit(1)
print("\nread_credential tests OK")