Files
felhom.eu/documentation/audits/update-night-2026-09-21/failwalk.py
T
admin da20722e76
gates / gates (push) Successful in 27s
Update night 2026-09-21: Phase 0 and Phase 1 evidence, the drill method, and two instrument fixes
INTERIM CHECKPOINT — evidence off the machine at the end of the phase that produced it (R-320),
not at the end of the session. Phases 2-5 follow in a later commit.

Phase 0, all three mechanisms proven with their controls:
- the fleet floor to 0.261.0 with its declared MinAgent — both demo boxes in 13 s, the hub
  logging `managed floor SERVED ... from declared (golden 0.258.0)`.
- a PRIVATE DRILL CATALOG (admin/app-catalog-drill), so that broken, dummy, cross-repo and
  engine-major edges can be measured without the live catalog ever carrying one. Positive
  control quoted, and two negative controls: the live catalog's main and both real boxes'
  caches unchanged.
- a throwaway image store on the scratch guest, which is what makes an UNATTENDED HOLD
  measurable at all: an edge that PASSES the within-a-major test and still fails.
  CompareImageRefs was proven to order host:port/ references by RUNNING it (4 positive cases
  + 1 negative control), not by reading it.

Phase 1: real within-a-major upstream edges walked on guest 9202 through the product's own
guarded Update, each app seeded and read back through its OWN front door (R-156), with a
per-edge verdict record in 09's shape. `inconclusive` is never collapsed into `failed`.

TWO INSTRUMENT FIXES, both in this repo's own evidence code:
- 00-api-recipe.md said the app page is /app/<n>; it is /apps/<n>, and every call it described
  404s. Corrected, with the session-expiry note that cost the same time.
- unattended-caller.py's follow() read update_phase/updating off the API ENVELOPE, so both were
  always None and EVERY followed update ran to its 900 s timeout and was then recorded
  `timeout` and never-press-again. Fixed before B1 relied on it. R-623.

No controller, agent or hub code was written. The live catalog carries no broken reference.

Gates: repo_gates.py --fast — all 15 OK, exit 0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-21 21:17:46 +02:00

102 lines
4.7 KiB
Python

#!/usr/bin/env python3
"""failwalk.py <app> <sub> — what the brief says to do with a real edge that FAILED or HELD.
"A failed or HELD real edge is the most valuable result of the night. Do not fix it, do not
retry it twice. Evidence off, the hold sentence quoted in both languages, then walk the
household's way out (restore per the sentence), record whether the data came back."
So this does exactly that and nothing else. It does NOT re-press the Update.
It re-seeds nothing: it uses the token file the failed run left behind where there is one, and
otherwise says plainly that the pre-update seed could not be carried across process boundaries and
re-seeds AFTER the restore instead — which proves the app works again, but not that THIS row
survived. The difference is stated rather than glossed.
"""
import json, os, re, sys, time
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)
import walk as w # noqa: E402
from fixtures import FIXTURES # noqa: E402
def sentences(app):
out = {}
for lang, sfx in (("hu", ""), ("en", "?lang=en")):
html = w.page(f"/apps/{app}{sfx}")
open(os.path.join(HERE, "apps", app, f"held-page-{lang}.html"), "w").write(html)
t = re.sub(r"<script.*?</script>", " ", html, flags=re.S)
t = re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", t))
out[lang] = [s.strip() for s in re.split(r"(?<=[.!?]) ", t)
if any(k in s.lower() for k in
("friss", "update", "vissza", "restore", "ment", "backup",
"masolat", "másolat", "copy", "meghajt", "hib", "error"))][:12]
return out
def main():
app, sub = sys.argv[1], sys.argv[2]
d = os.path.join(HERE, "apps", app)
os.makedirs(d, exist_ok=True)
w.login()
w.say(f"==== FAILWALK {app}: the household's way out of a held update")
st = w.stack(app)
held = {"state": st.get("state"), "updating": st.get("updating"),
"update_phase": st.get("update_phase"),
"update_phase_label": st.get("update_phase_label"),
"update_error": st.get("update_error"), "hold_reason": st.get("hold_reason"),
"observables": w.observables(app)}
w.say(f" [1] held state: phase={held['update_phase']} hold={held['hold_reason']!r}")
w.say(f" update_error = {held['update_error']!r}")
w.say(f" pinned = {held['observables']['pinned_images']}")
w.say(f" installed = {held['observables']['installed_images']}")
w.say(f" compose = {held['observables']['live_compose_image_lines']}")
w.say(f" inspect = {held['observables']['docker_inspect']}")
sent = sentences(app)
w.say(f" [2] hold sentence HU: {sent['hu'][:4]}")
w.say(f" [2] hold sentence EN: {sent['en'][:4]}")
logs = w.app_logs(app, 400)
open(f"{d}/held-app-logs.txt", "w").write(logs)
mig = None
for line in logs.splitlines():
if re.search(r"Applying .*\.\.\. OK|migrat", line, re.I) and len(line) < 300:
mig = line.strip()
w.say(f" [3] the last migration line the app printed, verbatim: {mig!r}")
# THE WAY OUT — the button the sentence names
way = w.restore(app)
w.say(f" [4] restore: {way.get('http')} in {way.get('seconds')}s; "
f"state={way.get('state_after')} hold={way.get('hold_after')!r}")
w.say(f" pinned after = {(way.get('observables_after') or {}).get('pinned_images')}")
w.say(f" inspect after = {(way.get('observables_after') or {}).get('docker_inspect')}")
# did the data come back? asked of the APP
fx = FIXTURES.get(app)
after = {"route": None, "ok": None,
"note": "the pre-update seed token did not survive the failed run's process, so this "
"re-seeds AFTER the restore: it proves the app WORKS again on the old version, "
"not that the specific pre-update row survived. Stated rather than glossed."}
if fx:
tok = fx.seed(w, sub, w.say)
if tok is not None:
after["route"] = "re-seeded after the restore"
after["ok"] = fx.verify(w, sub, tok, w.say)
w.say(f" [5] the app answers its own front door again and holds data: {after['ok']}")
sent_after = sentences(app)
w.say(f" [6] sentence after the restore HU: {sent_after['hu'][:3]}")
rec = {"leg": "failwalk", "app": app, "held": held, "hold_sentences": sent,
"last_migration_line": mig, "way_out": way, "data_after_restore": after,
"sentences_after_restore": sent_after}
json.dump(rec, open(f"{d}/failwalk.json", "w"), indent=2, ensure_ascii=False)
open(f"{d}/failwalk-log.txt", "w").write("\n".join(w.LOG) + "\n")
w.say(f" written -> {d}/failwalk.json")
if __name__ == "__main__":
main()