diff --git a/documentation/architecture/where-felhom-stands.html b/documentation/architecture/where-felhom-stands.html index aaded1ba..9cd02b13 100644 --- a/documentation/architecture/where-felhom-stands.html +++ b/documentation/architecture/where-felhom-stands.html @@ -6,7 +6,7 @@

Where Felhom stands

What we built, what happens when things go wrong, and what is still missing. Generated from where-felhom-stands.yaml, which cites the capability map, register row or evidence document behind every claim — and which is checked by scripts/check_stands.py.
-
Verified 2026-08-22 against felhom-agent 40d857b527, felhom-controller 2da259af38, hub 877fcd2a38. 17 claim(s) carry a recorded status move (8 re-checked in this pass) — each is marked on the page.
+
Verified 2026-08-22 against felhom-agent 40d857b527, felhom-controller 2da259af38, hub 877fcd2a38. 17 claim(s) carry a recorded status move (7 re-checked in this pass) — each is marked on the page.
Walked (23)done end to end on real hardware, evidence on file
Built (13)shipped and tested, the real path never walked
Partial (15)some walked, some not — the note says which
Missing (4)does not exist
needs-hardwareonly a running box could settle it
The journey
Seven stages, left to right, as they happen to a person.
@@ -40,7 +40,7 @@
A fresh install picks up an old image
A fresh install picks up an old imageNarrowed 2026-08-09: the resume path fetched the vouched golden; a fresh install still takes the newest LOCAL archive with no manifest comparison.register R-274
The off-site provider account is deleted
The off-site provider account is deletedThe credential that holds the customer's documents can still delete.register R-95
The machine is switched off for an afternoon
The machine is switched off for an afternoon — no notion of expected downtimeConfirmed 2026-08-09: eight operator mails for deliberate, attended work.register R-285
-
A customer restores their own data with no help
A customer restores their own data with no helpcontestedCONTESTED-RESOLVED: the map's MISSING is about a NON-OPERATOR performing it; the walks prove the route, not the person. Neither record was wrong. SINCE 2026-08-21 there is a second, harder blocker and it is not about the person at all: for the 40 of 53 apps that declare no data drive the off-site restore REFUSES before it starts, telling the customer a running app 'nincs telepítve' and to reinstall it to the same place — which those apps give them no way to choose (R-356). Those are exactly the apps whose whole dataset is a named volume. Until that is fixed, most customers cannot self-restore off-site even in principle.capability-map A customer (not the operator) performs a restore via UI alone · register R-201 · register R-356 · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md
+
A customer restores their own data with no help
A customer restores their own data with no helpWalked 2026-10-29 night on scratch 9202 through the household's restore button (the page's own POST): vikunja from the 02:30 local point in 79 s — the project made before the point came back, the one made after it is gone, the app runs, no hold; a restore that meets a busy volume puts the live data back (R-1256 H5). The route needs no help. Still open: no person who is not the operator has done it (claim.never-by-non-operator). The old off-site blocker (R-356) shipped in controller v0.219.0.capability-map A customer (not the operator) performs a restore via UI alone · register R-201 · register R-356 · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md · evidence audits/night-2026-10-28/part5-claims/selfrestore-walk.log
Generated by scripts/render_stands.py from where-felhom-stands.yaml. Do not hand-edit this file — edit the data and regenerate. The 2026-08-09 React bundle is kept as where-felhom-stands-2026-08-09-snapshot.html and is not maintained.
\ No newline at end of file diff --git a/documentation/architecture/where-felhom-stands.yaml b/documentation/architecture/where-felhom-stands.yaml index 82ef2c48..fb1d07f6 100644 --- a/documentation/architecture/where-felhom-stands.yaml +++ b/documentation/architecture/where-felhom-stands.yaml @@ -769,13 +769,14 @@ claims: band: failures title: "A customer restores their own data with no help" status: partial - note: "CONTESTED-RESOLVED: the map's MISSING is about a NON-OPERATOR performing it; the walks prove the route, not the person. Neither record was wrong. SINCE 2026-08-21 there is a second, harder blocker and it is not about the person at all: for the 40 of 53 apps that declare no data drive the off-site restore REFUSES before it starts, telling the customer a running app 'nincs telepítve' and to reinstall it to the same place — which those apps give them no way to choose (R-356). Those are exactly the apps whose whole dataset is a named volume. Until that is fixed, most customers cannot self-restore off-site even in principle." + note: "Walked 2026-10-29 night on scratch 9202 through the household's restore button (the page's own POST): vikunja from the 02:30 local point in 79 s — the project made before the point came back, the one made after it is gone, the app runs, no hold; a restore that meets a busy volume puts the live data back (R-1256 H5). The route needs no help. Still open: no person who is not the operator has done it (claim.never-by-non-operator). The old off-site blocker (R-356) shipped in controller v0.219.0." sources: - capability-map: "A customer (not the operator) performs a restore via UI alone" - register: "R-201" - register: "R-356" - evidence: "audits/REPORT-DRILL-backup-truth-2026-08-21.md" + - evidence: "audits/night-2026-10-28/part5-claims/selfrestore-walk.log" verified: - date: 2026-08-22 - verdict: contested + date: 2026-10-29 + verdict: confirmed depth: source-read diff --git a/documentation/audits/night-2026-10-28/bench/wanderer-edge/run/evidence/MV-wanderer/to-full.log b/documentation/audits/night-2026-10-28/bench/wanderer-edge/run/evidence/MV-wanderer/to-full.log index 11a420be..1ae39ee7 100644 --- a/documentation/audits/night-2026-10-28/bench/wanderer-edge/run/evidence/MV-wanderer/to-full.log +++ b/documentation/audits/night-2026-10-28/bench/wanderer-edge/run/evidence/MV-wanderer/to-full.log @@ -16,7 +16,7 @@ wanderer-db | 2026/10/28 22:36:58 Settings synced for index [lists] wanderer-db | └─ Dashboard: http://0.0.0.0:8090/_/ wanderer-db | wanderer-db | (!) Launch the URL below in the browser if it hasn't been open already to create your first superuser account: -wanderer-db | http://0.0.0.0:8090/_/#/pbinstall/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjb2xsZWN0aW9uSWQiOiJwYmNfMzE0MjYzNTgyMyIsImV4cCI6MTc5MzIyODgxOCwiaWQiOiJyZ3ljaHVzOGhnMnhpZmUiLCJyZWZyZXNoYWJsZSI6ZmFsc2UsInR5cGUiOiJhdXRoIn0.__xnQemwXA-zL0OMmzWpG4C0Zh0Bll-zJpnaqbpzbDU +wanderer-db | http://0.0.0.0:8090/_/#/pbinstall/ wanderer-db | (you can also create your first superuser by running: /pocketbase superuser upsert EMAIL PASS) wanderer-db | wanderer-search | 8888b d8888 Y8P 888 Y8P 888 diff --git a/documentation/audits/night-2026-10-28/part5-claims/selfrestore-walk.log b/documentation/audits/night-2026-10-28/part5-claims/selfrestore-walk.log new file mode 100644 index 00000000..f8bc03e4 --- /dev/null +++ b/documentation/audits/night-2026-10-28/part5-claims/selfrestore-walk.log @@ -0,0 +1,40 @@ +02:34:52 gate: tasks is gated — passed as the household (cookie set) +02:34:53 vikunja: readback of the seeded project http=200 ok=True +02:34:53 vikunja: readback of the seeded project http=200 ok=True +02:34:53 before: C1 True, C2 True +02:34:53 restoring from the newest point {'time': '2026-10-29T01:30:30Z', 'short_id': 'helyi', 'tier': 1, 'drive_label': 'Belső SSD (rendszer)'} +02:34:53 [R] restoring vikunja from snapshot 'helyi' (of 1 offered) +02:34:53 [R] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started'] +02:34:53 + 0.0s restore (True, None, None) +02:36:12 + 78.9s restore (False, None, None) +02:36:12 [R] after restore: state=running hold=None phase=None +02:36:16 restore result: {"ok": true, "snapshot_id": "helyi", "snapshots": [{"time": "2026-10-29T01:30:30Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}], "http": "HTTP/2 302", "location": ["location: /backups/restore?flash=flash.restore.started"], "seconds": 79.0, "state_after": "running", "hold_after": null, "observables_after": {"pinned_images": {"vikunja": "vikunja/vikunja:2.7.1"}, "installed_images": {"vikunja": "vikunja/vikunja:2.7.1"}, "catalog_images": {"vikunja": "vikunja/vikunja:2.7.1"}, "live_compose_image_lines": ["image: vikunja/vikunja:2.7.1@sha256:f7dc8428d10aef17f3075886d3c1c022f566ec456f88985f02c137f4fc534576"], "docker_inspect": ["vikunja vikunja/vikunja:2.7.1@sha256:f7dc8428d10aef17f3075886d3c1c022f566ec456f88985f02c137f4fc534576 running=true restarts=0"]}} +02:36:19 controller log: 2026/10/29 01:34:53 handlers.go:1896: [WARN] [web] Restore requested (async): stack=vikunja, snapshot=helyi from 172.18.0.2:38118 +2026/10/29 01:34:53 restore_unit.go:410: [INFO] [backup] Restoring vikunja from recovery unit /mnt/sys_drive/felhom-data/backups/primary/vikunja: images=1, secrets recovered=1/1, data_keys=0 +2026/10/29 01:34:54 manager.go:1446: [INFO] [stacks] Stopping stack: vikunja +2026/10/29 01:34:54 manager.go:1455: [INFO] [stacks] Stack vikunja stopped successfully (took 0.3s) +2026/10/29 01:35:07 offbox_restore_undo.go:169: [INFO] [offbox] vikunja: the live volumes are copied before the restore (2 copied, 0 new from the snapshot) in 12.296s — /mnt/sys_drive/felhom-data/backups/restore-undo/vikunja/20261029T013454Z (R-893) +2026/10/29 01:35:07 restore_crash_putback.go:89: [INFO] [backup] vikunja: the restore copy is recorded — a power cut from here on puts the app back on start (R-1195) +2026/10/29 01:35:07 restore.go:225: [INFO] [backup] Restoring Docker volume vikunja_vikunja_data for vikunja +2026/10/29 01:35:07 restore.go:225: [INFO] [backup] Restoring Docker volume vikunja_vikunja_db for vikunja +2026/10/29 01:35:17 restore.go:274: [INFO] [backup] Restored 2 Docker volume(s) for vikunja +2026/10/29 01:35:17 [INFO] [stacks] SaveAppConfig: saved config for vikunja +2026/10/29 01:35:17 deploy.go:911: [INFO] [stacks] Redeploying vikunja from recovery unit with 3 env vars +2026/10/29 01:35:17 pin.go:93: [INFO] [stacks] pin vikunja: vikunja=vikunja/vikunja:2.7.1 +2026/10/29 01:35:17 manager.go:1357: [INFO] [stacks] Starting stack: vikunja +2026/10/29 01:35:17 [INFO] [stacks] SaveAppConfig: saved config for vikunja +2026/10/29 01:35:18 manager.go:1372: [INFO] [stacks] Stack vikunja started successfully (took 0.4s) +2026/10/29 01:35:18 [INFO] [stacks] SaveAppConfig: saved config for vikunja +2026/10/29 01:35:18 installed.go:420: [INFO] [stacks] installed-images vikunja: recorded 1 service(s) (vikunja=vikunja/vikunja:2.7.1 (sha256:f7dc8428d10a…)) +2026/10/29 01:35:21 manager.go:1817: [INFO] [stacks] Stack vikunja post-start status: +2026/10/29 01:35:21 manager.go:1820: [INFO] [stacks] vikunja vikunja/vikunja:2.7.1@sha256:f7dc8428d10aef17f3075886d3c1c022f566ec456f88985f02c137f4fc534576 running Up 3 seconds +2026/10/29 01:36:12 restore_unit.go:591: [INFO] [backup] Restore-from-unit completed: vikunja — 2 volume(s) of 2 listed, 0 database(s) of 0 listed +2026/10/29 01:36:12 handlers.go:1911: [INFO] [web] Restore completed (async): stack=vikunja in 1m18.578891305s (volumes 2/2, dbs 0/0) + +02:36:19 after: state=running hold=None +02:36:19 vikunja: readback of the seeded project http=200 ok=True +02:36:19 C1 readback (want True): True +02:36:20 vikunja: the seeded account no longer authenticates :: http=403 {"code":1011,"message":"Wrong username or password."} + +02:36:20 C2 readback (want False — newer than the point): False +02:36:20 restore record: {"ok": true, "data": {"running": false, "op": "restore", "stack": "vikunja", "started_at": "2026-10-29T01:34:53.636426109Z", "last": {"op": "restore", "stack": "vikunja", "ok": true, "message": "A(z) vikunja: 2 adatk\u00f6tet vissza\u00e1ll\u00edtva \u2014 az alkalmaz\u00e1s \u00fajraindult.", "finished_at": "2026-10-29T01:36:12.215711729Z"}, "last_recent": true}} diff --git a/documentation/audits/night-2026-10-28/part5-claims/walk_selfrestore.py b/documentation/audits/night-2026-10-28/part5-claims/walk_selfrestore.py new file mode 100644 index 00000000..9993ed58 --- /dev/null +++ b/documentation/audits/night-2026-10-28/part5-claims/walk_selfrestore.py @@ -0,0 +1,29 @@ +#!/usr/bin/env python3 +"""walk_selfrestore.py — claim fail.customer-self-restore on scratch 9202 (night 2026-10-28), front doors only. +After the H5 proof: vikunja holds C1 (in the 02:30 restore point) and C2 (made after it). Press the household's restore +with nothing blocked. Want: the restore succeeds, C1 reads back, C2 is GONE (the point is older), the app runs, no hold.""" +import json, os, sys, time +sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts") +os.environ.setdefault("SC", "/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0b76c4d7-9b39-4e4e-869d-d0d2c6006331/scratchpad/rt-9202") +os.environ.setdefault("TARGET", "9202") +import box_walk as w +import upgrade_fixtures_box as fb +EV = "/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0b76c4d7-9b39-4e4e-869d-d0d2c6006331/scratchpad/night/p1/h5" +st = json.load(open(os.path.join(EV, ".state.json"))) +fx = fb.FIXTURES["vikunja"]; SUB = getattr(fx, "sub", "vikunja"); APP = "vikunja" +w.login() +w.say(f"before: C1 {fx.verify(w, SUB, st['c1'], w.say)}, C2 {fx.verify(w, SUB, st['c2'], w.say)}") +snaps = w.snapshots(APP) +newest = max(snaps, key=lambda x: x.get("time") or "") +w.say(f"restoring from the newest point {newest}") +t0 = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) +res = w.restore(APP, snapshot_id=newest.get("id") or newest.get("snapshot_id") or newest.get("short_id")) +w.say(f"restore result: {json.dumps(res)[:800]}") +w.say("controller log: " + w.guest(f"docker logs --since {t0} felhom-controller 2>&1 | grep -E '\\[(INFO|WARN|ERROR)\\]' | grep -E 'vikunja|restore|Restor' | grep -v ParseCompose | tail -25")) +s = w.stack(APP) +w.say(f"after: state={s.get('state')} hold={s.get('hold_reason')!r}") +w.say(f"C1 readback (want True): {fx.verify(w, SUB, st['c1'], w.say)}") +w.say(f"C2 readback (want False — newer than the point): {fx.verify(w, SUB, st['c2'], w.say)}") +code, d = w.ctl("GET", "/api/backup/restore-status") +w.say(f"restore record: {json.dumps(d)[:600]}") +open(os.path.join(EV, "selfrestore.log"), "a").write("\n".join(w.LOG) + "\n") diff --git a/documentation/audits/night-2026-10-28/r1238-plex/plex-fixture-try.log b/documentation/audits/night-2026-10-28/r1238-plex/plex-fixture-try.log new file mode 100644 index 00000000..e3351047 --- /dev/null +++ b/documentation/audits/night-2026-10-28/r1238-plex/plex-fixture-try.log @@ -0,0 +1,17 @@ +00:18:55 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['PLEX_CLAIM', 'HDD_PATH'] +00:18:55 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +00:19:26 [1] deployed, controller state=running, pinned={'plex': 'plexinc/pms-docker:1.43.5.11029-6bf8af3f0'} +00:19:26 deploy ok=True +00:19:26 gate: plex is gated — passed as the household (cookie set) +00:20:06 plex: POST /library/sections through the front door http=401 +00:20:06 plex: the front door was not taken as local (Unauthorized

401 Unauthorized

) — asking from Plex's own container +00:20:09 plex: POST /library/sections from its own container http=201 +00:20:09 plex: seeded library section felhomfa735b43 uuid=196fa995-e45f-4cb7-8a65-3f189ffe9b9b (route own-container, server 1.43.5.11029-6bf8af3f0) +00:20:09 seed -> {'name': 'felhomfa735b43', 'uuid': '196fa995-e45f-4cb7-8a65-3f189ffe9b9b', 'route': 'own-container'} +00:20:16 plex: section felhomfa735b43 read back (server 1.43.5.11029-6bf8af3f0, route own-container), same uuid=True, folders=['/media'] +00:20:16 verify -> True +00:20:32 pre-pull FROM: docker.io/plexinc/pms-docker:1.41.4.9463-630c9f557 + +00:20:46 [X] stop -> 200 {'ok': True, 'message': 'Stack plex stop completed'} +00:21:18 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'plex', 'volumes_removed': ['plex_plex_config', 'plex_plex_transcode'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'us +00:21:27 [X] after remove: deployed=False leftovers='' diff --git a/documentation/audits/night-2026-10-28/r1238-plex/plex_try.py b/documentation/audits/night-2026-10-28/r1238-plex/plex_try.py new file mode 100644 index 00000000..250d68f6 --- /dev/null +++ b/documentation/audits/night-2026-10-28/r1238-plex/plex_try.py @@ -0,0 +1,19 @@ +import os, sys, json +sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts") +os.environ.setdefault("SC", "/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0b76c4d7-9b39-4e4e-869d-d0d2c6006331/scratchpad/rt-9202"); os.environ.setdefault("TARGET", "9202") +import box_walk as w +import upgrade_fixtures_box as fb +EV = "/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0b76c4d7-9b39-4e4e-869d-d0d2c6006331/scratchpad/night/plex" +w.login() +fx = fb.FIXTURES["plex"] +try: + ok = w.deploy("plex", "plex") + w.say(f"deploy ok={ok}") + t = fx.seed(w, "plex", w.say) + w.say(f"seed -> {t}") + if t: + w.say(f"verify -> {fx.verify(w, 'plex', t, w.say)}") + w.say("pre-pull FROM: " + w.guest("docker pull -q plexinc/pms-docker:1.41.4.9463-630c9f557 2>&1 | tail -1", timeout=900)) +finally: + w.remove("plex") + open(os.path.join(EV, "plex-fixture-try.log"), "w").write("\n".join(w.LOG) + "\n") diff --git a/documentation/audits/night-2026-10-28/r1256-restore/h5-live/h5_live.py b/documentation/audits/night-2026-10-28/r1256-restore/h5-live/h5_live.py new file mode 100644 index 00000000..47028fb9 --- /dev/null +++ b/documentation/audits/night-2026-10-28/r1256-restore/h5-live/h5_live.py @@ -0,0 +1,66 @@ +#!/usr/bin/env python3 +"""h5_live.py seed | prove — R-1256 H5 live proof on scratch 9202 (night 2026-10-28), through the front doors. +seed : install vikunja, seed project C1 through its API, put a 1 GiB ballast file in its second volume (vikunja_db, so + the restore's second volume takes seconds — the put-back then comes long after the first volume's refusal). + The night backup (02:30) then makes the restore point. +prove: seed project C2 (newer than the backup); a created-only container holds the FIRST volume (vikunja_data) so + `docker volume rm -f` fails; a watcher removes it as soon as the controller logs the refusal; press the + household's restore. Want: the first volume is NOT written into (log), the restore fails, the put-back brings the + live data back (C1 AND C2 read back), the app runs, NO hold. +State (passwords) stays in the scratchpad, never printed.""" +import json, os, sys, time +sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts") +os.environ.setdefault("SC", "/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0b76c4d7-9b39-4e4e-869d-d0d2c6006331/scratchpad/rt-9202") +os.environ.setdefault("TARGET", "9202") +import box_walk as w +import upgrade_fixtures_box as fb +EV = "/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0b76c4d7-9b39-4e4e-869d-d0d2c6006331/scratchpad/night/p1/h5" +STATE = os.path.join(EV, ".state.json") +fx = fb.FIXTURES["vikunja"] +SUB = getattr(fx, "sub", "vikunja") +APP = "vikunja" +w.login() +if sys.argv[1] == "seed": + assert w.deploy(APP, SUB), "deploy failed" + t1 = fx.seed(w, SUB, w.say) + assert t1, "seed C1 failed" + vols = w.guest("docker volume ls --format '{{.Name}}' | grep -E '^vikunja' | sort").split() + w.say(f"volumes: {vols}") + dbv = [v for v in vols if v.endswith("_db")][0] + w.say(w.guest(f"docker run --rm -v {dbv}:/x alpine sh -c 'dd if=/dev/urandom of=/x/r1256-ballast.bin bs=1M count=1024 2>&1 | tail -1; ls -la /x'", timeout=900)) + json.dump({"c1": t1, "vols": vols}, open(STATE, "w")); os.chmod(STATE, 0o600) + w.say(f"C1 readback: {fx.verify(w, SUB, t1, w.say)}") +elif sys.argv[1] == "prove": + st = json.load(open(STATE)) + snaps = w.snapshots(APP) + w.say(f"snapshots offered: {json.dumps(snaps)[:400]}") + assert snaps, "no restore point yet" + t2 = fx.seed(w, SUB, w.say) # C2: newer than the backup — the put-back must bring it back + assert t2, "seed C2 failed" + st["c2"] = t2; json.dump(st, open(STATE, "w")) + datav = [v for v in st["vols"] if v.endswith("_data")][0] + w.say("blocker: " + w.guest(f"docker rm -f r1256-blocker >/dev/null 2>&1; docker create --name r1256-blocker -v {datav}:/x alpine true")) + w.say("rm -f with the blocker (premise): " + w.guest(f"docker volume rm -f {datav} 2>&1; echo rc=$?")) + w.guest("rm -f /root/r1256-watch.log; nohup sh -c 'timeout 1800 docker logs -f --since 1s felhom-controller 2>&1 | " + f"grep -m1 \"Failed to remove volume {datav} before the restore\" > /root/r1256-watch.log; date -u +%T.%N >> /root/r1256-watch.log; " + "docker rm -f r1256-blocker >> /root/r1256-watch.log 2>&1; date -u +%T.%N >> /root/r1256-watch.log' >/dev/null 2>&1 &") + time.sleep(2) + t0 = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) + newest = max(snaps, key=lambda x: x.get("time") or "") + w.say(f"restoring from the newest point {newest}") + res = w.restore(APP, snapshot_id=newest.get("id") or newest.get("snapshot_id") or newest.get("short_id")) + w.say(f"restore result: {json.dumps(res)[:1500]}") + w.say("watcher: " + w.guest("cat /root/r1256-watch.log")) + w.say("controller log: " + w.guest(f"docker logs --since {t0} felhom-controller 2>&1 | grep -E 'vikunja|R-1256|put-back|put back|Failed to remove' | tail -40")) + st_now = w.stack(APP) + w.say(f"after: state={st_now.get('state')} hold={st_now.get('hold_reason')!r}") + w.say(f"C1 readback: {fx.verify(w, SUB, st['c1'], w.say)}") + w.say(f"C2 readback: {fx.verify(w, SUB, t2, w.say)}") + code, d = w.ctl("GET", "/api/backup/restore-status") + w.say(f"restore record: {json.dumps(d)[:800]}") + w.say("blocker left: " + w.guest("docker ps -a --filter name=r1256-blocker --format '{{.Names}}'")) + w.say("copies: " + w.guest("find /mnt/felhom-drives/scratch_hdd/backups/restore-undo -maxdepth 2")) +elif sys.argv[1] == "teardown": + w.guest("docker rm -f r1256-blocker >/dev/null 2>&1; rm -f /root/r1256-watch.log") + w.remove(APP) +open(os.path.join(EV, f"{sys.argv[1]}.log"), "a").write("\n".join(w.LOG) + "\n") diff --git a/documentation/audits/night-2026-10-28/r1256-restore/h5-live/prove.log b/documentation/audits/night-2026-10-28/r1256-restore/h5-live/prove.log new file mode 100644 index 00000000..2e88fbff --- /dev/null +++ b/documentation/audits/night-2026-10-28/r1256-restore/h5-live/prove.log @@ -0,0 +1,77 @@ +02:32:48 snapshots offered: [{"time": "2026-10-29T01:30:30Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}] +02:32:48 gate: tasks is gated — passed as the household (cookie set) +02:32:48 vikunja: register http=200 +02:32:49 vikunja: create project http=201 +02:32:52 blocker: 227a359b77ee3d016002043a0a1fa3bb54d990bf50954535d2dd65337bb8336c + +02:32:55 rm -f with the blocker (premise): Error response from daemon: remove vikunja_vikunja_data: volume is in use - [072afe32e15994b46c232d1850bd6bf04fe5edb6bf27ffd18238f35911124402, 227a359b77ee3d016002043a0a1fa3bb54d990bf50954535d2dd65337bb8336c] +rc=1 + +02:33:00 restoring from the newest point {'time': '2026-10-29T01:30:30Z', 'short_id': 'helyi', 'tier': 1, 'drive_label': 'Belső SSD (rendszer)'} +02:33:00 [R] restoring vikunja from snapshot 'helyi' (of 1 offered) +02:33:00 [R] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started'] +02:33:00 + 0.0s restore (True, None, None) +02:33:59 + 59.0s restore (False, None, None) +02:33:59 [R] after restore: state=running hold=None phase=None +02:34:03 restore result: {"ok": true, "snapshot_id": "helyi", "snapshots": [{"time": "2026-10-29T01:30:30Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}], "http": "HTTP/2 302", "location": ["location: /backups/restore?flash=flash.restore.started"], "seconds": 59.0, "state_after": "running", "hold_after": null, "observables_after": {"pinned_images": {"vikunja": "vikunja/vikunja:2.7.1"}, "installed_images": {"vikunja": "vikunja/vikunja:2.7.1"}, "catalog_images": {"vikunja": "vikunja/vikunja:2.7.1"}, "live_compose_image_lines": ["image: vikunja/vikunja:2.7.1@sha256:f7dc8428d10aef17f3075886d3c1c022f566ec456f88985f02c137f4fc534576"], "docker_inspect": ["vikunja vikunja/vikunja:2.7.1@sha256:f7dc8428d10aef17f3075886d3c1c022f566ec456f88985f02c137f4fc534576 running=true restarts=0"]}} +02:34:06 watcher: 2026/10/29 01:33:31 restore.go:231: [ERROR] [backup] Failed to remove volume vikunja_vikunja_data before the restore: Error response from daemon: remove vikunja_vikunja_data: volume is in use - [227a359b77ee3d016002043a0a1fa3bb54d990bf50954535d2dd65337bb8336c] — exit status 1 (not written into) +01:33:33.292357758 +r1256-blocker +01:33:41.548622640 + +02:34:09 controller log: 2026/10/29 01:33:31 restore_crash_putback.go:89: [INFO] [backup] vikunja: the restore copy is recorded — a power cut from here on puts the app back on start (R-1195) +2026/10/29 01:33:31 restore.go:225: [INFO] [backup] Restoring Docker volume vikunja_vikunja_data for vikunja +2026/10/29 01:33:31 restore.go:231: [ERROR] [backup] Failed to remove volume vikunja_vikunja_data before the restore: Error response from daemon: remove vikunja_vikunja_data: volume is in use - [227a359b77ee3d016002043a0a1fa3bb54d990bf50954535d2dd65337bb8336c] — exit status 1 (not written into) +2026/10/29 01:33:31 restore.go:225: [INFO] [backup] Restoring Docker volume vikunja_vikunja_db for vikunja +2026/10/29 01:33:46 restore.go:269: [DEBUG] [backup] Volume vikunja_vikunja_db restored successfully +2026/10/29 01:33:46 restore.go:274: [INFO] [backup] Restored 1 Docker volume(s) for vikunja +2026/10/29 01:33:46 restore_unit.go:554: [ERROR] [backup] volume restore for vikunja: failed to restore 1 volume(s): [vikunja_vikunja_data] — NOT replaying the database copy; putting the live data back +2026/10/29 01:33:46 manager.go:1443: [DEBUG] [stacks] StopStack vikunja: current state=stopped deployed=true containers=0 +2026/10/29 01:33:46 manager.go:1446: [INFO] [stacks] Stopping stack: vikunja +2026/10/29 01:33:46 manager.go:1701: [DEBUG] Running: docker compose down (in /opt/docker/stacks/vikunja) +2026/10/29 01:33:46 manager.go:1455: [INFO] [stacks] Stack vikunja stopped successfully (took 0.1s) +2026/10/29 01:33:46 restore.go:225: [INFO] [backup] Restoring Docker volume vikunja_vikunja_data for vikunja +2026/10/29 01:33:47 restore.go:269: [DEBUG] [backup] Volume vikunja_vikunja_data restored successfully +2026/10/29 01:33:47 restore.go:225: [INFO] [backup] Restoring Docker volume vikunja_vikunja_db for vikunja +2026/10/29 01:33:58 restore.go:269: [DEBUG] [backup] Volume vikunja_vikunja_db restored successfully +2026/10/29 01:33:58 restore.go:274: [INFO] [backup] Restored 2 Docker volume(s) for vikunja +2026/10/29 01:33:58 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/vikunja — 3 env vars, 1 encrypted, 1 sensitive fields +2026/10/29 01:33:58 [INFO] [stacks] SaveAppConfig: saved config for vikunja +2026/10/29 01:33:58 deploy.go:911: [INFO] [stacks] Redeploying vikunja from recovery unit with 3 env vars +2026/10/29 01:33:58 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/vikunja — 3 env vars, 0 encrypted, 1 sensitive fields +2026/10/29 01:33:58 [INFO] [stacks] SaveAppConfig: saved config for vikunja +2026/10/29 01:33:58 pin.go:93: [INFO] [stacks] pin vikunja: vikunja=vikunja/vikunja:2.7.1 +2026/10/29 01:33:58 offbox_restore_undo.go:253: [WARN] [offbox] vikunja: the restore failed (failed to restore 1 volume(s): [vikunja_vikunja_data]) — the live volumes and definition are put back; starting the app (R-893) +2026/10/29 01:33:58 manager.go:1354: [DEBUG] [stacks] StartStack vikunja: current state=stopped deployed=true +2026/10/29 01:33:58 manager.go:1357: [INFO] [stacks] Starting stack: vikunja +2026/10/29 01:33:58 manager.go:1364: [DEBUG] [stacks] StartStack vikunja: prepared 9 env vars for compose +2026/10/29 01:33:58 manager.go:1701: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/vikunja) +2026/10/29 01:33:58 manager.go:1372: [INFO] [stacks] Stack vikunja started successfully (took 0.4s) +2026/10/29 01:33:58 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/vikunja — 3 env vars, 0 encrypted, 1 sensitive fields +2026/10/29 01:33:58 [INFO] [stacks] SaveAppConfig: saved config for vikunja +2026/10/29 01:33:58 installed.go:420: [INFO] [stacks] installed-images vikunja: recorded 1 service(s) (vikunja=vikunja/vikunja:2.7.1 (sha256:f7dc8428d10a…)) +2026/10/29 01:33:58 handlers.go:1907: [ERROR] [web] Restore failed (async): stack=vikunja: a(z) vikunja visszaállítása sikertelen. Nem változott semmi: az alkalmazás adatai visszakerültek a visszaállítás előtti állapotba, és az alkalmazás újra fut +2026/10/29 01:33:59 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/stacks/vikunja +2026/10/29 01:33:59 router.go:78: [DEBUG] [api] GET /api/stacks/vikunja (path=/stacks/vikunja) +2026/10/29 01:33:59 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/stacks/vikunja +2026/10/29 01:33:59 router.go:78: [DEBUG] [api] GET /api/stacks/vikunja (path=/stacks/vikunja) +2026/10/29 01:34:01 manager.go:1701: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/vikunja) +2026/10/29 01:34:01 manager.go:1817: [INFO] [stacks] Stack vikunja post-start status: +2026/10/29 01:34:01 manager.go:1820: [INFO] [stacks] vikunja vikunja/vikunja:2.7.1@sha256:f7dc8428d10aef17f3075886d3c1c022f566ec456f88985f02c137f4fc534576 running Up 3 seconds +2026/10/29 01:34:04 healthprobe.go:187: [DEBUG] Health probe vikunja: API GET :3456/api/v1/info → 200 (3ms) + +02:34:09 after: state=running hold=None +02:34:10 vikunja: readback of the seeded project http=200 ok=True +02:34:10 C1 readback: True +02:34:10 vikunja: readback of the seeded project http=200 ok=True +02:34:10 C2 readback: True +02:34:10 restore record: {"ok": true, "data": {"running": false, "op": "restore", "stack": "vikunja", "started_at": "2026-10-29T01:33:00.84310832Z", "last": {"op": "restore", "stack": "vikunja", "ok": false, "message": "Vissza\u00e1ll\u00edt\u00e1s sikertelen: a(z) vikunja vissza\u00e1ll\u00edt\u00e1sa sikertelen. Nem v\u00e1ltozott semmi: az alkalmaz\u00e1s adatai visszaker\u00fcltek a vissza\u00e1ll\u00edt\u00e1s el\u0151tti \u00e1llapotba, \u00e9s az alkalmaz\u00e1s \u00fajra fut", "finished_at": "2026-10-29T01:33:58.681576827Z"}, "last_recent": true}} +02:34:13 blocker left: +02:34:16 copies: /mnt/felhom-drives/scratch_hdd/backups/restore-undo +/mnt/felhom-drives/scratch_hdd/backups/restore-undo/r1256probe-old +/mnt/felhom-drives/scratch_hdd/backups/restore-undo/r1256probe-old/20261001T000000Z +/mnt/felhom-drives/scratch_hdd/backups/restore-undo/r1256probe-young +/mnt/felhom-drives/scratch_hdd/backups/restore-undo/r1256probe-young/20261028T213014Z +/mnt/felhom-drives/scratch_hdd/backups/restore-undo/paperless +/mnt/felhom-drives/scratch_hdd/backups/restore-undo/paperless/20261005T010000Z + diff --git a/documentation/audits/night-2026-10-28/r1256-restore/h5-live/seed.log b/documentation/audits/night-2026-10-28/r1256-restore/h5-live/seed.log new file mode 100644 index 00000000..689c15a7 --- /dev/null +++ b/documentation/audits/night-2026-10-28/r1256-restore/h5-live/seed.log @@ -0,0 +1,17 @@ +22:34:34 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +22:34:44 [1] deployed, controller state=running, pinned={'vikunja': 'vikunja/vikunja:2.7.1'} +22:34:44 gate: tasks is gated — passed as the household (cookie set) +22:34:44 vikunja: register http=200 +22:34:44 vikunja: create project http=201 +22:34:47 volumes: ['vikunja_vikunja_data', 'vikunja_vikunja_db'] +22:35:14 1073741824 bytes (1.0GB) copied, 4.619925 seconds, 221.6MB/s +total 1051464 +drwxr-xr-x 2 root root 4096 Oct 28 21:34 . +drwxr-xr-x 1 root root 4096 Oct 28 21:34 .. +-rw-r--r-- 1 root root 1073741824 Oct 28 21:34 r1256-ballast.bin +-rw-r--r-- 1 root root 4096 Oct 28 21:34 vikunja.db +-rw-r--r-- 1 root root 32768 Oct 28 21:34 vikunja.db-shm +-rw-r--r-- 1 root root 2908752 Oct 28 21:34 vikunja.db-wal + +22:35:15 vikunja: readback of the seeded project http=200 ok=True +22:35:15 C1 readback: True diff --git a/documentation/audits/night-2026-10-28/r1338-meili/bench-edge2/run/evidence/MV-wanderer/to-full.log b/documentation/audits/night-2026-10-28/r1338-meili/bench-edge2/run/evidence/MV-wanderer/to-full.log index ef8cc042..6d59f244 100644 --- a/documentation/audits/night-2026-10-28/r1338-meili/bench-edge2/run/evidence/MV-wanderer/to-full.log +++ b/documentation/audits/night-2026-10-28/r1338-meili/bench-edge2/run/evidence/MV-wanderer/to-full.log @@ -17,7 +17,7 @@ wanderer-db | 2026/10/28 22:59:30 Settings synced for index [actors] wanderer-db | └─ Dashboard: http://0.0.0.0:8090/_/ wanderer-db | wanderer-db | (!) Launch the URL below in the browser if it hasn't been open already to create your first superuser account: -wanderer-db | http://0.0.0.0:8090/_/#/pbinstall/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjb2xsZWN0aW9uSWQiOiJwYmNfMzE0MjYzNTgyMyIsImV4cCI6MTc5MzIzMDE3MCwiaWQiOiJpb2hzaHpjeXp6bGE3b2oiLCJyZWZyZXNoYWJsZSI6ZmFsc2UsInR5cGUiOiJhdXRoIn0.66F82yj7TdPBVesWeuZf2Gi5XFXY0VnF_yIKY-98f_g +wanderer-db | http://0.0.0.0:8090/_/#/pbinstall/ wanderer-db | (you can also create your first superuser by running: /pocketbase superuser upsert EMAIL PASS) wanderer | wanderer | > wanderer@0.21.2 start diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 04ff719c..49f983f6 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -152,7 +152,7 @@ stopping line that lies. | **R-1331** | Apps & catalog | P3 | **Paperless-ngx 2.20.15 carries Python Brotli 1.1.0 (CVE-2026-111497, fixed 1.1.1/1.2.1)** — its web side only COMPRESSES (whitenoise, compression_middleware); the DECODERS are urllib3's and httpx's, for replies to requests Paperless itself makes (its own outbound calls), never for what a visitor sends. Read in the running container on demo-hp 9201, 2026-10-27. 2026-10-28: the newest release 3.3.4 still carries Brotli 1.2.0 and is a new main version — no fixed release yet (`audits/runtime-fixes-2026-10-28/TABLE.md`). | OPEN — no step tonight (decided by CC unattended — operator may reverse): the bad path needs a hostile server Paperless calls | — | take the first Paperless release with Brotli ≥ 1.2.1 as a normal step | CC | | **R-1336** | Apps & catalog | P3 | **ActualBudget has no release with the HTTP2Shell fix (CVE-2026-111497): 26.9.0 carries Node.js 24.20.0 and the newest 26.10.0 still Node.js 24.21.0 (fixed: 24.22.1), read from both images.** `node app.js` serves the app; the setup gate covers it until the first account exists, then its own password. No box runs it. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) | upstream: an image with Node ≥ 24.22.1 | the watch lists it nightly (R-1327); take that release as a lane step | CC | | **R-1337** | Apps & catalog | P3 | **Calibre-Web (v4.0.8, the newest) loads a bad Brotli decoder: its `cps.py` maps Python Brotli 1.2.0 (fixed 1.2.1), Pillow's libbrotlidec 1.2.0 and Debian's libbrotlidec 1.1.0** — read from the process's maps on demo-hp 9201 (`audits/runtime-fixes-2026-10-28/calibre-web-brotli.txt`). Exposure: the Python decoder is urllib3's, for replies to the app's own outbound calls (book metadata), not for what a visitor sends — the Paperless shape (R-1331). No fixed release exists. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) | upstream: an image with Brotli ≥ 1.2.1 | take that release as a lane step; Debian's libbrotli1 fixed version is not known (R-1327's table has no Debian line) | CC | -| **R-1338** | Apps & catalog | P3 | **Wanderer's Meilisearch fills whatever memory limit it gets: 92–100 % anon of 256 MB (bench and 9202), still 88.8 % of 384 MB on the bench (2026-10-28)** — it sizes its indexing memory from the memory it sees (its default: 2/3 of the available RAM), so a higher limit alone does not end the `memory_tight` mark of Wanderer's step. No kill in any run. `audits/dockerhub-bigsteps-2026-10-28/wanderer/` | OPEN — **2026-10-28 evening: `MEILI_MAX_INDEXING_MEMORY` = 128 MiB (a third of 384 MB) measured, NOT pushed** (`audits/gitea-wanderer-2026-10-28/wanderer-meili/`): on the bench, under the harness's load, the search container peaked at 17.7 % (84 MB; 88.8 % without the cap, same load); on 9202 the cap was in the container (read back) and it still held 95.7 % anon (363 MB steady over 6 min, no kill). The box result decides: not shown to help, so not pushed. Also: the catalog's ladder has no entry type for a re-measure of a definition change at the SAME digests (`ladder.py`: a re-test whose digest equals its digest_from tests nothing new), so the v0.21.2 entry's `memory_tight` mark cannot be replaced without an image move. | — | find why the box and the bench differ (9202 has 512 MB swap, the bench none; Meilisearch's other memory settings); then decide the cap — the mark changes with the next Wanderer image move | CC | +| **R-1338** | Apps & catalog | P3 | **Wanderer's Meilisearch fills whatever memory limit it gets: 92–100 % anon of 256 MB (bench and 9202), still 88.8 % of 384 MB on the bench (2026-10-28)** — it sizes its indexing memory from the memory it sees (its default: 2/3 of the available RAM), so a higher limit alone does not end the `memory_tight` mark of Wanderer's step. No kill in any run. `audits/dockerhub-bigsteps-2026-10-28/wanderer/` | OPEN — **2026-10-28 night: MEASURED — it is the measurement, not Wanderer, and the cap is not needed** (`audits/night-2026-10-28/r1338-meili/`; `09` §3 decision 22 refined). Meilisearch frees with MADV_FREE: the given-back pages stay in `anon` but sit on the inactive FILE list (`file` ~1 MB, `inactive_file` ~290 MB) and the kernel takes them first — 83–96 % `anon` on the bench (no swap) AND on 9202, with and without the cap, 0 kills; it uses ~100 MB. The evening's 17.7 % was the container after the swap, not the cap. **Fixed in the tools:** the mark reads the live anon (catalog `scripts/memfigure.py`, the larger of two estimates, errs high; `upgrade-test.py`, `box_walk.py` soak; `test_memory_live.py` red-proved). Re-run: bench Wanderer v0.20.0 → v0.21.2 `proven`, **no marks** (search 83.3 % anon, 27.7 % live); 9202 95.2 % anon, 25.1 % live. **Prepared, NOT pushed** (the night's catalog fence): the tool change + Wanderer's entry without `memory_tight` (`audits/night-2026-10-28/catalog-morning/`; the catalog gates passed in a scratch clone). **Earlier, 2026-10-28 evening: `MEILI_MAX_INDEXING_MEMORY` = 128 MiB (a third of 384 MB) measured, NOT pushed** (`audits/gitea-wanderer-2026-10-28/wanderer-meili/`): on the bench, under the harness's load, the search container peaked at 17.7 % (84 MB; 88.8 % without the cap, same load); on 9202 the cap was in the container (read back) and it still held 95.7 % anon (363 MB steady over 6 min, no kill). The box result decides: not shown to help, so not pushed. Also: the catalog's ladder has no entry type for a re-measure of a definition change at the SAME digests (`ladder.py`: a re-test whose digest equals its digest_from tests nothing new), so the v0.21.2 entry's `memory_tight` mark cannot be replaced without an image move. | the catalog push (the morning list) | push the prepared catalog change (the live figure + Wanderer's entry re-measured, `memory_tight` off — an existing entry's memory fields edited at the SAME digests, which the gates accept); then close | CC | ## App updates — 10 rows (P2 1, P3 9) @@ -177,7 +177,7 @@ stopping line that lies. | **R-232** | Backup & restore | P2 | **DooPlex's backup makes every copy inside the same box — and nothing tells anyone when it fails.** Surveyed read-only 2026-08-06 (`audits/RECON-dooplex-backup-2026-08-06.md`). **What works:** five sets, 14/14 successful runs in 14 days; a file was restored from the `data` repo and matched the live original **byte for byte**; every set except two is cross-disk; k3s is integrity-checked on every run. **What the matrix exposes, ranked:** (a) **`notify_failure` is a no-op** — `NOTIFY_ON_FAILURE=true` but `NOTIFY_WEBHOOK_URL` is commented out, so a failed backup notifies **nobody**; the project already has a working Resend path that CI uses. Cheapest item, and it makes every other failure visible. (b) **Nothing leaves the box** — no rclone, no remote repo, no off-site target anywhere; Longhorn's target is `nfs://192.168.0.180:` pointing at DooPlex itself, and the only outbound-looking cron pulls *inbound* from Hetzner for a different project. The machine that runs the hub managing the customers' off-site chain has no off-site copy of its own. (c) **The backup tree is a single writable path** and the restic repos are not append-only — one bad script or ransomware destroys every copy at once. (d) **Two same-disk sets**: `.claude-memory` and the PostgreSQL dumps, whose source directory sits *inside* the backup tree. (e) **Longhorn `retain=1`** — one generation per volume, so a corruption noticed a day late has no earlier copy. (f) **`/opt/backup/docs/BACKUP-RESTORE.md` does not exist** though the systemd unit advertises it. (g) **`secrets/restic-repo` has never held a snapshot** — `backup-secrets.sh` contains no `restic` call; the secrets are GPG files on `sda1` only. (h) **No restore has ever been run** beyond today's single-file probe — the matrix's "ever demonstrated?" column is otherwise entirely empty. **Not a finding:** the restic passphrase. The on-box copy is on `sdb1`, a different disk from the backups, and the **operator holds an offline copy out of band** — so a disk loss is recoverable. The narrow residual is that it is operator-held rather than system-held, unlike the customer case's hub-vaulted escrow, so it should be confirmed current and findable by someone else. **Nothing was changed by the recon.** | NARROWED — **2026-10-23 (operator yes D33, D34):** the hub DB now leaves every 6 hours (hub 0.167.0 + `scripts/hub-db-backup/`, alarm at 13 h; restored back byte-identical) and the released images and agent packages leave nightly (`scripts/release-copy/`, R-1227 closed). (c)–(g) unchanged. **NARROWED 2026-10-08 — owner Viktor.** (a) DONE with the operator's yes in chat: `notify_failure` now mails admin@felhom.eu through Resend; proven by one test mail that reached the inbox (`audits/day-2026-10-08/r232/`; no backup was started). (b) partly: the hub database leaves DooPlex nightly to ep0 (R-173); everything else stays on the box. (c)–(h) unchanged. **2026-10-09 (operator yes in chat): (b) and (h) DONE for Gitea and the secrets.** Nightly 00:20 `felhom-dooplex-offsite` pushes Gitea (614 MB of repositories, the `gitea` database dump taken first, `app.ini`) and the nightly GPG secrets export, encrypted with a NEW key, to ep0 `operator` (`host/dooplex-gitea`) on the hub-DB write-only token — no change on ep0; first push 57 s; restore test weekly (manifest, `git fsck` every repo, `pg_restore --list`), run once: 27 805 files, 10 repos OK; `DooplexGiteaOffsiteStale`/`DooplexGiteaRestoreTestStale` in force (`absent()` included, promtool + 2 red-proofs); every backup unit now mails on failure (`OnFailure=`, dry run reached the inbox). **First real Gitea restore:** into a throwaway on the bench, 10/10 repos, product `main` = live, a file byte for byte, a login (`runbooks/gitea-restore.md`). The hub came back too (R-173, closed). **The registry is left out on purpose:** its images rebuild from the code. **Correction to the recon §7:** the API mirror holds ONE repo (`homelab-manifests`), not all — Gitea had a single copy, on DooPlex. `audits/dooplex-survival-2026-10-09/` **LEFT:** (c) append-only for the on-box tree, (d)–(g) as written; the paper copy of the new key (operator). **READY** for the rest **2026-10-24 (attended): `DooplexGiteaOffsiteStale` fired from 00:57Z — not this path's fault.** The 00:20 push refused, as designed, a 12 h old Gitea dump: the PostgreSQL dump CronJob had failed every run since 2026-10-23 16:00Z (`28P01`) because the window reboot restarted the database and CNPG put the bootstrap secret's password in force (homelab `531023d`, operator's pick: the CronJob only). Fixed; dump, push and restore test run by hand (26 456 files, 12 repos `git fsck`, `gitea.dump` 1114 entries); cleared 10:51Z (`audits/dooplex-alarms-2026-10-24/`). Backup alarms now arrive as ONE mail every 4 h (homelab `22d7697`). **2026-10-23 night — survey again, read only (`audits/dooplex-survival-2026-10-22/PLAN.md`):** both nightly off-site jobs and both weekly restore tests green (copies 2026-10-22, tests 2026-10-18); the local tree is still the same box (2026-10-22 03:16 ok). New in the survey: the hub DB copy is up to 24 h old (G5: every 6 h — the operator's yes; a DooPlex change). **2026-10-25: the dated check moved to 2026-10-26** — the session due today (restore-truth) was briefed „no DooPlex work” and does not read the restic repository; the next session that may, measures it on the 2026-10-26 sweep. **2026-10-25 dated check DONE (read only, the attended release run):** the 03:25:53 local sweep saved snapshot `3973450f` of `/mnt/5_hdd/backup/postgresql/dumps` (75 new files, 106 MiB added); the 00:00 dump folder `20261024-220001` (SUCCESS, written 00:00:10) existed before it; no backup alarm mail since 2026-10-23 22:20Z (control: that channel delivered two DooPlex backup mails on 2026-10-23). | — | — | operator | | **R-895** | Backup & restore | P2 | **The hub's clean-up-window check trusts the snapshot counts the box sends, so a broken-into box (or past-dated fakes added through the add-only key) can shrink the real off-site history without an alarm.** READ 2026-10-06 night in source (R-822's design): the before/after comparison uses counts the box itself reports (`hub/internal/offsitekeys/service.go:284`, `:343`); new fakes keep the count level. Decision 68 already accepts a box-trusted count. | **NARROWED 2026-10-18 — slice 2 built, not deployed** (hub v0.158.0): the 07:10 audit compares the `snapshots/` listing with yesterday's by name — files gone that no clean-up window explains → `offsite_snapshots_vanished` (error), files no reported box run explains → `offsite_snapshots_unexplained` (warning); operator-only, hub-origin (not in `allowedEventTypes`). First listing = baseline; a move-aside or RESET drops it; vanishing judged on a pinned tier only; files younger than 1 h judged the next day. Red-proofed twice (`snapshot_watch_test.go`). **Not built:** the window's before/after on the hub's own count (design §3.1) and refusing a window while unexplained files exist (§3.2). Unmeasured: the provider shell's `ls -l` time zone (± 3 h slack until then). **Deployed 2026-10-18 12:23 (hub v0.158.0).** The first listing (tomorrow 07:10) is a baseline; the first verdict is 2026-10-20 07:10. **Known limit (independent review 2026-10-18, F2):** `_unexplained` is explained by the box's OWN reported `offsite.last_run`, so whoever holds a box's key can silence it; `_vanished` is judged on the hub's own listing and is not affected. **2026-10-20 03:5x UTC: due date moved 10-20 → 10-21** — the first verdict lands at 07:10 hub time (05:10 UTC), after the night session that reached it ended; that brief did not include this read. Nothing is lost: the hub keeps its listing; read the 2026-10-20 07:10 lines on 10-21 (or later on 10-20). **FIRST VERDICT READ 2026-10-21 (rehearsal session, read-only):** the 07:10 CEST run judged three tiers against yesterday — demo-felhom 5, demo-hp 14, tester-1 4 new snapshot files, „each inside a reported box run"; nothing vanished, nothing unexplained; Tester4 stored its first baseline (44 files); Tester-2's own listing failed (no `snapshots/` dir — the box has never pushed). No `offsite_snapshots_vanished` / `_unexplained` mail in the operator mailbox (second channel). Evidence `audits/due-checks-2026-10-21/R-895-hub-0710.txt`. The positive alarm path is still proven only by the unit red-proofs. | the operator's yes to deploy hub v0.158.0 | Deploy; read the first 07:10 lines (newest upload vs the box's last run = the time-zone calibration), shrink the slack; then decide §3.1/§3.2 | CC | | **R-923** | Backup & restore | P2 | **The password manager (Vaultwarden) runs on DooPlex and holds the keys that open DooPlex's own off-site copies — a loss of DooPlex loses the keys with the copies.** Operator finding, 2026-10-09 11:19: "the password manager also runs on DooPlex (Vaultwarden), so I probably need to back up its DB somewhere too." Measured: `vaultwarden-system/vaultwarden` (1.37.3), PVC `vaultwarden-data` 7.4 MB (`db.sqlite3` + WAL, `rsa_key.pem`, no attachments); its only backups were Longhorn on DooPlex. **A key in a password manager that runs on DooPlex is not off DooPlex** — R-173's "both keys are off DooPlex" (2026-10-05) and the hub-DB runbook Step 0 were wrong in that sense. Also seen: Vaultwarden warns its `ADMIN_TOKEN` is plain text (an Argon2 hash is its guidance). | NARROWED — **2026-10-23 (D32):** the sheet is current — names and places only, one A4, HU + EN (`runbooks/break-glass-sheet.md`); every place checked to exist. LEFT: the operator prints it, fills it by hand, keeps it outside the home. **NARROWED 2026-10-09 (operator yes in chat):** Vaultwarden's own `backup` copy + `rsa_key.pem` ride the nightly encrypted DooPlex→ep0 job (`scripts/dooplex-offsite/`), checked on every push and every Sunday (integrity, users, items — rows only). That copy opens only with the DooPlex off-site key, and the vault only with the master password — so the circle is broken only once both are on paper outside DooPlex: `runbooks/break-glass-sheet.md`, `runbooks/total-loss-of-dooplex.md`. **LEFT:** the operator prints the sheet and stores it away from home; the `ADMIN_TOKEN` hash (a DooPlex change, operator's word). **2026-10-23 night — survey (`audits/dooplex-survival-2026-10-22/PLAN.md`, G1/G3/G10):** unchanged; the sheet is still the one act that breaks the circle, and S5's offline form is unknown to CC — on the sheet too (decide list in STATUS). | operator: print the sheet | Operator: print + store the sheet; then close | operator | -| **R-1254** | Backup & restore | P2 | **The hub saves a household's new escrow package in TWO writes; a failure between them leaves the new repository hash beside the OLD identity package, and the box then marks itself escrowed — on a total loss the written code opens nothing.** Found 2026-10-24 (HUNT-3 H1), read at `file:line`: `hub/internal/api/handler.go:1415` `SaveHostEscrow` commits blob + hash, `:1445` `SaveHostDRBundle` sets `identity_blob` in a second statement (`internal/store/store.go:3788-3796`; the upsert `:3490-3511` never touches it); the controller's confirm reads the hash (`felhom-controller/controller/internal/report/escrow_confirm.go:195-217`); the agent sends once. Same family (H2): a lost or late answer (30 s) tells the household „the previous code still works” when the hub may already hold the new package (`felhom-agent/internal/localapi/escrow_ceremony.go:79,532-534,639-641`). Static. **The hub half is fixed on main 2026-10-24 night** (felhom.eu `dcf4f84c`; Part 3): key, hash and identity blob in ONE transaction (`SaveHostEscrowWithIdentity`), a bad identity refused before any write; hub image `0.170.0-rc.r1254` built locally (`67db584a45d7`), NOT pushed, NOT deployed. The agent ceremony half (H2: „the previous code still works” after a lost answer) is still open. | **NARROWED** — **the hub half CLOSED 2026-10-25:** hub **0.170.0** deployed on the operator's yes (manifest bump, ArgoCD sync of the hub Deployment only, image digest `6478498d…`; 30 minutes and more with no alarm mail, the mailbox as the control); Tester 1 ran the escrow ceremony end to end through the page's endpoints and the hub stored the new package in one transaction („the new package is on the hub", `audits/release-2026-10-26/escrow-ceremony-tester1.txt`). The ceremony half stays open | the hub release (attended); the agent half | deploy hub v0.170.0 (Monday list); then the ceremony asks the hub what it holds before it says „the previous code still works” | CC | +| **R-1254** | Backup & restore | P2 | **The hub saves a household's new escrow package in TWO writes; a failure between them leaves the new repository hash beside the OLD identity package, and the box then marks itself escrowed — on a total loss the written code opens nothing.** Found 2026-10-24 (HUNT-3 H1), read at `file:line`: `hub/internal/api/handler.go:1415` `SaveHostEscrow` commits blob + hash, `:1445` `SaveHostDRBundle` sets `identity_blob` in a second statement (`internal/store/store.go:3788-3796`; the upsert `:3490-3511` never touches it); the controller's confirm reads the hash (`felhom-controller/controller/internal/report/escrow_confirm.go:195-217`); the agent sends once. Same family (H2): a lost or late answer (30 s) tells the household „the previous code still works” when the hub may already hold the new package (`felhom-agent/internal/localapi/escrow_ceremony.go:79,532-534,639-641`). Static. **The hub half is fixed on main 2026-10-24 night** (felhom.eu `dcf4f84c`; Part 3): key, hash and identity blob in ONE transaction (`SaveHostEscrowWithIdentity`), a bad identity refused before any write; hub image `0.170.0-rc.r1254` built locally (`67db584a45d7`), NOT pushed, NOT deployed. The agent ceremony half (H2: „the previous code still works” after a lost answer) is still open. | **NARROWED** — **the hub half CLOSED 2026-10-25:** hub **0.170.0** deployed on the operator's yes (manifest bump, ArgoCD sync of the hub Deployment only, image digest `6478498d…`; 30 minutes and more with no alarm mail, the mailbox as the control); Tester 1 ran the escrow ceremony end to end through the page's endpoints and the hub stored the new package in one transaction („the new package is on the hub", `audits/release-2026-10-26/escrow-ceremony-tester1.txt`). The ceremony half stays open. **2026-10-28 night: the ceremony half FIXED ON MAIN** (agent `5ddd124` CI 2473, controller `e33cffa` CI 2474; tests red first; `03` §8 [DESIGN] 2026-10-28): after a failed upload the agent asks the hub which package it holds (the identity blob compared) — the new one → success, another → 502 and „the previous code still works", no answer → 504 that promises nothing; the controller words 504 and its own failed call as „Nem tudjuk biztosan… Őrizd meg mindkét kódot…" and waits 60 s for the claim/confirm (was 15 s, shorter than the agent's 30 s upload). | the hub release (attended); the agent half | deploy hub v0.170.0 (Monday list); then the ceremony asks the hub what it holds before it says „the previous code still works” | CC | | **R-314** | Backup & restore | P3 | **`StopAbandon` has no web route — a customer who telephones is served by a command line.** `--abandon-stop` exists on the controller binary (`cmd/controller/main.go:86`) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). **An operator who is telephoned instead has to reach a shell on the customer's machine.** Used this session on the operator's ruling, container stopped first so the running controller could not overwrite `settings.json` from memory — a sequencing subtlety that is itself an argument for a route **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-314-279-177.md` — the box's own 14-day countdown has no operator door (the hub's 7-day phase does, decision 74); one door for R-314 + R-279: a fixed list of operator actions carried in the report reply. Question D1 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D1 (`09` §3 decision 185):** yes — option A, a fixed list of operator actions carried in the box's report reply. | **VERIFY — 2026-10-09: DELIVERED (hub 0.144.0 + controller 0.304.0); `run_job fill-watch` proven live on the Tester 1 box (hub row „done — ran to its end"; control: the box log „checked 3 filesystem(s)"; an unknown action is refused 400). `abandon_stop` / `abandon_extend` NOT yet proven — no box has a deletion countdown; prove them on the first one.** **VERIFY — built on main 2026-10-08 (D1, decision 185), ships with tomorrow's controller + hub releases; closes after the live proof on 9202** (`abandon_stop` / `abandon_extend` through the hub's buttons). **READY (S) — NEW 2026-08-12, RANK 3** **2026-10-05 (burn-down night): NEEDS A DESIGN.** There is no operator door into a RUNNING controller (its HTTP surface is the household's session; the CLI runs in a separate process and cannot safely trigger a job in the running one). Same design as R-177 and R-279. | R-241, R-307 | An operator-authenticated POST that calls the same `StopAbandon`, so the telephone path and the code path converge | CC | | **R-401** | Backup & restore | P3 | **Revisit the off-site integrity depth when a real store is LARGE — the default rests on ONE measurement, on ONE 134 MB store.** Controller v0.228.0 (R-399) made `--read-data-subset=100%` the default for every box. The whole justification is a single data point: `demo-hp`, 2026-08-30, 140 829 678 B / 2 651 blobs / 67 snapshots, structure 35.0 s vs 100% **39.2 s** — four seconds. Re-proven live 2026-08-31 at 38.7 s. **It does not extrapolate, and the reason is structural: the structure check's cost tracks the INDEX, a read-data run's tracks the DATA.** A 50 GB store is ~370x the data and this curve says nothing about it. **Nothing was invented from that one point** — no rotation schedule, no size threshold, no bandwidth budget — because four production designs in this project were specced against unvalidated mechanisms and all four were wrong. **`readDataSubsetRe` already accepts `n/m`**, so a rotating schedule (`1/7` on a different seventh each week) needs no parser work when the time comes; the missing input is a measurement on a large store, not code. **THE TRIGGER IS AN EVENT, NOT A DATE:** the slow-check WARN from v0.228.0 firing on any box (`integritySlowNoticeThreshold`, 5 min) — that line names the duration, the depth and this row. **WHAT HAPPENS IF NOBODY ACTS:** every box re-reads its entire store every week, however large it grows, and the first person to notice is a customer whose upload is saturated. **Whoever acts must also revisit `integrityCheckTimeout` (30 min)**, which is now the number a large store meets first. | **OPEN — WATCHING** | — | When the WARN fires: measure the curve on that store, then choose between a rotation (`n/m`), a size-conditional default, or leaving it. Do NOT choose from this row's numbers — they are the small-store case. | CC | | **R-409** | Backup & restore | P3 | **Nothing in the product can vouch for the bytes of a restored recovery unit — the only hash record covers 0.002 % of it.** MEASURED on demo-hp 2026-08-31 against kimai's restored unit: `manifest.json`'s `checksums` object carries sha256 for `.felhom.yml` (2 235 B), `app.yaml` (488 B) and `docker-compose.yml` (2 195 B) — **4 918 bytes of a 213 231 242-byte unit**. The database dump (48 217 B) and the two named-volume tars (160 331 776 B + 52 845 056 B) — the recoverable data, 99.998 % of the bytes — have no recorded hash anywhere. **And nothing else supplies one:** restic 0.14.0's `restore --verify` is a size-and-mtime reconciliation (a size-and-mtime-preserving one-byte corruption of a 160 MB tar passed clean, red-proofed), and `restic ls --json` file nodes in 0.14.0 carry name, size, mode, uid/gid and three timestamps and **no content hash**. **So "the restore produced correct files" is currently unanswerable by any automated means.** **What is NOT claimed here:** `restic check --read-data-subset=100%` already proves the STORE's packs, and the config files that ARE hashed are the ones a wrong-content failure would be hardest to spot in. | **OPEN — MEDIUM** | R-87, R-361 | Cheapest fix, and it is already half-built: extend the capture's `checksums` to cover `db_dumps` and `volume_dumps` — R-361 already computes a canonical dump sha256 to prove itself, so the value exists at capture time. Then a restore-test has a real reference and R-87's narrow version becomes a content check rather than a completeness one. Evidence: `audits/SPIKE-restic-restore-test-2026-08-31.md` §Q2, §Q3. | CC | @@ -185,7 +185,7 @@ stopping line that lies. | **R-540** | Backup & restore | P3 | **[P3-LOW] The hub knows exactly ONE off-site pool box, so there is no rule for what happens when it fills.** Read from source 2026-09-16 while making off-site the default: `HETZNER_POOL_BOX_ID` is a single value, and every shared customer becomes a sub-account on that box. With off-site now ON for every new customer (hub v0.116.0) the box fills faster, and the fill warning (80%/90% of the box, `monitor/offsite.go`) tells the operator it is filling but nothing says which box a new customer should land on. **Needs a selection rule** (least-full, or explicit per-customer), not a bigger box. No customer is at risk today: the pool box read 0.3% full (2.7 GB of 1 TB), Σ shared quota 150 GB, oversub 0.15x. | **READY — rank P3-LOW; owner: CC (hub)** **2026-10-05 (burn-down night): NEEDS A DESIGN (and money).** A selection rule needs a multi-box configuration and a second box. Next: an operator pick of the rule and its trigger (e.g. add box 2 at 70 %). | — | — | CC | | **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC** **2026-10-23 night (demo-hp, read only):** the local tier was refused 4 times (00:59–02:39 UTC) by the space preflight — `local has 8.5 GiB free; the last archive of guest 9201 was 6.5 GiB, so a new one needs about 9.2 GiB` — and the controller stopped 11 apps for about a minute before each refused try (then backed off 30 m → 2 h). The 2026-10-22 run had fit; `local` now also holds 6.4 GB of installer ISOs (1.27.1–1.30.0, the newest uploaded for yesterday's proof installs). Two halves: the box (free space on demo-hp: remove the superseded ISOs — a demo-hp housekeeping step, not done tonight) and the product (ask the agent's preflight BEFORE stopping the apps — R-921's shape for the local tier). **2026-10-23 day — the box half DONE:** the three superseded installer ISOs (1.27.1, 1.28.0, 1.29.0; no guest referenced them) removed from demo-hp's `local` by name (`pvesm free`); `/` 8.5 GB → 14 GB free (`df` before/after, `audits/day-2026-10-23/09-demohp-space.txt`), above the 9.2 GiB the preflight asks. 1.30.0 stays. **The product half — the proposed fix (R-921's pattern):** the agent already computes the space verdict (R-685: free vs the last archive × the margin); it serves it per guest on `GET /backup/status` next to `host_busy` — e.g. `local_space: {fits, need_bytes, free_bytes}` — and the controller reads it before any stop: when the local tier cannot fit, it stops no app, records the tier as skipped for space (the same contention record as R-921, one alarm after the ceiling), and still runs the other due tiers. An agent without the field keeps the old order. Tests to write first and red-prove: agent `TestBackupStatus_R548_LocalSpace`, controller `TestR548_NoStopWhenTheLocalTierCannotFit`. Rides the next agent + controller pair. | — | — | CC | | **R-933** | Backup & restore | P3 | **Mirror the images of every catalog app we support into our own registry, before scale.** Operator, 2026-10-12 (S8, `09` §3 decision 203): *„Will need to mirror the apps we support later, just to be sure."* Today a restore re-pulls each image from its maker; a version the maker deleted cannot start (the accepted limit, `07` §6.6, R-698 closed). Size it first: the supported catalog's images (every ladder version a box can hold) × their size; then where the copy lives (the DooPlex registry is Tier 2, precious — the copy must not make DooPlex a single point of failure for every household's restore) and how a restore falls back to it. | **OPEN** — NEW 2026-10-12 | the size measurement | Measure the total size of the supported catalog's images; then a one-page design (where, how a restore falls back, cost) | CC (measure + design), operator (the go) | -| **R-1255** | Backup & restore | P3 | **Escrow edge cases (review round 2, HUNT-3 H3/H4):** the R-1176 carry moves the newest kept package even when it holds no repository password (`hub/internal/store/escrow_carry.go:100-110` checks `identity_blob` only), while the right one stays under the deleted host (`store.go:3533-3534`); two set-asides on the same UTC day reuse a name and the purge can delete a still-stored copy's package (`hub/internal/offsitekeys/offsitekeys.go:378-385`; `store.go:4295-4300`). Static. | OPEN | — | carry the newest package that has a hash; a unique set-aside name per run; store tests | CC | +| **R-1255** | Backup & restore | P3 | **Escrow edge cases (review round 2, HUNT-3 H3/H4):** the R-1176 carry moves the newest kept package even when it holds no repository password (`hub/internal/store/escrow_carry.go:100-110` checks `identity_blob` only), while the right one stays under the deleted host (`store.go:3533-3534`); two set-asides on the same UTC day reuse a name and the purge can delete a still-stored copy's package (`hub/internal/offsitekeys/offsitekeys.go:378-385`; `store.go:4295-4300`). Static. | OPEN — **fixed on main 2026-10-28 night, not released** (felhom.eu `1ee6d5a5`, CI 2472; store tests red first; `03` §8 [DESIGN] 2026-10-28): the carry takes the newest retained row with an identity AND a repository-password hash (the newest row only when none has a hash — a legacy package, as before); a set-aside is `.orphaned-`; a retained row is purged only when it was tagged no later than its path's deletion (`repo_tagged_at`, new column). No live proof: a hub deploy is not a night's. | the hub release (attended) | release it (the morning list), read the store back after one carry and one abandonment, then close | CC | | **R-1256** | Backup & restore | P3 | **Restore edge cases (review round 2, HUNT-1 H4/H5/H6/H8):** (H4) an off-site restore whose definition write fails AFTER the compose was written restarts the app on the snapshot's compose (`internal/backup/offbox_reconstitute.go:979-986`; `cmd/controller/main.go:3318-3331`); (H5) a put-back whose stop fails goes on and can merge (`restore_crash_putback.go:155-158`; `restore.go:218` ignores `volume rm -f`), then removes the copy; (H6) a second-drive mirror can start between the web refusal and the restore's own record (`web/restore_wizard.go:226`; `opstatus.go:56-66`), and the install-time kept-data load checks only RestoreStatus/IsRunning (`api/kept_install.go:89-92`); (H8) an unused restore copy is never cleaned up. Static. | OPEN | — | one attended pass over the restore paths with these four as tests | CC | | **R-1259** | Backup & restore | P3 | **The DooPlex→ep0 copy jobs: three edges and tomorrow's clock change (review round 2, HUNT-2 H6–H8, HUNT-4 H1).** The release copy keeps the newest 5 agent versions whether vouched or not, so five unvouched releases drop the vouched one from ep0 (`scripts/release-copy/felhom-release-copy:199,218,237`); its `rsync --delete` removes old versions during the transfer and mirrors a shrinking registry as deletions (`:190,237`); the hub-DB push has no lower age bound (`scripts/hub-db-backup/felhom-hub-db-backup:41,49-50`). VERIFIED on DooPlex: on 2026-10-25 the 02:30 CEST push runs at 00:30Z (`systemd-analyze calendar`) and the hub's 02:00 slot at 01:00Z (Go takes the repeated hour's second pass), so the push re-sends the 20:00 snapshot and the off-site hub DB is ~13.5 h old until 08:30. Operator data only. | OPEN | — | always keep the vouched agent; prune after a verified copy; a future-age bound; the push slot after the hub's in UTC | CC | | **R-1274** | Backup & restore | P3 | **The restore refusal tells the household to „turn on remote backup, or connect a second drive” when the box has both and their first copy has not run yet.** Catalog sweep 2026-10-24, Grimmory and Calibre-Web: „Mentés most” makes the local copy only (`07` §8, by design), so the drive files had no copy yet; the refusal's sentence is `restore.refuse.files.none` (`felhom-controller/controller/internal/web/handlers.go:1835`) whatever the box has. The Hungarian sentence is frozen by `i18n_go_parity`, so the fix is a NEW key („the first copy is made in tonight's backup”) chosen when a tier is set up but holds nothing yet. | OPEN | — | The new key + the branch + a test | CC | @@ -301,7 +301,7 @@ stopping line that lies. | **R-1310** | Monitoring & notifications | P3 | **The security watch (R-1303) flags notices that do not apply and misses fixes that do — measured by reading all 38 flagged pins by hand on 2026-10-27.** False: advisories of ecosystem `actions` (Home Assistant's CI), of another package in the repo (Gitea's runner), with a hostname or ≤-old range (cal.com 2023/2024, Grafana 2022), for the desktop/Electron client (Termix), for the rootless image only (Uptime Kuma); a release hit on a word (Dawarich „no longer receives security fixes", Rallly v4.16.0). Wrong „no fix": Vikunja's six (range `<= 2.6.0`, fixed in 2.7.0), karakeep-chrome (154/156 exist). Missed: Tandoor 2.7.0 names 8 GHSA ids (1 matched), Dawarich 1.16.0 (2FA, export deletion), Wishlist v0.69.0 (critical, no GHSA). BEHIND reads linuxserver `26.09.20260929` as newer than 26.09.2; Seerr follows the renamed-away `fallenbagel/jellyseerr`. `audits/security-app-updates-2026-10-27/TRIAGE.md` NOT-A-FINDING list and `triage-notes/` | OPEN — filed 2026-10-27; **two shapes FIXED the same day** (ecosystem `actions` not listed; an empty fix with a closed range reads „after X"; test W9, red first) | — | teach `security_watch.py`: skip a package that is not the pinned one; match every GHSA id a release body names; ignore `YY.MM.YYYYMMDD` tags; repoint seerr to `seerr-team/seerr`; each with a test seen failing | CC | | **R-1334** | Monitoring & notifications | P3 | **Docker Hub's tag API refuses the watch (HTTP 429) since the DooPlex group was added: 19 DooPlex images NOT CHECKED in the 2026-10-28 night run (job 2446, `hub.docker.com` = 211 requests), and the „could not check registry:docker.io" mail went out (streak 2).** The runtime part is not affected (HEAD + cache). ghcr.io also answered one blob with 429 in the dry run of 07:02 (Immich v3.3.5, while this session read images from the same address). | **VERIFY** — BUILT 2026-10-28 midday, measured first (`audits/dockerhub-bigsteps-2026-10-28/part1/`): the 429s came from the BURST, not the count — night 2446 sent 211 tag-list requests in ~2 min from six workers (15 refused), dry run 2450 sent 224 in 3.5 min (none). `fe677bae`: Docker Hub requests spaced (1/s across the workers), a 429 retried with its Retry-After (NOT CHECKED only after the 4th try), tag lists cached 24 h per registry path on the runner (two names of one path fetch once); `8644450d`: the image reader retries a registry 429 too (ghcr.io refused two blob reads in back-to-back dry runs). Tests W14 + retry checks, red first. Dry runs 2458 and 2459: **0 Docker Hub 429, 0 retries** (210 requests over ~7 min). **No Docker Hub login needed.** Each of the two CI runners keeps its own tag cache. `minio/minio:latest` answers 401 — Docker Hub no longer serves it (homelab, R-1328). | — | close after the first CI night with 0 Docker Hub 429 (the 03:40 run of 2026-10-29) | CC | | **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC | -| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING — DESIGN WRITTEN 2026-10-26 night (the brief's Part 7: build only if time is left; it was not built — the line test and the attended release came first).** The host-side counter EXISTS and is readable (read on demo-hp 2026-10-26): `/sys/fs/cgroup/lxc//ns/system.slice/docker-<64-hex id>.scope/memory.events`, line `oom_kill N`. **Design:** (1) agent: a read-only collector (no root CLI — a sysfs read under the agent's own rights; if not readable, the existing sudoers fence gets one read-only `Cmnd_Alias`) that, for each guest in the `felhom` pool, globs those scopes and keeps `{container_id: oom_kill}`; served on the guest's own local API as `GET /guest/oom-kills` (`withGuest`, the guest's containers only). (2) controller: the 30-s dead-app scan asks once per pass; a RISE of a container's counter = an OOM kill whatever Docker's `OOMKilled` says → the existing `app_oom` line (R-636's per-run grain), `source: cgroup` in the details. An agent without the route = today's behaviour. (3) Tests: a fake cgroup tree (agent), a rising counter → one `app_oom` (controller, red-first). (4) Live: 9202, a throwaway app capped at 64M with a memory hog: the counter rises, `app_oom` names the container while `OOMKilled` reads false. ~1 day, two repos, one agent release. **Earlier:** WATCHING — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent` `acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC | +| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **BUILT 2026-10-28 night** (agent `6f933d6` CI 2475: `GET /guest/oom-kills`, the host's per-container `oom_kill`; controller `eed23e7` CI 2476: the 30-s scan raises `app_oom` for a counted kill whatever Docker's flag says, `source` in the details; `08` §5 [DESIGN] 2026-10-28). **Live 2026-10-28 23:27 on Tester 1 with test builds of both (put back after):** Radicale capped at 128 MB, a hog inside it killed, the host counter 0 → 1, ONE `app_oom` „seen by docker,cgroup", the operator's mailbox held exactly one mail with `"source":"docker,cgroup"` (`audits/night-2026-10-28/r528-oom/`). Docker 29.8.2 also flagged it, so the cgroup-only path is proven by tests. Closes after the agent release (vouch) and the controller release. **Earlier: WATCHING — DESIGN WRITTEN 2026-10-26 night (the brief's Part 7: build only if time is left; it was not built — the line test and the attended release came first).** The host-side counter EXISTS and is readable (read on demo-hp 2026-10-26): `/sys/fs/cgroup/lxc//ns/system.slice/docker-<64-hex id>.scope/memory.events`, line `oom_kill N`. **Design:** (1) agent: a read-only collector (no root CLI — a sysfs read under the agent's own rights; if not readable, the existing sudoers fence gets one read-only `Cmnd_Alias`) that, for each guest in the `felhom` pool, globs those scopes and keeps `{container_id: oom_kill}`; served on the guest's own local API as `GET /guest/oom-kills` (`withGuest`, the guest's containers only). (2) controller: the 30-s dead-app scan asks once per pass; a RISE of a container's counter = an OOM kill whatever Docker's `OOMKilled` says → the existing `app_oom` line (R-636's per-run grain), `source: cgroup` in the details. An agent without the route = today's behaviour. (3) Tests: a fake cgroup tree (agent), a rising counter → one `app_oom` (controller, red-first). (4) Live: 9202, a throwaway app capped at 64M with a memory hog: the counter rises, `app_oom` names the container while `OOMKilled` reads false. ~1 day, two repos, one agent release. **Earlier:** WATCHING — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent` `acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC | | **R-1193** | Monitoring & notifications | P3 | **When the box froze (R-1192), the operator heard only „no report for 45 m" — 28 minutes in — and the household heard nothing at all.** Measured 2026-10-20 against the operator mailbox (the second channel): the freeze began ~13:05 UTC; the hub page read ONLINE / OK at 13:22 (correct inside the 45-minute threshold, R-549); `host_stale` at 13:33, `app_oom` at 13:34 (immich-machine-learning killed by its memory limit), `host_crash_restart` at 14:26 and 16:29 (the two plug pulls), `node_stale` 15:59 / `node_recovered` 16:03. No mail or banner tells the household that its box is not answering, and no operator mail says „the box answers ping but not its dashboard". CORRECTED the same day: the first version of this row said the hub showed OK „throughout" and „nobody was told" — that read only the hub page, not the mailbox. Evidence `audits/rehearsal-2026-10-20/journal.md`. | OPEN | — | Design note (2026-10-21, after R-1192's diagnosis): the freeze's signature is cheap to see from the HOST side — the guest cgroup's memory PSI „full" above ~50 % for a minute, or its `memory.events max` climbing, while the host still reports; the agent could raise `guest_memory_thrash` (operator mail in its own words, not „no report for 45 m") and the household banner „A dobozod most nagyon lassú…". With the swap fix (R-1192) the measured thrash ends within a minute, so this alarm should be built only if a freeze is seen again after the agent release; until then it stays a design. Decide whether a dashboard that does not answer for N minutes while the host still reports should alarm sooner and in its own words (guest-side liveness), and whether the household is told | CC | | **R-1202** | Monitoring & notifications | P3 | **A box whose clock is behind blames the hub and stops an app without telling anyone.** Rehearsal 2026-10-20 B6 #8: RTC set 2 years back (the OS raised it to 2026-04-13): the dashboard shows only „Hub nem elérhető" while the log says „x509: certificate … not yet valid: current time 2026-04-13…"; Paperless crash-looped and the box stopped it with a restore hold; the `app_stopped_unhealthy` event failed (hub unreachable), so neither the household nor the operator heard. After the real time came back the app stayed stopped until a Start (clear sentence, 1 min). | **OPEN** | — | say „the box's clock is wrong (it says )" when TLS fails with not-yet-valid/expired against a hub that answers; keep the stop event queued until the hub is reachable | CC | | **R-340** | Monitoring & notifications | P3 | **The new reachability check does not touch the surface that actually failed.** R-339 reports when the hub cannot READ ep0 — but the read it performs is the `usage` op, which is `proxmox-backup-manager` plus `df` over SSH, and therefore rides the **local API daemon**. The 2026-08-18 incident explicitly CLEARED that daemon: `proxmox-backup.service` was healthy throughout, and it was the **HTTPS proxy on 8007** that was wedged with a full accept queue. **So R-339's check would have returned green for all 9 h 37 m of that outage.** It closes the case where ep0 is unreachable *as a host*; it does not close the case that actually happened. This is not a defect in R-339 — it is the honest boundary of what it watches, recorded so a future reader does not mistake a green box gauge for a working off-site tier | **READY (M) — NEW 2026-08-18** | a tenantsync endpoint-script version bump (the op is added on ep0, so it needs the same version-gated rollout `ErrUsageUnsupported` already models) | Add a **health op** to `scripts/felhom-tenantsync.sh` that probes `https://127.0.0.1:8007/` on ep0 and reports the proxy's fd count and listen-queue depth, then surface it as a third signal. **Overlaps the connections spike (R-336's remaining half): both want the same observations from ep0, so whichever runs SECOND must reuse the first's evidence rather than re-measuring a protected machine twice** **REUSE, per this row's own instruction — the connections spike ran FIRST (2026-08-20) and already produced most of what the health op wants; do not re-measure a protected machine a third time.** Available in `audits/evidence-ep0-established-connections-2026-08-20/`: the proxy **fd count** and its type breakdown (`lsof` + `/proc//fd`), the **listen-queue depth** (`ss -lnt` — `Recv-Q 0`, `Send-Q 1024`), the **ESTAB/CLOSE-WAIT split**, the **per-peer** connection histogram, a **31-minute persistence diff** of full 4-tuples, and a **46.18 h** slope with Poisson bounds. What the health op would still add beyond these: a loopback `GET https://127.0.0.1:8007/` probe — the observation that distinguished "process problem" from "network problem" on 2026-08-18 and the one thing this spike did NOT take, because it is the surface R-339 cannot see. **And this spike sharpens what the op should report:** a rising **ESTAB** count is the live signal (CLOSE-WAIT was **0**, not merely flat), and per **R-344** the fd ceiling that matters may be the **agent's**, not only ep0's. | CC |