felhom.eu/REPORT-night-2026-10-28.md

REPORT — night 2026-10-28 → 29 (unattended)

Safest readings taken (decided by CC unattended — operator may reverse)

1. An unused restore copy is deleted after 14 days (R-1256 H8; 07 §6.3 [DESIGN] 2026-10-28; CONTEXT). Options: 14 days, for ever (as before), or 3 days. A held or waiting copy is never touched.
2. H4 and H5's failed-stop branch stay on main with test proofs only. Rule 1 asks for a live proof by morning or a revert.
   • H4 runs only on an off-site restore, and 9202 has no off-site target. A fault injected there races the write-back. It can leave an operator-only hold (R-379), which is not ours to clear.
   • H5's failed stop needs a docker stop that fails on demand.
   • Both fixes fail safe: they hold or keep, and never pour.
   • Reading: keep them, and say so first. The live proof is owed to an attended session. Or the operator reverts those two branches of controller 342a394.
3. The catalog changes are two prepared patches, not a push (the brief: drill only, no live catalog push). documentation/audits/night-2026-10-28/catalog-morning/. The catalog working tree is clean again.
4. Plex's catalog_since stays 2026-10-23 in the prepared patch. The ladder writer set it to the day it ran. But the date means "the pin last moved", and the pin did not move tonight (it moved on 10-23, the one --no-verify push). Put back by hand, and written in the patch README.
5. STATUS was at its 50-line limit. Yesterday's one-item do-list (the homelab list, R-1328) became item 5 of the morning list. Its words did not change.

Five lines

1. Restore edges: 4 of 4 fixed (H4, H5, H6, H8), each test red first. Proven live: 2 (H5's busy volume, H8's sweep).
2. Escrow edges: R-1255 (hub: the carry, the set-aside name, the purge) and R-1254 (the ceremony half, agent + controller) fixed on main, tests red first. No live proof: that needs a hub deploy, and no hub deploy runs at night.
3. OOM proven: yes — on Tester 1, one app_oom "seen by docker,cgroup" and one mail.
4. Plex route: works — unclaimed, from Plex's own container. Proven on the bench and on box 9202 (1.41.4 → 1.43.5, the section read back with the same uuid).
5. Claims walked: 2 walked, 2 proven, 0 failed. Not walked: 31 of 55 (was 32).

Morning release list (each one is yours; nothing was released)

1. Hub vX: R-1255 (felhom.eu 1ee6d5a5).
2. Agent vY + your vouch: R-1254 (5ddd124) and R-528 (6f933d6). Then pinned.day0_agent (D50).
3. Controller vZ: R-1256 (342a394), R-1254's controller half (e33cffa), R-528 (eed23e7), and an unknown app answers 404 (20c5877). MinAgent unchanged (0.158.1): an older agent answers 404, and the controller asks again after 10 minutes.
4. Catalog: the two prepared patches. R-1338 (the memory figure; Wanderer re-measured at 27.7 %, not tight), and the Plex fixtures + the Plex ladder entry (R-1238, proven on both venues).
5. Owed to an attended session: the live proof of H4 and H5's failed stop, or a revert of those two branches.

If nothing is done: the fixes stay on main only. No box changes.

[DESIGN] lines

• R-1256: documentation/architecture/07-backup-architecture.md §6.3, "[DESIGN] 2026-10-28 night — four restore edges", plus two [FACT] paragraphs (H5, H8 live).
• R-1255: 03-host-agent.md, "[DESIGN] 2026-10-28 night — WHICH retained row is carried".
• R-1254: 03-host-agent.md, "[DESIGN] 2026-10-28 night — a lost or late answer".
• R-528: 08-alarm-ladder.md, "[DESIGN] 2026-10-28 night — the host's counter beside Docker's flag", with its [FACT] of the Tester 1 proof.

What was proven live

• H5, scratch 9202, 01:33 UTC: Vikunja's first volume was held by a created-only container. The household's restore was pressed through the page. The result was "Failed to remove volume vikunja_vikunja_data before the restore … (not written into)". The restore failed, and the put-back poured the live copy of both volumes. The app ran, with no hold. Projects from before and after the restore point read back.
• H8, scratch 9202, 03:29 UTC: three copies were planted by hand. The first sweep removed the two older than 14 days and kept the young one (8 ms). Then the young one was removed by hand.
• R-528, Tester 1, 2026-10-28 23:27 UTC: a Radicale capped at 128 MB. The host counter went 0 → 1, with one app_oom "seen by docker,cgroup" and one operator mail.
• R-1238 Plex box walk, 9202, 03:45–03:59 UTC:
  • Deployed through the setup gate at 1.41.4.
  • A library section was seeded from Plex's own container. Its front door answers 401 to the proxy.
  • The household's Update ran in 41 s.
  • The section read back on 1.43.5 with the same uuid.
  • The 10-minute soak peaked at 10.6 %, with no kill. The verdict was proven.

Claims walked (household first; front doors only)

Claim	Result	Evidence
fail.customer-self-restore	PROVEN-LIVE (the route). Stays partial: no one other than the operator has done it yet.	audits/night-2026-10-28/part5-claims/selfrestore-walk.log
fault.selfheal	PROVEN-LIVE, built → walked. A crashed redis ran again in 1 s, and the probe read the app healthy. A guest reboot brought everything back in under 2 minutes. No event, no mail.	…/part5-claims/selfheal/WALK.md

Not walkable tonight, with the reason for each:

• backup.sikeres (the run half): needs an off-site target, and 9202 has none.
• backup.fill-warning: 9202's drive is a shared 938 GB host disk, which cannot be filled safely.
• The app-crash mail and fault.operator-email: 9202 has no hub, and a hub deploy is not a night's work.
• backup.restore-proof: needs every tier on both machines. Its block is R-366. CC's hub token reads no event list.

Commits and CI (by commit)

Repo	Commit	CI run id / #	Result
felhom.eu (hub)	1ee6d5a5 R-1255	2470 / #1331	green
felhom.eu	8a3df79f design notes	2476 / #1332	green
felhom.eu	d8752de9 R-1338 evidence	2477 / #1333	green
felhom.eu	0abed37f + 6a6dd94d H5, the self-restore walk, row states	2478 / #1334	green
felhom.eu	7a6b9ff1 07 [DESIGN] + facts, fault.selfheal walked	2482 / #1337	green
felhom.eu	close-out (this report, STATUS, CONTEXT, rows, Plex box evidence, patches)	see the end of the final report	—
controller	342a394 R-1256	2469 / #532	green
controller	e33cffa R-1254	2472 / #533	green
controller	eed23e7 R-528	2474 / #534	green
controller	20c5877 unknown app → 404	2475 / #535	green
controller	a815b37 docs	2483 / #536	green
agent	5ddd124 R-1254	2471 / #283	green
agent	6f933d6 R-528	2473 / #284	green
agent	91bd96c docs	2479 / #285	green
catalog	none pushed (2 prepared patches)	—	—
drill	41b37ad, 37736f0, then their reverts 8aaf198 + 3b66d35 (the tree equals its start)	—	—

Teardown (three layers)

• Machine — scratch 9202 (04:14 UTC):
  • The controller is back on 0.346.1 and healthy. The test image 0.347.0-rc.n1028 is removed (a local tag, never pushed).
  • The live catalog is back; the config is byte-identical to the saved copy.
  • The drill mirror is back at its start (320cff5d).
  • No throwaway app, no planted restore copy, no volume made tonight. The containers are as found: the controller, the file manager, paperless (3) and traefik.
• Machine — Tester 1 (23:30 UTC): controller 0.346.1, agent 0.167.0 (sha256 28bdd282…), the test app removed.
• Host — demo-hp: the bench 9401 is stopped. No new guest, and nothing on the thin pool.
• Hub: nothing provisioned.

Register

Before: 225. After: 226. Opened: 1 (R-1339). Closed: 0.

Observations

• FILED: R-1339 — at boot the controller logs "drive …/userdata/ not mounted", but the drive IS mounted. No data is lost; the log line misleads a diagnosis.
• NOT-A-FINDING: the Plex bench soak got 401s. An unclaimed Plex refuses the harness's request. There were 0 kills, and the peak was 12 % of 2 GiB.
• NOT-A-FINDING: Plex's front door answers 401 to the box walk. Plex does not take the proxy's address as local. The fixture asks from Plex's own container, which is the measured route (R-1238).
• NOT-A-FINDING: 9202 drops app_start_failed and controller_started, because it has no hub. This is by design on a scratch box.
• NOT-A-FINDING: the ladder writer sets catalog_since to the day it runs. It is made for a pin move, and this was the one entry written after a move made by hand (R-1238's history). Put back in the patch.
• NOT-A-FINDING: the security review flagged the drill's Plex downgrade. It was on purpose: drill only, scratch 9202, and reverted.
• NOT-A-FINDING: the 03:45 push was refused once. The nightly security watch pushed its own state during the gate run. A rebase, then the push went in.
• NOT-A-FINDING: the security watch's daily list has a new Brotli decoder hole in many images. That is its normal job, with its own mail. A catalog image move would be a live catalog push, which the night fences.
• NOT-A-FINDING: 9202 holds 359 Docker volumes, all made before 2026-10-28 16:00. None is from tonight. This is scratch hygiene from earlier sessions, not a product defect.
• Fixed (small): two dead bench Wanderer tokens were redacted from committed evidence. They stay in the history of d8752de9; the throwaway app is gone.
• Fixed (small): a stale R-356 sentence in the self-restore claim's note.

Other

• Instruction files: none edited.
• System poster: no listed fact changed. The restore copy is not on the poster, and nothing was released.
• Architecture read: 07-backup-architecture.md (restore), 03-host-agent.md (escrow), 08-alarm-ladder.md (OOM), 09-update-architecture.md §6.4 (the ladder).
• Secrets: the night's passwords stayed in the scratchpad. Every evidence folder was scanned; the only hits are setting names (cf_tunnel_token, PLEX_CLAIM, "password from settings.json"), never values.