felhom.eu/REPORT-night-2026-10-28.md

# REPORT — night 2026-10-28 → 29 (unattended)

## Safest readings taken (decided by CC unattended — operator may reverse)

1. **An unused restore copy is deleted after 14 days** (R-1256 H8; `07` §6.3 [DESIGN] 2026-10-28; CONTEXT). Options:
   14 days, for ever (as before), or 3 days. A held or waiting copy is never touched.
2. **H4 and H5's failed-stop branch stay on main with test proofs only.** Rule 1 asks for a live proof by morning or a
   revert.
   - H4 runs only on an off-site restore, and 9202 has no off-site target. A fault injected there races the write-back.
     It can leave an operator-only hold (R-379), which is not ours to clear.
   - H5's failed stop needs a `docker stop` that fails on demand.
   - Both fixes fail safe: they hold or keep, and never pour.
   - Reading: keep them, and say so first. The live proof is owed to an attended session. Or the operator reverts those
     two branches of controller `342a394`.
3. **The catalog changes are two prepared patches, not a push** (the brief: drill only, no live catalog push).
   `documentation/audits/night-2026-10-28/catalog-morning/`. The catalog working tree is clean again.
4. **Plex's `catalog_since` stays 2026-10-23** in the prepared patch. The ladder writer set it to the day it ran. But the
   date means "the pin last moved", and the pin did not move tonight (it moved on 10-23, the one `--no-verify` push).
   Put back by hand, and written in the patch README.
5. **STATUS was at its 50-line limit.** Yesterday's one-item do-list (the homelab list, R-1328) became item 5 of the
   morning list. Its words did not change.

## Five lines

1. Restore edges: 4 of 4 fixed (H4, H5, H6, H8), each test red first. Proven live: 2 (H5's busy volume, H8's sweep).
2. Escrow edges: R-1255 (hub: the carry, the set-aside name, the purge) and R-1254 (the ceremony half, agent +
   controller) fixed on main, tests red first. No live proof: that needs a hub deploy, and no hub deploy runs at night.
3. OOM proven: **yes** — on Tester 1, one `app_oom` "seen by docker,cgroup" and one mail.
4. Plex route: **works** — unclaimed, from Plex's own container. Proven on the bench and on box 9202 (1.41.4 → 1.43.5,
   the section read back with the same uuid).
5. Claims walked: 2 walked, 2 proven, 0 failed. Not walked: 31 of 55 (was 32).

## Morning release list (each one is yours; nothing was released)

1. **Hub vX:** R-1255 (felhom.eu `1ee6d5a5`).
2. **Agent vY + your vouch:** R-1254 (`5ddd124`) and R-528 (`6f933d6`). Then `pinned.day0_agent` (D50).
3. **Controller vZ:** R-1256 (`342a394`), R-1254's controller half (`e33cffa`), R-528 (`eed23e7`), and an unknown app
   answers 404 (`20c5877`). MinAgent unchanged (0.158.1): an older agent answers 404, and the controller asks again
   after 10 minutes.
4. **Catalog:** the two prepared patches. R-1338 (the memory figure; Wanderer re-measured at 27.7 %, not tight), and
   the Plex fixtures + the Plex ladder entry (R-1238, proven on both venues).
5. Owed to an attended session: the live proof of H4 and H5's failed stop, or a revert of those two branches.

If nothing is done: the fixes stay on main only. No box changes.

## [DESIGN] lines

- R-1256: `documentation/architecture/07-backup-architecture.md` §6.3, "[DESIGN] 2026-10-28 night — four restore
  edges", plus two [FACT] paragraphs (H5, H8 live).
- R-1255: `03-host-agent.md`, "[DESIGN] 2026-10-28 night — WHICH retained row is carried".
- R-1254: `03-host-agent.md`, "[DESIGN] 2026-10-28 night — a lost or late answer".
- R-528: `08-alarm-ladder.md`, "[DESIGN] 2026-10-28 night — the host's counter beside Docker's flag", with its [FACT]
  of the Tester 1 proof.

## What was proven live

- **H5, scratch 9202, 01:33 UTC:** Vikunja's first volume was held by a created-only container. The household's
  restore was pressed through the page. The result was "Failed to remove volume vikunja_vikunja_data before the restore
  … (not written into)". The restore failed, and the put-back poured the live copy of both volumes. The app ran, with
  no hold. Projects from before and after the restore point read back.
- **H8, scratch 9202, 03:29 UTC:** three copies were planted by hand. The first sweep removed the two older than 14
  days and kept the young one (8 ms). Then the young one was removed by hand.
- **R-528, Tester 1, 2026-10-28 23:27 UTC:** a Radicale capped at 128 MB. The host counter went 0 → 1, with one
  `app_oom` "seen by docker,cgroup" and one operator mail.
- **R-1238 Plex box walk, 9202, 03:45–03:59 UTC:**
  - Deployed through the setup gate at 1.41.4.
  - A library section was seeded from Plex's own container. Its front door answers 401 to the proxy.
  - The household's Update ran in 41 s.
  - The section read back on 1.43.5 with the same uuid.
  - The 10-minute soak peaked at 10.6 %, with no kill. The verdict was `proven`.

## Claims walked (household first; front doors only)

| Claim | Result | Evidence |
|---|---|---|
| fail.customer-self-restore | **PROVEN-LIVE** (the route). Stays `partial`: no one other than the operator has done it yet. | `audits/night-2026-10-28/part5-claims/selfrestore-walk.log` |
| fault.selfheal | **PROVEN-LIVE**, built → walked. A crashed redis ran again in 1 s, and the probe read the app healthy. A guest reboot brought everything back in under 2 minutes. No event, no mail. | `…/part5-claims/selfheal/WALK.md` |

Not walkable tonight, with the reason for each:

- backup.sikeres (the run half): needs an off-site target, and 9202 has none.
- backup.fill-warning: 9202's drive is a shared 938 GB host disk, which cannot be filled safely.
- The app-crash mail and fault.operator-email: 9202 has no hub, and a hub deploy is not a night's work.
- backup.restore-proof: needs every tier on both machines. Its block is R-366. CC's hub token reads no event list.

## Commits and CI (by commit)

| Repo | Commit | CI run id / # | Result |
|---|---|---|---|
| felhom.eu (hub) | `1ee6d5a5` R-1255 | 2470 / #1331 | green |
| felhom.eu | `8a3df79f` design notes | 2476 / #1332 | green |
| felhom.eu | `d8752de9` R-1338 evidence | 2477 / #1333 | green |
| felhom.eu | `0abed37f` + `6a6dd94d` H5, the self-restore walk, row states | 2478 / #1334 | green |
| felhom.eu | `7a6b9ff1` 07 [DESIGN] + facts, fault.selfheal walked | 2482 / #1337 | green |
| felhom.eu | close-out (this report, STATUS, CONTEXT, rows, Plex box evidence, patches) | see the end of the final report | — |
| controller | `342a394` R-1256 | 2469 / #532 | green |
| controller | `e33cffa` R-1254 | 2472 / #533 | green |
| controller | `eed23e7` R-528 | 2474 / #534 | green |
| controller | `20c5877` unknown app → 404 | 2475 / #535 | green |
| controller | `a815b37` docs | 2483 / #536 | green |
| agent | `5ddd124` R-1254 | 2471 / #283 | green |
| agent | `6f933d6` R-528 | 2473 / #284 | green |
| agent | `91bd96c` docs | 2479 / #285 | green |
| catalog | none pushed (2 prepared patches) | — | — |
| drill | `41b37ad`, `37736f0`, then their reverts `8aaf198` + `3b66d35` (the tree equals its start) | — | — |

## Teardown (three layers)

- **Machine — scratch 9202 (04:14 UTC):**
  - The controller is back on 0.346.1 and healthy. The test image `0.347.0-rc.n1028` is removed (a local tag, never
    pushed).
  - The live catalog is back; the config is byte-identical to the saved copy.
  - The drill mirror is back at its start (`320cff5d`).
  - No throwaway app, no planted restore copy, no volume made tonight. The containers are as found: the controller,
    the file manager, paperless (3) and traefik.
- **Machine — Tester 1 (23:30 UTC):** controller 0.346.1, agent 0.167.0 (sha256 `28bdd282…`), the test app removed.
- **Host — demo-hp:** the bench 9401 is stopped. No new guest, and nothing on the thin pool.
- **Hub:** nothing provisioned.

## Register

Before: 225. After: 226. Opened: 1 (R-1339). Closed: 0.

## Observations

- FILED: R-1339 — at boot the controller logs "drive …/userdata/<app> not mounted", but the drive IS mounted. No data
  is lost; the log line misleads a diagnosis.
- NOT-A-FINDING: the Plex bench soak got 401s. An unclaimed Plex refuses the harness's request. There were 0 kills, and
  the peak was 12 % of 2 GiB.
- NOT-A-FINDING: Plex's front door answers 401 to the box walk. Plex does not take the proxy's address as local. The
  fixture asks from Plex's own container, which is the measured route (R-1238).
- NOT-A-FINDING: 9202 drops `app_start_failed` and `controller_started`, because it has no hub. This is by design on a
  scratch box.
- NOT-A-FINDING: the ladder writer sets `catalog_since` to the day it runs. It is made for a pin move, and this was the
  one entry written after a move made by hand (R-1238's history). Put back in the patch.
- NOT-A-FINDING: the security review flagged the drill's Plex downgrade. It was on purpose: drill only, scratch 9202,
  and reverted.
- NOT-A-FINDING: the 03:45 push was refused once. The nightly security watch pushed its own state during the gate run.
  A rebase, then the push went in.
- NOT-A-FINDING: the security watch's daily list has a new Brotli decoder hole in many images. That is its normal job,
  with its own mail. A catalog image move would be a live catalog push, which the night fences.
- NOT-A-FINDING: 9202 holds 359 Docker volumes, all made before 2026-10-28 16:00. None is from tonight. This is scratch
  hygiene from earlier sessions, not a product defect.
- Fixed (small): two dead bench Wanderer tokens were redacted from committed evidence. They stay in the history of
  `d8752de9`; the throwaway app is gone.
- Fixed (small): a stale R-356 sentence in the self-restore claim's note.

## Other

- Instruction files: none edited.
- System poster: no listed fact changed. The restore copy is not on the poster, and nothing was released.
- Architecture read: `07-backup-architecture.md` (restore), `03-host-agent.md` (escrow), `08-alarm-ladder.md` (OOM),
  `09-update-architecture.md` §6.4 (the ladder).
- Secrets: the night's passwords stayed in the scratchpad. Every evidence folder was scanned; the only hits are setting
  names (`cf_tunnel_token`, `PLEX_CLAIM`, "password from settings.json"), never values.