# REPORT — release run 2026-10-29 (attended): D62 + D63, then hub, agent, controller, catalog

## Readings taken

1. **D62's place: A** (the restore page's kept-copies list), the brief's default; no link on the app's page.
2. **D63's fault switch: a `-tags faultinject` test build** (the brief's first choice for both). Why not a read-only folder
   for H4: it races the compose write. Why not a SIGTERM-ignoring container for H5: `docker stop` then kills it and
   succeeds, so the stop never FAILS. The switch fails the exact step, on demand, once.
3. **H4's version split** was staged honestly: the snapshot held the catalog's 3.8.1; just before the restore the live
   compose was set to the TRUE 3.8.0 (its own digest) and restarted. A first try that edited only the tag (keeping 3.8.1's
   digest) was dropped — the backup's R-645 guard rightly refused it.
4. **The test boxes' leftovers:** the empty anonymous volumes this session's backup presses left (R-1267's class, 21 in all)
   were removed by name; nothing was pruned.

## Five lines

1. **H4 and H5's failed stop: proven live** on Tester 1 (08:10 and 08:18 UTC). Nothing reverted.
2. **Hub 0.175.0: live** (09:39 CET); one escrow ceremony on Tester 1 end to end; 30 minutes with no unexpected alarm.
3. **Agent 0.168.0: vouched** (10:21:56 CET), on all three test boxes; the OOM line proven on the release.
4. **Controller 0.347.0: on every box** — the global floor since 10:31 CET (MinAgent 0.158.1); the three test boxes and 9202
   run it.
5. **Catalog: pushed** (`313eaf6`); Tester 1 read it.

## Versions, digests, CI by commit

| Repo | Commit | CI run / # | What |
|---|---|---|---|
| felhom.eu (hub) | `19afefab` | 2485 / #1339 green | `restore_copy_deleted` allowed + operator-only (D62) |
| felhom.eu | `bf69bb91` | 2487 / #1340 green | `09` §3 decisions 281/282, `07` §6.3 [DESIGN] — **hub 0.175.0 built from here** (`sha256:642e94b6…`) |
| felhom.eu | `41f8f670` | 2488 / #1341 | D63 evidence, R-1340, `07` [FACT] |
| felhom.eu | `d6352cd8` | — | hub manifest → 0.175.0 (synced: Deployment/hub only) |
| felhom.eu | `650aa0cf` | — | release evidence, R-1341, rows moved to their sections |
| controller | `e3fce09` | 2486 / #537 green | D62 + D63 — rc.1 (`b6640c0a…`) |
| controller | `fd75c7c` | 2491 / #538 green | v0.347.0 heading — **0.347.0 built from here** (`sha256:c79b06e6…`) |
| agent | `91bd96c` | 2479 / #285 green | **0.168.0 built from here** (agent `1e70196c…d2ca`, bundle `3415c034…8cb8`, tag `v0.168.0`) |
| agent | `f4e63a5`, `6b0b093` | — | CHANGELOG; `pinned.day0_agent` = 0.168.0 |
| catalog | `3671802`, `ddf20f4`, `313eaf6` | — | R-1338, R-1238 (the night's patches), CHANGELOG/REPORT |

The close-out commits' CI runs are read at the end of the session (the final message names them).

## What was proven live (evidence: `documentation/audits/d63-2026-10-29/`)

- **H4** (test build): snapshot 3.8.1, live 3.8.0 → „comes back at the snapshot's version" → FAULT after the compose write →
  „the live definition is written back before the start" → start; the app on 3.8.0, no hold, both seeds read back.
- **H5's failed stop** (test build): a busy volume „not written into"; the put-back's stop FAILED → „nothing is put back; the
  copy stays …; holding the app"; the app page's hold sentence; the held copy's button off with its reason.
- **D62**: on the test build (the list line, wrong name 400, right name 200, gone; the old hub answered the event 400), and
  end to end on the release build + hub 0.175.0 (the hub stored `restore_copy_deleted`).
- **Hub**: a ceremony on Tester 1 — claimed, two words typed back, „the new package is on the hub"; the hub logged the DR
  bundle and the blob with the same fingerprint; the box stays `escrowed`, no stale-package warning.
- **Agent**: three signed jobs, each committed, self-check 76/76, sha256 matching; on Tester 1 a hog in a 128 MB Gokapi —
  host counter 1, one `app_oom` „seen by docker,cgroup", one operator mail.
- **Controller**: walk green on all three (controller healthy, all containers up, 0 refusals, 0 errors, login and file
  manager 200, https-only); a local restore of Gokapi on Tester 1 (2/2 volumes, running, front door 200).
- **Catalog**: Tester 1's sync — „catalog moved 0baf8f9 → 313eaf68", plex and wanderer updated.

## Teardown (three layers)

- **Machine — Tester 1:** throwaway Radicale and Gokapi removed through the product; 21 empty anonymous volumes removed by
  name (268, as before); the empty `restore-undo` folder removed; the test image removed; controller 0.347.0, agent
  0.168.0 (the releases). **Left: Radicale's restore hold record — the operator's to clear** (STATUS, the command).
  Recovery code re-minted and stored (`TESTER1_RECOVERY_CODE_2026_10_29`); the 10-25 one opens only older packages.
- **Machine — scratch 9202:** controller 0.347.0 (rc.1 then the release); its old images removed; 359 volumes as before.
- **Hosts:** demo-hp, felhom-pve, Tester 1's host run agent 0.168.0. No guest made.
- **Hub:** 0.175.0; the three per-box floors cleared at 11:07 CET after the global floor's 30-minute watch; nothing else.

## Register

Before 226. After 226. Opened 4 (R-1340, R-1341, R-1342, R-1343). Closed 4 (R-1238, R-1338, R-1256, R-1255). R-528 is WATCHING (released
and proven; its dated check 2026-11-12 closes it). R-1254 stays NARROWED (released; the lost-answer branch has tests only).

## Observations

- FILED: R-1343 — the operator's hold clear at 11:42 CET was written back by the running controller's next settings
  save; the CLI's restart hint does not say „inside the guest, at once" (it was run on the host, where that unit skipped
  itself — no harm). The safe form is clear + restart in one command.

- FILED: R-1342 — at 10:39 CET Tester 1 mailed the operator „Gokapi did not start … the pre-restore copy is kept": false.
  My test deleted the copy by hand during the 30-minute stable-start watch (a gap in D62: that watch is an operation on the
  app) and then removed the app (the watch did not end). Both fixed on main with tests red first; not released (one release
  per session) — it rides the next controller release.

- FILED: R-1340 — the mixed-hold operator mail says „database restore failed … rolled back … files and volumes are the
  snapshot's" for any cause; in H5 nothing moved and the app has no database.
- FILED: R-1341 — the OOM WARN repeats every 30 s while the container lives (the event goes once).
- NOT-A-FINDING: the box writes Plex's compose with the ladder entry's digest — the exact-digest pin, by design.
- NOT-A-FINDING: Tester 1 reports Radicale as held after its removal — the hold record exists; the hub learns held apps by
  design (D53).
- NOT-A-FINDING: the first seeds after a restart got 401 — the harness sent an empty password from a new process (the
  generated login lives in the deploying process); fixed in the driver, not the product.
- NOT-A-FINDING: setup tries pushed one `backup_run_failures` event (Radicale not protected off-site) — caused by the
  first, discarded version split; the R-645 guard was right.
- NOT-A-FINDING: the first agent watch read `None` for `host_id` — the key is nested; read with grep.
- NOT-A-FINDING: `GET /configuration/artifacts` with CC's token → 403 — session only, by design.

## Other

- Architecture read: `07-backup-architecture.md` §6.3/§8, `03-host-agent.md` (escrow), `05-hub-architecture.md`,
  `10-localisation.md`, `09-update-architecture.md` §3.
- Instruction files: none edited. System poster: no listed fact changed (a retention age inside a box is not on it).
- Secrets: every evidence file scanned — 0 hits for the hub token, the hub key, Tester 1's password, the throwaway apps'
  passwords; the recovery codes never left memory except into `sec put` (file shredded by it).