felhom.eu/REPORT-night-2026-10-19.md

# REPORT — the night briefs of 2026-10-19 and 2026-10-20 (part 2 run by day): one stop less per night, the safe undo for a failed off-site restore, hub 0.162.0 / agent 0.161.0 / installer 1.35.0 delivered, downloads out of the backups, a scanner on every push

# PART 2 — the night brief part 2 (run by day, 2026-10-20): your four answers, the safe undo, downloads out of the backups, Immich, the update test

**Safest readings taken alone (written first, as the brief asks):**
1. **R-1083 is not in agent 0.161.0.** An agent key would make 9202's controller book the agent's whole-guest backups,
   one of which writes to ep0 — a change on ep0 the brief forbids; 9202 is also outside the `felhom` pool.
2. **All five downloaders' folders are `excluded`, not only the two named** (qBittorrent, Transmission; also NZBGet,
   SABnzbd, MeTube): the brief's gate rule — a download folder is never optional or mandatory — covers all five.
3. **No update-test edge was run:** the bench (guest 9401) and the drill catalog are outside the brief's machines and
   repos. The list done and the list left are written; the ask is on the morning list.
4. The brief was written for a night run and ran by day; its „stop by 05:00" was read as a time box, not a clock.

## Part 2 — in plain words

- **Morning list:** vouch agent 0.161.0 · decide D21 (Immich) · MeTube's videos out of the second-drive copy — keep? ·
  the bench and the drill catalog for the update test — yes?
- **Hub 0.162.0 is live.** Its own deploy was the real restart test: Tester-2 (down since 4 October) got no repeat mail
  — read in the hub and, separately, in your mailbox.
- **Agent 0.161.0 runs on demo-hp, Tester 1 and demo-felhom.** On demo-hp the agent said „busy" during a backup and
  „free" before and after. **Not vouched — that is yours.**
- **Installer 1.35.0 is published.** The served file equals the tag's file.
- **A failed off-site restore now puts the app back exactly as it was** (your D20 = A, controller 0.336.0, the floor).
  Proven on the scratch box: after a forced failure the database matched its fingerprints from before, the app ran,
  nothing was held; a normal restore then worked.
- **Downloads are out of the backups** (torrent, usenet, MeTube): the controller's own check copies 0 of them now.
- **Immich's off-site copy:** 1.33 GB, of which 102 MB must survive. Two options; pick B (leave out the AI model files).
- **ep0's first weekly clean-up:** done this morning (Part 10 of part 1 below).

## Part 2 — each part

| Part | Done? | Why / how |
|---|---|---|
| 1a hub deploy | **done** | `manifests/hub.yaml` → 0.162.0 (`e1d349f1`), only Deployment/hub synced (the only OutOfSync); image `sha256:69285e64…` = the build; Synced/Healthy; reports from all three boxes; floor read 0.335.0; Day-0 agent list has no `-step1`. R-1001 + R-1011 closed. |
| 1b agent release | **done** | R-1083 left out (above). `release-agent.sh 0.161.0` from a clean clone; signed `agent_update` per box: demo-hp 06:38, Tester 1 06:54, demo-felhom 06:55 UTC, each COMPLETED, 76/76, committed. `host_busy` live; tunnel `absent` = the fixture row (9202's real Docker 29 answer); R-1010 by its tests. R-1070 + R-1010 closed. |
| 1b R-921 live | **not done** | The no-stop case happens only on the night schedule (a press backs up the local tier only). Row R-1173 with a dated check after demo-hp's next off-site night. |
| 1c installer | **done** | tag `installer-v1.35.0`, both webpage pins (`b0ae2c57`), webpage Deployment synced alone; served sha256 `f7a196fd…` = tag. R-1167 closed. |
| 2 safe undo | **done** | below. R-893 + R-1169 closed. |
| 3 downloads | **done, one read left** | catalog `c50adfc`, gate `download-binds`; classifier proof. demo-felhom (qBittorrent) runs its next second-drive copy 2026-10-21 03:30 — dated check on R-1019 (NARROWED). |
| 4 Immich | **done (plan only)** | below. R-49 → READY, D21. |
| 5 update test | **not done** | fixture side complete for every installed app; the edges need the bench / drill catalog (above). R-462 NARROWED. |
| 6 ep0 clean-up | **done** | part 1, Part 10. |
| 7 controller release | **done** | 0.336.0 below. |

## Part 2 — the safe undo, the 9202 proof (`07` §6.3; `audits/night2-2026-10-20/P2/`)

- Code `93fb251`: the fit check before the stop, the copy right after it (`backups/restore-undo/<app>/`), the put-back on
  every failure branch, the hold only when the put-back fails, the copy removed on success. 11 tests; nine mutations red.
- **9202, rc.1** (a helper with the fences in full; Rallly + postgres 18; the same forced failure as on 19 October):

| | Live before | After the forced failure |
|---|---|---|
| public tables | 36 | 36 |
| markers | A, B, C, D | A, B, C, D |
| `r893_only_old` (backup only) | absent | **absent** (on 19 October: present) |
| schema / data / row-count md5 | `05828c77` / `e5c1a72f` / `d8bcbc4b` | **equal** |
| app | running | **running**, no hold |
| the copy | — | 69 MB in 0.9 s, removed after |

  The sentence: „…visszaállítása sikertelen. Nem változott semmi: az alkalmazás adatai visszakerültek a visszaállítás
  előtti állapotba, és az alkalmazás újra fut". Then a normal restore of the same snapshot: the snapshot's content
  (A–C, the backup-only table present), the app running, the copy removed. 9202 back on 0.331.0 as found.
- Not measured live (each pinned by a red-proved test): the version-change branch, the `_files` sentence, the refusals
  before the stop, a failing put-back, the operator event (9202 has no hub).

## Part 2 — download binds (`15` §7; `audits/night2-2026-10-20/P3/`)

| App | Bind (userdata) | Class now | Copied by a tier |
|---|---|---|---|
| qBittorrent | media/torrents | excluded (was optional) | none (was the second drive) |
| Transmission | media/transmission, media/transmission/watch | excluded, excluded | none |
| NZBGet | media/nzbget | excluded (was optional) | none |
| SABnzbd | media/usenet | excluded (was optional) | none |
| MeTube | media/metube | excluded (was optional) | none |
| Sonarr, Radarr | media (rw) | excluded | none |
| Jellyfin, Plex, Emby | media (:ro) | excluded | none |
| Prowlarr | no userdata bind | — | — |

## Part 2 — Immich (`audits/night2-2026-10-20/P4/immich-offsite-plan.md`)

Model cache 824 MB (62 %) · raw database folder 283 MB (21 %) · Immich's own backups 115 MB (9 %) · logical dump 54 MB
· originals 49 MB · thumbnails 2 MB. **A** (leave out all three re-fetchable parts, −92 %) needs a never-run restore
from the dump alone on the vectorchord image; **B** (only the model cache, −62 %) needs one restore with an empty cache.
**Pick B now**, A's other parts after their proof.

## Part 2 — update-test edges

None run (above). Installed apps: 16; with a fixture: 15; without: bentopdf (keeps no data). Upstream within a major,
waiting for a tested step: PrivateBin 2.0.6→2.1.0, Home Assistant 2026.9.4→2026.10.2, RomM 5.3.1→5.4.0, Plex
1.41.4→1.43.4, Kimai 2.67.0→2.70.0 (`audits/night2-2026-10-20/P5/r462-list.txt`).

## Part 2 — versions, commits, rows

- Hub 0.162.0 deployed (`e1d349f1`). Agent 0.161.0 released (`5152123`, tag `v0.161.0`; notes `4e1621c`). Installer
  1.35.0 published (`b0ae2c57`). Controller 0.336.0 the floor (`93fb251`; notes `aa9e09e`, `7ca59e2`), image
  `sha256:93cd67d8…8d73`. Catalog `c50adfc`. Evidence `301f4e9e`, `bcaa9576`.
- i18n: no page template changed; seven Go-side keys (HU + EN), one rewritten (R-1169); no fixture changed.
- **System poster:** no listed fact changed — none of the three.
- **Register 160 before, 156 after. Closed 7:** R-1001, R-1011, R-1167, R-1070, R-1010, R-893, R-1169. **Opened 3:**
  R-1171 (demo-hp's local copy never restore-tested), R-1172 (the put-back logs at ERROR; an empty folder stays),
  R-1173 (R-921's no-stop not yet seen live). Re-stated: R-1019 (NARROWED, dated 2026-10-21), R-1083, R-462 (NARROWED),
  R-49 (READY, D21). **Both briefs together: 161 before, 156 after; closed 11, opened 6.**

## Observations — part 2

1. demo-hp's local whole-guest copy is never restore-tested: one copy kept, a 24 h settle, replaced at 24 h. FILED: R-1171
2. A successful put-back is logged at ERROR, and an empty `backups/restore-undo/` folder stays after a restore. FILED: R-1172
3. R-921's no-stop branch has no live observation yet (it needs a night schedule). FILED: R-1173
4. R-462's „only three apps are tested" was out of date: 58 apps have fixtures on both venues. NOT-A-FINDING: the row is re-stated with today's list.
5. The hub deploy at 06:03 UTC removed the old pod's log, which held the 07:10 Budapest off-site audit lines of today (R-895). NOT-A-FINDING: R-895's dated check is 2026-10-21; that run happens in the new pod.
6. In step 4 of the 9202 drill the data-only md5 differed from a read taken before snapshot 2, while the schema and every row count were equal. NOT-A-FINDING: the two reads were at different moments (the read, then the snapshot's own dump); the claim under test — before and after the forced failure — matched on all three fingerprints.
7. The helper's brief expected 37 tables after the normal restore; snapshot 2 has 36. NOT-A-FINDING: last night's 37 was the mixed state, not the snapshot (my arithmetic, corrected).
8. The first background wait loop for demo-hp's agent update printed a time and stopped before the box had updated. NOT-A-FINDING: the version was read directly and a foreground loop used instead.
9. A Python edit of the controller CHANGELOG failed on a quote character, and the commit carried only REPORT.md. NOT-A-FINDING: fixed by the next commit (`7ca59e2`).

## Part 2 — teardown

- Machine: 9202 back on 0.331.0 as found (Rallly removed through the product, the SFTP target, images and files removed
  by name, settings and the off-box folder put back); demo-hp 9201's probe script removed.
- Host: demo-hp `/tmp` copies removed; nothing on `pve/data`; no whole-guest backup of 9202.
- Hub: per-box floors set and cleared; the global floor raised; nothing else.
- ep0: read only.

---

# PART 1 — the evening/night brief of 2026-10-19

## In plain words

- **Morning list:** deploy hub 0.162.0? · release agent 0.161.0? · publish installer 1.35.0? · decide D20 (R-893).
- **One app stop less per night** (R-921): the controller (0.335.0, the floor) now asks the agent „are you busy?" before
  it stops the apps. The agent's answer is written but not released, so it starts working with the agent release.
- **No false error for a box without a tunnel** (R-1070): the real cause was Docker 29 writing its „not found" message
  in small letters. Fixed in the agent (not released).
- **A failed off-site restore, measured** (R-893): the box puts the newer rows over the backup's older volume; a table
  only the backup had stays. The app is held stopped. Two options written; pick A (save the app's volumes first).
- **Hub fixes built, not installed:** no repeat mail after a restart; the agent list offers only real agent packages.
  The ArgoCD danger to three hub secrets is already gone (ArgoCD no longer tracks them).
- **Installer message written, not published:** a re-run on a live box says „delete the old host first", EN and HU.
- **A scanner on every push, in all four repos** (warn-first). First scan: no real secret, no called weakness.
- **The scratch box cannot get its agent key without an agent change** (R-1083): the exact gap is written down.
- **ep0's first weekly clean-up ran as predicted:** only tester-1's 4 October copy went; every household has a rule.
  R-1162 closed.

Decisions CC took: none beyond the brief. The operator answered one question (R-921, 19:2x): „add one field".

## Part 1 — R-921 (`07` §6.4; `03` local API) — DONE (live once the agent ships)

- No existing agent answer served the host-wide gate (read: `/backup/due`, `/backup/status`, the 409 shape). Operator:
  add a read-only field to the existing route, no new route.
- Agent `56aabc7`: `host_busy` `{busy, what}` on `GET /backup/status`. Controller 0.335.0: `precheckSkips` → held →
  no stop, contention for each due tier (15-minute skip, one „BLOCKED" alarm after 3 h); free / older agent / failed
  read → the old order; the window's own tier in flight → attach as before.
- **The race** (free at the read, busy at the start): one short stop, resumed at once — today's behaviour, pinned.
- Tests, red-proved: agent `TestBackupStatus_R921_HostBusy` (assignment removed → `free gate: host_busy = <nil>`);
  controller `TestR921_NoStopWhileTheHostIsBusy` (branch removed → 3 apps stopped, the start refused),
  `…SkipsStillAlarmAfterTheCeiling`, `…OtherAnswersKeepTheOldOrder` (4 rows), `TestHostBusy_ReadsTheAgentsGate`.
- Live: demo-hp's agent answers no `host_busy` (read inside 9201) — the old order, as designed. A live busy state was
  not made (it needs the new agent on the box). R-921 closed per the brief.

## Part 2 — R-1070 (`11` §8.2) — DONE in the agent (not released)

- Where: `felhom-agent/internal/osupdate/leg.go` `HostHealthVerdict` (the tunnel check) and
  `internal/hub/cloudflared.go` `ClassifyTunnel`. Measured on demo-hp: `docker inspect` of a missing name prints
  `error: no such object: …` (Docker 29); the match wanted `No such object` → `unknown` → the host rule failed.
- Fix: case-insensitive match → new state `absent`; the host rule passes `absent` only when the reading from before
  the step (Docker answered) also had no cloudflared; `not_running` and `unknown` still fail. Hub 0.162.0 shows
  `absent` as „none set up". `11` §8.2 had said `unknown` does not fail — corrected (the code always failed it).
- Tests, red-proved: `TestHostHealthVerdict_R1070_NoTunnelSetUp` (7 rows), three `TestClassifyTunnel` rows,
  `TestSystemRows_R1070_AbsentTunnelIsNotAWarning`.

## Part 3 — R-893: measured and designed, no restore code changed

Helper on 9202 (fences carried in full). Evidence and the note: `documentation/audits/night-2026-10-19/P3/`.

- Rallly 4.15.3 + postgres 18; a scratch SFTP off-site on 9202's own disk; one failing SQL line in the prepared copy
  of the dump, so the replay failed AFTER the volume was replaced.
- **What it left:** the newest rows (the undo dump) over the backup's volume; a table only the backup had stayed
  (37 tables against 36); definition unchanged (same version); app held (`restore_mixed`), a start → 409.
- The restore page's sentence is false on both halves for this app → R-1169.
- **Options:** A — copy the app's live volumes before the replay, put them back on failure, start the app (one copy's
  disk for minutes; Rallly 69 MB in ~1 s; a fit check before the stop). B — empty the DB volume before the undo load
  (no disk; a bad undo copy leaves an empty database; usually still a support call). **Pick: A** (the operator's D8
  direction). Not measured: the version-change branch (needs a drill-catalog commit).
- Teardown: 9202 back on 0.331.0 as found, every throwaway removed (checked again by CC); host: `/tmp` only; hub: nothing.

## Part 4 — hub 0.162.0, built, NOT deployed (`05`)

- R-1001: start-up seeding from the DB for `host_stale`/`host_down`; `restore_test_stale` edge from the DB. Tests
  red-proved (each restart = a new checker over the same DB).
- R-1011: exact-file sha (`ErrNoFile`); step bundles leave the list; a vouch of one → `artifact_file_missing`. Registry
  read: 6 step-bundle versions, every release holds `felhom-agent`. Tests red-proved.
- R-953: **no change needed** — read back 18:21 UTC: ArgoCD tracks none of the three Secrets (no tracking mark), so the
  next deploy does nothing with them. `P4/r953-readback.txt`.
- Image `sha256:69285e640a9c5e4ce59b1fbb5cd94623fa47a71798e9d86fa0767ee6c69a6ac8` from a clean clone at `d92a712e`.

## Part 5 — agent, committed, NOT released (`03`)

R-1010 (`56aabc7`): on „Failed to find logical volume …_vzdump" one retry with the API's `force=1` (`pvesh usage`
lists `--force`; the same token privilege; no sudoers line, no root command); never a second force. Red-proved. Plus
R-921's field and R-1070. CHANGELOG „Unreleased". Not exercised live (it needs a half-removed snapshot on `pve/data`).

## Part 6 — installer 1.35.0, written, NOT published

R-1167: `die_live_box` in EN and HU at the preflight, host-enroll and the controller.yaml fetch. Harness: a static case
and two behaviour cases (the real functions against a local fake hub), red-proved on a mutated copy. **CI red twice**
(2106, 2108): the CI runner has no curl; the behaviour cases now SKIP out loud there (`5d0dfeac`).

## Part 7 — the scanner (R-986), warn-first — DONE

- Shared `scripts/security_scan_gate.py` in all four runners' fast sets (hook + CI); `.security-accepted` per repo;
  8 decoys in scratch repos, red-proved; CI fetches the pinned gitleaks (both sha256 checked — seen working in CI 2108).
- govulncheck pin v1.1.4 → **v1.8.0** (v1.1.4 cannot read the Go 1.27 code). CI has no Go: govulncheck NOT CHECKED
  there, out loud; the hook runs it.
- **First scan of today's `main`** (`P7/triage.md`): govulncheck 0 called in all three modules. gitleaks history:
  controller 12, agent 15, catalog 1 (all as on 2026-10-12), felhom.eu 50 (+7: the usenet bench 5, a boot journal 1,
  a test 1). Today's tree: controller 4, felhom.eu 42, agent 15, catalog 0 — every one triaged: **0 real**; accepted =
  throwaway demo/bench app values (the bench LXC 9401 stopped, the apps gone from 9202); false = test constants, public
  keys, an SSH fingerprint, a tunnel ID, hashes.

## Part 8 — R-1083: the exact gap (no agent change)

The agent mints a per-guest token only inside `--selftest=provision`, for a guest it brings up from a golden in the same
call. No product way exists for an existing guest. Row stays OPEN with the gap and two options.

## Part 9 — controller 0.335.0, the floor

rc.1 on demo-hp 18:08 UTC (0 errors) → `0.335.0` `sha256:262831f9…` → per-box floors 18:22 → all three moved by
18:37 → box walk green (0 unhealthy, 0 refusals, catalog in sync; https-only 28 OK, 0 FAIL; „Fájlok ↗" one sign-in on
demo-hp and Tester 1) → **global floor 0.335.0 / MinAgent 0.158.1 at 18:39:26 UTC**, read back; per-box floors cleared.
i18n: no page changed, no fixture changed. Evidence `P9/`.

## Part 10 — ep0's first weekly clean-up (`07` §plane-2 retention) — DONE, R-1162 closed

Read only on ep0, 03:45–03:47 UTC. Evidence `P10/01-before.txt` (19:49 UTC), `P10/02-after.txt`.

| Household | Task 03:30 UTC | Removed | Copies after |
|---|---|---|---|
| demo-hp | OK, keep-weekly 2 | nothing | 2 (10-08, 10-15) |
| demo-felhom | OK, keep-weekly 2 | nothing | 2 (10-06, 10-13) |
| Tester-2 | OK, keep-weekly 2 | nothing | 1 (10-04) |
| tester-1 | OK, keep-weekly 2 | 2026-10-04 | 2 (10-11, 10-18) |

- Exactly the dry runs' prediction. The operator job (03:45, keep-daily 14 + keep-weekly 8) removed one hub copy
  (10-06) — its own rule.
- **No „household without a rule" alarm.** Positive check from a second channel: ep0's read-only `usage` op, run
  directly, answers `unpruned_checked: true, unpruned: []` (tenantsync v1.3.0, sha256 same as the repo file). The hub
  (0.161.0) logs its ep0 read every 15 minutes (03:31 UTC the last before my read) and no „no prune job" line since its
  start (2026-10-19 14:10 UTC). The alarm is operator-channel only, so it is not in the events table (none there).

## Versions, commits

- Controller 0.335.0 (`15ea002`, REPORT `c4b3b0a`, scanner `a03c25a`). Hub 0.162.0 built (`5f31da04`). Agent
  `56aabc7`, `84ef581` (not released). Installer 1.35.0 (`d92a712e`, `5d0dfeac`, not published). felhom.eu scanner
  `15f74288`, docs `f92fa5a8`. Catalog `87f5588`, `57ddb20`.
- **System poster:** no listed fact changed tonight — none of the three.
- **Instruction/doc corrections:** `11-os-updates.md` §8.2 (the `unknown` tunnel sentence was false); `nodes.md` and
  `nightly-rotation.md` (9202 is DHCP, .121 now).

## Rows

Register **161 before, 160 after.** **Closed 4:** R-921, R-953, R-986, R-1162. **Opened 3:** R-1168 (warn →
fail), R-1169 (the false restore sentence), R-1170 (a vanished tunnel raises no alarm). Re-stated: R-1001, R-1011,
R-1010, R-1070, R-1167 (READY, waiting on the release/deploy/publish), R-893 (READY, D20), R-1083 (the gap).
**Dates moved** (the due-checks gate fired on the new date): R-895 and R-435, 2026-10-20 → 2026-10-21. Both are reads
this brief did not name; R-895's first verdict lands at 07:10 hub time, after this session. The reason is in each row.

## Observations

1. The restore page's held-mixed sentence is false for the measured app. FILED: R-1169
2. A tunnel container that disappears on a box with a tunnel configured raises no alarm. FILED: R-1170
3. The scanner warns only. FILED: R-1168
4. `11` §8.2 said an `unknown` tunnel does not fail the host rule; the code always failed it. NOT-A-FINDING: corrected in the session (the size rule); it is R-1070's cause.
5. govulncheck v1.1.4 (the 2026-10-12 pin) cannot read Go 1.27 code. NOT-A-FINDING: the pin moved to v1.8.0 tonight, recorded in `SCANNERS.md` §5.
6. The CI runner has no curl; two new harness cases failed CI 2106 and 2108. NOT-A-FINDING: fixed the same night (`5d0dfeac`), they skip out loud there.
7. R-953's tracking marks went after 2026-10-12 with no record of who removed them. NOT-A-FINDING: the danger the row named is gone, nothing to fix.
8. 9202's DHCP address moved (.114 → .121) and the walk named the old one. NOT-A-FINDING: fixed (`box_walk.py` reads it; two docs corrected).
9. The brief said the installer already prints HU and EN; it printed English only. NOT-A-FINDING: the HU lines are new, named in the scripts CHANGELOG.
10. The agent's backup cap (R-973) can also refuse a POST after the stop. NOT-A-FINDING: it answers 409, so the apps resume at once (the BUSY path), and it fires only after a flood of presses that each stopped the apps anyway.
11. The manual off-site run dumped and restarted every app, not only the one switched on. NOT-A-FINDING: by design, the coherent app-data pair (`07`).
12. On CI's depth-1 clone, gitleaks scans the whole tree (the root commit), ~100 s for felhom.eu. NOT-A-FINDING: a stronger check than the diff, all current hits accepted; recorded in `CONTEXT.md`.
13. My 19:49 before-file listed the operator job's keep-weekly but not its keep-daily 14. NOT-A-FINDING: my read printed only some fields; the job is unchanged.

## Teardown

- Machine: 9202 back as found (0.331.0, no throwaway left — checked); 9201 on the release 0.335.0; probe scripts removed.
- Host: `/tmp` copies only, removed. Nothing on `pve/data`.
- Hub: per-box floors cleared; nothing else created.
- ep0: read only (task list, task logs, snapshot list, prune-job list, the read-only `usage` op). Nothing changed.