felhom.eu/REPORT-night-2026-10-22.md

# REPORT — night 2026-10-22: seven decisions, the data-loss candidates, the 03:30 pressure, the tunnel alarm, private repositories

Evening → night brief, started ~16:00 CEST. Nothing was released, deployed or published, except the catalog change and
the scratch box's catalog login (both allowed by the brief). Design read: `07-backup-architecture.md` §6.1.1, §6.2,
§10.2; `03-host-agent.md` (tunnel, bundle); `05-hub-architecture.md` §15; `06-offsite-connectivity.md`;
`09-update-architecture.md` §6.5.

## 1. Five lines

1. **D8:** done before 18:00 — DooPlex's Docker settings come from git (no change, no restart), and a weekly build-cache
   prune timer runs Sunday 23:30.
2. **The seven decisions:** live — D6, D8, D27 (report only). Ready, waiting for a release or publish — D21
   (controller 0.340.0), D3 + D5 (one website patch), D4 (agent bundle patch + installer patch).
3. **Data-loss candidates:** 2 proven and fixed in code (R-1187 c, R-1189 d); 5 real in code, now rows with a written
   fix (R-1218, R-1219, R-1220, R-1221, R-1222); 1 closed half (R-1189 b, fixed in 0.304.0); 3 to the operator
   (R-1205 as D31, R-1189 f, R-1190 d); the rest stay as named candidates.
4. **03:30 pressure:** the second-drive copy's file cache is charged to the guest and pushes the apps out of memory.
   The fix runs the copy in a 512 MB memory-capped container: full stall 5.7 s → 0.37 s on the scratch box.
5. **Private repositories:** only felhom-controller can go private (after one CI change). felhom.eu, the agent and the
   catalog must stay public: the website, every install and every box read them with no login.

## 2. The morning list (also in STATUS)

1. Publish the website patch (D3 wording, D5 hardware page).
2. D4: the agent bundle patch with the next agent release (your vouch) + the installer patch with a new installer tag.
   Discard appliance 49 on the hub's Hosts page (tonight's test install).
3. Deploy hub 0.166.0 (tunnel alarm) and release the agent with `handshake_future_s` — one agent release with item 2.
4. Release controller 0.340.0 (R-49, R-1187 c, R-1189 d, R-1214), MinAgent 0.158.1.
5. Decide D29 — the controller repository private? CC: No, not now.
6. Decide D30 — the tunnel repair. CC: the agent makes a new key and registers it on the path that exists.
7. (D31, R-1205, when there is time.) Tester 2's „go" from yesterday stays on the list.

## 3. Per candidate

| Item | Result | Reason |
|---|---|---|
| R-1187 a | REAL in code → row R-1187 (now P2) | an export drops a folder with a symlink and still says done; capture code: needs a red-proof + 9202 proof |
| R-1187 b | REAL in code → R-1187 | the import sets no owners and drops symlinks |
| R-1187 c | **PROVEN and FIXED** (`d5f88a2`) | 9202 rc.1: an import over a stopped app merged old + new data; rc.2: refused, app untouched |
| R-1187 d | PLAUSIBLE → R-1187 | no busy check; after (a) |
| R-1187 f, g | REAL in code → R-1187 | HDD_PATH trusted; postgis missed, psql without ON_ERROR_STOP |
| R-1188 a, c | already CLOSED (R-1195; 0.337.0) | — |
| R-1188 b, d, e | R-1188, re-ranked P4 | unlikely or a sentence |
| R-1189 a | REAL, half-live → R-1189 | agent code; rides an agent release |
| R-1189 b | SPLIT: skipped nights CLOSED (R-899, 0.304.0); creep → **R-1221** (P4) | demo-hp journal: a night run after each of 3 off-cycle runs; creep 1.5–5 min/night measured |
| R-1189 c, g, i | REAL in code → R-1189 (P4-shaped) | quiesce vs restore; spring forward 2027; double click |
| R-1189 d | **FIXED** (`d5f88a2`) | the catch-up now runs the proof + integrity once; red-proved |
| R-1189 e | REAL in code → **R-1222** (hub, P3) | an evening backup time gives a false „dump missed" every morning |
| R-1189 f | operator → R-1189 | design question |
| R-1190 a | REAL in code → **R-1218** (P2) | proposed fix, not built: key code needs a live proof, and 9202 has no hub |
| R-1190 b | REAL in code → **R-1219** (P2) | the top bar links to a ceremony the page hides during an offer |
| R-1190 c | PLAUSIBLE → R-1190 | a read-only fingerprint compare decides it |
| R-1190 d | operator → R-1190 | design question |
| R-1190 g | REAL in code → **R-1220** (P2) | an unreadable app.yaml lists a live folder as kept (deletable) |
| R-1205 | operator → **D31** | the F-14 gate is deliberate (ruling 2026-07-13) |
| R-1206 | NARROWED | Felhom tier: no loss, by design (R-822 guard). NAS tier: a unit test is owed |

Research input: `documentation/audits/night-2026-10-22/P3-candidates/analysis.md`.

## 4. Records

**Commits.**
- felhom-controller: `6a17c95` (R-49), `d5f88a2` (R-1187 c, R-1189 d), `03c0a69` + `ae39467` (R-1214), `df78b6c` (docs).
  CI jobs 2214, 2220, 2221, 2222, 2223 — all success, attempt 1.
- felhom-agent: `a0c88e5` (R-1204 + D4), `7651461` (D4 moved to a patch), `e803ad2` (docs, a correction). CI 2217,
  2218, 2224 success.
- felhom.eu: `0639cf07`, `e267c5e2` + `1100a344` (one push), `acc78637`. CI 2215, 2219, 2225 success (2219 covers `e267c5e2`).
- app-catalog-felhom.eu: `ce33e3b` (Immich `backup.volumes`). CI 2216 success.
- homelab-manifests: `b35ba83` (D8), `c7506df` + `b0cce29` (REPORT). No CI in that repo.
- Drill catalog (scratch, no CI by design): `2ba4b53`, `9ebdba6`, `b0350a3` (tree = live `ce33e3b` again).

**Images pushed.** `felhom-controller:0.340.0-rc.1`..`rc.4` (never offered to a household: not X.Y.Z).
`felhom-hub:0.166.0` (built, NOT deployed: the live hub and `manifests/hub.yaml` are 0.165.0, read 18:2x).

**Teardown, three layers.**
- **Machine (9202):** the throwaway uptime-kuma removed through the product (its `.fab` deleted); the hog container and
  19 GB of test files removed; memory 25898 MB / swap 512 MB again; Immich's `.felhom.yml` back to its pre-night file;
  controller back on 0.339.0 (healthy, every app up). Kept on purpose: the `drill-reader` catalog login (D6).
  Left: the four rc images in 9202's own Docker store, like earlier rc images there.
- **Host (demo-hp):** VM 9343 destroyed (own id); `/tmp/r1214*` removed; two sampler loops stopped. The scratch
  storage and VM 341 / CT 9401 are standing items, not tonight's.
- **Hub:** appliance 49 from VM 9343 is still „registered" — Discard is the operator's (session-only). Nothing else.
- **DooPlex:** the D8 timer (intended). Nothing else changed.

**Rows:** before 178, after 180. Opened 5: R-1218, R-1219, R-1220, R-1221, R-1222. Closed 3: R-206 (D8), R-992 (D6),
R-1207 (D27). Updated: R-49, R-925, R-982, R-985, R-991, R-1187 (P3→P2), R-1188 (P3→P4), R-1189, R-1190, R-1204,
R-1205, R-1206, R-1214, R-1215.

**Unproven claims:** 35 of 55 not walked — unchanged. **System poster:** no listed fact changed (nothing that the
poster draws shipped tonight) — none of the three actions was needed.

**Observations.**
- 9202's sync refused Immich's template (R-971: Immich's data sits on a bench mount, not a household drive). —
  NOT-A-FINDING: R-971 by design; the file was placed by hand for the proof and put back.
- The local unit restore of Immich was refused (R-538). — NOT-A-FINDING: by design; the proof used an empty-cache start.
- A non-`--fast` catalog gate run started the runtime gate on DooPlex; its canary refused and left nothing. —
  NOT-A-FINDING: the canary is the control; the rule (`--fast` on DooPlex) exists.
- Images and packages stay public when a repository goes private. — NOT-A-FINDING: they follow the user `admin`, and
  boxes must pull them; recorded in R-925 and D29.
- A CI read token is a precondition of D29. — NOT-A-FINDING: a step of that decision, written in it.
- R-925 said „32 source comments"; the served count is 61 on four pages. — fixed in the row.
- **CC's own mistake 1:** the agent CHANGELOG and felhom.eu commit `1100a344`'s message say a push of the installer
  publishes it. Wrong: `/scripts/` is served from the tag `installer-v1.35.0` (R-110). Corrected in the agent CHANGELOG
  (`e803ad2`) and R-982; the commit message cannot change (no force-push). Keeping D4 as a patch still holds: both
  halves need a release that was forbidden tonight.
- **CC's own mistake 2:** controller commit `df78b6c` carries a „Co-Authored-By" line, which the brief forbids. It stays
  (no force-push); no other commit tonight has one.
- **CC's own mistake 3:** homelab-manifests' REPORT first claimed a separate daemon.json backup. There was none — the
  file did not change and is byte-identical to the role's copy in git (`cmp`). Corrected in `b0cce29`.
- The guest has no rsync. — NOT-A-FINDING: rsync lives in the controller image; the capped copy uses that image.
- R-1215's evidence is gone (Immich restarted at 02:16). — said in the row; it stays a watch.