felhom.eu/REPORT-day-2026-10-18.md

# REPORT — day 2026-10-18

## In plain words

- **The hub update is live.** Its own access log works and writes no secret (tested live with a fake token). You can
  now vouch an agent without a new golden. The off-site snapshot check gives its first verdict on 20 October.
- **A fresh reviewer read last night's code.** One medium finding: a stale host key could have left the emergency
  password unknown. Today's installer change (save in the hub first, then set) already closes it. Three low findings:
  two fixed today, one is a known design limit, now written in its row.
- **„Retire anyway":** when the host says no, the drive's apps start again. Where the host's disk list already says no,
  nothing is stopped at all. Controller 0.330.0 is the floor for every box.
- **Agent 0.159.2** runs on demo-hp, Tester 1 and demo-felhom (your yes). A whole-box backup ran with no pool check.
  **The vouch is yours** (STATUS).
- **Installer 1.34.0 is published** (your yes): the emergency password is saved in the hub first, then set, then
  checked with a login. Proven on demo-hp. A fresh install with it has not run yet (row R-1148).
- **The calculator** shows Immich's bigger figure (your yes). Immich alone on an 8 GB machine now reads „does not fit".
- **The old-restic check has not run.** demo-hp's clean-up comes only after tonight's backup (2026-10-19 about 04:16
  Budapest). The STATUS item stays open.

## 1. Hub 0.158.0 deploy

ArgoCD: only `Deployment/hub` was OutOfSync; synced that resource alone → Synced/Healthy, image `felhom-hub:0.158.0`,
startup log `felhom-hub 0.158.0 starting` + `access log: /data/access-log …, kept 90 days`. Floor read back from the hub
DB with its WAL: 0.329.0 / 0.158.1 at deploy time (`audits/day-2026-10-18/P1/hub-deploy-readback.txt`). Reports from all
three boxes in the access log within two minutes (sources .101, .149, .155).
- **Access log (R-1140, closed):** today's file `access-2026-10-18.log`, mode 0600. A fresh 48-hex fake token sent in a
  query, a `/bind/<token>` path, a `session=` query, a Bearer header, a Cookie and a POST body: found 0 times; control
  (the same file with the token appended) 1. Secret shapes (Bearer value, a 32+ character run, `password=`, `token=`,
  `session=`): 0; control 1. `/bind/<token>` is written `/bind/[redacted]` (`P1/access-log-live-test.txt`).
- **Agent-only vouch:** read-only — `/configuration` loads with CC's token (Day-0 section, golden 0.329.0, agent 0.159.1);
  the save (`/configuration/artifacts`) answers 403 to the token, by design. No manifest change by CC.
- **Off-site alarm (R-895, NARROWED):** no fixture route exists in the hub for it; the unit tests are the proof so far.
  Tomorrow's 07:10 run is the baseline (no verdict by design); the first verdict is 2026-10-20 07:10 — a DUE-CHECKS row.

## 2. The independent review

A fresh subagent with no context from the night reviewed controller v0.328.0→v0.329.0, hub 0.157.0→0.158.0, the agent's
orphan-check change and the installer's `--rotate-recovery-only`. Its report was written to a file and read back
(`audits/day-2026-10-18/P2/independent-review.md`, 54 lines). Verdicts (`P2/verdicts.md`):

| # | Finding | Verdict |
|---|---|---|
| F1 medium | rotate-only: unauthenticated healthz, then chpasswd, then a vault that a stale key fails → an unvaulted password | REAL in the night's code → **fixed** by R-1145 (vault first); harness `R1145-e`, fails against HEAD |
| F2 low–medium | `offsite_snapshots_unexplained` is explained by the box's own reported run time | REAL, by design (decision 68) → written into R-895; the custody alarm `_vanished` is not affected |
| F3 low | „anyway" stops apps before the agent is asked | REAL, was R-1144 → **fixed** (controller 0.330.0) |
| F4 low | host key in curl's argv during the vault PUT | REAL, pre-existing → **fixed** (header file) |

## 3. R-1144

`decommissionAnyway`: reads the agent's disk list first (a role other than user-data, or the backup target → the plain
refusal, nothing stopped); after a refusal past the stops (policy refusal or a failed agent call) starts exactly the apps
that were running; a drive the agent let go keeps its apps stopped. Household text HU + EN. Seam `decomStacks`, so no
test reaches docker. Three tests, two red-proofs (`P3/redproof-r1144.txt`). Not pressed live: no test box has a drive
that is safe to retire.

## 4. Agent 0.159.2

Released with `release-agent.sh` (code pushed first, CHANGELOG after — the `release-complete` hook refuses the other
order): sha256 `4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9`, bundle
`477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9` (read back from the registry). **demo-hp proof:** a
whole-guest backup of 9201 through the dashboard (`/api/guest-backup/trigger`, snapshot mode, completed in 7.5 min):
**0** `felhom-thin-check` sudo lines; control — the night's backup on 0.159.1 at 04:33 had **1**
(`P4/demo-hp-backup-journal.txt`). The failure path stays covered by the agent's fixture tests. **Operator: yes** →
Tester 1 and demo-felhom by signed jobs (agent_update + agent_config_update): both `signed op COMPLETED`, capability
probe 75/75, `update committed` 0.159.2, and a host report on 0.159.2 from each (`P4/`). **Not vouched** — STATUS.

## 5. The password order (R-1145, R-1146)

Step 4b (install and rotate-only): generate → PUT to the vault (200 = the hub's DB write is done) → chpasswd (one retry)
→ a PVE API login with the new value. **The states that can remain, and why each is safe or not:**

| State | When | Safe? |
|---|---|---|
| nothing changed | the vault refuses or is unreachable (a stale key included) | safe — root and the vault keep their old, matching pair |
| root = vault = new, PVE login 200 | the normal end | safe |
| vault = new, root = previous | chpasswd fails twice (or the run is killed between vault and set) | **not fully safe:** the vaulted value does not open the console; root keeps its previous password (the shadow line is fingerprinted before/after to prove it). The host key cannot read the vault back (by design), so the previous value cannot be re-vaulted; the log says the state in words, and the next run fixes it (it vaults first again). |
| root = vault = new, PVE login refused | pveproxy trouble | safe for the shell; the run dies and says the web login refused |

The old order's dangerous state (root = new, vault = old) is no longer reachable. The hub cannot tell the installer its
„last vaulted" time (no host-readable route); the installer trusts the PUT's 200, and CC checked the time moved in the
live proof. Live on demo-hp: vault `set_at` 2026-07-21 → 2026-10-18T10:43:33Z, the new value gets a PVE ticket (200),
the old one 401 (`P5/demo-hp-rotation.txt`). Harness: 52 passed / 0 failed (was 45/2); R1145-a..e red-proofed against
HEAD. Installer **1.34.0 published** on the operator's yes: tag `installer-v1.34.0`, both webpage pins, ArgoCD synced
the webpage Deployment only; the served file's sha256 = the tag's (`df722ee4…`). R-1146: SC2066 and GL8-F6 fixed, the
harness is gate `hostinstall-harness` with two decoys.

## 6. Calculator (R-1147)

Before/after (`audits/day-2026-10-18/P6-before-after.md`): Immich peak 3,290 → 3,983 MB (typical unchanged 1,557). The
one changed verdict: **Immich alone on 8 GB: fits, but tight → does not fit**. The family starter set is unchanged.
**Operator: yes** → published (cache-bust v5); live `sizing-data.json?v=5` serves 3,983. SIZING.md's table and the
pinned estimator tests moved with it. R-1036 closed (D14 = no).

## 7. The old-restic check

**Not run — the prune has not happened yet.** The hub allows demo-hp a clean-up window 6 days 12 hours after the last
one (2026-10-12 02:16:53 UTC → from 2026-10-18 14:16:53 UTC). But the box asks for a window only after its own nightly
off-site run, and that runs at 04:15 CEST. So the first real 0.19.1 prune comes **2026-10-19 at about 02:16 UTC**
(04:16 Budapest). Read at 14:33 UTC: no `[offbox]` line since the 11:09 restart, the scheduler's next run 2026-10-19
04:15 CEST, the hub's last window still id 9 (`P8/cleanup-window-not-yet.txt`, the hub DB read with its WAL). CC did not
grant a window by hand (that is the operator's). The STATUS item stays open, with the corrected time; the check script
is ready (`rx.sh` + a wrapper that lists and restores the newest, the oldest and the newest 0.14-made snapshot with both
restics and compares tree hashes).

## 8. Versions, commits, fixtures

- Hub **0.158.0** deployed (felhom.eu `4c319dde`).
- Controller **0.330.0** — `6c6bd87`; floor 0.330.0 / MinAgent 0.158.1 at 11:10:59 UTC (303 `floor_set`, read back with
  the WAL); per-box floors set and cleared; box walk green on the three test boxes (0 refusals, all healthy, login and
  files 200, https-only OK).
- Agent **0.159.2** — `65c1994` (code), tag `v0.159.2`, CHANGELOG `e25454c`.
- Installer **1.34.0** — felhom.eu `88aecf55` (vault-first + harness), `624bcdc3` (1.34.0, tagged), `3332eb09` (pins).
- Estimator — `37cb83c1`. Fixtures: none changed (the new texts travel in JSON).

## 9. Register — 173 before, 168 after; opened 1, closed 6

Closed: R-1140, R-1144, R-1145, R-1146, R-1147, R-1036. Opened: R-1148. Narrowed: R-895 (dated check 2026-10-20).

## 10. Observations

1. Installer 1.34.0's default path has not run in a fresh install. FILED: R-1148.
2. The review's F2 (box-reported run times explain new snapshots). NOT-A-FINDING as a new row: a known design limit,
   recorded in R-895.
3. `release-agent.sh` vs the `release-complete` hook: the CHANGELOG must follow the release. NOT-A-FINDING: the documented
   order (memory and skill); I tried the other order first and the hook refused it, as designed.
4. The estimator's new Immich figure is `memory.current` (with cache), not the `used` measure it normally takes.
   NOT-A-FINDING: on the safe side, said in SIZING.md, and approved by the operator with the table in front of them.
5. A safety check refused a `sh -c` with an unguarded `rm "$H"`. NOT-A-FINDING: rewritten with `${H:?}` in a helper.

## Teardown — three layers

- **Machines:** demo-hp 9201 — one whole-guest backup taken (kept, like any backup); no app installed or removed today.
  demo-hp host — the installer copy shredded after the rotation; root password rotated on purpose (new value in the
  vault). Tester 1, demo-felhom — agent 0.159.2 and controller 0.330.0 through their normal paths; nothing else.
- **Hub:** hub 0.158.0; global floor 0.330.0 / 0.158.1; no per-box floor; signed jobs consumed. No vouch by CC.
- DooPlex: builds only; ArgoCD synced two Deployments (hub, webpage), each alone.

Instruction files: none edited. System poster: none of the three (the poster shows backup retentions; the hub's access
log is not on it). Architecture documents read: `02`, `03`, `05`, `07`, `09` §3 decision 250; `break-glass.md`.