From 7f7081d1279cdf8b98734c4d68a37bb7e68f4885 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 18 Oct 2026 16:36:53 +0200 Subject: [PATCH] Day 2026-10-18: report, CONTEXT; old-restic check not run yet (demo-hp's clean-up window opens after the 2026-10-19 04:15 CEST run), STATUS time corrected Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS --- CONTEXT.md | 5 + REPORT-day-2026-10-18.md | 139 ++++++++++++++++++ STATUS.md | 2 +- .../P8/cleanup-window-not-yet.txt | 20 +++ 4 files changed, 165 insertions(+), 1 deletion(-) create mode 100644 REPORT-day-2026-10-18.md create mode 100644 documentation/audits/day-2026-10-18/P8/cleanup-window-not-yet.txt diff --git a/CONTEXT.md b/CONTEXT.md index a99ecff2..a3dda61b 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -16,6 +16,11 @@ > and holds nothing of its own; this file does hold its own content, namely the standing rulings below. +> **2026-10-18 (day) — hub 0.158.0 DEPLOYED (access log live, agent-only vouch, R-895 first verdict 2026-10-20); +> controller 0.330.0 is the floor (R-1144); agent 0.159.2 released and on the three test hosts, NOT vouched; installer +> 1.34.0 published (vault-first step 4b, R-1145; fresh install unproven, R-1148); estimator Immich peak 3,983 MB +> (R-1147).** Independent review of the night: `audits/day-2026-10-18/P2/`. `REPORT-day-2026-10-18.md`. + > **2026-10-18 (night) — controller 0.329.0 is the floor (MinAgent 0.158.1); golden 0.329.0 baked, NOT vouched; hub 0.158.0 > built, NOT deployed.** Each app's own `appdata/` (R-1142), empty allow-list (R-981), drive retirement asks the > agent first (R-1063), plain no-durable-id refusal (R-1069), `:latest` for releases only (R-1143), Immich 3072M/2048M diff --git a/REPORT-day-2026-10-18.md b/REPORT-day-2026-10-18.md new file mode 100644 index 00000000..5882d50c --- /dev/null +++ b/REPORT-day-2026-10-18.md @@ -0,0 +1,139 @@ +# REPORT — day 2026-10-18 + +## In plain words + +- **The hub update is live.** Its own access log works and writes no secret (tested live with a fake token). You can + now vouch an agent without a new golden. The off-site snapshot check gives its first verdict on 20 October. +- **A fresh reviewer read last night's code.** One medium finding: a stale host key could have left the emergency + password unknown. Today's installer change (save in the hub first, then set) already closes it. Three low findings: + two fixed today, one is a known design limit, now written in its row. +- **„Retire anyway":** when the host says no, the drive's apps start again. Where the host's disk list already says no, + nothing is stopped at all. Controller 0.330.0 is the floor for every box. +- **Agent 0.159.2** runs on demo-hp, Tester 1 and demo-felhom (your yes). A whole-box backup ran with no pool check. + **The vouch is yours** (STATUS). +- **Installer 1.34.0 is published** (your yes): the emergency password is saved in the hub first, then set, then + checked with a login. Proven on demo-hp. A fresh install with it has not run yet (row R-1148). +- **The calculator** shows Immich's bigger figure (your yes). Immich alone on an 8 GB machine now reads „does not fit". +- **The old-restic check has not run.** demo-hp's clean-up comes only after tonight's backup (2026-10-19 about 04:16 + Budapest). The STATUS item stays open. + +## 1. Hub 0.158.0 deploy + +ArgoCD: only `Deployment/hub` was OutOfSync; synced that resource alone → Synced/Healthy, image `felhom-hub:0.158.0`, +startup log `felhom-hub 0.158.0 starting` + `access log: /data/access-log …, kept 90 days`. Floor read back from the hub +DB with its WAL: 0.329.0 / 0.158.1 at deploy time (`audits/day-2026-10-18/P1/hub-deploy-readback.txt`). Reports from all +three boxes in the access log within two minutes (sources .101, .149, .155). +- **Access log (R-1140, closed):** today's file `access-2026-10-18.log`, mode 0600. A fresh 48-hex fake token sent in a + query, a `/bind/` path, a `session=` query, a Bearer header, a Cookie and a POST body: found 0 times; control + (the same file with the token appended) 1. Secret shapes (Bearer value, a 32+ character run, `password=`, `token=`, + `session=`): 0; control 1. `/bind/` is written `/bind/[redacted]` (`P1/access-log-live-test.txt`). +- **Agent-only vouch:** read-only — `/configuration` loads with CC's token (Day-0 section, golden 0.329.0, agent 0.159.1); + the save (`/configuration/artifacts`) answers 403 to the token, by design. No manifest change by CC. +- **Off-site alarm (R-895, NARROWED):** no fixture route exists in the hub for it; the unit tests are the proof so far. + Tomorrow's 07:10 run is the baseline (no verdict by design); the first verdict is 2026-10-20 07:10 — a DUE-CHECKS row. + +## 2. The independent review + +A fresh subagent with no context from the night reviewed controller v0.328.0→v0.329.0, hub 0.157.0→0.158.0, the agent's +orphan-check change and the installer's `--rotate-recovery-only`. Its report was written to a file and read back +(`audits/day-2026-10-18/P2/independent-review.md`, 54 lines). Verdicts (`P2/verdicts.md`): + +| # | Finding | Verdict | +|---|---|---| +| F1 medium | rotate-only: unauthenticated healthz, then chpasswd, then a vault that a stale key fails → an unvaulted password | REAL in the night's code → **fixed** by R-1145 (vault first); harness `R1145-e`, fails against HEAD | +| F2 low–medium | `offsite_snapshots_unexplained` is explained by the box's own reported run time | REAL, by design (decision 68) → written into R-895; the custody alarm `_vanished` is not affected | +| F3 low | „anyway" stops apps before the agent is asked | REAL, was R-1144 → **fixed** (controller 0.330.0) | +| F4 low | host key in curl's argv during the vault PUT | REAL, pre-existing → **fixed** (header file) | + +## 3. R-1144 + +`decommissionAnyway`: reads the agent's disk list first (a role other than user-data, or the backup target → the plain +refusal, nothing stopped); after a refusal past the stops (policy refusal or a failed agent call) starts exactly the apps +that were running; a drive the agent let go keeps its apps stopped. Household text HU + EN. Seam `decomStacks`, so no +test reaches docker. Three tests, two red-proofs (`P3/redproof-r1144.txt`). Not pressed live: no test box has a drive +that is safe to retire. + +## 4. Agent 0.159.2 + +Released with `release-agent.sh` (code pushed first, CHANGELOG after — the `release-complete` hook refuses the other +order): sha256 `4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9`, bundle +`477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9` (read back from the registry). **demo-hp proof:** a +whole-guest backup of 9201 through the dashboard (`/api/guest-backup/trigger`, snapshot mode, completed in 7.5 min): +**0** `felhom-thin-check` sudo lines; control — the night's backup on 0.159.1 at 04:33 had **1** +(`P4/demo-hp-backup-journal.txt`). The failure path stays covered by the agent's fixture tests. **Operator: yes** → +Tester 1 and demo-felhom by signed jobs (agent_update + agent_config_update): both `signed op COMPLETED`, capability +probe 75/75, `update committed` 0.159.2, and a host report on 0.159.2 from each (`P4/`). **Not vouched** — STATUS. + +## 5. The password order (R-1145, R-1146) + +Step 4b (install and rotate-only): generate → PUT to the vault (200 = the hub's DB write is done) → chpasswd (one retry) +→ a PVE API login with the new value. **The states that can remain, and why each is safe or not:** + +| State | When | Safe? | +|---|---|---| +| nothing changed | the vault refuses or is unreachable (a stale key included) | safe — root and the vault keep their old, matching pair | +| root = vault = new, PVE login 200 | the normal end | safe | +| vault = new, root = previous | chpasswd fails twice (or the run is killed between vault and set) | **not fully safe:** the vaulted value does not open the console; root keeps its previous password (the shadow line is fingerprinted before/after to prove it). The host key cannot read the vault back (by design), so the previous value cannot be re-vaulted; the log says the state in words, and the next run fixes it (it vaults first again). | +| root = vault = new, PVE login refused | pveproxy trouble | safe for the shell; the run dies and says the web login refused | + +The old order's dangerous state (root = new, vault = old) is no longer reachable. The hub cannot tell the installer its +„last vaulted" time (no host-readable route); the installer trusts the PUT's 200, and CC checked the time moved in the +live proof. Live on demo-hp: vault `set_at` 2026-07-21 → 2026-10-18T10:43:33Z, the new value gets a PVE ticket (200), +the old one 401 (`P5/demo-hp-rotation.txt`). Harness: 52 passed / 0 failed (was 45/2); R1145-a..e red-proofed against +HEAD. Installer **1.34.0 published** on the operator's yes: tag `installer-v1.34.0`, both webpage pins, ArgoCD synced +the webpage Deployment only; the served file's sha256 = the tag's (`df722ee4…`). R-1146: SC2066 and GL8-F6 fixed, the +harness is gate `hostinstall-harness` with two decoys. + +## 6. Calculator (R-1147) + +Before/after (`audits/day-2026-10-18/P6-before-after.md`): Immich peak 3,290 → 3,983 MB (typical unchanged 1,557). The +one changed verdict: **Immich alone on 8 GB: fits, but tight → does not fit**. The family starter set is unchanged. +**Operator: yes** → published (cache-bust v5); live `sizing-data.json?v=5` serves 3,983. SIZING.md's table and the +pinned estimator tests moved with it. R-1036 closed (D14 = no). + +## 7. The old-restic check + +**Not run — the prune has not happened yet.** The hub allows demo-hp a clean-up window 6 days 12 hours after the last +one (2026-10-12 02:16:53 UTC → from 2026-10-18 14:16:53 UTC). But the box asks for a window only after its own nightly +off-site run, and that runs at 04:15 CEST. So the first real 0.19.1 prune comes **2026-10-19 at about 02:16 UTC** +(04:16 Budapest). Read at 14:33 UTC: no `[offbox]` line since the 11:09 restart, the scheduler's next run 2026-10-19 +04:15 CEST, the hub's last window still id 9 (`P8/cleanup-window-not-yet.txt`, the hub DB read with its WAL). CC did not +grant a window by hand (that is the operator's). The STATUS item stays open, with the corrected time; the check script +is ready (`rx.sh` + a wrapper that lists and restores the newest, the oldest and the newest 0.14-made snapshot with both +restics and compares tree hashes). + +## 8. Versions, commits, fixtures + +- Hub **0.158.0** deployed (felhom.eu `4c319dde`). +- Controller **0.330.0** — `6c6bd87`; floor 0.330.0 / MinAgent 0.158.1 at 11:10:59 UTC (303 `floor_set`, read back with + the WAL); per-box floors set and cleared; box walk green on the three test boxes (0 refusals, all healthy, login and + files 200, https-only OK). +- Agent **0.159.2** — `65c1994` (code), tag `v0.159.2`, CHANGELOG `e25454c`. +- Installer **1.34.0** — felhom.eu `88aecf55` (vault-first + harness), `624bcdc3` (1.34.0, tagged), `3332eb09` (pins). +- Estimator — `37cb83c1`. Fixtures: none changed (the new texts travel in JSON). + +## 9. Register — 173 before, 168 after; opened 1, closed 6 + +Closed: R-1140, R-1144, R-1145, R-1146, R-1147, R-1036. Opened: R-1148. Narrowed: R-895 (dated check 2026-10-20). + +## 10. Observations + +1. Installer 1.34.0's default path has not run in a fresh install. FILED: R-1148. +2. The review's F2 (box-reported run times explain new snapshots). NOT-A-FINDING as a new row: a known design limit, + recorded in R-895. +3. `release-agent.sh` vs the `release-complete` hook: the CHANGELOG must follow the release. NOT-A-FINDING: the documented + order (memory and skill); I tried the other order first and the hook refused it, as designed. +4. The estimator's new Immich figure is `memory.current` (with cache), not the `used` measure it normally takes. + NOT-A-FINDING: on the safe side, said in SIZING.md, and approved by the operator with the table in front of them. +5. A safety check refused a `sh -c` with an unguarded `rm "$H"`. NOT-A-FINDING: rewritten with `${H:?}` in a helper. + +## Teardown — three layers + +- **Machines:** demo-hp 9201 — one whole-guest backup taken (kept, like any backup); no app installed or removed today. + demo-hp host — the installer copy shredded after the rotation; root password rotated on purpose (new value in the + vault). Tester 1, demo-felhom — agent 0.159.2 and controller 0.330.0 through their normal paths; nothing else. +- **Hub:** hub 0.158.0; global floor 0.330.0 / 0.158.1; no per-box floor; signed jobs consumed. No vouch by CC. +- DooPlex: builds only; ArgoCD synced two Deployments (hub, webpage), each alone. + +Instruction files: none edited. System poster: none of the three (the poster shows backup retentions; the hub's access +log is not on it). Architecture documents read: `02`, `03`, `05`, `07`, `09` §3 decision 250; `break-glass.md`. diff --git a/STATUS.md b/STATUS.md index 5a5bf636..e49835d1 100644 --- a/STATUS.md +++ b/STATUS.md @@ -41,7 +41,7 @@ ## What we work on - **CC's secrets** (R-1118): done 2026-10-17 — the store, `sec`, the guards in every CC session (option B: `bash -x` and `curl -v` allowed inside `sec run`), the nightly check (linger on). The old credentials file and its three copies are deleted. -- **After 2026-10-18 14:16 UTC** (demo-hp's next clean-up window): check that the old restic (image 0.322.0) still lists and restores demo-hp's store after its first real 0.19.1 prune (`audits/night-restore-2026-10-17/tools/rx.sh`). Everything else of R-1105/R-1107 is done. +- **After 2026-10-19 02:15 UTC** (demo-hp's next clean-up window: the hub allows one from 2026-10-18 14:16 UTC, but the box asks only after its 04:15 CEST night run; not open on 2026-10-18 at 14:33 UTC, `audits/day-2026-10-18/P8/`): check that the old restic (image 0.322.0) still lists and restores demo-hp's store after its first real 0.19.1 prune (`audits/night-restore-2026-10-17/tools/rx.sh`). Everything else of R-1105/R-1107 is done. - The power-cut findings still open (R-1004, R-1005, R-1010, R-1013). R-1012 closed: agent 0.159.1 names an orphan thin snapshot in the backup failure. - Media apps on every drive (R-1023): live with the 0.313.0 floor — every drive in every media app, a new drive made ready by the box, a pulled drive loses nothing, a safe removal lets go first and the drive comes back by itself. diff --git a/documentation/audits/day-2026-10-18/P8/cleanup-window-not-yet.txt b/documentation/audits/day-2026-10-18/P8/cleanup-window-not-yet.txt new file mode 100644 index 00000000..800ca423 --- /dev/null +++ b/documentation/audits/day-2026-10-18/P8/cleanup-window-not-yet.txt @@ -0,0 +1,20 @@ +# P8 — old-restic check: NOT RUN (the clean-up window has not opened). Read 2026-10-18T14:33:43Z. + +## demo-hp's clean-up windows (hub DB with its WAL, one tar) +id|customer|opened_at|closes_by|closed_at|box_result +9|demo-hp|2026-10-12 02:16:53|2026-10-12 02:36:53|2026-10-12 02:17:06|pruned +6|demo-hp|2026-10-05 05:25:53|2026-10-05 05:45:53|2026-10-05 05:26:04|pruned +2|demo-hp|2026-10-04 05:36:14|2026-10-04 05:56:14|2026-10-04 05:36:17|nothing + +## The rule (hub/internal/offsitekeys/service.go:255) + windowCadence = 6*24*time.Hour + 12*time.Hour // "weekly", with slack for the night chain's drift +Last window 2026-10-12 02:16:53 UTC + 6 d 12 h = 2026-10-18 14:16:53 UTC (the STATUS time). +The box asks for a window only AFTER its nightly off-site run (controller backup/offbox_window.go header). + +## demo-hp guest 9201 controller (docker logs since 06:00 UTC, [offbox] lines + scheduler) +felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.330.0 Up 3 hours (healthy) +2026/10/18 11:09:36 scheduler.go:150: [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-10-19 04:15 CEST +2026/10/18 11:09:36 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="offbox-backup" schedule="04:15" nextRun=2026-10-19T04:15:00+02:00 totalJobs=15 +2026/10/18 11:09:36 scheduler.go:85: [DEBUG] [scheduler] daily job offbox-backup: next run at 2026-10-19 04:15:00 CEST (waiting 15h5m23s) + +Next run: 2026-10-19 04:15 CEST = 02:15 UTC; the window opens right after it. No window was asked for or granted by CC.