diff --git a/CONTEXT.md b/CONTEXT.md index 6f3d141f..6df0bb5c 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -16,6 +16,16 @@ > and holds nothing of its own; this file does hold its own content, namely the standing rulings below. +> **2026-10-26 (attended release run) — released: hub 0.171.0, agent 0.166.0 (vouched 10:04:55 CET), controller 0.344.0 +> (global floor 09:07:47 UTC, MinAgent 0.158.1), catalog `73c165d`; golden 0.344.0 baked and vouched (10:26:23 CET).** +> Live on the release builds (`audits/release-monday-2026-10-26/`): R-1287 (the set-aside mails „⚠️ … nothing failed"), +> R-1289 (the vouch line names 0.344.0), R-1280 (a 25 h job REJECTED on demo-hp, a 30 min one ran), R-1281 (forged header +> 000 with a working control; the household's files 200; OnlyOffice fetches the document over `felhom-files`), R-1282, +> R-1278, R-1286, R-1283 — closed. **R-1285 back to OPEN:** in the normal two-job delivery the bundle runs in the OLD +> agent process, so the policy pass waits for the next bundle. R-1261 stays VERIFY until Tuesday's live line test (R-1250). +> New: R-1294 (demo-hp's weekly off-site clean-up refused by its own cap, 209 > 179). The golden waiver file is removed — +> the gate is green on the new golden alone. D50 pin `day0_agent` → 0.166.0. + > **2026-10-26 (attended release run) — D51 = Start; D52 = bake the golden today on v0.344.0.** (`09` §3 decisions 263, > 264.) D51: the operator pressed Start on demo-hp's Kimai (07:57 local); the crash-loop hold lifted, both containers > healthy, 0 restarts (R-1286). D52: the Sunday 2026-10-25 bake was missed (not in that night's brief); the golden waiver diff --git a/REPORT-release-monday-2026-10-26.md b/REPORT-release-monday-2026-10-26.md new file mode 100644 index 00000000..4d21813a --- /dev/null +++ b/REPORT-release-monday-2026-10-26.md @@ -0,0 +1,75 @@ +# REPORT — the 2026-10-26 release: hub 0.171.0, agent 0.166.0, controller 0.344.0, catalog, golden 0.344.0 + +**Plain words first.** +1. **The push is unblocked.** The waiver covered one day; the three waiting felhom.eu commits went out (CI 2352 ✓ on + `7ad7c996`, which carries `b00d5c61` and `1550b1da`). The waiver is now removed: the golden gate is green on the new + golden alone. +2. **Hub 0.171.0 is live.** A set-aside now mails „⚠️ … nothing failed" as a warning (seen in your mailbox), and the + vouch line names a release version as „the fleet's newest". +3. **Agent 0.166.0 is vouched** (10:04:55 CET) and runs on the three test boxes; a 25-hour signed job is refused on a box. +4. **Controller 0.344.0 runs on every live box** (3 of 3; the global floor 09:07:47 UTC). Tester-2 has no box (R-862). +5. **Golden 0.344.0 is baked and vouched** (10:26:23 CET); a fresh install from the public ISO came up on 0.344.0 with no + update step. + +## The release-build checks (one line each) + +- **R-1287** — Tester 1 set-aside → mail „⚠️ tester-1: a backup of an earlier install was set aside — nothing failed", + warning (hub log + mailbox; the 10-25 mails of the same event read 🔴 „failed"). Closed. +- **R-1289** — the agent vouch logged „behind the fleet's 0.344.0" while `0.344.0-rc.3` reports sit in the same table. Closed. +- **R-1280** — demo-hp: 25 h job REJECTED, 30 min job ran; `felhom-opsign -ttl 25h` refuses. Closed. +- **R-1285** — the policy pass is wired (seen under 0.166.0), but in the normal delivery the bundle runs in the OLD agent + process, so a BYO host still loses our drop-in one release late. **Back to OPEN.** +- **R-1281** — FileBrowser only on its own networks on all three boxes; forged header from four apps → 000 (control 404/301); + household files 200, no cookie 401; OnlyOffice linked: joins `felhom-files`, editor config 200, the document fetched 200. + The file manager restarted once at the update. Closed. +- **R-1261 / R-1250** — the line hold is in 0.344.0; the live test is Tuesday 10:00–16:00 (dated check stays). R-1261 VERIFY. +- **R-1282** — „Később" survives a controller restart (start-up line + settings.json). Closed. +- **R-1278** — the install page shows the kept-backup hint and link; none before or after (Tester 1). Closed. +- **R-1286** — Kimai has its install id (07:01:55Z); the hold sentences are released. Closed. +- **R-1288** — released (proven on the night's rig). Closed. +- **R-1283** — catalog `73c165d` pushed; Tester 1 synced at once, MeTube's folder `optional`. Closed. +- **D47** — 0 set aside after the update; every app has an install id. + +## Versions, digests, CI by commit + +| What | Version / digest | Commit → CI job | +|---|---|---| +| hub | 0.171.0, `sha256:742d2e69ce75d0d96598e8ee9a9acb11e677a3663348571e34abbd3819c7c46e` | `7ad7c996` → 2352 ✓ (code `f6328486` → 2342 ✓); manifest `6efb4760` | +| agent | 0.166.0, binary `3f22c65b…bfea9c5c`, bundle `093772c1…5156003a3` | `aa69114` → 2351 ✓; record `b890b57` → 2356 ✓ | +| controller | 0.344.0, `sha256:9435165dab85b1200d94157ec49d8be065889139e1831a5476dc68f4b9474b8a` | `0a59044` → 2355 ✓ | +| catalog | `73c165d` | gates green locally (catalog CI runs on push) | +| golden | 0.344.0, `767a90fe07ed47c85a3e08a16b85926e30531b65608f9856aa25855b4dd53210` (637,293,448 bytes) | `documentation/tests/golden-0.344.0-2026-10-26/` | + +## Rows + +Before **219**, after **212**; opened **2** (R-1294 demo-hp's off-site clean-up refused by its own cap; R-1295 PBS DR +provisioning rolled back at a re-enrolled install, log only); closed **9** (R-1278, R-1280, R-1281, R-1282, R-1283, R-1286, +R-1287, R-1288, R-1289). R-1285 back to OPEN. + +## Observations + +- FILED: R-1294 — demo-hp's weekly off-site clean-up refused (209 > 179) at 03:17Z; each refused week grows the next plan. +- FILED: R-1295 — at the fresh install the PBS DR auto-provision for Tester4 rolled back (`self_check_failed`); no mail. +- R-1285 reopened (delivery order), see above. +- NOT-A-FINDING: the vouch's `wrapper_sha=false` — the PBS-DR wrapper was never vouched (empty before and after). +- NOT-A-FINDING: 9202's rc start logged one WARN while FileBrowser was re-created in the same second; joined 2 s later, log + only; the release boxes logged none. +- NOT-A-FINDING: the 07:23Z self-bind mail to tester4@ — the operator's delete of `Tester4-84bbe7`, as designed. +- NOT-A-FINDING: the 02:57Z DooPlex sweep found `RIG_2026_10_25_SETUP_CODE` in last night's session record — that box is + destroyed, so the code opens nothing; the sweep worked as designed. +- NOT-A-FINDING: R-1282's status API read „LOST" on Tester 1 — the API shows `later_at` only on a box with media apps; + the stored value and the start-up line both kept it (instrument, recorded in the evidence). +- Deviation from the brief, said once: the fresh-install proof ran AFTER the golden vouch, because an install refuses any + golden but the vouched one (R-297); the vouch was reversible and the proof passed. + +## Teardown + +- **Machines:** fresh-install VM 9373 on demo-hp destroyed with its disks (nvme-scratch); Tester 1's throwaway Tandoor (×2) + and OnlyOffice (×2) removed through the product, nothing left; 9202 back on 0.343.0 with FileBrowser on `traefik-public` + and no `felhom-files` network, the rc image deleted; the drill VM back to `virgin` (CT 9100 purged). +- **Host:** demo-hp's thin pool never used; `nvme-scratch` (standing) untouched otherwise. +- **Hub:** host `Tester4-eb7371` (the fresh-install VM) stays for the operator to delete (STATUS); per-box floors cleared. +- **Secrets:** bake token/user, bind token, pairing code shredded; the pairing code blacked out in the committed + screenshot; evidence scanned (6 sec values + the 3 one-time values: 0 hits). New sec entry `REL_2026_10_26_ROOTPW` + (the test VM's typed root password, replaced at day-0; the VM is gone). +- One test side effect: at 2026-10-27 03:30 the „Később" timer fires on Tester 1, which has no media app — it restarts nothing. diff --git a/STATUS.md b/STATUS.md index 2e7d16b6..6d6f1b67 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-10-26 (morning).** Installer 1.36.1 is live: a wrong passphrase now says so. Overnight, commit only: the torrent line test holds every seeder; FileBrowser is off the app network; a set-aside mails as a set-aside; the four D44 picks are built and proven on a test VM. +**Updated 2026-10-26 (release day).** Every box runs hub 0.171.0, agent 0.166.0 and controller 0.344.0, and new installs start on golden 0.344.0 — all vouched: the torrent line hold, FileBrowser off the app network, a set-aside mails „nothing failed", a signed job lasts at most 24 h. One leftover: on a BYO host the SSH clean-up still comes one release late (R-1285). ## What works - **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241). @@ -23,6 +23,7 @@ - DooPlex: something sets the shared checkout to „bare" (R-935). ## What we work on +- Next for CC: demo-hp's weekly off-site clean-up is refused by its own cap (R-1294); the BYO SSH clean-up runs one release late (R-1285). - Dated checks, run by CC. 2026-10-27: the qBittorrent line test on a seeding box, with DooPlex's qBittorrent as the peer (R-1250, your yes); the weekly clean-up window (R-435; the logs of the last one were lost to restarts). 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes). ## Waiting on you — decide @@ -47,7 +48,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te 6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. Since 2026-10-14. 7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12. -**Monday 2026-10-26 (attended; each a STOP).** 1. Hub v0.171.0 (R-1287, R-1289). 2. Agent v0.166.0 + your vouch (R-1280, R-1285). 3. Controller v0.344.0 — before any household gets agent 0.165.0 (R-1261, R-1281, R-1288, R-1282, R-1286, R-1278). 4. Catalog: MeTube (R-1283). 5. Dated: R-1250 Tuesday. 6. Hub Hosts page: delete the dead `Tester4-84bbe7`. If not: the fixes stay unreleased; the host alarms. +**2026-10-26.** On the hub's Hosts page, delete `Tester4-eb7371` (CC's fresh-install test VM, destroyed). If not: the hub alarms that the box went silent. **2026-10-23 do list.** 1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens. 2. On the hub's Hosts page: Discard appliance 49. If not: a dead entry alarms. diff --git a/documentation/audits/release-monday-2026-10-26/agent-0166/boxes.txt b/documentation/audits/release-monday-2026-10-26/agent-0166/boxes.txt new file mode 100644 index 00000000..10e6340f --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/agent-0166/boxes.txt @@ -0,0 +1,33 @@ +# agent 0.166.0 delivery by signed agent_update + agent_config_update (felhom-opsign, key felhom-op-1), 2026-10-26 +== demo-hp +felhom-agent 0.166.0 +time=2026-10-26T09:03:07.108+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=d70489200b46088d op=agent_update +time=2026-10-26T09:03:08.570+01:00 level=WARN msg="osupdate: config bundle INSTALLED" op=agent_config_update agent_version=0.166.0 bundle="{\"agent_version\": \"0.166.0\", \"authority\": \"signed\", \"kept\": [\"/etc/felhom/crash-guard.conf\"], \"prev_dir\": \"/var/lib/felhom-os-apply/bundle-prev/20261026T080307Z-before-0.166.0\ +time=2026-10-26T09:03:09.616+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=81bbc6a41cef76cb op=agent_config_update +time=2026-10-26T09:03:10.331+01:00 level=INFO msg="felhom-agent daemon starting" version=0.166.0 host_id=demo-hp-bb76ea hub_url=https://hub.felhom.eu interval_s=900 +time=2026-10-26T09:03:11.465+01:00 level=INFO msg="capabilities self-check" ok=76 total=76 degraded=0 inactive=0 +time=2026-10-26T09:04:12.557+01:00 level=WARN msg="selfupdate: update committed" version=0.166.0 wrapper="" +time=2026-10-26T09:18:23.594+01:00 level=WARN msg="signedjobs: REJECTED signed op — executor not called" job=4702af39db454f15 op=agent_config_update reason=rejected err="authz: op validity window too long: issued_at=2026-10-26 08:11:55 +0000 UTC expires_at=2026-10-27 09:11:55 +0000 UTC (25h0m0s, the most is 24h0m0s)" +time=2026-10-26T09:33:25.398+01:00 level=WARN msg="osupdate: config bundle INSTALLED" op=agent_config_update agent_version=0.166.0 bundle="{\"agent_version\": \"0.166.0\", \"authority\": \"signed\", \"kept\": [\"/etc/felhom/crash-guard.conf\"], \"prev_dir\": \"/var/lib/felhom-os-apply/bundle-prev/20261026T083324Z-before-0.166.0\ +os-apply: BUNDLE POLICY DONE install_mode=appliance removed=0 +time=2026-10-26T09:33:25.500+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE POLICY DONE install_mode=appliance removed=0" +time=2026-10-26T09:33:25.500+01:00 level=INFO msg="osupdate: bundle policy pass done" op=agent_config_update agent_version=0.166.0 bundle_policy="{\"install_mode\": \"appliance\", \"removed\": []}" +time=2026-10-26T09:33:26.543+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=a22eca000de7bfb6 op=agent_config_update +== felhom-pve +felhom-agent 0.166.0 +time=2026-10-26T09:10:07.742+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=afca32508fe8d923 op=agent_update +time=2026-10-26T09:10:08.529+01:00 level=WARN msg="osupdate: config bundle INSTALLED" op=agent_config_update agent_version=0.166.0 bundle="{\"agent_version\": \"0.166.0\", \"authority\": \"signed\", \"kept\": [\"/etc/felhom/crash-guard.conf\"], \"prev_dir\": \"/var/lib/felhom-os-apply/bundle-prev/20261026T081007Z-before-0.166.0\ +time=2026-10-26T09:10:09.181+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=430f09b61278c214 op=agent_config_update +time=2026-10-26T09:10:10.456+01:00 level=INFO msg="felhom-agent daemon starting" version=0.166.0 host_id=demo-felhom-8363b5 hub_url=https://hub.felhom.eu interval_s=900 +time=2026-10-26T09:10:11.126+01:00 level=INFO msg="capabilities self-check" ok=76 total=76 degraded=0 inactive=0 +time=2026-10-26T09:11:12.434+01:00 level=WARN msg="selfupdate: update committed" version=0.166.0 wrapper="" +== root@192.168.0.154 +felhom-agent 0.166.0 +time=2026-10-26T09:16:10.590+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=665accf7f3a71a24 op=agent_update +time=2026-10-26T09:16:11.762+01:00 level=WARN msg="osupdate: config bundle INSTALLED" op=agent_config_update agent_version=0.166.0 bundle="{\"agent_version\": \"0.166.0\", \"authority\": \"signed\", \"kept\": [\"/etc/felhom/crash-guard.conf\"], \"prev_dir\": \"/var/lib/felhom-os-apply/bundle-prev/20261026T081610Z-before-0.166.0\ +time=2026-10-26T09:16:12.801+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=8dabce5865f2c029 op=agent_config_update +time=2026-10-26T09:16:13.668+01:00 level=INFO msg="felhom-agent daemon starting" version=0.166.0 host_id=tester-1-d70be4 hub_url=https://hub.felhom.eu interval_s=900 +time=2026-10-26T09:16:14.714+01:00 level=INFO msg="capabilities self-check" ok=76 total=76 degraded=0 inactive=0 +time=2026-10-26T09:17:15.564+01:00 level=WARN msg="selfupdate: update committed" version=0.166.0 wrapper="" + +## R-1280 signer side (felhom-opsign built from v0.166.0): -ttl 25h -> rc=2 'refused: a box accepts a window of at most 24h0m0s (R-1280)'; -ttl 24h -> rc=0 (envelope deleted, never sent) diff --git a/documentation/audits/release-monday-2026-10-26/agent-0166/release.txt b/documentation/audits/release-monday-2026-10-26/agent-0166/release.txt new file mode 100644 index 00000000..6a5f3823 --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/agent-0166/release.txt @@ -0,0 +1,29 @@ +[release-agent] building 0.166.0 … +[release-agent] built ok: sha256 3f22c65b78914632fa476292f6b58e5af3ed9e9900f3a5a7a347a9c0bfea9c5c +[release-agent] config bundle built: sha256 093772c1bda8b974e86ed45472f8fadc8992f247563f5739876102a5156003a3 +[release-agent] tagging v0.166.0 at aa69114 … +[release-agent] publishing … +[publish-agent] publishing /tmp/felhom-agent-MuCo1A (16291150 bytes, sha256 3f22c65b78914632…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/0.166.0/felhom-agent +[publish-agent] pre-delete existing artifact: HTTP 404 (404/204 expected) +[publish-agent] upload OK (HTTP 201) +[publish-agent] round-trip GET verified (sha256 matches) +[publish-agent] DONE. Record in the hub operator UI (Configs → Day-0 artifacts): agent 0.166.0 / 3f22c65b78914632fa476292f6b58e5af3ed9e9900f3a5a7a347a9c0bfea9c5c +[release-agent] config bundle published (HTTP 201) +[release-agent] pushing v0.166.0 … +remote: . Processing 1 references +remote: Processed 1 references in total +To https://gitea.dooplex.huAGENT_VERSION=0.166.0 +AGENT_SHA256=3f22c65b78914632fa476292f6b58e5af3ed9e9900f3a5a7a347a9c0bfea9c5c + + RELEASED — and installable, verified by download, not by this script's own say-so. + + version : 0.166.0 + tag : v0.166.0 + sha256 : 3f22c65b78914632fa476292f6b58e5af3ed9e9900f3a5a7a347a9c0bfea9c5c + bundle : 093772c1bda8b974e86ed45472f8fadc8992f247563f5739876102a5156003a3 (felhom-config-bundle.json — vouch it with the agent) + + NOT VOUCHED. Vouching is what points machines at this version and stays your deliberate act: + hub operator UI → Configs → Day-0 artifacts. Until then boxes keep installing the previous one. +/admin/felhom-agent.git + * [new tag] v0.166.0 -> v0.166.0 +[release-agent] verifying by independent download … diff --git a/documentation/audits/release-monday-2026-10-26/fresh-install/install/01-boot.png b/documentation/audits/release-monday-2026-10-26/fresh-install/install/01-boot.png new file mode 100644 index 00000000..6f4a47c4 Binary files /dev/null and b/documentation/audits/release-monday-2026-10-26/fresh-install/install/01-boot.png differ diff --git a/documentation/audits/release-monday-2026-10-26/fresh-install/install/02-disk.png b/documentation/audits/release-monday-2026-10-26/fresh-install/install/02-disk.png new file mode 100644 index 00000000..8b78dedd Binary files /dev/null and b/documentation/audits/release-monday-2026-10-26/fresh-install/install/02-disk.png differ diff --git a/documentation/audits/release-monday-2026-10-26/fresh-install/install/03-after-eula.png b/documentation/audits/release-monday-2026-10-26/fresh-install/install/03-after-eula.png new file mode 100644 index 00000000..65aa5f1f Binary files /dev/null and b/documentation/audits/release-monday-2026-10-26/fresh-install/install/03-after-eula.png differ diff --git a/documentation/audits/release-monday-2026-10-26/fresh-install/install/04-location.png b/documentation/audits/release-monday-2026-10-26/fresh-install/install/04-location.png new file mode 100644 index 00000000..fef13541 Binary files /dev/null and b/documentation/audits/release-monday-2026-10-26/fresh-install/install/04-location.png differ diff --git a/documentation/audits/release-monday-2026-10-26/fresh-install/install/05-password.png b/documentation/audits/release-monday-2026-10-26/fresh-install/install/05-password.png new file mode 100644 index 00000000..0c45d46a Binary files /dev/null and b/documentation/audits/release-monday-2026-10-26/fresh-install/install/05-password.png differ diff --git a/documentation/audits/release-monday-2026-10-26/fresh-install/install/06-pw-email.png b/documentation/audits/release-monday-2026-10-26/fresh-install/install/06-pw-email.png new file mode 100644 index 00000000..43ab3c69 Binary files /dev/null and b/documentation/audits/release-monday-2026-10-26/fresh-install/install/06-pw-email.png differ diff --git a/documentation/audits/release-monday-2026-10-26/fresh-install/install/07-network.png b/documentation/audits/release-monday-2026-10-26/fresh-install/install/07-network.png new file mode 100644 index 00000000..b44b1ffd Binary files /dev/null and b/documentation/audits/release-monday-2026-10-26/fresh-install/install/07-network.png differ diff --git a/documentation/audits/release-monday-2026-10-26/fresh-install/install/08-summary.png b/documentation/audits/release-monday-2026-10-26/fresh-install/install/08-summary.png new file mode 100644 index 00000000..f3b77046 Binary files /dev/null and b/documentation/audits/release-monday-2026-10-26/fresh-install/install/08-summary.png differ diff --git a/documentation/audits/release-monday-2026-10-26/fresh-install/install/08b-after-ok.png b/documentation/audits/release-monday-2026-10-26/fresh-install/install/08b-after-ok.png new file mode 100644 index 00000000..b929d843 Binary files /dev/null and b/documentation/audits/release-monday-2026-10-26/fresh-install/install/08b-after-ok.png differ diff --git a/documentation/audits/release-monday-2026-10-26/fresh-install/install/08c-hostname.png b/documentation/audits/release-monday-2026-10-26/fresh-install/install/08c-hostname.png new file mode 100644 index 00000000..13f96b3f Binary files /dev/null and b/documentation/audits/release-monday-2026-10-26/fresh-install/install/08c-hostname.png differ diff --git a/documentation/audits/release-monday-2026-10-26/fresh-install/install/09-summary.png b/documentation/audits/release-monday-2026-10-26/fresh-install/install/09-summary.png new file mode 100644 index 00000000..7d45ab65 Binary files /dev/null and b/documentation/audits/release-monday-2026-10-26/fresh-install/install/09-summary.png differ diff --git a/documentation/audits/release-monday-2026-10-26/fresh-install/install/10-installing.png b/documentation/audits/release-monday-2026-10-26/fresh-install/install/10-installing.png new file mode 100644 index 00000000..ddab12fa Binary files /dev/null and b/documentation/audits/release-monday-2026-10-26/fresh-install/install/10-installing.png differ diff --git a/documentation/audits/release-monday-2026-10-26/fresh-install/install/11-now.png b/documentation/audits/release-monday-2026-10-26/fresh-install/install/11-now.png new file mode 100644 index 00000000..799c5e43 Binary files /dev/null and b/documentation/audits/release-monday-2026-10-26/fresh-install/install/11-now.png differ diff --git a/documentation/audits/release-monday-2026-10-26/fresh-install/install/install-pressed.txt b/documentation/audits/release-monday-2026-10-26/fresh-install/install/install-pressed.txt new file mode 100644 index 00000000..ddbf9b96 --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/fresh-install/install/install-pressed.txt @@ -0,0 +1 @@ +Install pressed 09:23:16Z diff --git a/documentation/audits/release-monday-2026-10-26/fresh-install/proof.txt b/documentation/audits/release-monday-2026-10-26/fresh-install/proof.txt new file mode 100644 index 00000000..41c7e084 --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/fresh-install/proof.txt @@ -0,0 +1,20 @@ +# Fresh install on golden 0.344.0 (D52 proof), 2026-10-26. VM 9373 on demo-hp (6 cores, 12 GiB, nvme-scratch), public ISO felhom-installer-1.30.0-pve9.2-1.iso (sha256 590d7b9b… = the published one), graphical installer driven by vmctl.sh (screens in install/, the pairing code blacked out). +Install pressed 09:23:16Z +golden vouched by the operator 10:26:23 CET (Artifact manifest set: agent=0.166.0 golden=0.344.0 min_agent=0.158.1) +bind: POST to the public https://hub.felhom.eu/bind/ (the 07:23Z mail) with the console's pairing code + TESTER4_RETRIEVAL (sec) 09:29:41Z -> 200 'Sikeres összekötés' + +## channel A — the hub (box-originated events and reports) +2026/10/26 10:29:41 [INFO] self-bind SUCCESS: appliance 52 bound to customer Tester4 by customer self-service (token 16bfbd89…) +2026/10/26 10:30:41 [INFO] host enrolled: Tester4-eb7371 (customer Tester4) +2026/10/26 10:30:55 [INFO] Artifact manifest served for customer Tester4 (agent=0.166.0 golden=0.344.0) +2026/10/26 10:31:14 [WARN] escrow CARRIED for Tester4: the sealed package of deleted host Tester4-84bbe7 is now the current package of Tester4-eb7371 (repo-key hash 5501e4bf2cda…); the box will ask for the recovery code +2026/10/26 10:31:15 [ERROR] pbsdr auto-provision for Tester4 (WG-registration hook): pbsdr: F-14 auto re-issue for Tester4: tenantsync: endpoint refused: own-namespace list as felhom@pbs!Tester4 failed twice — rolled back (code self_check_failed) — save th +2026/10/26 10:32:25 [INFO] Notification preferences updated for Tester4: address set=true, household=false, events=[backup_failed db_dump_failed disk_warning disk_critical storage_disconnected node_down health_critical expected_backup_missed expected_dbdump_mi +2026/10/26 10:32:30 [INFO] Event from Tester4: controller_started (info) — Controller elindult (0.344.0) +2026/10/26 10:32:49 [INFO] Event from Tester4: controller_started (info) — Controller elindult (0.344.0) + +## channel B — the box itself (guest 192.168.0.157, LAN): /claim page asset tag = sha256("felhom-asset-tag-v1:"+version+":"+buildID)[:4] +192.168.0.157 (felhom.tester4.test/claim): style.css?v=38ef5ac9 +192.168.0.148 (felhom.enkicsifelhom.hu/login): style.css?v=38ef5ac9 +192.168.0.155 (felhom.enkisfelhom.hu/login): style.css?v=38ef5ac9 +(.148 = Tester 1, .155 = demo-hp: both on the 0.344.0 release image) -> the new box runs the same build. diff --git a/documentation/audits/release-monday-2026-10-26/fresh-install/teardown.txt b/documentation/audits/release-monday-2026-10-26/fresh-install/teardown.txt new file mode 100644 index 00000000..e1c573c7 --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/fresh-install/teardown.txt @@ -0,0 +1,8 @@ +name: release1026-box +scsi0: nvme-scratch:9373/vm-9373-disk-0.raw,size=64G +scsi1: nvme-scratch:9373/vm-9373-disk-1.raw,size=48G +scsihw: virtio-scsi-single +purging VM 9373 from related configurations.. +0 +ls: cannot access '/mnt/hdd_1/images/9373': No such file or directory +Mem: 29 13 9 0 7 15 diff --git a/documentation/audits/release-monday-2026-10-26/fresh-install/vm-create.txt b/documentation/audits/release-monday-2026-10-26/fresh-install/vm-create.txt new file mode 100644 index 00000000..9d4c4ab7 --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/fresh-install/vm-create.txt @@ -0,0 +1,21 @@ +Formatting '/mnt/hdd_1/images/9373/vm-9373-disk-0.raw', fmt=raw size=68719476736 preallocation=off +scsi0: successfully created disk 'nvme-scratch:9373/vm-9373-disk-0.raw,size=64G' +Formatting '/mnt/hdd_1/images/9373/vm-9373-disk-1.raw', fmt=raw size=51539607552 preallocation=off +scsi1: successfully created disk 'nvme-scratch:9373/vm-9373-disk-1.raw,size=48G' +update VM 9373: -boot order=scsi0;ide2 +boot: order=scsi0;ide2 +cores: 6 +cpu: host +ide2: local:iso/felhom-installer-1.30.0-pve9.2-1.iso,media=cdrom,size=1665356K +memory: 12288 +meta: creation-qemu=11.0.3,ctime=1793006231 +name: release1026-box +net0: virtio=BC:24:11:90:FC:9F,bridge=vmbr0 +ostype: l26 +scsi0: nvme-scratch:9373/vm-9373-disk-0.raw,size=64G +scsi1: nvme-scratch:9373/vm-9373-disk-1.raw,size=48G +scsihw: virtio-scsi-single +smbios1: uuid=d5d44035-5eec-4883-8824-bea3ea84c174 +vga: std +vmgenid: 019eab9b-6f9d-455c-ab35-230a3d4620a5 +STARTED diff --git a/documentation/audits/release-monday-2026-10-26/r1278/_kepthint/record.json b/documentation/audits/release-monday-2026-10-26/r1278/_kepthint/record.json new file mode 100644 index 00000000..c361ce0e --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/r1278/_kepthint/record.json @@ -0,0 +1,41 @@ +{ + "app": "tandoor", + "page_before": { + "http": "200", + "hint": false, + "kept": 0, + "link": false + }, + "install": true, + "seed": true, + "backup": { + "code": "200", + "seconds": 55, + "status": "{\"db_dump\": {\"count\": 3, \"duration\": \"52.044590795s\", \"last_run\": \"2026-10-26T08:22:02.417593112Z\", \"success\": true}, \"enabled\": true, \"running\": false}" + }, + "page_kept": { + "http": "200", + "hint": true, + "kept": 1, + "link": true + }, + "reinstall": true, + "set_aside_line": "2026/10/26 08:25:18 [INFO] [backup] tandoor: the recovery unit of an earlier install (0035d2587546ee84; the install ids differ) is SET ASIDE whole → /mnt/sys_drive/felhom-data/backups/set-aside/tand", + "after_remove": { + "containers": "", + "volumes": "", + "appyaml": "", + "appdata": "", + "kept": "", + "subdomain": "404" + }, + "set_aside_after_remove": "ls: cannot access '/mnt/*/felhom-data/backups/set-aside/tandoor': No such file or directory", + "primary_after_remove": "ls: cannot access '/mnt/*/felhom-data/backups/primary/tandoor': No such file or directory", + "page_after": { + "http": "200", + "hint": false, + "kept": 0, + "link": false + }, + "controller_lines": "2026/10/26 08:22:40 [INFO] [stacks] image retention (remove of tandoor): pass over 24 image(s) of [ghcr.io/tandoorrecipes/recipes postgres] — 2 candidate(s), 2 deleted, the rest kept\n2026/10/26 08:25:18 [INFO] [settings] kept copy recorded: kind=unit source=\"tandoor\" reason=earlier_install path=/mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261026T082518Z\n2026/10/26 08:25:18 [INFO] [backup] copy KEPT (earlier_install): unit \"tandoor\" at /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261026T082518Z\n2026/10/26 08:25:18 [INFO] [backup] tandoor: the recovery unit of an earlier install (0035d2587546ee84; the install ids differ) is SET ASIDE whole → /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261026T082518Z; this install (6ce5\n2026/10/26 08:25:18 [INFO] Event pushed: backup_run_failures (error) [hu-only] — App \"tandoor\": its backup belonged to an earlier install (0035d2587546ee84; this install 6ce5be52fde71858), so it was SET ASIDE whole at /mnt/sys_drive/felho\n2026/10/26 08:26:01 [INFO] [backup] kept copy /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261026T082518Z was deleted with its app's backups — record dropped\n2026/10/26 08:26:03 [INFO] [stacks] image retention (remove of tandoor): pass over 24 image(s) of [ghcr.io/tandoorrecipes/recipes postgres] — 2 candidate(s), 2 deleted, the rest kept\n" +} \ No newline at end of file diff --git a/documentation/audits/release-monday-2026-10-26/r1278/_kepthint/walk.log b/documentation/audits/release-monday-2026-10-26/r1278/_kepthint/walk.log new file mode 100644 index 00000000..c170438e --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/r1278/_kepthint/walk.log @@ -0,0 +1,14 @@ +08:18:36Z (a) install page, nothing kept: {'http': '200', 'hint': False, 'kept': 0, 'link': False} +08:18:36Z [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +08:20:52Z [1] deployed, controller state=running, pinned={'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.15', 'tandoor-postgres': 'postgres:17-alpine'} +08:20:52Z gate: recipes is gated — passed as the household (cookie set) +08:21:05Z tandoor: createsuperuser :: Running django-vite in production mode (no HMR) Superuser created successfully. +08:21:10Z tandoor: seeded superuser drillaa3ccf +08:21:10Z [4] Mentés most -> 200 {'ok': True, 'message': 'Mentés elindítva'} +08:22:38Z [5] stop 200, remove KEEPING data -> 200 {'ok': True, 'data': {'removed': 'tandoor', 'volumes_removed': ['tandoor_tandoor_media', 'tandoor_tandoor_postgres_data', 'tandoor_tandoor_static'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], +08:22:48Z (b) install page, a backup kept: {'http': '200', 'hint': True, 'kept': 1, 'link': True} +08:22:49Z [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +08:24:59Z [1] deployed, controller state=running, pinned={'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.15', 'tandoor-postgres': 'postgres:17-alpine'} +08:25:26Z (c) clean-up: set aside seen +08:26:01Z [7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'tandoor', 'volumes_removed': ['tandoor_tandoor_media', 'tandoor_tandoor_postgres_data', 'tandoor_tandoor_static'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], +08:26:17Z (d) install page after the clean-up: {'http': '200', 'hint': False, 'kept': 0, 'link': False} diff --git a/documentation/audits/release-monday-2026-10-26/r1278/run.txt b/documentation/audits/release-monday-2026-10-26/r1278/run.txt new file mode 100644 index 00000000..c170438e --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/r1278/run.txt @@ -0,0 +1,14 @@ +08:18:36Z (a) install page, nothing kept: {'http': '200', 'hint': False, 'kept': 0, 'link': False} +08:18:36Z [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +08:20:52Z [1] deployed, controller state=running, pinned={'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.15', 'tandoor-postgres': 'postgres:17-alpine'} +08:20:52Z gate: recipes is gated — passed as the household (cookie set) +08:21:05Z tandoor: createsuperuser :: Running django-vite in production mode (no HMR) Superuser created successfully. +08:21:10Z tandoor: seeded superuser drillaa3ccf +08:21:10Z [4] Mentés most -> 200 {'ok': True, 'message': 'Mentés elindítva'} +08:22:38Z [5] stop 200, remove KEEPING data -> 200 {'ok': True, 'data': {'removed': 'tandoor', 'volumes_removed': ['tandoor_tandoor_media', 'tandoor_tandoor_postgres_data', 'tandoor_tandoor_static'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], +08:22:48Z (b) install page, a backup kept: {'http': '200', 'hint': True, 'kept': 1, 'link': True} +08:22:49Z [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +08:24:59Z [1] deployed, controller state=running, pinned={'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.15', 'tandoor-postgres': 'postgres:17-alpine'} +08:25:26Z (c) clean-up: set aside seen +08:26:01Z [7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'tandoor', 'volumes_removed': ['tandoor_tandoor_media', 'tandoor_tandoor_postgres_data', 'tandoor_tandoor_static'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], +08:26:17Z (d) install page after the clean-up: {'http': '200', 'hint': False, 'kept': 0, 'link': False} diff --git a/documentation/audits/release-monday-2026-10-26/r1281/demo-felhom-forged-header.txt b/documentation/audits/release-monday-2026-10-26/r1281/demo-felhom-forged-header.txt new file mode 100644 index 00000000..722d3ee9 --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/r1281/demo-felhom-forged-header.txt @@ -0,0 +1,5 @@ +# R-1281 on controller 0.344.0, demo-felhom (felhom-pve guest 9201), 2026-10-26 ~08:25Z. fbprobe.sh (night tool, APPS env added): an app container asks FileBrowser directly; the header name is read inside the box, never printed. +qbittorrent/sonarr/radarr/prowlarr (traefik-public): no header -> 000 | forged household header -> 000 +filebrowser networks: felhom-filebrowser felhom-files; traefik: felhom-files felhom-tunnel traefik-public +CONTROL (same containers, same curl): -> felhom-controller:8080 = 404, -> traefik:80 = 301 (curl works; the network answers) +Tester 1 and demo-hp: filebrowser on felhom-filebrowser + felhom-files only (docker inspect). diff --git a/documentation/audits/release-monday-2026-10-26/r1281/tester-1-household-signin.txt b/documentation/audits/release-monday-2026-10-26/r1281/tester-1-household-signin.txt new file mode 100644 index 00000000..73254055 --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/r1281/tester-1-household-signin.txt @@ -0,0 +1,4 @@ +1. dashboard /__files/open -> HTTP/2 302 | to files host: True +2. files host /_felhom/sso -> HTTP/2 302 | files cookie set: True +3. a listing through traefik with the files cookie -> 200 +4. the same without any cookie (control) -> 401 diff --git a/documentation/audits/release-monday-2026-10-26/r1281/tester-1-onlyoffice-run1-wrong-path.txt b/documentation/audits/release-monday-2026-10-26/r1281/tester-1-onlyoffice-run1-wrong-path.txt new file mode 100644 index 00000000..9490d68f --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/r1281/tester-1-onlyoffice-run1-wrong-path.txt @@ -0,0 +1,15 @@ + [1] deploy -> 202 {'ok': True, 'data': {'warning': 'Az alkalmazások csúcsterhelése meghaladhatja a rendelkezésre álló memóriát. Ilyenkor a + [1] deployed, controller state=running, pinned={'onlyoffice': 'onlyoffice/documentserver:9.4.0'} +1. OnlyOffice installed and answering: True +2. link switched on -> 200 {"ok":true,"data":{"enabled":true,"enabled_at":"2026-10-26T08:32:53Z","status":"active"},"message":"Integráció engedélye +3. onlyoffice networks: felhom-files traefik-public | filebrowser: felhom-filebrowser felhom-files +4. filebrowser -> onlyoffice /healthcheck: true +5. onlyoffice -> filebrowser /health: 200 +6. control: paperless-webserver -> filebrowser: 000 +7. household uploads the .docx -> 200 +8. editor config -> 404 | document url: (none) +10. delete the .docx -> 200 +11. link off -> 200 + [7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'onlyoffice', 'volumes_removed': ['onlyoffice_onlyoffice_data', 'onlyoffice_onlyoffice_logs'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alka +12. leftovers: {'containers': '', 'volumes': '', 'appyaml': '', 'appdata': '', 'kept': '', 'subdomain': '404'} +13. filebrowser networks after: felhom-filebrowser felhom-files diff --git a/documentation/audits/release-monday-2026-10-26/r1281/tester-1-onlyoffice.txt b/documentation/audits/release-monday-2026-10-26/r1281/tester-1-onlyoffice.txt new file mode 100644 index 00000000..d6035c60 --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/r1281/tester-1-onlyoffice.txt @@ -0,0 +1,16 @@ + [1] deploy -> 202 {'ok': True, 'data': {'warning': 'Az alkalmazások csúcsterhelése meghaladhatja a rendelkezésre álló memóriát. Ilyenkor a + [1] deployed, controller state=running, pinned={'onlyoffice': 'onlyoffice/documentserver:9.4.0'} +1. OnlyOffice installed and answering: True +2. link switched on -> 200 {"ok":true,"data":{"enabled":true,"enabled_at":"2026-10-26T09:07:29Z","status":"active"},"message":"Integráció engedélye +3. onlyoffice networks: felhom-files traefik-public | filebrowser: felhom-filebrowser felhom-files +4. filebrowser -> onlyoffice /healthcheck: true +5. onlyoffice -> filebrowser /health: 200 +6. control: paperless-webserver -> filebrowser: 000 +7. household uploads the .docx -> 200 +8. editor config -> 200 | document url: http://filebrowser:80/api/resources/download +9. OnlyOffice fetches that document -> 200 +10. delete the .docx -> 200 +11. link off -> 200 + [7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'onlyoffice', 'volumes_removed': ['onlyoffice_onlyoffice_data', 'onlyoffice_onlyoffice_logs'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alka +12. leftovers: {'containers': '', 'volumes': '', 'appyaml': '', 'appdata': '', 'kept': '', 'subdomain': '404'} +13. filebrowser networks after: felhom-filebrowser felhom-files diff --git a/documentation/audits/release-monday-2026-10-26/r1282/tester-1-later.txt b/documentation/audits/release-monday-2026-10-26/r1282/tester-1-later.txt new file mode 100644 index 00000000..decd7b24 --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/r1282/tester-1-later.txt @@ -0,0 +1,14 @@ +1. status before -> ('200', '') +2. POST „Később" -> 200 {"data":{"later_at":"2026-10-27 03:30"},"ok":true} + +3. status after Később -> ('200', '') +4. docker restart felhom-controller -> restarted +5. status after restart -> ('200', '') +6. start-up lines -> 2026/10/26 08:26:52 [INFO] [media] media restart scheduled for 2026-10-27T03:30:00+01:00 (the household's „later"; stored — a controller restart keeps it, R-1282) +2026/10/26 08:26:56 [INFO] [media] the household's „later" media restart is kept from before the restart: 2026-10-27T02:30:00Z (R-1282) +VERDICT line above: an INSTRUMENT miss, not the fix — MediaNotice returns before LaterAt on a box with no media apps (Tester 1 has none). Second channel below: settings.json after the restart, and the start-up line „kept from before the restart". +Up 31 seconds (healthy) + +## second channel, 08:27:34Z (docker exec, settings.json inside the controller): "media_restart_later_at": "2026-10-27T02:30:00Z" (= 03:30 CET, the time the POST answered) +## one restart only: started=2026-10-26T08:26:55Z restarts=0 (the probe's docker restart) +## side effect: at 2026-10-27 03:30 CET the later restart fires on Tester 1 — it has no media app, so it restarts nothing (MediaNotice/mediaApps empty). diff --git a/documentation/audits/release-monday-2026-10-26/r1283-catalog-sync.txt b/documentation/audits/release-monday-2026-10-26/r1283-catalog-sync.txt new file mode 100644 index 00000000..b02b2ebf --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/r1283-catalog-sync.txt @@ -0,0 +1,3 @@ +# R-1283 catalog, 2026-10-26: pushed 73c165d (operator yes). Tester 1, POST /api/sync 09:10:4xZ: +[sync] catalog moved cd0cf48fabd4 -> 73c165d60f63 (0 new, 1 stack dir(s) changed); Sablonok frissítve — frissítve: metube +templates/metube/.felhom.yml on the box: class: optional (media/metube) diff --git a/documentation/audits/release-monday-2026-10-26/r1287-setaside/_setaside/record.json b/documentation/audits/release-monday-2026-10-26/r1287-setaside/_setaside/record.json new file mode 100644 index 00000000..2f148085 --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/r1287-setaside/_setaside/record.json @@ -0,0 +1,23 @@ +{ + "app": "tandoor", + "install": true, + "seed": true, + "backup": { + "code": "200", + "seconds": 55, + "status": "{\"db_dump\": {\"count\": 3, \"duration\": \"52.047194809s\", \"last_run\": \"2026-10-26T08:04:57.081005936Z\", \"success\": true}, \"enabled\": true, \"running\": false}" + }, + "reinstall": true, + "set_aside_line": "2026/10/26 08:06:54 [INFO] [backup] tandoor: the recovery unit of an earlier install (b8b08ea4d47c14a9; the install ids differ) is SET ASIDE whole → /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261026T080654Z; this install (82f96221066d4834) gets its own unit from its first backup (D47)", + "set_aside_at": "2026-10-26T08:08:08Z", + "controller_lines": "2026/10/26 08:05:33 [INFO] Event pushed: app_removed (info) [hu-only] — Alkalmazás eltávolítva: tandoor\n2026/10/26 08:06:54 [INFO] [backup] tandoor: the recovery unit of an earlier install (b8b08ea4d47c14a9; the install ids differ) is SET ASIDE whole → /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261026T080654Z; this install (82f96221066d4834) gets its own unit from its first backup (D47)\n2026/10/26 08:06:55 [INFO] Event pushed: backup_run_failures (error) [hu-only] — App \"tandoor\": its backup belonged to an earlier install (b8b08ea4d47c14a9; this install 82f96221066d4834), so it was SET ASIDE whole at /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261026T080654Z (D47) and \n", + "after_remove": { + "containers": "", + "volumes": "", + "appyaml": "", + "appdata": "", + "kept": "", + "subdomain": "404" + }, + "set_aside_after_remove": "ls: cannot access '/mnt/*/felhom-data/backups/set-aside/tandoor': No such file or directory" +} \ No newline at end of file diff --git a/documentation/audits/release-monday-2026-10-26/r1287-setaside/_setaside/walk.log b/documentation/audits/release-monday-2026-10-26/r1287-setaside/_setaside/walk.log new file mode 100644 index 00000000..60dd26cb --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/r1287-setaside/_setaside/walk.log @@ -0,0 +1,11 @@ +08:01:25Z [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +08:03:46Z [1] deployed, controller state=running, pinned={'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.15', 'tandoor-postgres': 'postgres:17-alpine'} +08:03:46Z gate: recipes is gated — passed as the household (cookie set) +08:04:00Z tandoor: createsuperuser :: Running django-vite in production mode (no HMR) Superuser created successfully. +08:04:05Z tandoor: seeded superuser drill1bdbd5 +08:04:05Z [4] Mentés most -> 200 {'ok': True, 'message': 'Mentés elindítva'} +08:05:33Z [5] stop 200, remove KEEPING data -> 200 {'ok': True, 'data': {'removed': 'tandoor', 'volumes_removed': ['tandoor_tandoor_media', 'tandoor_tandoor_postgres_data', 'tandoor_tandoor_static'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], +08:05:38Z [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +08:07:54Z [1] deployed, controller state=running, pinned={'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.15', 'tandoor-postgres': 'postgres:17-alpine'} +08:08:08Z set aside: 2026/10/26 08:06:54 [INFO] [backup] tandoor: the recovery unit of an earlier install (b8b08ea4d47c14a9; the install ids differ) is SET ASIDE whole → /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261026T080654Z; this install (82f96221066d4834) gets its +08:08:47Z [7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'tandoor', 'volumes_removed': ['tandoor_tandoor_media', 'tandoor_tandoor_postgres_data', 'tandoor_tandoor_static'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], diff --git a/documentation/audits/release-monday-2026-10-26/r1287-setaside/hub-and-mail.txt b/documentation/audits/release-monday-2026-10-26/r1287-setaside/hub-and-mail.txt new file mode 100644 index 00000000..f3efa703 --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/r1287-setaside/hub-and-mail.txt @@ -0,0 +1,11 @@ +# R-1287 live on hub 0.171.0 (2026-10-26). Channel A: the hub log. Channel B: the operator mailbox (Gmail connector, admin@felhom.eu). +## hub log +2026/10/26 09:06:55 [INFO] Event from tester-1: backup_run_failures (error) — App "tandoor": its backup belonged to an earlier install (b8b08ea4d47c14a9; this install 82f96221066d4834), so it was SET ASIDE whole at /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261026T080654Z (D47) and listed on the restore page. Expected after a reinstall; not expected on an upgrade — look if no one reinstalled the app. +2026/10/26 09:06:55 [INFO] error backup_run_failures for tester-1 holds only set-aside legs — sent as warning (R-1287) +2026/10/26 09:06:55 [INFO] Operator email sent for tester-1/backup_run_failures + +## mailbox (subjects; read with the Gmail connector 08:12Z) +2026-10-26T08:06:55Z [Felhom] ⚠️ tester-1: a backup of an earlier install was set aside — nothing failed (tandoor) Severity: warning +control, hub 0.170.0: +2026-10-25T17:31:56Z [Felhom] 🔴 tester-1: 1 of 1 apps failed to back up (backup run) Severity: error +2026-10-25T17:36:55Z [Felhom] 🔴 tester-1: 1 of 1 apps failed to back up (backup run) Severity: error diff --git a/documentation/audits/release-monday-2026-10-26/r1287-setaside/run.txt b/documentation/audits/release-monday-2026-10-26/r1287-setaside/run.txt new file mode 100644 index 00000000..60dd26cb --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/r1287-setaside/run.txt @@ -0,0 +1,11 @@ +08:01:25Z [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +08:03:46Z [1] deployed, controller state=running, pinned={'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.15', 'tandoor-postgres': 'postgres:17-alpine'} +08:03:46Z gate: recipes is gated — passed as the household (cookie set) +08:04:00Z tandoor: createsuperuser :: Running django-vite in production mode (no HMR) Superuser created successfully. +08:04:05Z tandoor: seeded superuser drill1bdbd5 +08:04:05Z [4] Mentés most -> 200 {'ok': True, 'message': 'Mentés elindítva'} +08:05:33Z [5] stop 200, remove KEEPING data -> 200 {'ok': True, 'data': {'removed': 'tandoor', 'volumes_removed': ['tandoor_tandoor_media', 'tandoor_tandoor_postgres_data', 'tandoor_tandoor_static'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], +08:05:38Z [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +08:07:54Z [1] deployed, controller state=running, pinned={'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.15', 'tandoor-postgres': 'postgres:17-alpine'} +08:08:08Z set aside: 2026/10/26 08:06:54 [INFO] [backup] tandoor: the recovery unit of an earlier install (b8b08ea4d47c14a9; the install ids differ) is SET ASIDE whole → /mnt/sys_drive/felhom-data/backups/set-aside/tandoor/20261026T080654Z; this install (82f96221066d4834) gets its +08:08:47Z [7] stop 200, remove with data -> 200 {'ok': True, 'data': {'removed': 'tandoor', 'volumes_removed': ['tandoor_tandoor_media', 'tandoor_tandoor_postgres_data', 'tandoor_tandoor_static'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], diff --git a/documentation/audits/release-monday-2026-10-26/tools/files_signin.py b/documentation/audits/release-monday-2026-10-26/tools/files_signin.py new file mode 100644 index 00000000..cf033b95 --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/tools/files_signin.py @@ -0,0 +1,36 @@ +#!/usr/bin/env python3 +"""files_signin.py — R-1281 on a release box: the household still reaches its files through traefik after FileBrowser +left traefik-public. The walk the dashboard's „Fájlok ↗" makes: GET /__files/open (household session) → a one-time +pass → GET /_felhom/sso on files. → the files cookie → a listing through traefik. A no-cookie control. +Night version: audits/night-2026-10-25/tools/r1281household.py (the rig); this one drives box_walk's TARGET. +No secret is printed. + +usage: SC= TARGET=tester-1 python3 files_signin.py""" +import os +import re +import sys + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../catalog-sweep-2026-10-24/tools")) +import sweep as sw # noqa: E402 + +w = sw.w +w.login() +sess = open(f"{w.SC}/sess{os.getpid()}.txt").read().strip() +files_host = "files." + w.DOMAIN +r = w.sh(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", w.HOSTHDR, "-H", f"Cookie: {sess}", + f"{w.BASE}/__files/open?to=%2F"]) +loc = w._loc(r.stdout) +print("1. dashboard /__files/open ->", (r.stdout or "").split("\n")[0].strip(), "| to files host:", files_host in loc) +m = re.search(r"https?://[^/]+(/_felhom/sso\?[^\s]+)", loc) +if not m: + sys.exit("no pass handed out") +r = w.sh(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", f"Host: {files_host}", f"{w.BASE}{m.group(1)}"]) +ck = re.search(r"(?im)^set-cookie:\s*(felhom_files=[^;\r\n]+)", r.stdout or "") +print("2. files host /_felhom/sso ->", (r.stdout or "").split("\n")[0].strip(), "| files cookie set:", bool(ck)) +if not ck: + sys.exit("no files cookie") +url = f"{w.BASE}/api/resources?path=%2F&" + "sou" + "rce=" + os.environ.get("FB_SOURCE", "Adatlemez") +r = w.sh(["curl", "-sk", "-o", "/dev/null", "-w", "%{http_code}", "-H", f"Host: {files_host}", "-H", f"Cookie: {ck.group(1)}", url]) +print("3. a listing through traefik with the files cookie ->", r.stdout) +r = w.sh(["curl", "-sk", "-o", "/dev/null", "-w", "%{http_code}", "-H", f"Host: {files_host}", url]) +print("4. the same without any cookie (control) ->", r.stdout) diff --git a/documentation/audits/release-monday-2026-10-26/tools/kept_hint_probe.py b/documentation/audits/release-monday-2026-10-26/tools/kept_hint_probe.py new file mode 100644 index 00000000..75e3b751 --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/tools/kept_hint_probe.py @@ -0,0 +1,77 @@ +#!/usr/bin/env python3 +"""kept_hint_probe.py — R-1278 on a release box: the install page of an app that is NOT installed says when a backup of +an earlier install is kept, and links the restore page. A THROWAWAY Tandoor: (a) the install page before anything +exists (control: no hint), (b) install, seed, back up, remove KEEPING the backups, the install page again (the hint and +the link must be there), (c) clean-up the way setaside_probe.py proved: reinstall (its first backup sets the kept unit +aside), then remove with data and backups. Endpoint-level, the requests the pages send. No secret is printed. + +usage: SC= SWEEP_EV= TARGET=tester-1 python3 kept_hint_probe.py""" +import json, os, re, sys, time +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../catalog-sweep-2026-10-24/tools")) +import sweep as sw # noqa: E402 + +w, fxm = sw.w, sw.fxm +APP = "tandoor" + + +def install_page(): + sess = open(f"{w.SC}/sess{os.getpid()}.txt").read().strip() + r = w.sh(["curl", "-skL", "-H", w.HOSTHDR, "-H", f"Cookie: {sess}", "-w", "\n%{http_code}", f"{w.BASE}/stacks/{APP}/deploy"]) + body = r.stdout or "" + code = body.rsplit("\n", 1)[-1] + m = re.search(r'class="alert alert-info kept-backup-offer" data-kept-backups="(\d+)"', body) + link = bool(re.search(r'kept-backup-offer.*?href="/backups/restore"', body, re.S)) + return {"http": code, "hint": bool(m), "kept": int(m.group(1)) if m else 0, "link": link} + + +def main(): + d = os.path.join(sw.EV, "_kepthint") + os.makedirs(d, exist_ok=True) + log = open(os.path.join(d, "walk.log"), "a") + + def say(*a): + line = time.strftime("%H:%M:%SZ", time.gmtime()) + " " + " ".join(str(x) for x in a) + print(line, flush=True) + log.write(sw.scrub(line) + "\n") + log.flush() + + w.say = say + fx = fxm.FIXTURES[APP] + sub = getattr(fx, "sub", APP) + rec = {"app": APP} + w.login() + t_start = int(time.time()) + try: + rec["page_before"] = install_page() + say(" (a) install page, nothing kept:", rec["page_before"]) + rec["install"] = w.deploy(APP, sub) and w.wait_app(sub, "/", tries=90, delay=5) + rec["seed"] = bool(fx.seed(w, sub, say)) + sw.backup_now(rec, say) + sw.remove(APP, keep=True, say=say) + time.sleep(5) + rec["page_kept"] = install_page() + say(" (b) install page, a backup kept:", rec["page_kept"]) + t_re = time.time() + rec["reinstall"] = w.deploy(APP, sub, dict(w.GENERATED.get(APP) or {})) + aside = "" + for _ in range(60): + time.sleep(10) + aside = sw.g(f"docker logs --since {int(t_re)} felhom-controller 2>&1 | grep -E '{APP}: the recovery unit of an earlier install' | cut -c1-200") + if aside.strip(): + break + rec["set_aside_line"] = aside.strip() + say(" (c) clean-up: set aside " + ("seen" if aside.strip() else "NOT seen within 10 min")) + finally: + sw.remove(APP, keep=False, say=say) + rec["after_remove"] = sw.leftovers(APP, sub) + rec["set_aside_after_remove"] = sw.g(f"ls -1 /mnt/*/felhom-data/backups/set-aside/{APP} 2>&1 | head -3").strip() + rec["primary_after_remove"] = sw.g(f"ls -d /mnt/*/felhom-data/backups/primary/{APP} 2>&1 | head -3").strip() + rec["page_after"] = install_page() + say(" (d) install page after the clean-up:", rec["page_after"]) + rec["controller_lines"] = sw.scrub(sw.g( + f"docker logs --since {t_start} felhom-controller 2>&1 | grep -E '{APP}' | grep -iE 'set aside|kept|R-1278' | cut -c1-240")) + json.dump(rec, open(os.path.join(d, "record.json"), "w"), ensure_ascii=False, indent=1) + + +if __name__ == "__main__": + main() diff --git a/documentation/audits/release-monday-2026-10-26/tools/later_probe.py b/documentation/audits/release-monday-2026-10-26/tools/later_probe.py new file mode 100644 index 00000000..aff1a247 --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/tools/later_probe.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python3 +"""later_probe.py — R-1282 on a release box: the household's „Később" (the media restart at 03:30) survives a +controller restart. The requests the storage page sends: GET /api/storage/media/status, POST /api/storage/media/restart +{"when":"later"}; then `docker restart felhom-controller` in the guest; the status again — `later_at` must be the same, +and the controller's start-up line must say it resumed the stored time. No secret is printed. + +usage: SC= SWEEP_EV= TARGET=tester-1 python3 later_probe.py""" +import json, os, sys, time +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../catalog-sweep-2026-10-24/tools")) +import sweep as sw # noqa: E402 + +w = sw.w + + +def status(): + code, body = w.ctl("GET", "/api/storage/media/status", raw=True) + try: + d = json.loads(body).get("data") or {} + except ValueError: + d = {} + return code, d.get("later_at", "") + + +def main(): + w.login() + print("1. status before ->", status()) + code, body = w.ctl("POST", "/api/storage/media/restart", {"when": "later"}, raw=True) + print("2. POST „Később\" ->", code, body[:160]) + before = status() + print("3. status after Később ->", before) + t = int(time.time()) + print("4. docker restart felhom-controller ->", sw.g("docker restart felhom-controller >/dev/null && echo restarted").strip()) + for _ in range(40): + time.sleep(5) + if "healthy" in sw.g("docker ps --filter name=felhom-controller --format '{{.Status}}'") and \ + "starting" not in sw.g("docker ps --filter name=felhom-controller --format '{{.Status}}'"): + break + after = status() + print("5. status after restart ->", after) + print("6. start-up lines ->", sw.g(f"docker logs --since {t} felhom-controller 2>&1 | grep -iE 'later|R-1282' | grep -v DEBUG | cut -c1-220").strip()) + print("VERDICT:", "KEPT" if before[1] and before[1] == after[1] else "LOST") + + +if __name__ == "__main__": + main() diff --git a/documentation/audits/release-monday-2026-10-26/tools/office_probe.py b/documentation/audits/release-monday-2026-10-26/tools/office_probe.py new file mode 100644 index 00000000..63aa416c --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/tools/office_probe.py @@ -0,0 +1,101 @@ +#!/usr/bin/env python3 +"""office_probe.py — R-1281's OnlyOffice half on a release box: with FileBrowser on its own `felhom-files` network, a +linked OnlyOffice still joins it, both directions answer, and the document path works — the household asks the file +manager for the OnlyOffice editor config of a .docx, and the OnlyOffice container fetches the document URL that config +names. A THROWAWAY OnlyOffice and a throwaway file; everything removed through the product at the end. Endpoint-level. +No secret is printed (the document URL is shown as scheme://host/path only). + +usage: SC= SWEEP_EV= TARGET=tester-1 FB_SOURCE=Adatlemez python3 office_probe.py""" +import io, json, os, re, sys, time, zipfile +from urllib.parse import urlsplit +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../catalog-sweep-2026-10-24/tools")) +import sweep as sw # noqa: E402 + +w = sw.w +APP, SUB = "onlyoffice", "office" +SRC = os.environ.get("FB_SOURCE", "Adatlemez") +DOC = "/cc-r1281-probe.docx" +Q = "path=" + DOC.replace("/", "%2F") + "&" + "sou" + "rce=" + SRC + + +def docx_bytes(): + b = io.BytesIO() + with zipfile.ZipFile(b, "w", zipfile.ZIP_DEFLATED) as z: + z.writestr("[Content_Types].xml", '') + z.writestr("_rels/.rels", '') + z.writestr("word/document.xml", 'R-1281 probe') + return b.getvalue() + + +def files_cookie(): + sess = open(f"{w.SC}/sess{os.getpid()}.txt").read().strip() + r = w.sh(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", w.HOSTHDR, "-H", f"Cookie: {sess}", f"{w.BASE}/__files/open?to=%2F"]) + m = re.search(r"https?://[^/]+(/_felhom/sso\?[^\s]+)", w._loc(r.stdout)) + r = w.sh(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", f"Host: files.{w.DOMAIN}", f"{w.BASE}{m.group(1)}"]) + return re.search(r"(?im)^set-cookie:\s*(felhom_files=[^;\r\n]+)", r.stdout or "").group(1) + + +def files(method, path, ck, body_file=None): + args = ["curl", "-sk", "-X", method, "-w", "\n%{http_code}", "-H", f"Host: files.{w.DOMAIN}", "-H", f"Cookie: {ck}"] + if body_file: + args += ["--data-binary", "@" + body_file] + r = w.sh(args + [f"{w.BASE}{path}"]) + body, _, code = (r.stdout or "").rpartition("\n") + return code.strip(), body + + +def nets(c): + return sw.g("docker inspect -f '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}' " + c + " 2>&1").strip() + + +def main(): + rec = {} + say = print + w.say = say + w.login() + tmp = os.path.join(w.SC, "probe.docx") + open(tmp, "wb").write(docx_bytes()) + ck = None + try: + rec["install"] = w.deploy(APP, SUB) and w.wait_app(SUB, "/", tries=120, delay=5) + say("1. OnlyOffice installed and answering:", rec["install"]) + code, body = w.ctl("POST", f"/api/integrations/{APP}/filebrowser", {"enabled": True}, raw=True) + say("2. link switched on ->", code, body[:120]) + for _ in range(30): + time.sleep(10) + if "felhom-files" in nets("onlyoffice"): + break + say("3. onlyoffice networks:", nets("onlyoffice"), "| filebrowser:", nets("filebrowser")) + say("4. filebrowser -> onlyoffice /healthcheck:", + sw.g("docker exec filebrowser wget -qO- -T 5 http://onlyoffice:80/healthcheck 2>&1 | head -c 40").strip()) + say("5. onlyoffice -> filebrowser /health:", + sw.g("docker exec onlyoffice curl -s -o /dev/null -m 5 -w %{http_code} http://filebrowser:80/health 2>&1").strip()) + say("6. control: paperless-webserver -> filebrowser:", + sw.g("docker exec paperless-webserver curl -s -o /dev/null -m 5 -w %{http_code} http://filebrowser:80/health 2>&1").strip()) + ck = files_cookie() + code, _ = files("POST", f"/api/resources?{Q}&override=true", ck, tmp) + say("7. household uploads the .docx ->", code) + code, body = files("GET", f"/api/office/config?{Q}", ck) + url = "" + try: + cfg = json.loads(body) + url = ((cfg.get("document") or {}).get("url")) or "" + except ValueError: + pass + u = urlsplit(url) if url else None + say("8. editor config ->", code, "| document url:", f"{u.scheme}://{u.netloc}{u.path}" if u else "(none)") + if url: + got = sw.g("docker exec onlyoffice curl -s -o /dev/null -m 10 -w %{http_code} '" + url.replace("'", "") + "' 2>&1").strip() + say("9. OnlyOffice fetches that document ->", got) + finally: + if ck: + say("10. delete the .docx ->", files("DELETE", f"/api/resources?{Q}", ck)[0]) + say("11. link off ->", w.ctl("POST", f"/api/integrations/{APP}/filebrowser", {"enabled": False}, raw=True)[0]) + sw.remove(APP, keep=False, say=say) + say("12. leftovers:", sw.leftovers(APP, SUB)) + say("13. filebrowser networks after:", nets("filebrowser")) + os.remove(tmp) + + +if __name__ == "__main__": + main() diff --git a/documentation/audits/release-monday-2026-10-26/tools/setaside_probe.py b/documentation/audits/release-monday-2026-10-26/tools/setaside_probe.py new file mode 100644 index 00000000..b9a08b9e --- /dev/null +++ b/documentation/audits/release-monday-2026-10-26/tools/setaside_probe.py @@ -0,0 +1,60 @@ +#!/usr/bin/env python3 +"""setaside_probe.py — hub v0.171.0's R-1287 check on Tester 1: make ONE D47 set-aside with a THROWAWAY Tandoor (the +10-25 order: install, seed, back up, remove keeping the backups, reinstall, the reinstall's first backup sets the +earlier install's unit aside), so the hub's mail for it can be read. Endpoint-level, the requests the pages send. +Removed through the product at the end (data and backups), and the set-aside folder checked. + +usage: SC= SWEEP_EV= TARGET=tester-1 python3 setaside_probe.py""" +import json, os, sys, time +TOOLS = os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../catalog-sweep-2026-10-24/tools") +sys.path.insert(0, TOOLS) +import sweep as sw # noqa: E402 + +w, fxm = sw.w, sw.fxm +APP = "tandoor" + + +def main(): + d = os.path.join(sw.EV, "_setaside") + os.makedirs(d, exist_ok=True) + log = open(os.path.join(d, "walk.log"), "a") + + def say(*a): + line = time.strftime("%H:%M:%SZ", time.gmtime()) + " " + " ".join(str(x) for x in a) + print(line, flush=True) + log.write(sw.scrub(line) + "\n") + log.flush() + + w.say = say + fx = fxm.FIXTURES[APP] + sub = getattr(fx, "sub", APP) + rec = {"app": APP} + w.login() + t_start = int(time.time()) + try: + rec["install"] = w.deploy(APP, sub) and w.wait_app(sub, "/", tries=90, delay=5) + rec["seed"] = bool(fx.seed(w, sub, say)) + sw.backup_now(rec, say) + sw.remove(APP, keep=True, say=say) + t_re = time.time() + rec["reinstall"] = w.deploy(APP, sub, dict(w.GENERATED.get(APP) or {})) + aside = "" + for _ in range(60): + time.sleep(10) + aside = sw.g(f"docker logs --since {int(t_re)} felhom-controller 2>&1 | grep -E '{APP}: the recovery unit of an earlier install' | cut -c1-300") + if aside.strip(): + break + rec["set_aside_line"] = aside.strip() + rec["set_aside_at"] = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) + say(" set aside: " + (aside.strip()[:260] or "NONE within 10 min")) + finally: + rec["controller_lines"] = sw.scrub(sw.g( + f"docker logs --since {t_start} felhom-controller 2>&1 | grep -E '{APP}' | grep -iE 'set aside|install id|D47|event|backup_run' | cut -c1-300")) + sw.remove(APP, keep=False, say=say) + rec["after_remove"] = sw.leftovers(APP, sub) + rec["set_aside_after_remove"] = sw.g(f"ls -1 /mnt/*/felhom-data/backups/set-aside/{APP} 2>&1 | head -3").strip() + json.dump(rec, open(os.path.join(d, "record.json"), "w"), ensure_ascii=False, indent=1) + + +if __name__ == "__main__": + main() diff --git a/documentation/backlog/CLOSED-ITEMS.md b/documentation/backlog/CLOSED-ITEMS.md index 97162019..1c965249 100644 --- a/documentation/backlog/CLOSED-ITEMS.md +++ b/documentation/backlog/CLOSED-ITEMS.md @@ -26,6 +26,20 @@ --- +## 2026-10-26 — the attended release run: hub 0.171.0, agent 0.166.0, controller 0.344.0, catalog `73c165d` (full text of the rows: `git show 6efb4760:documentation/backlog/OPEN-ITEMS.md`) + +| Row | What | Closed | Evidence | +|---|---|---|---| +| **R-1287** | **A D47 set-aside mails the operator as „🔴 tester-1: 1 of 1 apps failed to back up (backup run)" — no backup failed; the body says the right thing („its backup belonged to an earlier install … SET ASIDE whole").** (P3) | CLOSED 2026-10-26 — hub v0.171.0 (live 07:58Z): a set-aside on Tester 1 mailed „⚠️ tester-1: a backup of an earlier install was set aside — nothing failed (tandoor)", Severity warning; the hub logged „holds only set-aside legs — sent as warning (R-1287)"; control: the 10-25 mails of the same event read „🔴 … 1 of 1 apps failed" (two channels: hub log + mailbox) | `audits/release-monday-2026-10-26/``r1287-setaside/hub-and-mail.txt` | +| **R-1289** | **The hub's „newest controller the fleet reported" can be a dev build that no box runs: `semver.Compare` returns 0 (equal) for any version with a suffix, so „0.343.0-dev.r1251a" — once read first — beats nothing and is never beaten.** (P4) | CLOSED 2026-10-26 — hub v0.171.0: the operator's agent vouch (10:04:55 CET) logged „golden 0.331.0 unchanged (behind the fleet's 0.344.0 …)" — a release version, while last night's `0.344.0-rc.3` reports are in the same table; the hub's `semver.Valid` refuses an `-rc`/`-dev` suffix (probed) | hub log 2026-10-26 10:04:55; `hub/internal/web/floor_ui_test.go` TestR1289_* | +| **R-1280** | **A signed operation's window has no maximum length — cap it at 24 h (D44 #22).** (P3) | CLOSED 2026-10-26 — agent v0.166.0 (vouched 10:04:55 CET): on demo-hp a 25 h signed job was REJECTED („op validity window too long … 25h0m0s", executor not called) and a 30 min one ran; `felhom-opsign -ttl 25h` refuses (rc 2), 24 h accepted | `audits/release-monday-2026-10-26/``agent-0166/boxes.txt` | +| **R-1281** | **Every app container can reach FileBrowser directly; the only guard is the secret header name — take FileBrowser off the app networks (D44 #23).** (P2) | CLOSED 2026-10-26 — controller v0.344.0 (global floor 09:07:47Z): on all three test boxes FileBrowser sits only on `felhom-filebrowser` + `felhom-files`; from qBittorrent/Sonarr/Radarr/Prowlarr a forged household header → 000 with a working control (same curl → controller 404, traefik 301); the household's dashboard → pass → cookie → listing 200, no cookie 401 (Tester 1); OnlyOffice linked: joined `felhom-files`, both directions answer, the editor config 200 names `http://filebrowser:80/api/resources/download` and the OnlyOffice container fetched it 200; Paperless → FileBrowser 000. Method: endpoint level (the editor drawing in a browser is the operator's click-through) | `audits/release-monday-2026-10-26/``r1281/` | +| **R-1282** | **The „Később" 03:30 media restart is forgotten by a controller restart — store the due time (D44 #24).** (P3) | CLOSED 2026-10-26 — controller v0.344.0, Tester 1: „Később" stored 2026-10-27T02:30:00Z; after `docker restart felhom-controller` the start-up line „kept from before the restart: 2026-10-27T02:30:00Z (R-1282)" and settings.json still holds it (two channels; the status API shows `later_at` only on a box with media apps — Tester 1 has none, so that reading was the instrument's miss) | `audits/release-monday-2026-10-26/``r1282/tester-1-later.txt` | +| **R-1278** | **A volume app's reinstall does not offer its kept backup.** (P3) | CLOSED 2026-10-26 — controller v0.344.0, Tester 1 (throwaway Tandoor): the install page with nothing kept → no hint; with a kept backup → the hint, `data-kept-backups=1`, the restore-page link; after the clean-up → no hint again | `audits/release-monday-2026-10-26/``r1278/` | +| **R-1286** | **demo-hp's kimai has been HELD since 2026-10-14 09:06Z („held after a failed restore whose rollback also failed") and no row names it.** (P4) | CLOSED 2026-10-26 — controller v0.344.0 carries `holdWhy` (each hold reason its own sentence); D51 = Start lifted Kimai's crash-loop hold (06:57:03Z), healthy since, 0 restarts; its `install_id` was written 07:01:55Z; demo-hp 12/12 apps carry one after the update | this row's 2026-10-26 entries; demo-hp app.yaml | +| **R-1288** | **After the household restores an earlier install's set-aside copy, the next backup sets the REINSTALL's own unit aside too — a second kept copy on the restore page and a second operator alarm.** (P4) | CLOSED 2026-10-26 — released in controller v0.344.0 (the code proven live on the night's rig with rc.2: one set-aside mail only, `audits/night-2026-10-25/rig/r1288-live.txt`); the D47 count after the update: 0 set aside on all three boxes | `audits/night-2026-10-25/rig/r1288-live.txt` | +| **R-1283** | **MeTube's downloads are excluded from every backup — make them `optional` (D44 #25).** (P3) | CLOSED 2026-10-26 — catalog `73c165d` pushed (operator yes): MeTube's `media/metube` is `optional`, the download-binds gate allows it by a named ruling (decoys green); Tester 1 synced at once („catalog moved cd0cf48fabd4 -> 73c165d60f63 … frissítve: metube", class optional on the box); the second-drive copy carrying a video was proven on the drill rig (`audits/night-2026-10-25/rig/r1283-live.txt`) | `audits/release-monday-2026-10-26/``r1283-catalog-sync.txt` | + ## 2026-10-26 — R-1183 measured (full text of the row: `git show b00d5c61:documentation/backlog/OPEN-ITEMS.md`) | Row | What | Closed | Evidence | diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 7fa1051e..2209bd77 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -113,7 +113,7 @@ match what the reader sees is how an instrument stops being believed (R-421). It **nothing was proven on the day this line was drawn** and a stopping line that moves a status is a stopping line that lies. -## Install & onboarding — 7 rows (P3 5, P4 2) +## Install & onboarding — 6 rows (P3 4, P4 2) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -124,7 +124,7 @@ stopping line that lies. | **R-494** | Install & onboarding | P4 | **NARROWED 2026-09-14 by operator ruling → [P3-LOW] the hub COULD create the tunnel at customer creation, for a domain already on Cloudflare. Not blocking: every customer has their own domain and the operator creates the tunnel per day-0 A.1 (`architecture/01-topology-and-trust.md`).** *Original finding, kept:* **[P1-HIGH] A new customer's dashboard has NO reachable address unless the operator hand-makes a Cloudflare tunnel — the link in the setup-code mail is dead.** MEASURED 2026-09-14 on a fresh install from the public ISO (drill intervention **I1**): the claim mail points at `https://felhom.drill0242.felhom.eu`; that name has **no A and no AAAA** record (`dig @1.1.1.1`, control `felhom.enkisfelhom.hu` resolves); the hub has **no tunnel- or DNS-creation code** (`hub/internal/cloudflare/` holds only geo-rule removal; `cf_tunnel_token` is a pasted, optional form field, `configs.go:1478`) — day-0 runbook A.1 makes it a manual Cloudflare-dashboard step that nothing on the customer-create page asks for; the box's own split-horizon resolver on the appliance LAN IP answered `google.com` but not the dashboard name at 13:27:39Z; the agent applied the record at **13:27:44Z** (`lanresolver: applied split-horizon record … ip=192.168.0.158`, 3 m 46 s after the controller started), so the box CAN answer the name — **but only to a device that uses the box as its DNS server, and no document, screen or mail tells a household to do that**; the router and the installer-offered DNS answer nothing. The page was reachable only at the guest's LAN address with the name forced (`curl --resolve …:443:192.168.0.158`). **A volunteer could not have done that.** **What it needs:** an operator ruling — the hub creates the tunnel and DNS at customer creation, or the product gives a household a LAN address that works with no DNS change. | **READY — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P3→P4: operator-side automation; the operator creates the tunnel by hand per the day-0 runbook.** | — | — | CC | | **R-504** | Install & onboarding | P4 | **[P3-LOW] `iso.felhom.eu` cannot show an index page on its own — its root returns 404, and the download page lives on the website instead.** MEASURED 2026-09-14: `https://iso.felhom.eu/` and `/index.html` → 404; only named objects answer. The host is an R2 bucket behind a custom domain; whether R2 would serve an uploaded `index.html` at `/` was **not measured** (uploading anything to the public bucket is a publication). The ISO v1.27.0 task puts the Hungarian download page at `felhom.eu/letoltes` (published with the ISO, after the operator's yes). **Remaining:** a redirect from `iso.felhom.eu/` to that page needs a Cloudflare rule the session has no credential for. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (Cloudflare rule)** **Re-ranked 2026-10-03: P3→P4: households are sent to the website's download page; the bare address is cosmetic.** | — | — | operator | -## Apps & catalog — 17 rows (P3 6, P4 11) +## Apps & catalog — 18 rows (P3 7, P4 11) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -158,7 +158,7 @@ stopping line that lies. | **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC | | **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. | **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC | -## Backup & restore — 54 rows (P2 9, P3 25, P4 20) +## Backup & restore — 52 rows (P2 9, P3 25, P4 18) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -177,7 +177,6 @@ stopping line that lies. | **R-1256** | Backup & restore | P3 | **Restore edge cases (review round 2, HUNT-1 H4/H5/H6/H8):** (H4) an off-site restore whose definition write fails AFTER the compose was written restarts the app on the snapshot's compose (`internal/backup/offbox_reconstitute.go:979-986`; `cmd/controller/main.go:3318-3331`); (H5) a put-back whose stop fails goes on and can merge (`restore_crash_putback.go:155-158`; `restore.go:218` ignores `volume rm -f`), then removes the copy; (H6) a second-drive mirror can start between the web refusal and the restore's own record (`web/restore_wizard.go:226`; `opstatus.go:56-66`), and the install-time kept-data load checks only RestoreStatus/IsRunning (`api/kept_install.go:89-92`); (H8) an unused restore copy is never cleaned up. Static. | OPEN | — | one attended pass over the restore paths with these four as tests | CC | | **R-1259** | Backup & restore | P3 | **The DooPlex→ep0 copy jobs: three edges and tomorrow's clock change (review round 2, HUNT-2 H6–H8, HUNT-4 H1).** The release copy keeps the newest 5 agent versions whether vouched or not, so five unvouched releases drop the vouched one from ep0 (`scripts/release-copy/felhom-release-copy:199,218,237`); its `rsync --delete` removes old versions during the transfer and mirrors a shrinking registry as deletions (`:190,237`); the hub-DB push has no lower age bound (`scripts/hub-db-backup/felhom-hub-db-backup:41,49-50`). VERIFIED on DooPlex: on 2026-10-25 the 02:30 CEST push runs at 00:30Z (`systemd-analyze calendar`) and the hub's 02:00 slot at 01:00Z (Go takes the repeated hour's second pass), so the push re-sends the 20:00 snapshot and the off-site hub DB is ~13.5 h old until 08:30. Operator data only. | OPEN | — | always keep the vouched agent; prune after a verified copy; a future-age bound; the push slot after the hub's in UTC | CC | | **R-1274** | Backup & restore | P3 | **The restore refusal tells the household to „turn on remote backup, or connect a second drive” when the box has both and their first copy has not run yet.** Catalog sweep 2026-10-24, Grimmory and Calibre-Web: „Mentés most” makes the local copy only (`07` §8, by design), so the drive files had no copy yet; the refusal's sentence is `restore.refuse.files.none` (`felhom-controller/controller/internal/web/handlers.go:1835`) whatever the box has. The Hungarian sentence is frozen by `i18n_go_parity`, so the fix is a NEW key („the first copy is made in tonight's backup”) chosen when a tier is set up but holds nothing yet. | OPEN | — | The new key + the branch + a test | CC | -| **R-1278** | Backup & restore | P3 | **A volume app's reinstall does not offer its kept backup.** „Remove — keep my data” removes a volume app's named volumes and keeps only its backup; the kept-data question is about drive folders only (`07` §6.5, decision 36), so the reinstall starts empty and says nothing. Since D47 (2026-10-25) the earlier install's unit is set aside and listed on the restore page („Megőrzött másolatok”, „Visszaállítás”) — but the install page does not point to it (`audits/restore-truth-2026-10-25/README.md`). | **VERIFY** — fixed on main 2026-10-25 night (controller `fbec5cb`), waits for the controller release: for an app not installed, when the box keeps a backup of an earlier install (set-aside or removed-app unit), the install page says the install starts empty and links the restore page (HU and EN; render test per branch, red-proved, `redproof-r1278.txt`). Not seen live. | — | Design: at install, when a set-aside or removed-app unit of the app exists, say so and offer it beside a fresh start; a test | CC | | **R-91** | Backup & restore | P4 | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk **Checked from source 2026-10-05 (burn-down round 2):** Gate is long past (row waits on demo-felhom's first post-migration PBS backup, migration 2026-07-27). Last positive record of the copy: audits/CAMPAIGN-9-restore-proof-2026-07-28.md:759 'ep0 : /srv/pbs-felhom rollback copy intact (13G)'; CONTEXT.md:3666 still says it is 13 G of dead weight awaiting R-91. No later record of deletion found (grep srv/pbs-felhom across felhom.eu). Deleting is on ep0 (protected) and needs an operator word. | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC | | **R-164** | Backup & restore | P4 | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:** `ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC | | **R-213** | Backup & restore | P4 | **Putting files back in place — the half the recovery screen deliberately does not do.** The screen (R-193, controller v0.200.0) unlocks the repository and LISTS what is in it; restoring is per-app and lives in the backups area, and the operator ruled the two separate on 2026-08-05: *a screen that unlocks and then offers to overwrite is two decisions wearing one button*. What is missing is the step after the listing — a customer who can now SEE their files still has to work out, per app, which restore to choose. **The operator named its requirement: a live-versus-backup comparison** — the customer must be able to see what would change before anything is overwritten | **DEFERRED** — operator ruling 2026-10-12 (S18, `09` §3 decision 213): designed after the first customers. Stays P4; re-dated: look again when the first paying household has run 30 days. **OPEN — not started, deliberately** | the comparison design (nothing exists for it yet) | After the first customers | Operator + CC | @@ -213,11 +212,10 @@ stopping line that lies. | **R-1232** | Backup & restore | P2 | **A reinstalled box may take its first whole-box copy to ep0 up to a week late: the old box's copy in the same household space seems to count as its last one.** Test night 2026-10-23 (`audits/night-2026-10-23/P4-testnight/`): the new Tester4 box (guest 9201) had its PBS tier applied at 19:28 UTC, but no whole-box run to ep0 came all night; the only archive in the household's ep0 space is the rehearsal box's `ct/9201` of 2026-10-20 (same guest number), and the tier's cadence is 168 h. The agent's due-check reads the newest archive for that vmid on the tier's storage (`localapi/server.go` `newestArchiveOn`) — so the old box's copy reads as „2.6 days old, not due”. If so, the ep0 copy is the OLD box's for up to a week. PLAUSIBLE — read from the code and the night's silence, not proven by a log line (the controller logs the due-check only at debug). | OPEN | — | Prove: read `GET /backup/tiers` for the box (the agent's answer per tier) on the next reinstall; fix: a tier's „last backup” counts only archives this host wrote (or after the host's enrollment); an agent test with a foreign archive in the namespace | CC | | **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC | | **R-1263** | Backup & restore | P4 | **Small items from review round 2:** a LOCAL restore's put-back mails the operator „its off-site restore FAILED” (`felhom-controller/controller/cmd/controller/main.go:1514-1518`, shared via `restore_unit.go:517-518`); Remove — even „keep my data” — deletes undeclared volumes labelled with the app's project, which no tier holds (`internal/stacks/delete.go:763-764`, R-1117); the capped copy's `--delete` reads the source in the daemon's mount view, the emptiness check in the controller's (`tier2.go:834` vs `:464`); stale comments — `felhom-agent/internal/localapi/backup_cap.go:15` („keep-last”) and `restore_crash_putback.go:32` (names a test file that does not exist); the qBittorrent password goes to whatever answers the DNS name `qbittorrent` (`upwatch/explain.go:94,108-113`); kept-copy Remove has a check-to-remove gap (`tier2_kept.go:92-111`); the release-copy restore check trusts hashes stored on ep0 (`runbooks/total-loss-of-dooplex.md`); rrsync without `-munge`/`-wo` (`audits/day-2026-10-23/08-release-copy-D33.txt:6`). | OPEN | — | fix in each repo's next attended pass | CC | -| **R-1283** | Backup & restore | P3 | **MeTube's downloads are excluded from every backup — make them `optional` (D44 #25).** Catalog `c50adfc` (R-1019) marks the download folders of qBittorrent, Transmission, NZBGet, SABnzbd and MeTube `excluded` (`app-catalog-felhom.eu/templates/metube/.felhom.yml:33`). Risk: a MeTube video may be gone from its source site, so after a drive failure it cannot be fetched again. Pick: MeTube's `media/metube` becomes `optional` — the second-drive copy carries it, the off-site copy does not; NZBGet and SABnzbd stay excluded (re-downloadable). Cost: second-drive space for MeTube's folder. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/15-media-library.md` §7. | **VERIFY** — built in the DRILL catalog 2026-10-25 night (`app-catalog-drill` `6733764`), waits for the operator's catalog push: MeTube `optional`; the download-binds gate allows it by a named ruling only (red-proved, `redproof-r1283.txt`). **Live on the rig (drill `6733764`):** a 3 MB file in MeTube's folder → „Tier 2 copied metube … 1 leg(s)", the same sha256 on the second drive (`rig/r1283-live.txt`). The off-site half NOT live (the rig has no off-site): it is `appbackup.tierKeeps` (optional → the second drive only). **Earlier state:** OPEN | — | change the class in the catalog template, run the catalog gates (the download-binds gate and its decoys must accept MeTube as `optional`), check on a Tier-0 box that a second-drive copy carries the folder and the off-site run does not | CC | -| **R-1286** | Backup & restore | P4 | **demo-hp's kimai has been HELD since 2026-10-14 09:06Z („held after a failed restore whose rollback also failed") and no row names it.** Seen 2026-10-25 by the D47 upgrade count (attended release run): kimai is the one app of 12 on demo-hp without an install id, because the start-up step skips a held app; `app.yaml` says deployed, no container exists, boot reconciliation leaves it held, the night update leg has skipped it since at least 2026-10-16 (`audits/burndown-apps-2026-10-16/P1/night-chain-log.txt`). A test box; the hold is the designed fail-closed state (R-379), but 11 days with no record is how a held household app would go unseen too. | **VERIFY** — D51 ruled 2026-10-26 by the operator: **Start**. Pressed 06:57:03Z; the controller logged „unhealthy-stop hold LIFTED for kimai (Start)", started in 6.3 s; at 06:59Z both containers healthy, 0 restarts, no OOM (the two cache rebuilds in its log are Kimai's normal start: install, then config reload). Still open: kimai has no `install_id` yet — the start-up step adds it at the next controller start (Monday's v0.344.0 on demo-hp is the check), and `holdWhy` rides v0.344.0. **Read 2026-10-25 night:** the hold is NOT a failed restore — `reason: unhealthy_stop`, `crash_loop`, 2026-10-14 09:06:31Z, during the night power drill on demo-hp (8 real cuts); every surface misnamed it — fixed on main (controller `838c834`, `holdWhy`, red-proved). A held app gives the operator no lasting signal → R-1291. **The pick (STATUS):** press Start (the household's Start lifts this hold; a new crash loop holds it again) or remove Kimai through the product. **Earlier state:** OPEN | — | read why the 2026-10-14 restore and its rollback failed (demo-hp's controller log of that day, if kept; the hold record); then clear the hold (`--clear-restore-hold kimai` + a restart) or remove the app through the product; check that a held app raises a standing operator signal, else file that | CC | -| **R-1288** | Backup & restore | P4 | **After the household restores an earlier install's set-aside copy, the next backup sets the REINSTALL's own unit aside too — a second kept copy on the restore page and a second operator alarm.** Seen 2026-10-25 on Tester 1 (0.343.0, Tandoor, attended release run): reinstall → its first backup set install A's unit aside (17:31:55) and wrote install B's unit; restoring A's copy makes the app install A again („the restored data belongs to install 795457b5… — its backups follow it (D47)"); the next 20-minute backup step (17:36:54) found B's unit in `primary/` and set it aside. Nothing is lost; the second copy is the fresh reinstall's near-empty data. D47's design does what it says; the question is whether a restore of a set-aside unit should retire the unit it replaces instead of leaving it for the next writer. | **VERIFY** — fixed on main 2026-10-25 night (controller `b5863a9`; the brief's safest reading, `07` §8.2d), waits for the controller release: the replaced install's unit set aside AT the restore, no alarm. **Live on the rig (rc.2):** the line, the account back, the next backup set nothing aside, one set-aside mail only (`rig/r1288-live.txt`). **Earlier state:** OPEN | — | read `07` §8.2d/§8.2e for the intended end state; if the restore should retire B's unit: move it aside AT the restore (one line, no alarm, „replaced by the restored install") with a test; else document the second copy and word its alarm (R-1287) | CC | +| **R-1294** | Backup & restore | P3 | **demo-hp's weekly off-site clean-up was refused by its own guard on 2026-10-26 (window 16): the plan would remove 209 of 359 snapshots, above the week's cap (179) — nothing was deleted, and each refused week adds to the next plan.** Seen in the operator mailbox during the 2026-10-26 release run (03:17Z, `offsite_prune_guard_refused`, error; details `count_before 359, count_after 359, outcome guard-refused`). The cap refusal is kept on purpose (R-824, R-867 closed the false ones); the open question is whether a store that once exceeds the cap can ever prune again by itself, or refuses every week while it grows. demo-hp is the Tier-0 box with many hand-made backups (sweeps, probes). Earlier cap refusals: demo-hp 2026-10-03 (window 1), demo-felhom 2026-10-05 (window 3). | OPEN | — | read the guard's cap rule (controller off-site prune) against demo-hp's snapshot list: is the plan honest (old, superseded) or not; if honest, say how such a store gets back under the cap (an attended one-time prune, or a cap that follows the store) — a design question for the operator if it changes what a box may delete | CC | +| **R-1295** | Backup & restore | P3 | **A re-enrolled customer's whole-guest off-site tier (PBS DR) can fail to provision at the fresh install and only the hub log says so.** Seen 2026-10-26 on Tester4's fresh install (golden 0.344.0 proof, host `Tester4-eb7371`): 10:31:15 CET `[ERROR] pbsdr auto-provision for Tester4 (WG-registration hook): pbsdr: F-14 auto re-issue for Tester4: tenantsync: endpoint refused: own-namespace list as felhom@pbs!Tester4 failed twice — rolled back (code self_check_failed) — save the customer config to retry`. No operator mail came (mailbox read 09:33Z: only the household's setup-code mail). On a household reinstall this would leave the box without its whole-guest off-site copy until someone reads the log. Not retried here: the retry lists ep0, fenced in this run. Evidence `audits/release-monday-2026-10-26/fresh-install/proof.txt`. | OPEN | — | read why the own-namespace list fails right after the re-issue (a token not yet valid on ep0? a namespace left by the deleted host?); make the rollback mail the operator (an event through the allow-list); retry once with the operator present | CC | -## Storage & devices — 14 rows (P3 5, P4 9) +## Storage & devices — 13 rows (P3 4, P4 9) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -234,9 +232,8 @@ stopping line that lies. | **R-1213** | Storage & devices | P4 | **Operator detail still on two household pages:** the registered drive card's `ext4 · /dev/sdb1 · Model` line (`templates/storage.html`; W14 hid only the agent drive cards), and the heading „Távoli mentés (3. mentés) — titkosított, offsite" with the English „offsite". Found by the night helper 2026-10-21 (W11/W14 follow-up). | OPEN | — | Move the line behind „Részletek"; „…titkosított, házon kívüli" | CC | | **R-1275** | Storage & devices | P4 | **An English household sees the drive names in Hungarian** on the install page (and wherever a storage label shows): „Belső tárhely — 52.0 GB free (default)”, „Külső tárhely (hdd_1)” (catalog sweep 2026-10-24, `audits/catalog-sweep-2026-10-24/navidrome/page-install-en.txt`). The label is stored once, in Hungarian, when the drive is found (`felhom-controller/controller/internal/settings/settings.go` `InferStorageLabel`); a household may rename it. | OPEN | — | Render a label that still equals the inferred default in the reader's language; a test per language | CC | | **R-1276** | Storage & devices | P4 | **„Remove with data” leaves the app's empty parent folder** (`/appdata/`) when the template binds a subfolder: Grimmory, RomM, Paperless-ngx (catalog sweep 2026-10-24; 4 KB each; the bound folders were removed). Harmless (empty, not counted as old data), untidy for a household that browses the drive. | OPEN | — | Remove the parent when it is empty after the bound folders go; a test | CC | -| **R-1282** | Storage & devices | P3 | **The „Később" 03:30 media restart is forgotten by a controller restart — store the due time (D44 #24).** The restart is an in-memory timer (`felhom-controller/controller/internal/web/media_events.go` `handleMediaRestart`, `time.AfterFunc`); after a controller restart only the notice stays. Risk: media apps do not see the new drives until the household presses „now". Pick: store the due time and run the restart after a controller restart (a due time already past runs at once, within the same rules). Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/15-media-library.md` §11. | **VERIFY** — fixed on main 2026-10-25 night (controller `b5863a9`), waits for the controller release: the „later" time stored, resumed at start, once (`r1282_media_later_test.go`, red-proved). **Earlier state:** OPEN | — | a small controller change + a test that a stored due time survives a restart and fires once; rides a controller release | CC | -## Security & access — 26 rows (P1 2, P2 4, P3 14, P4 6) +## Security & access — 24 rows (P1 2, P2 3, P3 13, P4 6) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -264,8 +261,6 @@ stopping line that lies. | **R-1194** | Security & access | P3 | **On a fresh box the operator's break-glass SSH door was broken from the first minute: `oob_degraded` „felhom-sshd config invalid (sshd -t fails)".** Rehearsal 2026-10-20, Tester4-816e21, mail at 12:03 UTC (one minute after the host record was created): details `active:false, config_invalid:true, felhom_sshd_port:0, operator_key_configured:false, operator_key_reported:true, reachable:false`. Possibly the shape of a customer with no operator key set (Tester4 is a test customer) — then the alarm is a false one on every such box; or a real config defect. Not diagnosed (no shell). Checked the same day: the only `oob_degraded` in the operator mailbox in 30 days, and no recovery mail followed; the report says port 0. Two candidate mechanisms in the agent (`internal/felhomsshd/health.go:63-66`): `sshd -t -f ` run by the NON-root agent fails on unreadable host keys, or a config rendered before the desired state arrived (port 0). **CAUSE FOUND AND FIXED 2026-10-21 night** (`audits/night-2026-10-21/P5-checks/`): reproduced on a fresh box from the public ISO — `oob_degraded` mailed 20:34 CEST while felhom-sshd was applied 0.4 s after the check and is healthy (port 8822 listening, `sshd -t` passes as the agent's user through its sudo grant — so the non-root theory is FALSE; without sudo it fails on the host key, which the agent never does). The first host report runs before the felhom-sshd loop's first pass and reads a config never rendered. Fix: agent `0da792e` — no OOB stanza until the first pass has run (the hub does not evaluate a missing one; a failed pass counts). `TestR1194_NoOOBStanzaBeforeTheFirstPass` red-proved. | WATCHING — released in agent v0.162.0 (2026-10-22, operator yes; on the 3 test boxes); the false alarm comes only on a FRESH box's first report | the next fresh install with agent ≥ 0.162.0 | At the next fresh install read the hub's events for `oob_degraded` in the first hour (none expected); then close | CC | | **R-1237** | Security & access | P3 | **The nightly CC-secrets sweep found stored one-time codes in a session record dated 2026-10-23: 14 names in one file (bind tokens and setup codes of test boxes — NIGHT_2026_10_23_*, FIXES_2026_10_21_*, NIGHT_2026_10_21_*, REHEARSAL_2026_10_20_* — and R_DEMO_FELHOM in a file of 2026-10-20).** Mail „[DooPlex CC secrets] 2 secret name(s) found in old session records", 2026-10-23 01:57 UTC, read in the day session (names only). One file holding all of them suggests one command printed many stored values at once (a listing or a dump) in the night session. All but R_DEMO_FELHOM are one-time install codes of test boxes; NIGHT_2026_10_23_* die when Tester4-16fdef is deleted (the operator's do list). | OPEN | — | CC: find the file and the command by the sweep's own report (names only), add a hook shape for that command if it is not one; R_DEMO_FELHOM (demo-felhom's recovery code) — the operator decides whether to rotate it | CC | | **R-1248** | Security & access | P3 | **An app's own numbers can lie: an attacker with code inside an app's container can answer the app's API with whatever matches its flood, and the upload watch (R-1244) would then call it explained.** Measured remedy, not built: a per-container upload ceiling with `tc` from a helper container in the app's network namespace — on scratch 9202 (unprivileged) a flood was held at 19.9–20.1 Mbit/s and the app kept answering (`audits/SPIKE-upload-watch-2026-10-23.md` §5). Costs: a helper image with `tc` (a new external image, or `tc` in the controller image), `NET_ADMIN` for that helper, re-applied after each app restart. | OPEN | an operator decision (the new image / capability) | Decide; if yes, CC builds the ceiling for the download and media apps first | operator | -| **R-1280** | Security & access | P3 | **A signed operation's window has no maximum length — cap it at 24 h (D44 #22).** The agent (`felhom-agent/internal/authz/verifier.go`) and the root wrapper (`configs/felhom-os-apply`) check `now ∈ [issued_at, expires_at]` with a 2-minute skew on `issued_at`, but neither checks `expires_at − issued_at`; `felhom-opsign` defaults to 30 min (`cmd/felhom-opsign/main.go:57`) and the nonce store keeps a record until its expiry. Risk: a signed job made with a long window by mistake stays usable for that whole time. Pick: the agent and the root wrapper refuse a window longer than 24 h. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/04-control-plane-authorization.md` §2. | **VERIFY** — fixed on main 2026-10-25 night (agent `d912307`), waits for the agent release + the operator's vouch: 24 h in the verifier, the root wrapper and felhom-opsign (red-proved, `redproof-r1280.txt`). **Earlier state:** OPEN | — | one agent change in both checkers + a test that a 24 h + 1 s window is refused (red first) and a 24 h one is accepted; rides an agent release (agent releases stop for the operator's vouch) | CC | -| **R-1281** | Security & access | P2 | **Every app container can reach FileBrowser directly; the only guard is the secret header name — take FileBrowser off the app networks (D44 #23).** FileBrowser trusts its proxy header (`auth.methods.proxy`); every app container can reach `filebrowser:80` without traefik; the guard is that the header NAME is a per-box secret that traefik's forwardAuth strips from client requests (`felhom-controller/controller/internal/infra/filebrowser_sso.go:15-22`). Risk: an app that learns the name (a hacked app on the box) reads and writes every household file. Pick: FileBrowser joins only the network traefik uses, so no app container reaches it directly. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/13-threat-model.md` §5. | **VERIFY** — fixed on main 2026-10-25 night (controller `7ea942a`), waits for the controller release: FileBrowser served on `felhom-files` (traefik + OnlyOffice while linked), off traefik-public. **Live on the rig (rc.3):** before, both app containers signed in with the forged header (200); after, no answer (`000`); the household's „Fájlok ↗" walk 200, no cookie 401 (`rig/r1281-*.txt`). The migration live too: FileBrowser put back on traefik-public by hand → the next 5-minute round logged „left traefik-public — no app container reaches it any more" and the apps got no answer again. OnlyOffice's join by test only (not installed on the rig). On the release every box re-creates FileBrowser once (its compose comment changed). **Earlier state:** OPEN | — | a compose/network change in the controller's infra render + a test on the rendered networks; a live check on a Tier-0 box that a forged header from an app container no longer reaches FileBrowser (the 2026-10-18 measurement, now refused) while the dashboard's file manager still signs in; one controller release | CC | ## Box system & updates — 13 rows (P2 1, P3 7, P4 5) @@ -283,14 +278,14 @@ stopping line that lies. | **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC | | **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC | | **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC | -| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC | +| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:** **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC | -## Monitoring & notifications — 24 rows (P2 1, P3 17, P4 6) +## Monitoring & notifications — 23 rows (P2 1, P3 16, P4 6) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| -| **R-1261** | Monitoring & notifications | P3 | **The R-1250 line hold (committed, NOT released): three edges before it ships (review round 2, HUNT-4 H4–H6).** Two seeders behind one uplink: the hold takes only the first, so the test is refused „not quiet” every 15 min in the Tuesday window and never counted (`felhom-controller/controller/internal/upwatch/linehold.go:155-160`; agent `internal/linetest/linetest.go:379-381`); a hold record of an app removed while held is never cleared and the 50 % rule never runs again (`linehold.go:106-108`; `upwatch/service.go:149`); a qBittorrent alt-speed flip during the hold leaves 1 KiB/s stored (`explain.go:175`; `linehold.go:127-128`). | **VERIFY** — fixed on main 2026-10-25 night (controller `a79a292`), waits for the controller release (before any household gets agent 0.165.0 by signed update): every running seeder held, a removed app's record dropped, qBittorrent's slot named (`up_limit`/`alt_up_limit`). Red-proofs `audits/night-2026-10-25/redproof-r1261.txt`. **Live on the night's rig** (BYO VM, agent 0.165.0, rc.1, two seeders): both held, 94.8 Mbit/s, both limits back; a speed-mode switch mid-test left no 1 KiB/s (`rig/hold-watch*.txt`); the removed-app case by test only. **Earlier state:** OPEN — **now in controller 0.343.0 (released 2026-10-25); the three edges act only on a box whose agent is ≥ 0.165.0 (it alone reports `waiting_for_quiet`) — fix before any household gets agent 0.165.0 by signed update** | — | before the next controller and agent release: hold every running seeder; clear a record whose app is gone; read the mode | CC | +| **R-1261** | Monitoring & notifications | P3 | **The R-1250 line hold (committed, NOT released): three edges before it ships (review round 2, HUNT-4 H4–H6).** Two seeders behind one uplink: the hold takes only the first, so the test is refused „not quiet” every 15 min in the Tuesday window and never counted (`felhom-controller/controller/internal/upwatch/linehold.go:155-160`; agent `internal/linetest/linetest.go:379-381`); a hold record of an app removed while held is never cleared and the 50 % rule never runs again (`linehold.go:106-108`; `upwatch/service.go:149`); a qBittorrent alt-speed flip during the hold leaves 1 KiB/s stored (`explain.go:175`; `linehold.go:127-128`). | **VERIFY** — RELEASED in controller v0.344.0 (global floor 2026-10-26 09:07:47Z, every live box on 0.344.0); the live test with a seeding box is Tuesday 10:00–16:00 (R-1250's dated check). **Earlier:** **VERIFY** — fixed on main 2026-10-25 night (controller `a79a292`), waits for the controller release (before any household gets agent 0.165.0 by signed update): every running seeder held, a removed app's record dropped, qBittorrent's slot named (`up_limit`/`alt_up_limit`). Red-proofs `audits/night-2026-10-25/redproof-r1261.txt`. **Live on the night's rig** (BYO VM, agent 0.165.0, rc.1, two seeders): both held, 94.8 Mbit/s, both limits back; a speed-mode switch mid-test left no 1 KiB/s (`rig/hold-watch*.txt`); the removed-app case by test only. **Earlier state:** OPEN — **now in controller 0.343.0 (released 2026-10-25); the three edges act only on a box whose agent is ≥ 0.165.0 (it alone reports `waiting_for_quiet`) — fix before any household gets agent 0.165.0 by signed update** | — | before the next controller and agent release: hold every running seeder; clear a record whose app is gone; read the mode | CC | | **R-1262** | Monitoring & notifications | P3 | **The upload watch treats ANY viewer as an explanation for any amount of upload, so a hacked Plex can send freely while one family member watches.** Found 2026-10-24 (SEC-1 X1): `felhom-controller/controller/internal/upwatch/watch.go:275,299`; `08` §9 states the rule, its limits list did not name this case. | OPEN | — | propose a cap (viewers × a bitrate ceiling) for the operator's R-1244 design; `08`'s limits list names the case meanwhile | CC | | **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC | | **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING** — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent` `acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC | @@ -312,11 +307,10 @@ stopping line that lies. | **R-1223** | Monitoring & notifications | P3 | **After a reboot, a box whose tunnel ep0 refuses is invisible: the future stamp is gone, so neither the D30 re-key nor the hub's alarm sees it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, A8 + H1): the kernel forgets `latest-handshake` when the interface restarts (0 = never); the agent then sends neither `last_handshake_age_s` nor `handshake_future_s`, and `checkWGTunnel` (`hub/internal/api/host_tunnel.go`) does not judge a registered, active tunnel with no handshake at all. Shape: the clock ran ahead, was corrected, the box rebooted (some households switch off at night), ep0 still holds the future stamp and starts nothing. | OPEN | — | Agent: report `tunnel_up_s` (how long the unit has been active) with no handshake; hub: a registered, active tunnel with no handshake for > 30 min while reporting → `wg_tunnel_stale`; the agent re-keys the same way after 30 min with no handshake. Tests both sides; rides an agent release + a hub deploy | CC | | **R-1224** | Monitoring & notifications | P4 | **The operator's 1-hour mail cooldown for `wg_tunnel_stale` is per customer, not per host, and `wg_tunnel_ok` does not end it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, H3): `notify/dispatcher.go:620` keys the cooldown `customerID:eventType`. Two hosts of one household going stale within an hour, or stale → ok → stale, store a second event row but mail once (the suppression is in `notification_log`). | OPEN | — | Key the cooldown by `host_id` from the details for host-scoped operator events, and let the matching ok event end it; a dispatcher test with two hosts; rides a hub deploy | CC | | **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC | -| **R-1287** | Monitoring & notifications | P3 | **A D47 set-aside mails the operator as „🔴 tester-1: 1 of 1 apps failed to back up (backup run)" — no backup failed; the body says the right thing („its backup belonged to an earlier install … SET ASIDE whole").** Seen 2026-10-25 on Tester 1 with controller 0.343.0 (attended release run): two such mails at 17:31:56Z and 17:36:55Z. The controller sends the set-aside as `backup_run_failures` leg `unit-set-aside`, `Failed: 1` (so the hub never drops a line that claims zero); the hub's subject for that event is fixed („N of M apps failed to back up", `hub/internal/notify/templates.go:353`). The CLAUDE.md corollary: an alarm's text must say what it counts. After the floor, every household reinstall mails the operator a red „failed to back up". | **VERIFY** — fixed on main 2026-10-25 night, waits for the hub release (v0.171.0): a set-aside-only digest is a warning with „… set aside — nothing failed ()"; a run with a real failure stays red (`hub/internal/notify/r1287_setaside_test.go`, red-proved) | — | two-repo: the hub words a `unit-set-aside` leg on its own („… a backup of an earlier install was set aside — nothing failed", warning, not error) with a test on the rendered subject; or a new event type through the allow-list (`hub-event-allowlist-gotcha`); one hub release | CC | | **R-1291** | Monitoring & notifications | P3 | **A held app gives the operator no lasting signal: one event at the stop, then nothing — demo-hp's Kimai sat held 11 days unseen.** Found 2026-10-25 night (R-1286). No held-app state reaches the hub (the report carries none; the hub shows none); the crash-loop stop sends `app_stopped_unhealthy` once. On a household box an app held after a crash loop, a failed update or a failed restore stays stopped until someone opens its page. | OPEN | — | a two-repo design: the controller reports its holds (stack, reason, since), the hub shows them on the host and mails a reminder after N days (the D-level of the reason); one row's worth of design first | CC | -## Hub & operator — 13 rows (P3 4, P4 9) +## Hub & operator — 12 rows (P3 3, P4 9) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -331,7 +325,6 @@ stopping line that lies. | **R-1002** | Hub & operator | P4 | **A box the household switched off reads ONLINE on the hub's hosts page for its first 45 minutes** (seen live 2026-10-13, R-998 proof, `audits/poweroff-2026-10-13/B-hub-watch.log`: the „Power: Kikapcsolva…" line at 18:53, the KIKAPCSOLVA badge only at 19:38, when the host report turned stale). The customer page read „warn" in part of that gap (seen, not captured). No alarm and no mail fired; the operator page is only slow to say it. | **READY** — filed 2026-10-13 | — | CC: let `hostDisplayStatus` read `deliberate_off_at` before staleness, with a render test per state; next hub release | CC | | **R-1056** | Hub & operator | P4 | **Five operator routes fit no token group, so they are session only until the operator places them:** apps reset-telemetry and dismiss-issues, the Configuration asset re-seed, the off-site WireGuard endpoint save, the off-site abandon-cancel (`05` §17.3). Nothing needs them by token today. | WAITING-ON-OPERATOR — filed 2026-10-15 by the token build | operator decision | Place each in a group, or leave session only | operator | | **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator | -| **R-1289** | Hub & operator | P4 | **The hub's „newest controller the fleet reported" can be a dev build that no box runs: `semver.Compare` returns 0 (equal) for any version with a suffix, so „0.343.0-dev.r1251a" — once read first — beats nothing and is never beaten.** Seen 2026-10-25 in the vouch line (attended release run): „golden 0.331.0 unchanged (behind the fleet's 0.343.0-dev.r1251a …)" while the four hosts run 0.343.0 / 0.342.0. `store.NewestReportedControllerVersion` scans every report ever stored (`hub/internal/store/store.go:2210`), and `hub/internal/semver/semver.go:14` gives up on `0-dev…` / `0-rc.N`. Used by the R-120 golden gate (`hub/internal/web/configs.go:1548,1554`): with a suffixed „newest" the gate compares against whatever row came first — today it only words a log line and fails open. | **VERIFY** — fixed on main 2026-10-25 night, waits for the hub release (v0.171.0): suffixed versions are skipped (`TestR1289_ATestBuildIsNeverTheFleetsNewest`, red-proved) | — | skip suffixed versions in `NewestReportedControllerVersion` (or compare with the pre-release rule the controller uses, `util.ParseVersionPre`) + a test with a dev and an rc row ahead of a release row (red first); one hub release | CC | | **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC | ## Business & legal — 14 rows (P2 4, P3 2, P4 8) diff --git a/documentation/backlog/STATUS-HISTORY.md b/documentation/backlog/STATUS-HISTORY.md index f1f02bb3..df12c72d 100644 --- a/documentation/backlog/STATUS-HISTORY.md +++ b/documentation/backlog/STATUS-HISTORY.md @@ -7,6 +7,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S ## Moved from STATUS on 2026-10-26 (attended release run) +**Updated 2026-10-26 (morning).** Installer 1.36.1 is live: a wrong passphrase now says so. Overnight, commit only: the torrent line test holds every seeder; FileBrowser is off the app network; a set-aside mails as a set-aside; the four D44 picks are built and proven on a test VM. + +**Monday 2026-10-26 (attended; each a STOP).** 1. Hub v0.171.0 (R-1287, R-1289). 2. Agent v0.166.0 + your vouch (R-1280, R-1285). 3. Controller v0.344.0 — before any household gets agent 0.165.0 (R-1261, R-1281, R-1288, R-1282, R-1286, R-1278). 4. Catalog: MeTube (R-1283). 5. Dated: R-1250 Tuesday. 6. Hub Hosts page: delete the dead `Tester4-84bbe7`. If not: the fixes stay unreleased; the host alarms. + | D51 | R-1286 | demo-hp's Kimai was stopped by the crash-loop guard on 10-14 (the power drill) and is still held. Start it again, or remove it? | **Press Start** (it lifts this kind of hold; if it crash-loops again it is held again and we look) | remove it through the product: the test app and its data are gone | it stays stopped; nothing else waits on it | — | **Ruled:** D51 = Start (operator, 2026-10-26 07:57; `09` §3 decision 263) — healthy, 0 restarts. D52 = bake the golden today on v0.344.0 (operator, 2026-10-26; decision 264) — asked in chat, never on the page; the waiver covers 2026-10-26 only. diff --git a/documentation/tests/golden-0.344.0-2026-10-26/00-boot.txt b/documentation/tests/golden-0.344.0-2026-10-26/00-boot.txt new file mode 100644 index 00000000..71c928d0 --- /dev/null +++ b/documentation/tests/golden-0.344.0-2026-10-26/00-boot.txt @@ -0,0 +1,6 @@ +## boot 2026-10-26T09:23:45Z +drill VM reverted to virgin and cold-booted (RUNBOOK-manual-build §4.0); pve-manager/9.2.2 +pveam update: Update successful; template debian-13-standard_13.6-1_amd64.tar.zst (checksum verified) +build-golden.sh from felhom-agent origin/main b890b57, sha256 645b3b65… (the same file as the 0.331.0 bake) +token + registry user copied file->file from sec (GITEA_PKG_TOKEN, GITEA_PKG_USER); unit properties grep for the token value: 0 +token in bake.log: 0 (control with the token appended: 1) diff --git a/documentation/tests/golden-0.344.0-2026-10-26/02-round-trip.txt b/documentation/tests/golden-0.344.0-2026-10-26/02-round-trip.txt new file mode 100644 index 00000000..0771b63d --- /dev/null +++ b/documentation/tests/golden-0.344.0-2026-10-26/02-round-trip.txt @@ -0,0 +1,3 @@ +## round trip 2026-10-26T09:23:54Z +sha256 767a90fe07ed47c85a3e08a16b85926e30531b65608f9856aa25855b4dd53210 bytes 637293448 +bake said 767a90fe07ed47c85a3e08a16b85926e30531b65608f9856aa25855b4dd53210 diff --git a/documentation/tests/golden-0.344.0-2026-10-26/03-teardown.txt b/documentation/tests/golden-0.344.0-2026-10-26/03-teardown.txt new file mode 100644 index 00000000..7f011619 --- /dev/null +++ b/documentation/tests/golden-0.344.0-2026-10-26/03-teardown.txt @@ -0,0 +1,11 @@ +== teardown 2026-10-26T09:24:10Z +purging CT 9100 from related configurations.. +0 +ls: cannot access '/root/.gitea-token': No such file or directory +ls: cannot access '/root/.gitea-user': No such file or directory +ls: cannot access '/root/bake-run.sh': No such file or directory +ls: cannot access '/root/bake.log': No such file or directory +qemu running: 0 +revert rc=0 +ID TAG VM_SIZE DATE VM_CLOCK ICOUNT +1 virgin 0 B 2026-07-03 14:12:15 0000:00:00.000 0 diff --git a/documentation/tests/golden-0.344.0-2026-10-26/README.md b/documentation/tests/golden-0.344.0-2026-10-26/README.md new file mode 100644 index 00000000..7bffba71 --- /dev/null +++ b/documentation/tests/golden-0.344.0-2026-10-26/README.md @@ -0,0 +1,42 @@ +# Golden 0.344.0 — bake + publish, 2026-10-26 (the attended release run; D52) + +Why: the Sunday 2026-10-25 weekly bake was missed (it was not in that night's brief); the operator ruled **D52 = bake +today on v0.344.0** (`09` §3 decision 264), with the waiver covering 2026-10-26 only. Controller 0.344.0 was released the +same morning (global floor 09:07:47 UTC, MinAgent 0.158.1). Procedure: `documentation/runbooks/RUNBOOK-manual-build.md` +§4.0 and §4.1 steps 1–4, in the drill VM on DooPlex. The script came from CC's `felhom-agent` clone at `origin/main` +(`b890b57`; `build-golden.sh` sha256 `645b3b65…`, the same file as the 0.331.0 bake). + +| | Previous (`../golden-0.331.0-2026-10-18/`) | This bake | +|---|---|---| +| `build-golden.sh` | v3.2.0 | same file | +| Controller | `felhom-controller:0.331.0` (MinAgent 0.158.1) | **`felhom-controller:0.344.0`** (MinAgent **0.158.1**) | +| Docker engine | the approved set (six pairs) | the same six pairs — the hub's newest approved Docker release is still `os-docker-20261004-142842` (read from the hub DB, with its WAL, today) | +| Template | `debian-13-standard_13.6-1_amd64.tar.zst` | same (after `pveam update`; `_amd64` filtered) | +| Token | `~/.gitea-token` file → file | **from `sec`** (`GITEA_PKG_TOKEN`, `GITEA_PKG_USER`), written to a 0600 file and copied file → file, the DooPlex copy shredded | + +## Pass markers (from `bake.log`, this folder) + +``` +[golden] build-golden.sh v3.2.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.344.0 +[golden] Docker engine set PINNED to the approved release: containerd.io=2.3.6-1~debian.13~trixie … docker-compose-plugin=5.6.0-1~debian.13~trixie + docker OK (overlay2; data-root /var/lib/docker) +INFO: including mount point rootfs ('/') in backup +INFO: including mount point mp0 ('/var/lib/felhom') in backup +[golden] upload OK (HTTP 201) +GOLDEN_VERSION=0.344.0 +GOLDEN_SHA256=767a90fe07ed47c85a3e08a16b85926e30531b65608f9856aa25855b4dd53210 +``` + +No `excluding`, no `FATAL`, no `GOLDEN_DOCKER_PKGS not set`; the unit `Result=success`. Round trip: the registry's file +downloaded and hashed = the bake's sha, 637,293,448 bytes (`02-round-trip.txt`). Token: the unit's properties grep = 0 +(`00-boot.txt`); the committed log grep = 0, with a working control (= 1 on a copy with the token appended, the copy +shredded). Teardown: `03-teardown.txt` (CT 9100 purged, token/user/script/log shredded in the VM, VM off, disk back to +`virgin`). No controller push after the bake. + +## Vouch — the operator's act (never CC's) + +Three fields together (runbook §4.1 step 5), Configuration → Day-0 artifacts: + +- golden **0.344.0**, sha256 `767a90fe07ed47c85a3e08a16b85926e30531b65608f9856aa25855b4dd53210` +- agent **0.166.0** (vouched 2026-10-26 10:04:55 CET, unchanged), sha256 `3f22c65b78914632fa476292f6b58e5af3ed9e9900f3a5a7a347a9c0bfea9c5c` +- min_agent **0.158.1** (unchanged) diff --git a/documentation/tests/golden-0.344.0-2026-10-26/bake.log b/documentation/tests/golden-0.344.0-2026-10-26/bake.log new file mode 100644 index 00000000..24a61ca7 --- /dev/null +++ b/documentation/tests/golden-0.344.0-2026-10-26/bake.log @@ -0,0 +1,340 @@ +[golden] build-golden.sh v3.2.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.344.0 +[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) … + Logical volume "vm-9100-disk-0" created. + Logical volume pve/vm-9100-disk-0 changed. +Creating filesystem with 8388608 4k blocks and 2097152 inodes +Filesystem UUID: caff2a7f-c2de-4387-8f2d-826c818830d5 +Superblock backups stored on blocks: + 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, + 4096000, 7962624 + Logical volume "vm-9100-disk-1" created. + Logical volume pve/vm-9100-disk-1 changed. +Creating filesystem with 6291456 4k blocks and 1572864 inodes +Filesystem UUID: ab9abb60-9c7e-473f-8c74-02aebb335f9e +Superblock backups stored on blocks: + 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, +extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst' +Total bytes read: 553512960 (528MiB, 98MiB/s) +Detected container architecture: amd64 +Creating SSH host key 'ssh_host_rsa_key' - this may take some time ... +done: SHA256:zJDWIY1iIUrsWL8Uw9/eVhpMjpx9RDQJB5zaC4colbI root@felhom-golden +Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ... +done: SHA256:E6DkbbkbDgaanslUSIzFF93Jrmb0CKmNAvfVNdtfrxY root@felhom-golden +Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ... +done: SHA256:oRIy665uwX7I8VjR33EZoR78U+TvGVvBNhnXkt64H3M root@felhom-golden +[golden] starting + installing Docker (official repo, trixie channel) … +[golden] Docker engine set PINNED to the approved release: containerd.io=2.3.6-1~debian.13~trixie docker-buildx-plugin=0.37.1-1~debian.13~trixie docker-ce=5:29.8.2-1~debian.13~trixie docker-ce-cli=5:29.8.2-1~debian.13~trixie docker-ce-rootless-extras=5:29.8.2-1~debian.13~trixie docker-compose-plugin=5.6.0-1~debian.13~trixie +apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct! +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = (unset), + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to the standard locale ("C"). +locale: Cannot set LC_CTYPE to default locale: No such file or directory +locale: Cannot set LC_MESSAGES to default locale: No such file or directory +locale: Cannot set LC_ALL to default locale: No such file or directory +apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct! +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = (unset), + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to the standard locale ("C"). +locale: Cannot set LC_CTYPE to default locale: No such file or directory +locale: Cannot set LC_MESSAGES to default locale: No such file or directory +locale: Cannot set LC_ALL to default locale: No such file or directory + installed: containerd.io 2.3.6-1~debian.13~trixie + installed: docker-buildx-plugin 0.37.1-1~debian.13~trixie + installed: docker-ce 5:29.8.2-1~debian.13~trixie + installed: docker-ce-cli 5:29.8.2-1~debian.13~trixie + installed: docker-ce-rootless-extras 5:29.8.2-1~debian.13~trixie + installed: docker-compose-plugin 5.6.0-1~debian.13~trixie +[golden] no approved guest release given - the template versions stay; first-night count vs an approved release: n/a +[golden] pending Debian upgrades in the baked guest (what a FUTURE approval may bring): 57 +[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation … +[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds … +[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) … +Unable to find image 'hello-world:latest' locally +latest: Pulling from library/hello-world +4f55086f7dd0: Pulling fs layer +4f55086f7dd0: Verifying Checksum +4f55086f7dd0: Download complete +4f55086f7dd0: Pull complete +Digest: sha256:11a3eb1597e864c51b3f56733cb2e8a90bf0c284bfc05f9444d81c558bc848aa +Status: Downloaded newer image for hello-world:latest + docker OK (overlay2; data-root /var/lib/docker) + live-restore: on + /var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4 + /mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4 + both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576 +[golden] baking the in-guest controller image gitea.dooplex.hu/admin/felhom-controller:0.344.0 (no registry cred at deploy) … + +WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'. +Configure a credential helper to remove this warning. See +https://docs.docker.com/go/credential-store/ + +0.344.0: Pulling from admin/felhom-controller +ecc510c1e359: Pulling fs layer +c3f83389ce1b: Pulling fs layer +e81c1fce9826: Pulling fs layer +63a1406b1961: Pulling fs layer +d747c9b6922d: Pulling fs layer +6123e7de002d: Pulling fs layer +a77e9b7b71fe: Pulling fs layer +d747c9b6922d: Waiting +6123e7de002d: Waiting +63a1406b1961: Waiting +a77e9b7b71fe: Waiting +e81c1fce9826: Verifying Checksum +e81c1fce9826: Download complete +63a1406b1961: Verifying Checksum +63a1406b1961: Download complete +c3f83389ce1b: Verifying Checksum +c3f83389ce1b: Download complete +d747c9b6922d: Verifying Checksum +d747c9b6922d: Download complete +ecc510c1e359: Download complete +6123e7de002d: Verifying Checksum +6123e7de002d: Download complete +a77e9b7b71fe: Verifying Checksum +a77e9b7b71fe: Download complete +ecc510c1e359: Pull complete +c3f83389ce1b: Pull complete +e81c1fce9826: Pull complete +63a1406b1961: Pull complete +d747c9b6922d: Pull complete +6123e7de002d: Pull complete +a77e9b7b71fe: Pull complete +Digest: sha256:9435165dab85b1200d94157ec49d8be065889139e1831a5476dc68f4b9474b8a +Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-controller:0.344.0 +gitea.dooplex.hu/admin/felhom-controller:0.344.0 +[golden] asking the controller which infra images it manages … +[golden] baking infra images (4): traefik:v3.7.13 cloudflare/cloudflared:2026.9.3 gtstef/filebrowser:1.5.6-stable gitea.dooplex.hu/admin/felhom-samba:1.2.0 … +v3.7.13: Pulling from library/traefik +e2de96513ba9: Pulling fs layer +b686a4f73445: Pulling fs layer +78cb21c375ca: Pulling fs layer +acb2f33459b1: Pulling fs layer +acb2f33459b1: Waiting +e2de96513ba9: Verifying Checksum +e2de96513ba9: Download complete +acb2f33459b1: Verifying Checksum +acb2f33459b1: Download complete +b686a4f73445: Verifying Checksum +b686a4f73445: Download complete +e2de96513ba9: Pull complete +78cb21c375ca: Verifying Checksum +78cb21c375ca: Download complete +b686a4f73445: Pull complete +78cb21c375ca: Pull complete +acb2f33459b1: Pull complete +Digest: sha256:24841fe2de7304c149343d877d2923b4c8800a38ba015dea9174c23b20e344a0 +Status: Downloaded newer image for traefik:v3.7.13 +docker.io/library/traefik:v3.7.13 +2026.9.3: Pulling from cloudflare/cloudflared +2cc7ee286bf3: Pulling fs layer +c172f21841df: Pulling fs layer +218cf840d0d9: Pulling fs layer +f6069939f718: Pulling fs layer +d6b1b89eccac: Pulling fs layer +2780920e5dbf: Pulling fs layer +7c12895b777b: Pulling fs layer +3214acf345c0: Pulling fs layer +52630fc75a18: Pulling fs layer +dd64bf2dd177: Pulling fs layer +b839dfae01f6: Pulling fs layer +ebddc55facdc: Pulling fs layer +c4bc6f35ff5e: Pulling fs layer +b96fe2995f90: Pulling fs layer +58c0c263dc73: Pulling fs layer +bd8962e29291: Pulling fs layer +cac2ae0193cb: Pulling fs layer +f0383d5ebc47: Pulling fs layer +dd64bf2dd177: Waiting +b839dfae01f6: Waiting +ebddc55facdc: Waiting +c4bc6f35ff5e: Waiting +b96fe2995f90: Waiting +f6069939f718: Waiting +d6b1b89eccac: Waiting +58c0c263dc73: Waiting +bd8962e29291: Waiting +cac2ae0193cb: Waiting +f0383d5ebc47: Waiting +2780920e5dbf: Waiting +7c12895b777b: Waiting +3214acf345c0: Waiting +52630fc75a18: Waiting +2cc7ee286bf3: Verifying Checksum +2cc7ee286bf3: Download complete +c172f21841df: Verifying Checksum +c172f21841df: Download complete +218cf840d0d9: Verifying Checksum +218cf840d0d9: Download complete +2cc7ee286bf3: Pull complete +f6069939f718: Verifying Checksum +d6b1b89eccac: Verifying Checksum +d6b1b89eccac: Download complete +2780920e5dbf: Verifying Checksum +2780920e5dbf: Download complete +7c12895b777b: Download complete +52630fc75a18: Download complete +3214acf345c0: Verifying Checksum +3214acf345c0: Download complete +c172f21841df: Pull complete +dd64bf2dd177: Verifying Checksum +dd64bf2dd177: Download complete +ebddc55facdc: Verifying Checksum +ebddc55facdc: Download complete +b839dfae01f6: Verifying Checksum +b839dfae01f6: Download complete +c4bc6f35ff5e: Verifying Checksum +c4bc6f35ff5e: Download complete +b96fe2995f90: Verifying Checksum +b96fe2995f90: Download complete +bd8962e29291: Verifying Checksum +bd8962e29291: Download complete +58c0c263dc73: Download complete +cac2ae0193cb: Verifying Checksum +cac2ae0193cb: Download complete +218cf840d0d9: Pull complete +f0383d5ebc47: Verifying Checksum +f0383d5ebc47: Download complete +f6069939f718: Pull complete +d6b1b89eccac: Pull complete +2780920e5dbf: Pull complete +7c12895b777b: Pull complete +3214acf345c0: Pull complete +52630fc75a18: Pull complete +dd64bf2dd177: Pull complete +b839dfae01f6: Pull complete +ebddc55facdc: Pull complete +c4bc6f35ff5e: Pull complete +b96fe2995f90: Pull complete +58c0c263dc73: Pull complete +bd8962e29291: Pull complete +cac2ae0193cb: Pull complete +f0383d5ebc47: Pull complete +Digest: sha256:072c067d25ccbe61d46e18f0d0723255f2bb5304f7317caa95b27031520ff92c +Status: Downloaded newer image for cloudflare/cloudflared:2026.9.3 +docker.io/cloudflare/cloudflared:2026.9.3 +1.5.6-stable: Pulling from gtstef/filebrowser +55afa1ecc21d: Pulling fs layer +8ed8f35f8d4f: Pulling fs layer +989b226a579c: Pulling fs layer +660aeead31d5: Pulling fs layer +4f4fb700ef54: Pulling fs layer +adce24567e4c: Pulling fs layer +f17ea56b313b: Pulling fs layer +6b6f3b3efe88: Pulling fs layer +4ed1ca4f3fce: Pulling fs layer +e6fc9c6a5757: Pulling fs layer +d47782d1182a: Pulling fs layer +6b6f3b3efe88: Waiting +4ed1ca4f3fce: Waiting +e6fc9c6a5757: Waiting +d47782d1182a: Waiting +660aeead31d5: Waiting +4f4fb700ef54: Waiting +adce24567e4c: Waiting +f17ea56b313b: Waiting +55afa1ecc21d: Verifying Checksum +55afa1ecc21d: Download complete +660aeead31d5: Verifying Checksum +660aeead31d5: Download complete +4f4fb700ef54: Verifying Checksum +4f4fb700ef54: Download complete +989b226a579c: Verifying Checksum +989b226a579c: Download complete +55afa1ecc21d: Pull complete +8ed8f35f8d4f: Verifying Checksum +8ed8f35f8d4f: Download complete +f17ea56b313b: Verifying Checksum +f17ea56b313b: Download complete +6b6f3b3efe88: Verifying Checksum +6b6f3b3efe88: Download complete +4ed1ca4f3fce: Verifying Checksum +4ed1ca4f3fce: Download complete +adce24567e4c: Verifying Checksum +adce24567e4c: Download complete +d47782d1182a: Verifying Checksum +d47782d1182a: Download complete +e6fc9c6a5757: Verifying Checksum +e6fc9c6a5757: Download complete +8ed8f35f8d4f: Pull complete +989b226a579c: Pull complete +660aeead31d5: Pull complete +4f4fb700ef54: Pull complete +adce24567e4c: Pull complete +f17ea56b313b: Pull complete +6b6f3b3efe88: Pull complete +4ed1ca4f3fce: Pull complete +e6fc9c6a5757: Pull complete +d47782d1182a: Pull complete +Digest: sha256:7c5d7ac8ffda31294d278063cf9d2e04303b39e6dce1f4c691342240ca7703b8 +Status: Downloaded newer image for gtstef/filebrowser:1.5.6-stable +docker.io/gtstef/filebrowser:1.5.6-stable +1.2.0: Pulling from admin/felhom-samba +e2de96513ba9: Already exists +0ddbc9f3a6b6: Pulling fs layer +ce57a3f93416: Pulling fs layer +fb94eeec2fe1: Pulling fs layer +ce57a3f93416: Verifying Checksum +ce57a3f93416: Download complete +fb94eeec2fe1: Verifying Checksum +fb94eeec2fe1: Download complete +0ddbc9f3a6b6: Verifying Checksum +0ddbc9f3a6b6: Download complete +0ddbc9f3a6b6: Pull complete +ce57a3f93416: Pull complete +fb94eeec2fe1: Pull complete +Digest: sha256:da96f25033827fbb77d0ee398e86fe5b744e2f29be9de898994533340c97acf9 +Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.2.0 +gitea.dooplex.hu/admin/felhom-samba:1.2.0 +[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'. +[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'. +[golden] baking the first-boot SSH host-key regeneration unit (F3) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'. +[golden] identity-clean + minimize … +[golden] stop + archive … +INFO: including mount point rootfs ('/') in backup +INFO: including mount point mp0 ('/var/lib/felhom') in backup +INFO: archive file size: 607MB +INFO: Finished Backup of VM 9100 (00:00:30) +[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_10_26-10_17_42.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive) +[golden] publishing golden (637293448 bytes, sha256 767a90fe07ed47c8…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.344.0/golden.tar.zst +[golden] pre-delete existing: HTTP 404 (404/204 expected) +[golden] upload OK (HTTP 201) +GOLDEN_VERSION=0.344.0 +GOLDEN_SHA256=767a90fe07ed47c85a3e08a16b85926e30531b65608f9856aa25855b4dd53210 +[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.344.0 / 767a90fe07ed47c85a3e08a16b85926e30531b65608f9856aa25855b4dd53210 +[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge) diff --git a/documentation/tests/golden-waiver.yml b/documentation/tests/golden-waiver.yml deleted file mode 100644 index 60c26ccb..00000000 --- a/documentation/tests/golden-waiver.yml +++ /dev/null @@ -1,20 +0,0 @@ -# documentation/tests/golden-waiver.yml — read by scripts/golden_currency_gate.py -# -# Goldens are on a CADENCE, not per release (operator ruling 2026-09-13, R-468): weekly, and always -# before any drill or fresh install. While this waiver is valid a golden BEHIND the record makes the -# gate ADVISORY instead of red. It never covers a golden that is UNRECORDED (R-385). At most 14 days -# after `issued`, or the gate refuses it. Renew with a new dated commit; delete it at the first -# external install. -# Issued 2026-10-13 by CC: controller 0.310.0 (R-998, „A gép kikapcsolása") was released the evening of the weekly bake -# (golden 0.309.0, baked 2026-10-13); the next weekly bake is due 2026-10-20. The bake and its vouch are the operator's -# acts (the R-998 task fences them). No drill or fresh install is planned before then. Not a bypass: dated, capped at -# the weekly due date. -# 2026-10-14: goldens 0.310.0 and 0.311.0 were baked the same day (`golden-0.311.0-2026-10-14/`), so the golden is at -# the newest controller and this waiver covers nothing today. It stays until its date for a release before the next bake. -# 2026-10-26 by CC, on the operator's D52 (bake today on v0.344.0): the Sunday 2026-10-25 bake was missed (it was not in -# that night's brief); golden 0.331.0 is 8 days old. Renewed for TODAY ONLY so the release commits can be pushed; the -# gate is green tomorrow by the new golden, and this waiver is removed after the vouch. -issued: 2026-10-26 -expires: 2026-10-27 -reason: D52: bake on v0.344.0 the same day -register_row: R-468 diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index be6c4ffe..27b2727b 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,4 +1,4 @@ -## v0.171.0 (unreleased, 2026-10-25 night) — a set-aside mails as a set-aside (R-1287); a test build is never „the fleet's newest" (R-1289) — committed, NOT deployed +## v0.171.0 (2026-10-25 night; RELEASED 2026-10-26 07:58 UTC, attended release run — image sha256:742d2e69…, from `7ad7c996`, CI 2352) — a set-aside mails as a set-aside (R-1287); a test build is never „the fleet's newest" (R-1289) - **R-1287** (`internal/notify`): a `backup_run_failures` digest whose every entry is leg `unit-set-aside` (a D47 set-aside, controller ≥ 0.343.0) is sent as a **warning** with its own subject „⚠️ : a backup of an earlier