diff --git a/REPORT-release-1-dates-2026-11-14.md b/REPORT-release-1-dates-2026-11-14.md new file mode 100644 index 00000000..390d2a8a --- /dev/null +++ b/REPORT-release-1-dates-2026-11-14.md @@ -0,0 +1,93 @@ +# REPORT — release 1 of four, the „dates” release (2026-11-14, daytime, attended at two STOPs) + +## 1. Claims in the prompt that turned out wrong, named first + +1. **„The hub's box facts hold no list of files” — true for the PARSED facts, not for what the box sends.** The box's + host report carries a sha256 per installed root file (`system.facts.host.config_bundle.live`); the hub's parser + (`sysfacts.ConfigBundle`) drops it. The fix did not need it. +2. **„The vouch holds only one checksum of the bundle” — true.** But the hub can get more at any time: it already has + a read-only client for the package registry, and the bundle is a published file. That made a hub-only fix small. +3. **„Go 1.27.4 and its images are published” — true** (go.dev; Docker Hub `golang:1.27.4-trixie` and + `-alpine3.24`, both pinned by digest). +4. **The baseline's „golden 0.352.0 baked 2026-11-11 … plus 7 days” is right**, and moot: a new golden is vouched. +5. **CC's own wrong claim, in the 30-minute hub watch:** the second read was taken 12 seconds before the 30 minutes + were over (10:28:03 against 10:28:15 UTC) and its heading said they had passed. A third read is in the file. + +## 2. Five lines + +1. **Every box that reports runs controller 0.352.1** (Tester 1, demo-hp, demo-felhom, scratch 9202), built with + **Go 1.27.4** (read from the running binary). The HTTPS header is **unchanged: `max-age=86400`** on all 26 public + names — the one-year setting is NOT in this release (built on 11-14; the words „one day early is OK” were not given). +2. **Hub 0.181.0 is live** since 09:58 UTC, on the operator's yes; Synced/Healthy; 0 ERROR lines. +3. **The false „root files behind” warning is stopped by the fix, not the stopgap.** The hub now compares the files + of the two bundles. Proof that no mail is due on 11-17: the three „behind since” marks in the hub's database are + empty since 09:59:20 UTC, the alarm check sent 0 alarms, and the System page reads „All boxes look fine”. +4. **Golden 0.352.1 is baked, proven by one fresh install, and vouched by the operator** (10:25:43 UTC). + `golden_currency_gate.py`: OK, rc 0. +5. **Not proven:** that a box's report never timed out in the hub's slow first 8 minutes (R-1393); the vouch through + a second channel (read from the hub's log only); that no https trouble happened since 10-15 (not searched, because + the date condition already failed); the agent on Go 1.27.4 (release 2). + +## 3. Details + +**Rulings recorded (Part 0):** `09-update-architecture.md` §3 decisions 303 (D80 = the root CA copy), 304 (D81 = yes, +with the stopgap word), 305 (the N100's time service: no hand repair; 2026-12-15), 306 (the build order). D80 and D81 +are out of the decide table; dated lines on R-1380, R-1392, R-1301, R-1388. + +| | Version | Digest | Commit | CI (run id, by commit) | +|---|---|---|---|---| +| Controller | 0.352.1 (rc.1 first) | `sha256:1a6e8945211a…13b6bb` | felhom-controller `8747744b` | 2637 success | +| Hub | 0.181.0 | `sha256:be2e9e5676ee…d1f429` | felhom.eu `83316fc9` (image), `63f69b99` (manifest) | 2636 success, 2638 success (read before the sync) | +| Golden | 0.352.1 | `4f411cbeb4b2…01346d` | build-golden.sh v3.2.0 | — | +| Agent | 0.173.0, not released | — | — | — | + +**HSTS (R-1050).** Condition 1 (the date) failed, so the other two were not judged. The check date moved to +2026-11-21 with the reason in the row; `STATUS.md` says how to ask for it. + +**Go, the 20-minute reads.** Controller: seven `MaxBytesReader` places, each reads the body to its end in the +handler's own goroutine before it answers; no `TimeoutHandler`. Hub: no `MaxBytesReader` at all. Neither matches the +advisory's crash pattern. The agent has one place; its turn (release 2) is hygiene. Written on R-1391. + +**R-1392, the fix.** `hub/internal/bundlefiles` (new): fetch both bundles, accept each only by a sha256 already +trusted, compare everything but `agent_version`. Fail direction: anything unread → „behind”, as before. Seven tests; +the three the prompt asked for were seen red with the fix line switched off. The two real bundles (0.172.1, 0.173.0) +hold the same 32 files. Design text: `11-os-updates.md` §5.4.2 `[FACT]`. No `[DESIGN]` text was needed. + +**Rows:** 250 before, 250 after. Opened R-1393 (hub slow after a restart). Closed R-1392. Narrowed R-1391 (the agent +is left). Dated lines: R-1050, R-468, R-1380, R-1301, R-1388. (R-1391 and R-1392 were opened by this morning's health +check, before this run.) + +**Machines changed, and how each was left.** +- Tester 1, demo-hp, demo-felhom: controller 0.352.1 by the normal update path; per-box floors cleared. +- Scratch 9202: rc.1, then 0.352.1 by hand through its image file. +- demo-hp: VM 9385 made and destroyed with its disks; the installer ISO fetched and removed. +- DooPlex: two image builds, the bake in the drill VM (now off, disk back on `virgin`), the hub deploy, two copies + of the hub database for reads (shredded). Nothing else. +- Hub: host `Tester4-46836f` is left for the operator to delete (and its appliance entry, if listed). The hub + re-issued Tester4's off-site key and carried its sealed package by itself. No ep0 change by CC. +- The N100: took 0.352.1 like any box; no hand step. + +**Architecture documents read:** `09-update-architecture.md` (§3 decisions 247, 249, 250, 251, 304), `11-os-updates.md` +§5.4.2, `05-hub-architecture.md` §17, `runbooks/publish-train-rules.md`, `runbooks/RUNBOOK-manual-build.md` §4, +`tests/golden-0.352.0-2026-11-11/README.md`, the `felhom-build-deploy` skill. **Not read:** `08-alarm-ladder.md` +(the alarm's wording and audience were not changed). + +**Capability map:** no row changed — nothing a box newly proves. **System poster:** no listed fact changed; none of +the three acts was needed. **`unproven.py --summary`:** 32 walked, 15 partial, 4 built, 4 missing, 23 not walked — +the tool's numbers; no number was moved by this run. + +**Instruction files:** none edited. + +**Unreleased after this run:** felhom.eu hub source holds one commit after 0.181.0 — the „same files” log line is +written once per pair, not at every page view (`hub/CHANGELOG.md` „Unreleased”). It rides the next hub release. + +## 4. Observations + +- The hub is slow for 8 minutes after a restart (24 slow statements, the System page 12.7 s, an enrolment 11.4 s). `FILED: R-1393`. +- The hub logged the „same files” line 6 times per page view. Fixed in source, unreleased. `NOT-A-FINDING: small, fixed in the session (the size rule)`. +- Tester 1 pushed „household cleared its mail address” at the controller's start. `NOT-A-FINDING: the box's own stored setting says so (email_cleared=true) and the start only repeats it; demo-felhom kept its address through the same update. Not checked: when Tester 1's address was cleared.` +- demo-felhom's controller start reads 09:18:23Z, 24 s before the floor that caused it. `NOT-A-FINDING: the N100's clock, R-1301.` +- The hub keeps `bundle_behind_since` marks of two deleted Tester4 hosts. `NOT-A-FINDING: decision 293 (D73) already rules that a host delete removes such rows; not built yet.` +- `os_config_bundle_behind` fired 4 times before today (last 2026-11-06). `NOT-A-FINDING: the same defect, now closed (R-1392).` +- On the LAN the installer address answers 404. `NOT-A-FINDING: R-1389, already on the operator's list.` +- The claim's two status lines are not kept in a file (the script printed them to the session only). `NOT-A-FINDING: an evidence gap of this run, said in the golden's README.` diff --git a/STATUS.md b/STATUS.md index 6ae98c7e..4c72ca02 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for Tester 2? Yes — tell the friend go (`documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-11-14 (morning).** Health check: the three test boxes are healthy; last night's backups and restore proofs passed. CI was red three nights for one due check; it is read now. The yellow „root files behind" is a false alarm that mails on 11-17 unless you answer D81. +**Updated 2026-11-14 (midday).** Release 1 („dates”) is out: controller 0.352.1 and hub 0.181.0 with the Go security fix; golden 0.352.1 is vouched. The false „root files behind” warning is fixed; no alarm mail is due on 11-17. The one-year HTTPS setting is NOT in (next controller release, from 11-15). ## What works - **The Plex attack did not reach Felhom's secrets** (R-1241). @@ -39,7 +39,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | D7 | R-990, R-989 | Rule on the forum ideas and the selfh.st ideas now, or park them? | **Park them** (P4; nothing waits on them) | Rule now: about one hour of reading | nothing changes; the research gets old | — | ## Waiting on you — do -**2026-11-14.** Build order: four releases (dates, host, „hub tells you”, promises); no new gap-review list first. **2026-11-11.** 1. N100's time service: no hand step now; release 2 repairs it. If it is not live by 2026-12-15: `apparmor_parser -r -T -W /etc/apparmor.d/usr.sbin.chronyd && systemctl restart chrony` (R-1301). 2. Hub Hosts page: Discard appliance `3fb5fed3…` (11-11 test box). 3. Rotate login `kisfenyo`'s Gitea token into the store as `GITEA_GIT_TOKEN` (R-925); the push after `sec put` is in R-1390. 4. Home DNS: remove the `felhom.eu` override (R-1389). 5. Read the dates only you can read (R-1381). 6. Search Console read (R-1035, 11-18). +**2026-11-14.** Next builds: host, „hub tells you”, promises; no new gap list first. Hub Hosts page: delete host `Tester4-46836f` (after 12:15 CET; else it alarms). Say „HSTS now” from 11-15 (R-1050). **2026-11-11.** 1. N100's time service: no hand step now; release 2 repairs it. If not live by 2026-12-15: the hand step in R-1301. 2. Hosts page: discard `3fb5fed3…`. 3. Rotate login `kisfenyo`'s Gitea token into the store as `GITEA_GIT_TOKEN` (R-925); the push after `sec put` is in R-1390. 4. Home DNS: remove the `felhom.eu` override (R-1389). 5. Read the dates only you can read (R-1381). 6. Search Console read (R-1035, 11-18). **Still needed? (yes/no)** — answer per line. 1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). (since 10-16) diff --git a/documentation/audits/release-1-dates-2026-11-14/README.md b/documentation/audits/release-1-dates-2026-11-14/README.md new file mode 100644 index 00000000..7c3b190a --- /dev/null +++ b/documentation/audits/release-1-dates-2026-11-14/README.md @@ -0,0 +1,19 @@ +# Release 1 of four — the „dates” release — evidence (2026-11-14) + +Report: `REPORT-release-1-dates-2026-11-14.md` (repo root). Golden: `documentation/tests/golden-0.352.1-2026-11-14/`. + +| File | What it shows | +|---|---| +| `controller/08-boxwalk-after.txt` | Tester 1, demo-hp, demo-felhom on 0.352.1 two minutes after the per-box floors: containers 10 / 26 / 9, none unhealthy, 0 ERROR lines, the running binary reads `go1.27.4`, image digest `1a6e8945…` | +| `controller/10-https-only.txt` | `scripts/https_only_check.py`, exit 0 | +| `controller/10b-hsts-header.txt` | every public name of the three boxes: `strict-transport-security: max-age=86400` and nothing more (26 names) — unchanged, HSTS one year is not in this release | +| `controller/11-global-floor.txt` | the floor save (303 `floor_set`), the System page's read-back, the hub's log lines | +| `controller/12-watch.txt` | 35 minutes after the floor: the three boxes, the hub's log with a positive control, the dashboard | +| `controller/13-per-box-floors-cleared.txt` | the three per-box floors cleared | +| `hub/01-before-system-page.txt` | hub 0.180.0: three boxes „root files behind”, behind since 2026-11-10 16:14 UTC | +| `hub/02-deploy.txt` | the sync of Deployment/hub at `63f69b99`, Synced/Healthy, image digest `be2e9e56…`, the start line „bundles are compared by their files” | +| `hub/03-after-system-page.txt` | hub 0.181.0: „All boxes look fine”; the cells say „the same files … only the label differs” | +| `hub/04-behind-since-marks.txt` | the hub database with its WAL (newest host report 30 s old): the three marks emptied at 09:59:20 UTC; the alarm check „0 alarm(s) sent” is in `02`/`05` | +| `hub/05-watch.txt` | the hub's log from the start: 0 ERROR, 24 slow-statement WARN lines in the first 8 minutes (R-1393), none after; the events | + +Not kept as files: the rc.1 read on 9202 (healthy, 0 ERROR; in the session only), the CI answers (run ids in the report). diff --git a/documentation/audits/release-1-dates-2026-11-14/controller/08-boxwalk-after.txt b/documentation/audits/release-1-dates-2026-11-14/controller/08-boxwalk-after.txt new file mode 100644 index 00000000..7da46089 --- /dev/null +++ b/documentation/audits/release-1-dates-2026-11-14/controller/08-boxwalk-after.txt @@ -0,0 +1,33 @@ +### root@192.168.0.154 2026-11-14T09:20:35Z +containers=10 unhealthy=0 restarting=0 exited=[] +gitea.dooplex.hu/admin/felhom-controller:0.352.1 Up About a minute (healthy) +image file: gitea.dooplex.hu/admin/felhom-controller:0.352.1 +image digest: sha256:0d3c28fab7ae2d08e0c92c3c4318316d4920cd440c4c9e3fac4581896314543a repo: gitea.dooplex.hu/admin/felhom-controller@sha256:1a6e8945211a0f1995fd2bd0782053c7f33152fcb00ca465d0596d961a13b6bb +started=2026-11-14T09:18:58.892043895Z +go in the running binary: 2 go1.27.4 +errors=0 warn=0 info=139 panics=0 +2026/11/14 09:18:59 [INFO] felhom-controller 0.352.1 starting (customer: tester-1, domain: enkicsifelhom.hu) +2026/11/14 09:18:59 [INFO] [selfupdate] Post-update startup: update successful (0.352.0 → 0.352.1) +### demo-hp 2026-11-14T09:20:37Z +containers=26 unhealthy=0 restarting=0 exited=[] +gitea.dooplex.hu/admin/felhom-controller:0.352.1 Up About a minute (healthy) +image file: gitea.dooplex.hu/admin/felhom-controller:0.352.1 +image digest: sha256:0d3c28fab7ae2d08e0c92c3c4318316d4920cd440c4c9e3fac4581896314543a repo: gitea.dooplex.hu/admin/felhom-controller@sha256:1a6e8945211a0f1995fd2bd0782053c7f33152fcb00ca465d0596d961a13b6bb +started=2026-11-14T09:18:59.079611972Z +go in the running binary: 2 go1.27.4 +errors=0 warn=3 info=147 panics=0 +2026/11/14 09:18:59 main.go:335: [INFO] felhom-controller 0.352.1 starting (customer: demo-hp, domain: enkisfelhom.hu) +2026/11/14 09:19:00 deploy.go:1470: [WARN] [stacks] Stack bookstack: new field ADMIN_PASSWORD (type=password) requires manual configuration +2026/11/14 09:19:00 deploy.go:1470: [WARN] [stacks] Stack calibre-web: new field ADMIN_PASSWORD (type=password) requires manual configuration +2026/11/14 09:19:00 deploy.go:1470: [WARN] [stacks] Stack plex: new field PLEX_CLAIM (type=text) requires manual configuration +2026/11/14 09:19:00 updater.go:855: [INFO] [selfupdate] Post-update startup: update successful (0.352.0 → 0.352.1) +### felhom-pve 2026-11-14T09:20:39Z +containers=9 unhealthy=0 restarting=0 exited=[] +gitea.dooplex.hu/admin/felhom-controller:0.352.1 Up About a minute (healthy) +image file: gitea.dooplex.hu/admin/felhom-controller:0.352.1 +image digest: sha256:0d3c28fab7ae2d08e0c92c3c4318316d4920cd440c4c9e3fac4581896314543a repo: gitea.dooplex.hu/admin/felhom-controller@sha256:1a6e8945211a0f1995fd2bd0782053c7f33152fcb00ca465d0596d961a13b6bb +started=2026-11-14T09:18:23.620814926Z +go in the running binary: 2 go1.27.4 +errors=0 warn=0 info=154 panics=0 +2026/11/14 09:18:24 [INFO] felhom-controller 0.352.1 starting (customer: demo-felhom, domain: demo-felhom.eu) +2026/11/14 09:18:24 [INFO] [selfupdate] Post-update startup: update successful (0.352.0 → 0.352.1) diff --git a/documentation/audits/release-1-dates-2026-11-14/controller/10-https-only.txt b/documentation/audits/release-1-dates-2026-11-14/controller/10-https-only.txt new file mode 100644 index 00000000..8533ee01 --- /dev/null +++ b/documentation/audits/release-1-dates-2026-11-14/controller/10-https-only.txt @@ -0,0 +1,36 @@ +== 2026-11-14T09:20:49Z python3 scripts/https_only_check.py +control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -) +== tester-1 (enkicsifelhom.hu): 6 public name(s) + OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1 + OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1 + OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 + OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1 + OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1 + OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1 +== demo-hp (enkisfelhom.hu): 15 public name(s) + OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1 + OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1 + OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1 + OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1 + OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1 + OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1 + OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1 + OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 + OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1 + OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1 + OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1 + OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1 + OK http://time.enkisfelhom.hu/r1033-check?q=1 -> 301 https://time.enkisfelhom.hu/r1033-check?q=1 + OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1 + OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1 +== demo-felhom (demo-felhom.eu): 8 public name(s) + OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1 + OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1 + OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1 + OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 + OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1 + OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1 + OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1 + OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1 +== every public name redirects http -> https +exit=0 diff --git a/documentation/audits/release-1-dates-2026-11-14/controller/10b-hsts-header.txt b/documentation/audits/release-1-dates-2026-11-14/controller/10b-hsts-header.txt new file mode 100644 index 00000000..c052b0ff --- /dev/null +++ b/documentation/audits/release-1-dates-2026-11-14/controller/10b-hsts-header.txt @@ -0,0 +1,53 @@ +== 2026-11-14T09:20:59Z Strict-Transport-Security of each public name (GET https:///, header only; unchanged in 0.352.1 — HSTS one year is NOT in this release) +https://arcade.enkisfelhom.hu strict-transport-security: max-age=86400 + +https://books.enkisfelhom.hu strict-transport-security: max-age=86400 + +https://docs.enkisfelhom.hu strict-transport-security: max-age=86400 + +https://felhom.demo-felhom.eu strict-transport-security: max-age=86400 + +https://felhom.enkicsifelhom.hu strict-transport-security: max-age=86400 + +https://felhom.enkisfelhom.hu strict-transport-security: max-age=86400 + +https://files.demo-felhom.eu strict-transport-security: max-age=86400 + +https://files.enkicsifelhom.hu strict-transport-security: max-age=86400 + +https://files.enkisfelhom.hu strict-transport-security: max-age=86400 + +https://gist.demo-felhom.eu strict-transport-security: max-age=86400 + +https://gist.enkisfelhom.hu strict-transport-security: max-age=86400 + +https://ha.enkisfelhom.hu strict-transport-security: max-age=86400 + +https://paperless.enkicsifelhom.hu strict-transport-security: max-age=86400 + +https://papir.enkisfelhom.hu strict-transport-security: max-age=86400 + +https://paste.enkicsifelhom.hu strict-transport-security: max-age=86400 + +https://paste.enkisfelhom.hu strict-transport-security: max-age=86400 + +https://pdf.enkisfelhom.hu strict-transport-security: max-age=86400 + +https://plex.enkisfelhom.hu strict-transport-security: max-age=86400 + +https://prowlarr.demo-felhom.eu strict-transport-security: max-age=86400 + +https://radarr.demo-felhom.eu strict-transport-security: max-age=86400 + +https://sonarr.demo-felhom.eu strict-transport-security: max-age=86400 + +https://time.enkisfelhom.hu strict-transport-security: max-age=86400 + +https://torrent.demo-felhom.eu strict-transport-security: max-age=86400 + +https://travel.enkisfelhom.hu strict-transport-security: max-age=86400 + +https://wiki.enkicsifelhom.hu strict-transport-security: max-age=86400 + +https://wiki.enkisfelhom.hu strict-transport-security: max-age=86400 + diff --git a/documentation/audits/release-1-dates-2026-11-14/controller/11-global-floor.txt b/documentation/audits/release-1-dates-2026-11-14/controller/11-global-floor.txt new file mode 100644 index 00000000..e5e96b00 --- /dev/null +++ b/documentation/audits/release-1-dates-2026-11-14/controller/11-global-floor.txt @@ -0,0 +1,10 @@ +== 2026-11-14T09:21:15Z global floor 0.352.1, MinAgent 0.158.1 (CC's hub token) +== /configuration/global-floor +HTTP/1.1 303 See Other +Location: /configuration?flash=floor_set +Global controller floor: 0.352.1 · vouched agent: 0.173.0 +2026/11/14 10:18:51 [INFO] managed floor SERVED for tester-1: floor 0.352.1, agent requirement "0.158.1" from declared (golden 0.352.0) +2026/11/14 10:18:51 [INFO] managed floor SERVED for demo-felhom: floor 0.352.1, agent requirement "0.158.1" from declared (golden 0.352.0) +2026/11/14 10:18:51 [INFO] managed floor SERVED for demo-hp: floor 0.352.1, agent requirement "0.158.1" from declared (golden 0.352.0) +2026/11/14 10:21:15 [INFO] Global controller-version floor set to "0.352.1" (declared MinAgent "0.158.1") +2026/11/14 10:21:15 [INFO] access token "cc-releases-2": POST /configuration/global-floor → 303 diff --git a/documentation/audits/release-1-dates-2026-11-14/controller/12-watch.txt b/documentation/audits/release-1-dates-2026-11-14/controller/12-watch.txt new file mode 100644 index 00000000..94098c50 --- /dev/null +++ b/documentation/audits/release-1-dates-2026-11-14/controller/12-watch.txt @@ -0,0 +1,52 @@ +== controller watch: global floor 09:21:15Z; read 2026-11-14T09:56:21Z +-- the three boxes (read on the box, over SSH) +### root@192.168.0.154 2026-11-14T09:56:21Z +containers=10 unhealthy=0 restarting=0 exited=[] +gitea.dooplex.hu/admin/felhom-controller:0.352.1 Up 37 minutes (healthy) +image file: gitea.dooplex.hu/admin/felhom-controller:0.352.1 +image digest: sha256:0d3c28fab7ae2d08e0c92c3c4318316d4920cd440c4c9e3fac4581896314543a repo: gitea.dooplex.hu/admin/felhom-controller@sha256:1a6e8945211a0f1995fd2bd0782053c7f33152fcb00ca465d0596d961a13b6bb +started=2026-11-14T09:18:58.892043895Z +go in the running binary: 2 go1.27.4 +errors=0 warn=0 info=787 panics=0 +2026/11/14 09:18:59 [INFO] felhom-controller 0.352.1 starting (customer: tester-1, domain: enkicsifelhom.hu) +2026/11/14 09:18:59 [INFO] [selfupdate] Post-update startup: update successful (0.352.0 → 0.352.1) +### demo-hp 2026-11-14T09:56:22Z +containers=26 unhealthy=0 restarting=0 exited=[] +gitea.dooplex.hu/admin/felhom-controller:0.352.1 Up 37 minutes (healthy) +image file: gitea.dooplex.hu/admin/felhom-controller:0.352.1 +image digest: sha256:0d3c28fab7ae2d08e0c92c3c4318316d4920cd440c4c9e3fac4581896314543a repo: gitea.dooplex.hu/admin/felhom-controller@sha256:1a6e8945211a0f1995fd2bd0782053c7f33152fcb00ca465d0596d961a13b6bb +started=2026-11-14T09:18:59.079611972Z +go in the running binary: 2 go1.27.4 +errors=0 warn=3 info=808 panics=0 +2026/11/14 09:18:59 main.go:335: [INFO] felhom-controller 0.352.1 starting (customer: demo-hp, domain: enkisfelhom.hu) +2026/11/14 09:19:00 deploy.go:1470: [WARN] [stacks] Stack bookstack: new field ADMIN_PASSWORD (type=password) requires manual configuration +2026/11/14 09:19:00 deploy.go:1470: [WARN] [stacks] Stack calibre-web: new field ADMIN_PASSWORD (type=password) requires manual configuration +2026/11/14 09:19:00 deploy.go:1470: [WARN] [stacks] Stack plex: new field PLEX_CLAIM (type=text) requires manual configuration +2026/11/14 09:19:00 updater.go:855: [INFO] [selfupdate] Post-update startup: update successful (0.352.0 → 0.352.1) +### felhom-pve 2026-11-14T09:56:24Z +containers=9 unhealthy=0 restarting=0 exited=[] +gitea.dooplex.hu/admin/felhom-controller:0.352.1 Up 37 minutes (healthy) +image file: gitea.dooplex.hu/admin/felhom-controller:0.352.1 +image digest: sha256:0d3c28fab7ae2d08e0c92c3c4318316d4920cd440c4c9e3fac4581896314543a repo: gitea.dooplex.hu/admin/felhom-controller@sha256:1a6e8945211a0f1995fd2bd0782053c7f33152fcb00ca465d0596d961a13b6bb +started=2026-11-14T09:18:23.620814926Z +go in the running binary: 2 go1.27.4 +errors=0 warn=0 info=761 panics=0 +2026/11/14 09:18:24 [INFO] felhom-controller 0.352.1 starting (customer: demo-felhom, domain: demo-felhom.eu) +2026/11/14 09:18:24 [INFO] [selfupdate] Post-update startup: update successful (0.352.0 → 0.352.1) +-- hub log since the floor: events, alarms, error lines +-- hub dashboard +Demo Ügyfél OK — 7 min ago | 0.352.1 +Demo HP OK — 7 min ago | 0.352.1 +Tester 1 OK — 7 min ago | 0.352.1 +-- positive control for the empty list above: the hub log of the window has 169 lines; by kind: + 126 [INFO] + 43 [DEBUG] +-- lines that name an event or a mail: +2026/11/14 10:18:57 [INFO] Notification preferences updated for demo-felhom: address set=true, household=true, events=[app_stopped_unhealthy app_update_held app_update_undone backup_failed db_dump_fai +2026/11/14 10:19:00 [INFO] Event from tester-1: controller_updated (info) — Controller frissítve: 0.352.0 → 0.352.1 +2026/11/14 10:19:00 [INFO] Notification prefs push for tester-1: household cleared its mail address — deleted +2026/11/14 10:19:00 [INFO] Notification preferences updated for tester-1: address set=false, household=true, events=[] +2026/11/14 10:19:00 [INFO] Event from demo-hp: controller_updated (info) — Controller frissítve: 0.352.0 → 0.352.1 +2026/11/14 10:19:02 [INFO] Event from demo-felhom: controller_started (info) — Controller elindult (0.352.1) +2026/11/14 10:19:05 [INFO] Event from tester-1: controller_started (info) — Controller elindult (0.352.1) +2026/11/14 10:19:05 [INFO] Event from demo-hp: controller_started (info) — Controller elindult (0.352.1) diff --git a/documentation/audits/release-1-dates-2026-11-14/controller/13-per-box-floors-cleared.txt b/documentation/audits/release-1-dates-2026-11-14/controller/13-per-box-floors-cleared.txt new file mode 100644 index 00000000..b4367b4d --- /dev/null +++ b/documentation/audits/release-1-dates-2026-11-14/controller/13-per-box-floors-cleared.txt @@ -0,0 +1,10 @@ +== 2026-11-14T09:56:38Z per-box floors cleared +== /customers/tester-1/floor +HTTP/1.1 303 See Other +Location: /customers/tester-1?flash=floor_set +== /customers/demo-hp/floor +HTTP/1.1 303 See Other +Location: /customers/demo-hp?flash=floor_set +== /customers/demo-felhom/floor +HTTP/1.1 303 See Other +Location: /customers/demo-felhom?flash=floor_set diff --git a/documentation/audits/release-1-dates-2026-11-14/hub/01-before-system-page.txt b/documentation/audits/release-1-dates-2026-11-14/hub/01-before-system-page.txt new file mode 100644 index 00000000..3e32c355 --- /dev/null +++ b/documentation/audits/release-1-dates-2026-11-14/hub/01-before-system-page.txt @@ -0,0 +1,3 @@ +== 2026-11-14T09:54:09Z BEFORE the deploy (hub 0.180.0): the System page's Root files cells and Needs-attention lines + 9 title="behind the vouched agent 0.173.0's bundle — send it with a signed agent_config_update (behind since 2026-11-10 16:14 UTC)" +3 diff --git a/documentation/audits/release-1-dates-2026-11-14/hub/02-deploy.txt b/documentation/audits/release-1-dates-2026-11-14/hub/02-deploy.txt new file mode 100644 index 00000000..0338bb95 --- /dev/null +++ b/documentation/audits/release-1-dates-2026-11-14/hub/02-deploy.txt @@ -0,0 +1,13 @@ +== 2026-11-14T09:58:11Z sync: Deployment/hub only, revision 63f69b99 +application.argoproj.io/felhom patched +Waiting for deployment "hub" rollout to finish: 0 of 1 updated replicas are available... +deployment "hub" successfully rolled out +sync=Synced rev=63f69b99ceff84d83d44144c91b73e11bdc152d0 health=Healthy op=Succeeded +gitea.dooplex.hu/admin/felhom-hub:0.181.0 +hub-dbfb69758-mzvtk gitea.dooplex.hu/admin/felhom-hub@sha256:be2e9e5676eec7622622402a830d8f92913a93fe746e421db60f949138d1f429 ready=true restarts=0 +2026/11/14 10:58:15 [INFO] felhom-hub 0.181.0 starting +2026/11/14 10:58:17 [INFO] Root files: bundles are compared by their files (R-1392) +2026/11/14 10:58:21 [WARN] slow statement: 2570ms (over 1s): DELETE FROM events WHERE created_at < ? +2026/11/14 10:58:25 [WARN] slow statement: 3549ms (over 1s): DELETE FROM app_log_issues WHERE last_seen < ? +2026/11/14 10:58:28 [WARN] slow statement: 3051ms (over 1s): SELECT r.customer_id, r.received_at, r.report_json, r.health_status, r.cpu_percent, r.memory_percent, r.container_total, r.container_running, r.backup_last_sna +2026/11/14 10:58:32 [WARN] slow statement: 3816ms (over 1s): SELECT hr.host_id, hr.customer_id, hr.report_json FROM host_reports hr JOIN (SELECT host_id, MAX(id) AS mx FROM host_reports GROUP BY host_id) latest ON hr.id diff --git a/documentation/audits/release-1-dates-2026-11-14/hub/03-after-system-page.txt b/documentation/audits/release-1-dates-2026-11-14/hub/03-after-system-page.txt new file mode 100644 index 00000000..9e4af634 --- /dev/null +++ b/documentation/audits/release-1-dates-2026-11-14/hub/03-after-system-page.txt @@ -0,0 +1,7 @@ +== 2026-11-14T09:59:20Z AFTER the deploy (hub 0.181.0): the System page +lines 'root files behind the vouched': 0 +cells titled 'behind the vouched agent': 0 + 6 title="the same files as the vouched agent 0.173.0's bundle (only the label differs)" +Needs attention + One line per box that is amber or red, and why. Amber: worth a look. Red: an operator alarm fires (`08` §6.3). + All boxes look fine. diff --git a/documentation/audits/release-1-dates-2026-11-14/hub/04-behind-since-marks.txt b/documentation/audits/release-1-dates-2026-11-14/hub/04-behind-since-marks.txt new file mode 100644 index 00000000..2b9b3ff5 --- /dev/null +++ b/documentation/audits/release-1-dates-2026-11-14/hub/04-behind-since-marks.txt @@ -0,0 +1,17 @@ +== hub DB copy with its WAL, read 2026-11-14T10:00:21Z +freshness: quick_check ok | newest host report 2026-11-14 09:59:52 | db now 2026-11-14 10:00:22 +hub_settings columns ['key', 'value', 'updated_at'] +-- bundle_behind_since marks (empty value = not behind): + ('bundle_behind_since:Tester4-16fdef', '2026-10-23T08:05:18Z', '2026-10-23 08:05:20') + ('bundle_behind_since:Tester4-84bbe7', '2026-10-25T22:55:53Z', '2026-10-25 22:55:54') + ('bundle_behind_since:demo-felhom-8363b5', '', '2026-11-14 09:59:20') + ('bundle_behind_since:demo-hp-bb76ea', '', '2026-11-14 09:59:20') + ('bundle_behind_since:tester-1-d70be4', '', '2026-11-14 09:59:20') +-- control, same table: agent_behind_since marks and the floor: + ('agent_behind_since:Tester-2-be8404', '2026-10-05T09:15:29Z', '2026-10-05 09:15:30') + ('agent_behind_since:Tester4-16fdef', '2026-10-23T08:05:18Z', '2026-10-23 08:05:20') + ('agent_behind_since:demo-felhom-8363b5', '', '2026-10-14 12:43:27') + ('agent_behind_since:tester-1-d70be4', '', '2026-10-14 12:43:27') + ('min_controller_version', '0.352.1', '2026-11-14 09:21:15') + ('min_controller_version_declared_min_agent', '0.352.1=0.158.1', '2026-11-14 09:21:15') +-- os_config_bundle_behind events ever: (4, '2026-11-06 10:53:20') diff --git a/documentation/audits/release-1-dates-2026-11-14/hub/05-watch.txt b/documentation/audits/release-1-dates-2026-11-14/hub/05-watch.txt new file mode 100644 index 00000000..55360e41 --- /dev/null +++ b/documentation/audits/release-1-dates-2026-11-14/hub/05-watch.txt @@ -0,0 +1,55 @@ +== hub watch: 0.181.0 live since 09:58:15Z; read 2026-11-14T10:26:37Z +sync=Synced health=Healthy +hub-dbfb69758-mzvtk ready=true restarts=0 +log lines since the start: 244; by kind: + 181 [INFO] + 38 [DEBUG] + 24 [WARN] + 1 [offsite] re-issued shared credentials for Tester4 (subaccount 343429) +-- every [ERROR] and [WARN] line: +2026/11/14 10:58:21 [WARN] slow statement: 2570ms (over 1s): DELETE FROM events WHERE created_at < ? +2026/11/14 10:58:25 [WARN] slow statement: 3549ms (over 1s): DELETE FROM app_log_issues WHERE last_seen < ? +2026/11/14 10:58:28 [WARN] slow statement: 3051ms (over 1s): SELECT r.customer_id, r.received_at, r.report_json, r.health_status, r.cpu_percent, r.memory_percent, r.container_total, r.container_running, r.backup_last_snapshot, r.c +2026/11/14 10:58:32 [WARN] slow statement: 3816ms (over 1s): SELECT hr.host_id, hr.customer_id, hr.report_json FROM host_reports hr JOIN (SELECT host_id, MAX(id) AS mx FROM host_reports GROUP BY host_id) latest ON hr.id = latest.m +2026/11/14 10:59:11 [WARN] slow statement: 4053ms (over 1s): SELECT id, host_id, layer, received_at, trigger, mode, outcome, healthy, release_id, report_json FROM os_reports WHERE host_id = ? AND layer = ? ORDER BY id DESC LIMIT ? +2026/11/14 10:59:15 [WARN] slow statement: 3888ms (over 1s): SELECT id, host_id, layer, received_at, trigger, mode, outcome, healthy, release_id, report_json FROM os_reports WHERE host_id = ? AND layer = ? ORDER BY id DESC LIMIT ? +2026/11/14 10:59:19 [WARN] slow statement: 1152ms (over 1s): SELECT id, host_id, layer, received_at, trigger, mode, outcome, healthy, release_id, report_json FROM os_reports WHERE host_id = ? AND layer = ? ORDER BY id DESC LIMIT ? +2026/11/14 10:59:19 [WARN] slow statement: 3911ms (over 1s): SELECT id, host_id, layer, received_at, trigger, mode, outcome, healthy, release_id, report_json FROM os_reports WHERE host_id = ? AND layer = ? ORDER BY id DESC LIMIT ? +2026/11/14 10:59:20 [WARN] slow request: GET /system 200 12656ms (over 5s) +2026/11/14 11:00:05 [WARN] slow statement: 32362ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC +2026/11/14 11:00:18 [WARN] slow statement: 12036ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC +2026/11/14 11:01:39 [WARN] slow statement: 7245ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC +2026/11/14 11:01:43 [WARN] slow statement: 2425ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC +2026/11/14 11:01:47 [WARN] slow statement: 2490ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC +2026/11/14 11:02:37 [WARN] slow statement: 4590ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC +2026/11/14 11:02:39 [WARN] slow statement: 1045ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC +2026/11/14 11:02:42 [WARN] slow statement: 1355ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC +2026/11/14 11:03:02 [WARN] slow statement: 1857ms (over 1s): SELECT id, customer_id, event_type, severity, message, details_json, source, created_at FROM events WHERE customer_id = ? AND event_type IN ('wg_tunnel_stale', 'wg_tunne +2026/11/14 11:03:35 [WARN] slow statement: 2580ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC +2026/11/14 11:03:37 [WARN] slow statement: 1254ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC +2026/11/14 11:03:39 [WARN] slow statement: 1233ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC +2026/11/14 11:06:33 [WARN] slow statement: 1407ms (over 1s): SELECT id, customer_id, event_type, severity, message, details_json, source, created_at FROM events WHERE customer_id = ? AND event_type IN ('wg_tunnel_stale', 'wg_tunne +2026/11/14 11:12:13 [WARN] slow request: POST /api/v1/host-enroll 201 11357ms (over 5s) +2026/11/14 11:12:34 [WARN] escrow CARRIED for Tester4: the sealed package of deleted host Tester4-d35551 is now the current package of Tester4-46836f (repo-key hash 5501e4bf2cda…); the box will ask for the recovery code +-- events and mails: +2026/11/14 10:58:18 [INFO] osupdates: alarms = no OS leg 168h0m0s, reboot needed 336h0m0s, ring 0 stalled 168h0m0s, not covered 336h0m0s +2026/11/14 10:58:28 [INFO] Host staleness checker initialized: 3 ok, 0 already alarmed (seeded silently, R-1001), 0 stale, 0 down (left unseeded → first Check emits; host_stale after 45m0s, host_dow +2026/11/14 10:59:20 [INFO] osupdates: alarm check done in 1.847s, 0 alarm(s) sent [] +2026/11/14 11:12:02 [INFO] claim reenroll email sent to the registered address of Tester4 +2026/11/14 11:13:51 [INFO] Event from Tester4: controller_started (info) — Controller elindult (0.352.1) +2026/11/14 11:14:11 [INFO] Event from Tester4: controller_started (info) — Controller elindult (0.352.1) +2026/11/14 11:15:37 [INFO] Golden vouch check: golden 0.352.1 is published, the hub vouches "0.352.0" — alarm in 2h0m0s if still so +2026/11/14 11:19:06 [INFO] Event from Tester4: backup_tier_skipped (warning) — Whole-guest backup tier felhom-pbs skipped: its storage does not exist on the host (never provisioned or removed). No a +2026/11/14 11:22:07 [INFO] Event from Tester4: app_deploy_started (info) — Alkalmazás telepítése elindult: BentoPDF +2026/11/14 11:23:49 [INFO] Event from Tester4: app_deployed (info) — Alkalmazás telepítve: BentoPDF +== second read 2026-11-14T10:28:03Z (12 seconds BEFORE the 30 minutes after 09:58:15Z were over — see the third read) +sync=Synced health=Healthy +hub-dbfb69758-mzvtk ready=true restarts=0 +[ERROR] lines since the start: 0 panic: 0 slow statements in the last 20 min: 0 +-- events of the three test boxes and operator mails since the deploy: +(end of list) +== third read 2026-11-14T10:29:35Z +sync=Synced health=Healthy +hub-dbfb69758-mzvtk ready=true restarts=0 +lines 251 | [ERROR] 0 | panic 0 | slow statements after 10:10Z: 0 +operator mailbox (label Monitoring, a second channel), read 10:28Z: no alarm mail since 09:18Z; the one new mail is Tester4's setup code of 10:12:03Z (this run's install) diff --git a/documentation/backlog/CLOSED-ITEMS.md b/documentation/backlog/CLOSED-ITEMS.md index 73c803da..312ef8da 100644 --- a/documentation/backlog/CLOSED-ITEMS.md +++ b/documentation/backlog/CLOSED-ITEMS.md @@ -26,6 +26,14 @@ --- +## 2026-11-14 — R-1392 closed: root files are „behind” only when the files differ (full text of the row: `git show 561a4d98:documentation/backlog/OPEN-ITEMS.md`) + +| Row | What | Closed | Evidence | +|---|---|---|---| +| **R-1392** | **Every agent release whose `configs/` did not change turned every box's „Root files” amber, and after 7 days the hub mailed `os_config_bundle_behind`, although the files on the box were the vouched ones.** (P3) | CLOSED 2026-11-14 — hub 0.181.0 (D81 = yes, `09` §3 decision 304): the hub fetches both bundles, accepts each only by a sha256 it already trusts, and compares everything but the label. Live since 09:58 UTC: the System page reads „All boxes look fine”; the three „behind since 2026-11-10 16:14” marks were cleared at 09:59:20 UTC (read from the hub database with its WAL), so no mail was due on 2026-11-17. The stopgap („send the bundle”) was not needed | `audits/release-1-dates-2026-11-14/hub/`; `11-os-updates.md` §5.4.2; hub `83316fc9` | + +**Reasoning kept.** Anything the hub cannot read, verify or parse answers „not the same” — the old verdict. A registry that is away never makes a box look current. + ## 2026-11-07 — R-1228 closed: DooPlex's copy of the website retired (full text of the row: `git show 7e56d166:documentation/backlog/OPEN-ITEMS.md`) | Row | What | Closed | Evidence | diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 084fd551..9264b351 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -268,12 +268,12 @@ stopping line that lies. | **R-779** | Security & access | P4 | **[P3-LOW] Part A's "two outside addresses seen as two" is proven through the simulated tunnel only; on the REAL tunnel the second outside address (ep0, one request allowed) was refused by Cloudflare's edge with 403 and never reached the box.** Measured 2026-10-01 19:51 UTC (`audits/visitors-2026-10-01/A/L2-demo-hp-real-tunnel.txt`): no log line on demo-hp; demo-hp's box has no geo restriction in its settings, so a Cloudflare ZONE rule (country or bot, not read) refused a German datacenter address. DooPlex's own address on the real tunnel was seen as itself. **Needs:** one sign-in from a second Hungarian address (the operator's phone off wifi) while DooPlex is locked out — 2 minutes; and say which Cloudflare rule refused ep0. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (a phone), CC reads the logs** **Re-ranked 2026-10-03: P3→P4: a proof gap on the real tunnel; operator-only follow-up.** | — | — | CC + operator | | **R-904** | Security & access | P4 | **Cloudflare can read every household's app traffic; replacing it with our own relay is a later item.** Facts (reviewer discussion 2026-10-08; `01`): app traffic and the dashboard reach the box through the Cloudflare Tunnel (`01` §5 trust table, rows end-user ↔ apps and customer ↔ controller UI; §7), and the tunnel's public end is Cloudflare's edge, where TLS ends — so Cloudflare can technically read that traffic (the FAQ says so since 2026-10-08, R-900; „TLS ends at the edge" is not written in `01` — add it there). What Cloudflare gives today, free: inbound reach with no router setup, the CGNAT answer (`01` §4, §7); certificates (`01` §7, the free tier covers one level below a zone); the geo-WAF the hub enforces (`01` §5 last row, §7); flood protection (not in `01`). The alternative named: our own EU relay over WireGuard with TLS passthrough by SNI, certificates on the box, the geo-block on the relay. Its costs: one more machine the operator keeps up, and a single point of reach for every box; weaker flood protection; about a week of work after a spike. Operator ruling 2026-10-08 09:07 (`09` §3 decision 184): a later item. | **DEFERRED** — **2026-10-10: the asked sentence is in `01` §7** („TLS ends at Cloudflare's edge …”); the spike stays deferred. **DEFERRED — after the first customers (operator ruling 2026-10-08 09:07)** | — | A spike after the first customers (the relay's reach, cost and flood behaviour, measured) | operator | | **R-1040** | Security & access | P4 | **Until the household makes Sonarr's or Radarr's first admin, any container on the box can read the app's API key.** Measured on the bench 2026-10-14 with the R-1030 seed in place: `AuthenticationRequired=Enabled`, `AuthenticationMethod=None` — `/initialize.json` from another container answers 200 with `apiKey`. Outsiders are blocked by the box's setup gate in that window; a compromised other app is not. | **OPEN** — filed 2026-10-14 | — | Seed the API key at install (`SONARR__AUTH__APIKEY`, measured to work) so the box knows it, and close the window by telling the household in the first steps; or seed a Forms user | CC | -| **R-1050** | Security & access | P4 | **Raise HSTS to one year on or after 2026-11-15, if no https trouble was reported.** `09` §3 decision 249 (D2 = A, operator 2026-10-15): HSTS stays `max-age=86400` now; after a month with no https trouble it becomes one year (`31536000`); never preload, never `includeSubDomains`. The value is `HSTSMaxAgeSeconds` in `felhom-controller/controller/internal/infra/infra.go`, pinned by `TestRenderForwardedHeaders_HTTPSOnly`. „Trouble” = any report of a name that will not open, or a box answering http after the floor. | **WATCHING** — filed 2026-10-15 | the date | On or after 2026-11-15: read the register and the operator's mail for https trouble since 2026-10-15; none → a controller release with one year (test updated), bake → vouch → floor | CC | +| **R-1050** | Security & access | P4 | **Raise HSTS to one year on or after 2026-11-15, if no https trouble was reported.** `09` §3 decision 249 (D2 = A, operator 2026-10-15): HSTS stays `max-age=86400` now; after a month with no https trouble it becomes one year (`31536000`); never preload, never `includeSubDomains`. The value is `HSTSMaxAgeSeconds` in `felhom-controller/controller/internal/infra/infra.go`, pinned by `TestRenderForwardedHeaders_HTTPSOnly`. „Trouble” = any report of a name that will not open, or a box answering http after the floor. | WATCHING — **2026-11-14: NOT in controller 0.352.1.** That release was built on 2026-11-14, one day before the ruled date, and the operator's words „one day early is OK” were not given. Check date moved to 2026-11-21 (one week: the next day a controller release is likely; a one-day move would only turn the nightly CI red with nothing to release). The search for https trouble since 2026-10-15 is NOT done yet. Earlier: **WATCHING** — filed 2026-10-15 | the date | On or after 2026-11-15: read the register and the operator's mail for https trouble since 2026-10-15; none → a controller release with one year (test updated), bake → vouch → floor | CC | | **R-1054** | Security & access | P3 | **The hub's operator/global API key is still one bearer that can do everything on `/api/v1`, including reading every box's break-glass root password (`GET /admin/hosts/{id}/recovery-credential`).** Hub v0.152.0 gave the web surface scoped tokens (`05` §17) but tokens are refused on `/api/v1` by design. Who uses the global key today is not listed in one place. Fix: list its consumers; move each operator use to a scoped token on the web side (or a token class on the API), and keep the global key only where a box path needs it — or retire it. | OPEN — filed 2026-10-15 by the token build | — | List the global key's consumers (pod env, scripts, runbooks) | CC | | **R-1194** | Security & access | P3 | **On a fresh box the operator's break-glass SSH door was broken from the first minute: `oob_degraded` „felhom-sshd config invalid (sshd -t fails)".** Rehearsal 2026-10-20, Tester4-816e21, mail at 12:03 UTC (one minute after the host record was created): details `active:false, config_invalid:true, felhom_sshd_port:0, operator_key_configured:false, operator_key_reported:true, reachable:false`. Possibly the shape of a customer with no operator key set (Tester4 is a test customer) — then the alarm is a false one on every such box; or a real config defect. Not diagnosed (no shell). Checked the same day: the only `oob_degraded` in the operator mailbox in 30 days, and no recovery mail followed; the report says port 0. Two candidate mechanisms in the agent (`internal/felhomsshd/health.go:63-66`): `sshd -t -f ` run by the NON-root agent fails on unreadable host keys, or a config rendered before the desired state arrived (port 0). **CAUSE FOUND AND FIXED 2026-10-21 night** (`audits/night-2026-10-21/P5-checks/`): reproduced on a fresh box from the public ISO — `oob_degraded` mailed 20:34 CEST while felhom-sshd was applied 0.4 s after the check and is healthy (port 8822 listening, `sshd -t` passes as the agent's user through its sudo grant — so the non-root theory is FALSE; without sudo it fails on the host key, which the agent never does). The first host report runs before the felhom-sshd loop's first pass and reads a config never rendered. Fix: agent `0da792e` — no OOB stanza until the first pass has run (the hub does not evaluate a missing one; a failed pass counts). `TestR1194_NoOOBStanzaBeforeTheFirstPass` red-proved. | WATCHING — released in agent v0.162.0 (2026-10-22, operator yes; on the 3 test boxes); the false alarm comes only on a FRESH box's first report | the next fresh install with agent ≥ 0.162.0 | At the next fresh install read the hub's events for `oob_degraded` in the first hour (none expected); then close | CC | | **R-1237** | Security & access | P3 | **The nightly CC-secrets sweep found stored one-time codes in a session record dated 2026-10-23: 14 names in one file (bind tokens and setup codes of test boxes — NIGHT_2026_10_23_*, FIXES_2026_10_21_*, NIGHT_2026_10_21_*, REHEARSAL_2026_10_20_* — and R_DEMO_FELHOM in a file of 2026-10-20).** Mail „[DooPlex CC secrets] 2 secret name(s) found in old session records", 2026-10-23 01:57 UTC, read in the day session (names only). One file holding all of them suggests one command printed many stored values at once (a listing or a dump) in the night session. All but R_DEMO_FELHOM are one-time install codes of test boxes; NIGHT_2026_10_23_* die when Tester4-16fdef is deleted (the operator's do list). | **OPEN** — 2026-10-27 (operator yes): the dead names were REMOVED from the store with `sec rm` (46, one commit each; `audits/r1299-window-button-2026-10-27/sec-rm.txt`, names only): every one-time code this row names is gone; R_DEMO_FELHOM stays (unsure). Consequence: the nightly sweep no longer watches for the removed values — they open nothing (their boxes are destroyed). The session transcripts themselves are unchanged — that question stays here. **Earlier:** OPEN | — | CC: find the file and the command by the sweep's own report (names only), add a hook shape for that command if it is not one; R_DEMO_FELHOM (demo-felhom's recovery code) — the operator decides whether to rotate it | CC | | **R-1248** | Security & access | P3 | **An app's own numbers can lie: an attacker with code inside an app's container can answer the app's API with whatever matches its flood, and the upload watch (R-1244) would then call it explained.** Measured remedy, not built: a per-container upload ceiling with `tc` from a helper container in the app's network namespace — on scratch 9202 (unprivileged) a flood was held at 19.9–20.1 Mbit/s and the app kept answering (`audits/SPIKE-upload-watch-2026-10-23.md` §5). Costs: a helper image with `tc` (a new external image, or `tc` in the controller image), `NET_ADMIN` for that helper, re-applied after each app restart. | OPEN | an operator decision (the new image / capability) | Decide; if yes, CC builds the ceiling for the download and media apps first | operator | -| **R-1391** | Security & access | P3 | **The hub, the controller and the agent are built on Go 1.27.3, which has GO-2026-6859 (an HTTP/1 server can crash with a runtime panic when a `MaxBytesReader` limit is hit while the handler reads the body and writes a header at the same time); fixed in Go 1.27.4.** Found 2026-11-14 (health-check session) from the security watch's 2026-11-11 mail, which names only the hub image (`golang 1.27.3-alpine3.24`); `go.mod` in all three repos reads `go 1.27.3`. Reach not measured: the crash needs a handler that reads the request body concurrently with writing the response header — whether any Felhom handler does is not read yet. A crash restarts the process (the controller's supervisor, the agent's unit, the hub's pod), so the effect is an outage of seconds, not data loss. Source: https://vuln.go.dev/ID/GO-2026-6859.json | OPEN — filed 2026-11-14 | — | build each component's next release on Go 1.27.4 (`go.mod` + the build image); until then, grep each repo for `MaxBytesReader`/`MaxBytesHandler` and a handler that writes headers while a goroutine reads the body | CC | +| **R-1391** | Security & access | P3 | **The hub, the controller and the agent are built on Go 1.27.3, which has GO-2026-6859 (an HTTP/1 server can crash with a runtime panic when a `MaxBytesReader` limit is hit while the handler reads the body and writes a header at the same time); fixed in Go 1.27.4.** Found 2026-11-14 (health-check session) from the security watch's 2026-11-11 mail, which names only the hub image (`golang 1.27.3-alpine3.24`); `go.mod` in all three repos reads `go 1.27.3`. Reach not measured: the crash needs a handler that reads the request body concurrently with writing the response header — whether any Felhom handler does is not read yet. A crash restarts the process (the controller's supervisor, the agent's unit, the hub's pod), so the effect is an outage of seconds, not data loss. Source: https://vuln.go.dev/ID/GO-2026-6859.json | NARROWED — **2026-11-14: the controller (0.352.1, global floor 09:21 UTC, on every box that reports) and the hub (0.181.0, live 09:58 UTC) are built on Go 1.27.4. LEFT: the agent (still 1.27.3), release 2.** Reach read the same day: no controller handler matches the crash pattern (seven `MaxBytesReader` places, each reads the body to its end before it answers; no `TimeoutHandler`); the hub has no `MaxBytesReader` at all; the agent has ONE (`internal/localapi/server.go`, a JSON decode in the handler's own goroutine — not read further). So the agent's turn is hygiene, not urgent. Evidence `audits/release-1-dates-2026-11-14/`. Earlier: OPEN — filed 2026-11-14 | — | build each component's next release on Go 1.27.4 (`go.mod` + the build image); until then, grep each repo for `MaxBytesReader`/`MaxBytesHandler` and a handler that writes headers while a goroutine reads the body | CC | | **R-1309** | Security & access | P4 | **What no automated source shows: a fix announced only by mail, forum or a blog we do not read; a security fix nobody calls one (cloudflared 2026.10.2's „bump distroless image"); closed-source apps (Plex, Emby).** The watch (R-1303) shows these only as BEHIND (weekly). The vendor mails the operator receives stay a source. `audits/security-watch-2026-10-26/SPIKE.md` | OPEN — filed 2026-10-26 (a stated limit) | — | none until a vendor offers a feed; add a vendor's feed to `security_watch.FEEDS` when one exists | CC | ## Box system & updates — 21 rows (P3 16, P4 5) @@ -292,7 +292,7 @@ stopping line that lies. | **R-1214** | Box system & updates | P3 | **On the 8 GB box with the swap fix, the night's second-drive backup still pushes memory „full" pressure above 20 % for about 3 minutes; the box keeps answering.** Night soak 2026-10-21→22 (`audits/night-2026-10-21/P1-soak/03-rec.jsonl`, 10-s samples; fresh box from the public ISO, agent with swap = RAM, 8 apps, the rehearsal uploads for 2 h): at 01:30–01:34 UTC (the 03:30 tier2 job) host 39 % / guest 45 % over 10 s, over 60 s at most 20 % / 25 %; IO „full" 66 %; guest swap 1.35 → 2.1 GB; `/api/health` answered every probe (slowest 4.1 s, largest interval 16 s); 0 OOM kills. Once more in the upload window (Immich loading its models, 31 % / 35 % for one sample). The brief's soak pass line was „full" < 20 %, so the soak missed that one line; the freeze itself (96 % for an hour, R-1192) did not come back. Not diagnosed: which part of the tier2 job (the copy's page cache, the volume dumps) causes it. | WATCHING — **2026-10-24 02:4xZ dated check read (weekend-night session, read only):** Tester 1 (8 GB host, guest 4 GB, controller 0.341.0) ran its 03:30 copy at 01:30:00–01:30:13Z (3 apps, 157 MB + 86 MB + 16 KB; no „capped copy container did not start" WARN, no leftover `felhom-tier2-copy-*`); the guest cgroup's memory „full" stall counter on the host (`/sys/fs/cgroup/lxc/9201/memory.pressure`) read **total 444 µs** over the whole 11.1 h since the host started (15:32Z), so „full" stayed far under 20 % (20 % of 10 s = 2 s); 0 OOM kills. **Not shown:** that the copy ran in the capped container (the INFO log does not name it; Docker keeps only ~200 events, from 02:36Z). **Weak load:** 0.24 GB, not the soak's large copy — so the row stays WATCHING; close after a night whose copy moves ≥ 1 GB, read from the same counter. WATCHING — **RELEASED 2026-10-23 in controller 0.340.0** (floor 06:41 UTC; on all three test boxes, start-up „no leftover”, 0 copy containers; Tester 1's manual second-drive run: 3 apps, 0 containers left). **Open only for the consequence:** dated check 2026-10-24 — Tester 1's first night copy ran in the capped container and memory „full” pressure stayed under 20 % (the box's own night log). READY — rides controller 0.340.0. READY — waits for the controller 0.340.0 release. **2026-10-22 night — cause measured, fix built and proven on 9202, waits for the controller release.** Cause: the copy's page cache is charged to the guest and evicts the apps' working set. On 9202 (4 GB RAM, 1.8 GB held by a hog, a 6 GB copy): plain rsync full stall 5 702 / 2 464 ms, peak „full” 16.8 / 6.9 %; the same copy in a 512 MB memory-capped container 374 / 320 ms, 0.9 / 1.0 %. Controller main `03c0a69` + `ae39467` (unreleased, rides 0.340.0): the copy runs capped only for clean `/mnt/` paths with no symlink (security review); a docker error (exit 125) falls back to plain rsync. A real `POST /api/backup/tier2` on rc.4: 6 capped containers, each exit 0; Immich's copy equals its source (dry-run 0 differences, 44 = 44 files). Evidence `audits/night-2026-10-22/P4-R1214/`. **2026-10-23 night (stray-copy fix, night brief Part 1).** The capped copy's container outlived the copy (a timeout or a controller restart killed only the `docker` CLI; `--delete` went on outside the R-1180 guard). Controller main `66a69d4` + `912e592` (unreleased, rides 0.340.0): a fixed name + label per copy, removed by name on any error, every leftover removed at start before the scheduler and the web server, a restore refused while a copy runs or may run; `--mount` (a path the daemon does not see is refused, not created empty) and the image ID with `--pull never` (review C4, C7). Proven on 9202 with rc.5 (`audits/night-2026-10-23/P1-straycopy/`): a restart in the middle of a 16 GB copy → the new controller removed `felhom-tier2-copy-immich-948f2280` at start, the restore probe was refused during the copy, the redone copy equals the source; a 15 s test build → the timed-out copy's container removed, the partial copy stopped growing. A second review of that code (N2–N5) and the commit's own security review (two races) → `9d9e404`, `1babfe9`: a stray clears when a later removal works, the start-up check never removes this process's own copy and holds the destination's lock, one copy per destination at a time, `RunTier2` refuses during a restore for every caller and records itself in one step with that check, every leg asks again. Re-proven on 9202 with rc.8 (`audits/night-2026-10-23/P1-straycopy/1*`). 9202 back on 0.339.0. | — | Release 0.340.0; then re-measure the 03:30 job on the 8 GB soak box and close if „full” stays low | CC | | **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a new install from the public download replaces the bootstrap and the off-site check; `pilot/TESTER2-handover.md`).** | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator | | **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC | -| **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. **2026-11-11:** the End-dates list leans on this row (`documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB21); the newest golden passed its 7-day limit on 2026-11-10 and no waiver is in force, so the gate refuses code pushes to `felhom.eu`. **2026-11-11: golden 0.352.0 baked and published; one fresh box installed from it (`documentation/tests/golden-0.352.0-2026-11-11/README.md`); the gate is green (`python3 scripts/golden_currency_gate.py`, rc 0). The operator vouched it on 2026-11-11 at 14:48:11 CET.** | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement **2026-11-03: the Sunday 2026-11-01 bake was missed (it was in no brief): golden 0.344.0 is 8 days old, the floor 0.349.0; a CODE push to felhom.eu is refused until a bake and the vouch or a dated waiver. The night's close-out was documents only, which the gate exempts; no waiver written. First on the morning list.** | — | — | CC | +| **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. **2026-11-11:** the End-dates list leans on this row (`documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB21); the newest golden passed its 7-day limit on 2026-11-10 and no waiver is in force, so the gate refuses code pushes to `felhom.eu`. **2026-11-11: golden 0.352.0 baked and published; one fresh box installed from it (`documentation/tests/golden-0.352.0-2026-11-11/README.md`); the gate is green (`python3 scripts/golden_currency_gate.py`, rc 0). The operator vouched it on 2026-11-11 at 14:48:11 CET.** | WATCHING — **2026-11-14: golden 0.352.1 baked (sha256 `4f411cbe…346d`), one fresh install proven on a nested VM on demo-hp, vouched by the operator 10:25:43 UTC (`documentation/tests/golden-0.352.1-2026-11-14/`). The gate judges the bake's age: the next bake is due by 2026-11-21.** Earlier: **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement **2026-11-03: the Sunday 2026-11-01 bake was missed (it was in no brief): golden 0.344.0 is 8 days old, the floor 0.349.0; a CODE push to felhom.eu is refused until a bake and the vouch or a dated waiver. The night's close-out was documents only, which the gate exempts; no waiver written. First on the morning list.** | — | — | CC | | **R-1358** | Box system & updates | P3 | **cloudflared 2026.10.4 misses two dependency fixes (VULN-183456 chi, 2026.10.5; VULN-184686 go-jose, 2026.10.6; the newest is 2026.10.7).** No public text on reach. Not pinned tonight: the tunnel is a box's only door, and 9202 has none to prove it on. | OPEN — filed 2026-11-03 | — | raise `CloudflaredImage` on the next controller release with a tunnel proof on Tester 1 | CC | | **R-1380** | Box system & updates | P3 | **Proxmox renews its own certificate and the agent's pin does not follow: from that day the agent cannot control or back up a guest (5 answers).** Found 2026-11-11 (the End-dates list, class T2, `documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB02-3 to DB02-7). MEASURED: the agent trusts one exact Proxmox certificate (a pin), and the certificate on the three test boxes ends about two years after install — the N100's about 2028-07-16, then demo-hp and Tester 1 within three months. Proxmox makes a new certificate by itself about 14 days before the old one ends (so about 2028-07-02 on the N100). No code path gives the agent the new fingerprint, no alarm names the cause, no runbook holds the hand step, and nobody has tried it. Every box installed in the same season fails in the same fortnight. P3 only because the first date is in 2028; it is certain, not a risk. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 297). Not built.** **2026-11-11 (afternoon): tried on demo-hp (`documentation/audits/certpin-spike-2026-11-11/README.md`).** A forced new certificate (`pvecm updatecerts --force`, then a reload of the web service; Proxmox's own renewal does that reload by itself) blinds the agent at once: `TLS pin mismatch`, no reconcile, no host report to the hub, 502 to the controller; guests and apps stay up. One operator mail names a symptom (`agent_channel_unknown`), none names the cause; the hub's Hosts page still read ONLINE eleven minutes after the last report (when it turns offline was not measured). The hand repair works and is written down (`documentation/runbooks/RUNBOOK-agent-proxmox-pin-repair.md`, 2.1 s from restart to a working call). **A CA file in place of the fingerprint works with no agent code:** with `proxmox.tls.ca_file` on a readable copy of the box's root CA and the fingerprint empty, the agent went through a second new certificate with no hand step; a wrong CA file was refused. Two facts for the fix: the agent's user cannot read Proxmox's certificate files, and a set fingerprint wins over the CA file. The design with both ways is `03-host-agent.md` §12a; CC's pick differs from the ruled shape, so it is decision D80. | READY — **2026-11-14 09:55 — RULED (D80, `09` §3 decision 303): the root CA copy.** Built in release 2 (the host release). OPEN — filed 2026-11-11 (end-dates list) | — | the operator rules D80 (CA file, or follow the fingerprint); then the design's open parts in `03-host-agent.md` §12a (the root step that refreshes the copy; an alarm that names the cause), then build, then prove on a scratch host with a forced renewal | CC | | **R-1382** | Box system & updates | P3 | **A box that was off past an end date: nobody knows if it comes back by itself (5 answers).** Found 2026-11-11 (the End-dates list, class T4, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; read in source and documents, never tried). After months without power: the web certificate has ended (DB01-6: does Traefik renew at the first start, and what does the household see until then), Debian's package keys may be too old to verify an update (DB16-6), the tunnel client may be below what Cloudflare accepts (DB19-6), a 30-day restore copy is past its date (DB24-6), and the 14-day countdown after a stopped off-site copy ran out while the box was off (DB04-6). Tester 2 has been off since 2026-10-04 and is the natural test. | OPEN — filed 2026-11-11 (end-dates list) | — | When Tester 2 comes back (or with a scratch host and a moved clock on Tier 0, as a task): record what renews by itself and what needs a hand; put the hand steps in a runbook. | CC | @@ -304,7 +304,7 @@ stopping line that lies. | **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — 2026-10-27: agent v0.167.0 RELEASED; in the NORMAL signed delivery the pass now runs right after the commit on all three boxes („running the bundle policy pass after the self-update commit (R-1285)” → `BUNDLE POLICY DONE install_mode=appliance removed=0`; `audits/release-2026-10-27/agent-0167-tester1-demofelhom.txt`). Not live: a BYO box coming FROM 0.164.0 (none in the fleet). **Earlier:** **VERIFY** — 2026-10-26: agent `b1bc63d` (unreleased) — the new agent runs the bundle-policy pass once right after its self-update commits (`selfupdate.SetOnCommitted`); test in the real order (commit, then one pass), red-proved. **Not live:** only a box coming FROM 0.164.0 discriminates (0.165.0's wrapper already applies D43), which needs a signed downgrade chain on a BYO box — the live proof rides the next agent release. **Earlier:** **OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:** **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC | -## Monitoring & notifications — 29 rows (P2 3, P3 18, P4 8) +## Monitoring & notifications — 28 rows (P2 3, P3 17, P4 8) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -335,10 +335,9 @@ stopping line that lies. | **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC | | **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC | | **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC | -| **R-1392** | Monitoring & notifications | P3 | **Every agent release whose `configs/` did not change turns every box's „Root files” amber, and after 7 days the hub mails `os_config_bundle_behind` for each box, although the files on the box are the vouched ones.** Found 2026-11-14 (health-check session): the System page shows „root files behind the vouched agent's” on demo-felhom, demo-hp and Tester 1 (bundle 0.172.1, sha `3e7acaee…`); the vouched agent 0.173.0 carries bundle sha `51ebc1f4…`; `git diff v0.172.1 v0.173.0 -- configs/` is EMPTY, and the 0.173.0 release delivered no bundle job on purpose („`configs/` did not change”, felhom-agent CHANGELOG). The hub compares the WHOLE bundle's sha256 (`hub/internal/web/system.go` `bundleCell`, `hub/internal/osupdates/service.go` ~line 1182), and the bundle carries its version, so the sha moves with every release. Vouched 2026-11-10 16:29 CET → the alarm is due about 2026-11-17 on all three boxes. **Two ways out:** (A) the release sends a signed `agent_config_update` every time, even with no file change (cheap, one job per box, keeps the hub as is); (B) the hub compares the files (the box reports `config_bundle.live` per-file sha256s) with the vouched bundle's files, not the bundle's own sha. | READY — **2026-11-14 09:55 — RULED (D81, `09` §3 decision 304): yes, the hub must not call vouched files „behind”; the stopgap „send the bundle” is allowed for the three test boxes.** OPEN — filed 2026-11-14 | operator: A or B | today: send the 0.173.0 bundle to the three boxes (clears the amber and the 11-17 alarm); then A in `felhom-build-deploy` or B in the hub | CC | | **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC | -## Hub & operator — 17 rows (P2 1, P3 5, P4 11) +## Hub & operator — 18 rows (P2 1, P3 5, P4 12) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -359,6 +358,7 @@ stopping line that lies. | **R-1375** | Hub & operator | P2 | **The outside part of three endings is done by hand or not at all — the worst: an off-site account can stay at the provider with no record in the hub.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH06, EH01, EA21 and the ending "a tester ends"; read in source and documents; no account was called). (1) **Unticking the off-site tier** of a household takes nothing down and leaves no reminder (`hub/internal/web/configs.go:1737-1743`). A later RESET takes the account down only when the tier is ticked (`hub/internal/web/customer_reset.go:203-245`), so the sub-account and its data stay, are paid for, and the hub holds no record that they exist. (2) **Switching the second off-site tier off** takes nothing down: storage entry, token and copies stay; the source calls the teardown 'a deliberate future op' (agent `internal/pbsdr/manager.go:274`). (3) **A tester ends**: no handler and no runbook; done by hand once (2026-09-25); no list says what to collect. **Members (answers marked N):** EH06-4, EH01-4, EA21-4, and 'a tester ends' in the table's head. Related: R-810 (a household leaves), R-32. | OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D71 = yes, D74 = yes; `09` §3 decisions 291, 294).** Not built. | — | Hub (D71): keep the account record after an untick; RESET and delete take the account down, ticked or not. Docs (D74): the runbook 'a household or a tester leaves', from the five code doors, in a later documents-only session. Parts (2) and (3) of this row have no ruling of their own. | CC | | **R-1376** | Hub & operator | P3 | **Records of a thing that is gone stay in six places.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH00, EH02, EH21, EA00, EA11, EA27; read in source, `documentation/audits/endings-list-2026-11-10/evidence/hub-delete-statements.txt`; no hub database was read). (1) **Hub:** thirteen tables have no delete statement anywhere. Deleting a box leaves its update settings, its update reports and its kernel notices; deleting a household leaves its off-site key rows, off-site window rows and abandon requests. Most are small; the update reports of each box grow with time. (2) **Host uninstall** leaves the mount units and host mounts of household drives (said when R-179 closed; no open row carried it). (3) **A drive wiped by a signed job** keeps a record and a mount unit that name the old filesystem (agent `internal/signedjobs/wipe.go:61-110`). (4) **The token of a destroyed guest** stays valid until a new guest takes its number: the token store has no remove (agent `internal/localapi/tokenstore.go:127-210`). **Members (answers marked N):** EH00-4, EH02-4, EH21-4, EA00-4, EA11-4, EA27-4. | OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D73 = yes, with one exception; `09` §3 decision 293).** Not built. | — | Hub (D73): add the tables with no delete to the two purges and to the daily clean-up; the delete journal (`host_deletions`, `customer_resets`) stays on purpose, and the privacy text says so. Agent: remove a guest's token when the guest is destroyed. Parts (2) and (3) have no ruling of their own. | CC | | **R-1386** | Hub & operator | P3 | **Timed jobs with no time limit and no measured duration (25 answers).** Found 2026-11-11 (the Clock list, class W4, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source; the hub was not measured, by ruling). The hub's 60-second sweep runs its checks one after the other with no limit for each: one slow check delays all the others, the liveness check too (CH19-3, CH19-8, CH01-3, CH08-3, CH08-8, CH10-3, CH13-8, CH18-3, CH06-8, CH02-3); the 07:10 audit has 10 minutes, measured with three boxes only (CH11-3, CH11-8). On the box: the interval jobs have no limit of their own (CC11-3, CC12-3, CC13-3, CC14-3, CC15-3, CC16-3, CC20-3, CA06-3, CA10-3, CA10-8, CA13-8, CA15-3, CA19-3). Every measured duration is from a small test store. | OPEN — filed 2026-11-11 (clock list) | — | Hub: log each sweep step's duration and alarm when one sweep takes longer than its interval; then read a week of numbers and set limits. Box: the same for the controller's interval jobs. | CC | +| **R-1393** | Hub & operator | P4 | **For about 8 minutes after a hub restart its database statements take 1 to 32 seconds; the System page took 12.7 s and a box's enrolment 11.4 s.** Measured 2026-11-14 at the 0.181.0 deploy (`audits/release-1-dates-2026-11-14/hub/05-watch.txt`): 24 `slow statement` lines from 09:58 to 10:06 UTC, none in the 20 minutes after. The database file is 678 MB; the slow ones read `host_reports`, `os_reports` and `events` (whole report JSON per row). One 32-second statement ran while CC copied the database out of the pod for a read (CC's own load). No report was lost that CC could see; not measured: whether a box's report timed out in that window. | OPEN — filed 2026-11-14 | — | measure a restart with no copy running; then decide: an index or a shorter keep on `host_reports`/`os_reports`, or accept | CC | ## Business & legal — 14 rows (P2 4, P3 2, P4 8) @@ -424,7 +424,7 @@ stopping line that lies. | item | due (UTC) | what to measure | |---|---|---| | R-1293 | 2027-03-29 | the spring clock change: the hub's 02:00 snapshot and the 02:30 push on 2027-03-28 (02:00–03:00 does not exist) — which file each push took; detail in the R-1293 row | -| R-1050 | 2026-11-15 | HSTS one year if no https trouble since 2026-10-15; detail in the R-1050 row | +| R-1050 | 2026-11-21 | HSTS one year in the next controller release if no https trouble since 2026-10-15; detail in the R-1050 row | | R-1035 | 2026-11-18 | „felhő tárhely” + „saját felhő tárhely” impressions and average position, and the new page's own impressions/clicks/position, 4 weeks after publishing; operator supplies the read; detail in the R-1035 row | | R-528 | 2026-11-28 | no further failed oom_check after the one on 2026-11-11 (demo-felhom), no unexplained app_oom; detail in the R-528 row | | R-1035 | 2026-12-09 | the same three numbers 8 weeks after publishing, compared with the 4-week read; a null at 4 weeks is not a verdict, both reads are needed; detail in the R-1035 row | diff --git a/documentation/backlog/STATUS-HISTORY.md b/documentation/backlog/STATUS-HISTORY.md index 1d997a81..a37ec165 100644 --- a/documentation/backlog/STATUS-HISTORY.md +++ b/documentation/backlog/STATUS-HISTORY.md @@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S **Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.** +## Moved from STATUS on 2026-11-14 (midday) + +**Updated 2026-11-14 (morning).** Health check: the three test boxes are healthy; last night's backups and restore proofs passed. CI was red three nights for one due check; it is read now. The yellow „root files behind" is a false alarm that mails on 11-17 unless you answer D81. + ## Moved from STATUS on 2026-11-14 (morning) **Updated 2026-11-11 (afternoon).** Golden 0.352.0 is vouched, so code pushes to `felhom.eu` work again. The N100 keeps no time: one hand step repairs it (below). A new Proxmox certificate blinds the agent; the hand repair is proven on demo-hp and one decision waits (D80). diff --git a/documentation/tests/golden-0.352.1-2026-11-14/00-boot.txt b/documentation/tests/golden-0.352.1-2026-11-14/00-boot.txt new file mode 100644 index 00000000..bb17cf32 --- /dev/null +++ b/documentation/tests/golden-0.352.1-2026-11-14/00-boot.txt @@ -0,0 +1,6 @@ +## boot 2026-11-14T10:01Z +drill VM reverted to virgin and cold-booted (RUNBOOK-manual-build §4.0); pve-manager/9.2.2 +pveam update: update successful; template debian-13-standard_13.6-1_amd64.tar.zst (checksum verified) +build-golden.sh v3.2.0 from felhom-agent 91a2924ad0, sha256 645b3b65… (the same file as the 0.352.0 bake) +token + registry user copied file->file from sec (GITEA_PKG_TOKEN, GITEA_PKG_USER); unit properties grep for the token value: 0 +bake started 10:02:02Z, ended before 10:07:53Z diff --git a/documentation/tests/golden-0.352.1-2026-11-14/02-round-trip.txt b/documentation/tests/golden-0.352.1-2026-11-14/02-round-trip.txt new file mode 100644 index 00000000..c3298fc6 --- /dev/null +++ b/documentation/tests/golden-0.352.1-2026-11-14/02-round-trip.txt @@ -0,0 +1,6 @@ +## round trip 2026-11-14T10:08:42Z (no login) +curl rc=0 +bytes: 638851360 +sha256: 4f411cbeb4b28a7dd65d4121e34e6ebb98f985bd07a48051502ac8d25c01346d +bake: 4f411cbeb4b28a7dd65d4121e34e6ebb98f985bd07a48051502ac8d25c01346d +registry login in the archive: tar listing for .docker/config.json = 0 (listing control: 8 other lines found) diff --git a/documentation/tests/golden-0.352.1-2026-11-14/03-teardown.txt b/documentation/tests/golden-0.352.1-2026-11-14/03-teardown.txt new file mode 100644 index 00000000..49d3d14f --- /dev/null +++ b/documentation/tests/golden-0.352.1-2026-11-14/03-teardown.txt @@ -0,0 +1,8 @@ +## in the drill VM 2026-11-14T10:08:06Z +token in bake.log: 0 (control with the token appended: 1) +pass markers: docker OK=1 including-mount-point=2 excluding=0 FATAL=0 not-set=0 upload=1 +purging CT 9100 from related configurations.. +pct destroy rc=0 +ls: cannot access '/root/.gitea-token': No such file or directory +ls: cannot access '/root/bake.log': No such file or directory +drill VM off; disk reverted to virgin 2026-11-14T10:08:29Z diff --git a/documentation/tests/golden-0.352.1-2026-11-14/04-vouch-readback.txt b/documentation/tests/golden-0.352.1-2026-11-14/04-vouch-readback.txt new file mode 100644 index 00000000..5dc9bc5a --- /dev/null +++ b/documentation/tests/golden-0.352.1-2026-11-14/04-vouch-readback.txt @@ -0,0 +1,5 @@ +## vouch read-back 2026-11-14T10:25:50Z (the hub's log; the pod's clock is CET) +2026/11/14 11:12:15 [INFO] Artifact manifest served for customer Tester4 (agent=0.173.0 golden=0.352.0) +2026/11/14 11:25:31 [INFO] artifact choices (felhom-agent): 1 version(s) without felhom-agent left out (R-1011): 0.163.0-step1 +2026/11/14 11:25:43 [INFO] Artifact manifest set: agent=0.173.0 golden=0.352.1 min_agent="0.158.1" wrapper_sha=false bundle_sha="51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1" +2026/11/14 11:25:43 [INFO] artifact vouch timing (R-1346): registry probes 94ms, sha lookups 15ms, bundle sha 10ms, fleet check + save 239ms, total 359ms diff --git a/documentation/tests/golden-0.352.1-2026-11-14/README.md b/documentation/tests/golden-0.352.1-2026-11-14/README.md new file mode 100644 index 00000000..d375feb6 --- /dev/null +++ b/documentation/tests/golden-0.352.1-2026-11-14/README.md @@ -0,0 +1,79 @@ +# Golden 0.352.1 — bake, publish, one fresh install, vouch — 2026-11-14 (release 1 „dates”, Part 3) + +Why: the weekly bake, done inside a brief so it is not missed (golden 0.352.0 was baked 2026-11-11; R-468). +Procedure: 0.352.0's README, the same recipe. Controller 0.352.1 is the global floor (MinAgent 0.158.1). +Documents read: `architecture/09-update-architecture.md` (§3 decisions 247, 250), `architecture/05-hub-architecture.md` +(Day-0 vouch, D50), `runbooks/publish-train-rules.md`, `runbooks/RUNBOOK-manual-build.md` §4, rows R-468, R-242. + +| | Previous (`../golden-0.352.0-2026-11-11/`) | This bake | +|---|---|---| +| `build-golden.sh` | v3.2.0 | same file (sha256 `645b3b65…`) | +| Controller | `felhom-controller:0.352.0` | **`felhom-controller:0.352.1`** (Go 1.27.4; MinAgent **0.158.1**) | +| Traefik baked | v3.7.17 | same | +| Docker engine | the six approved pairs (29.9.2) | same | +| Template | `debian-13-standard_13.6-1_amd64.tar.zst` | same | +| Token | from `sec`, file → file | same | + +## Pass markers (`bake.log`) + +``` +[golden] build-golden.sh v3.2.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.352.1 + docker OK (overlay2; data-root /var/lib/docker) +[golden] upload OK (HTTP 201) +GOLDEN_VERSION=0.352.1 +GOLDEN_SHA256=4f411cbeb4b28a7dd65d4121e34e6ebb98f985bd07a48051502ac8d25c01346d +``` + +`including mount point` 2, `excluding` 0, `FATAL` 0, `GOLDEN_DOCKER_PKGS not set` 0 (`03-teardown.txt`). The token is +not in `bake.log`: 0, with a control of 1, in the VM and in the committed copy. +Round trip: the registry's file, downloaded with no login and hashed, equals the bake's sha; 638,851,360 bytes; no +registry login in the archive (`02-round-trip.txt`). Build guest and teardown: `00-boot.txt`, `03-teardown.txt`. +**Golden gate:** `python3 scripts/golden_currency_gate.py` → OK, rc 0. No waiver was written. + +## The fresh install (`fresh-install/`) + +**The path, exactly.** The hub offered only the vouched golden (0.352.0) at that time. So the box ran the public +installer 1.36.1 (sha256 `e18cdd17…aa1f1`) with **one added line** after line 479, which names golden 0.352.1 and its +sha256 (`installer-the-one-line.diff.txt`), run by hand as root in the direct shape +(`--customer-id Tester4 --mode appliance --hub-url https://hub.felhom.eu --passphrase-file …`), not through the bind +page. **One more difference from a household's path:** the installer file was fetched on DooPlex and copied in, +because on this LAN the name `felhom.eu` answers 404 from the retired site (R-1389). All other steps are the +product's own: the hub's enrolment, the agent download, the golden's fetch and sha check, the guest build, the +setup-code mail, the claim page, the app page. + +| Step | Result | Proof | +|---|---|---| +| VM 9385 `golden1114-box` on demo-hp, disks on `nvme-scratch` (never the thin pool), Proxmox from the Felhom ISO 1.30.0 (sha256 checked against the published file) | installed; Install pressed 10:07:53Z, SSH answered 10:10:46Z | `install/` | +| CC's SSH key on the host **before** the installer ran (the 2026-11-11 lesson) | done; the key still opened the host after the installer closed it | `box-state-and-install-log.txt` | +| Time service on the fresh host, before the installer | `chrony` active, 0.0005 s off (10:10:58Z); active after it too | — | +| Installer run, started 10:11:24Z | „Day-0 provision SUCCESS — vmid=9201 host_id=Tester4-46836f” before 10:14:29Z | `box-state-and-install-log.txt` | +| Golden on the box | sha256 `4f411cbe…346d` = the bake and the registry | read on the box | +| Agent | **0.173.0**, active | read on the box | +| Controller | **0.352.1** from the golden: one `starting` line, no `controller_updated`; 0 ERROR lines | `box-state-and-install-log.txt` | +| Hub | host `Tester4-46836f`, agent 0.173.0, ONLINE | `app.txt` (last lines) | +| Setup-code mail | arrived 10:12:03Z | the test mailbox | +| Claim page | `POST /claim` → 302 `/` at 10:15:16Z; the box then offers the recovery page for Tester4's sealed package (not entered) | this README (the request's two status lines are not kept in a file) | +| One app: BentoPDF, through the page's own deploy request | 202 at 10:22:07Z; `running` at 10:23:57Z; `pdf.tester4.test` answers 200; a name with no app answers 404 | `app.txt` | + +**CC's two mistakes on the way, both in CC's own script:** the first two tries to install the app looked for the +session token in the wrong place and got `403 CSRF token missing or invalid` (nothing was installed; 6 minutes +lost). Not a product fault. + +**The hub's own acts on this customer:** the fresh install made the hub re-issue Tester4's off-site key and carry the +sealed package of the deleted host to the new one. CC touched nothing on ep0. + +## Vouch — done by the operator, 2026-11-14 10:25:43 UTC (`04-vouch-readback.txt`) + +- golden **0.352.1**, sha256 `4f411cbeb4b28a7dd65d4121e34e6ebb98f985bd07a48051502ac8d25c01346d` +- agent **0.173.0**, bundle `51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1` (both unchanged) +- min_agent **0.158.1** (unchanged) + +Read from the hub's log only (one channel). `pinned.day0_agent` in the agent's retention policy stays 0.173.0. + +## Teardown + +`fresh-install/teardown.txt`: VM 9385 stopped and destroyed with its two disks; the installer ISO removed from +demo-hp; the typed password, the session cookie and the setup-code file shredded on DooPlex. The drill VM is off and +its disk is back on `virgin`. The hub host `Tester4-46836f` (and its appliance entry, smbios uuid +`23d48cf6-1984-46a7-9954-557faf333f10`, if the Hosts page lists one) are the operator's to remove — the host from +about 45 minutes after its last report (R-599). Until then the hub mails „no report” for it. diff --git a/documentation/tests/golden-0.352.1-2026-11-14/bake.log b/documentation/tests/golden-0.352.1-2026-11-14/bake.log new file mode 100644 index 00000000..761dcfe3 --- /dev/null +++ b/documentation/tests/golden-0.352.1-2026-11-14/bake.log @@ -0,0 +1,339 @@ +[golden] build-golden.sh v3.2.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.352.1 +[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) … + Logical volume "vm-9100-disk-0" created. + Logical volume pve/vm-9100-disk-0 changed. +Creating filesystem with 8388608 4k blocks and 2097152 inodes +Filesystem UUID: 6a11faf4-eafe-462d-bc6b-d3f3b54f2f77 +Superblock backups stored on blocks: + 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, + 4096000, 7962624 + Logical volume "vm-9100-disk-1" created. + Logical volume pve/vm-9100-disk-1 changed. +Creating filesystem with 6291456 4k blocks and 1572864 inodes +Filesystem UUID: 63909107-2b40-4ed6-86fa-97320060f069 +Superblock backups stored on blocks: + 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, +extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst' +Total bytes read: 553512960 (528MiB, 135MiB/s) +Detected container architecture: amd64 +Creating SSH host key 'ssh_host_rsa_key' - this may take some time ... +done: SHA256:OoKz/bAime/Y82Y6PAbV2mkqNSKqdJKKJ7IkVyDg0hc root@felhom-golden +Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ... +done: SHA256:mWiYmA+1ncXTPjQllxy0KkZfQL9rwOTgPfPLlK4Xvbo root@felhom-golden +Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ... +done: SHA256:p/oUB5Zf4PkGKBF3zwqh9ysgXyoB/8/KvelW7ZWg2JU root@felhom-golden +[golden] starting + installing Docker (official repo, trixie channel) … +[golden] Docker engine set PINNED to the approved release: containerd.io=2.4.2-3~debian.13~trixie docker-buildx-plugin=0.38.1-1~debian.13~trixie docker-ce=5:29.9.2-1~debian.13~trixie docker-ce-cli=5:29.9.2-1~debian.13~trixie docker-ce-rootless-extras=5:29.9.2-1~debian.13~trixie docker-compose-plugin=5.6.1-1~debian.13~trixie +apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct! +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = (unset), + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to the standard locale ("C"). +locale: Cannot set LC_CTYPE to default locale: No such file or directory +locale: Cannot set LC_MESSAGES to default locale: No such file or directory +locale: Cannot set LC_ALL to default locale: No such file or directory +apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct! +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = (unset), + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to the standard locale ("C"). +locale: Cannot set LC_CTYPE to default locale: No such file or directory +locale: Cannot set LC_MESSAGES to default locale: No such file or directory +locale: Cannot set LC_ALL to default locale: No such file or directory + installed: containerd.io 2.4.2-3~debian.13~trixie + installed: docker-buildx-plugin 0.38.1-1~debian.13~trixie + installed: docker-ce 5:29.9.2-1~debian.13~trixie + installed: docker-ce-cli 5:29.9.2-1~debian.13~trixie + installed: docker-ce-rootless-extras 5:29.9.2-1~debian.13~trixie + installed: docker-compose-plugin 5.6.1-1~debian.13~trixie +[golden] no approved guest release given - the template versions stay; first-night count vs an approved release: n/a +[golden] pending Debian upgrades in the baked guest (what a FUTURE approval may bring): 63 +[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation … +[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds … +[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) … +Unable to find image 'hello-world:latest' locally +latest: Pulling from library/hello-world +4f55086f7dd0: Pulling fs layer +4f55086f7dd0: Download complete +4f55086f7dd0: Pull complete +Digest: sha256:cf1b20a0c89a666b6950fecda7ccbc1b1a86520e7c53f908f3a21e79d0d60458 +Status: Downloaded newer image for hello-world:latest + docker OK (overlay2; data-root /var/lib/docker) + live-restore: on + /var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4 + /mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4 + both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576 +[golden] baking the in-guest controller image gitea.dooplex.hu/admin/felhom-controller:0.352.1 (no registry cred at deploy) … + +WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'. +Configure a credential helper to remove this warning. See +https://docs.docker.com/go/credential-store/ + +0.352.1: Pulling from admin/felhom-controller +ecc510c1e359: Pulling fs layer +c3f83389ce1b: Pulling fs layer +e81c1fce9826: Pulling fs layer +63a1406b1961: Pulling fs layer +d747c9b6922d: Pulling fs layer +eb49b63afe2b: Pulling fs layer +c6c5608e8d7b: Pulling fs layer +63a1406b1961: Waiting +d747c9b6922d: Waiting +eb49b63afe2b: Waiting +c6c5608e8d7b: Waiting +e81c1fce9826: Verifying Checksum +e81c1fce9826: Download complete +c3f83389ce1b: Verifying Checksum +c3f83389ce1b: Download complete +d747c9b6922d: Download complete +eb49b63afe2b: Verifying Checksum +eb49b63afe2b: Download complete +63a1406b1961: Verifying Checksum +63a1406b1961: Download complete +c6c5608e8d7b: Verifying Checksum +c6c5608e8d7b: Download complete +ecc510c1e359: Verifying Checksum +ecc510c1e359: Download complete +ecc510c1e359: Pull complete +c3f83389ce1b: Pull complete +e81c1fce9826: Pull complete +63a1406b1961: Pull complete +d747c9b6922d: Pull complete +eb49b63afe2b: Pull complete +c6c5608e8d7b: Pull complete +Digest: sha256:1a6e8945211a0f1995fd2bd0782053c7f33152fcb00ca465d0596d961a13b6bb +Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-controller:0.352.1 +gitea.dooplex.hu/admin/felhom-controller:0.352.1 +[golden] asking the controller which infra images it manages … +[golden] baking infra images (4): traefik:v3.7.17 cloudflare/cloudflared:2026.10.4 gtstef/filebrowser:1.5.8-stable gitea.dooplex.hu/admin/felhom-samba:1.2.0 … +v3.7.17: Pulling from library/traefik +881fc932939e: Pulling fs layer +c901ba6ff9e0: Pulling fs layer +4f376259b112: Pulling fs layer +7705a4c1b3f8: Pulling fs layer +7705a4c1b3f8: Waiting +881fc932939e: Verifying Checksum +881fc932939e: Download complete +c901ba6ff9e0: Verifying Checksum +c901ba6ff9e0: Download complete +7705a4c1b3f8: Verifying Checksum +7705a4c1b3f8: Download complete +4f376259b112: Verifying Checksum +4f376259b112: Download complete +881fc932939e: Pull complete +c901ba6ff9e0: Pull complete +4f376259b112: Pull complete +7705a4c1b3f8: Pull complete +Digest: sha256:f4bb143d1135a579b630ed491e17ec50f6f7c9c6a30585ff3d385471d2a86f5a +Status: Downloaded newer image for traefik:v3.7.17 +docker.io/library/traefik:v3.7.17 +2026.10.4: Pulling from cloudflare/cloudflared +2cc7ee286bf3: Pulling fs layer +c172f21841df: Pulling fs layer +218cf840d0d9: Pulling fs layer +f6069939f718: Pulling fs layer +d6b1b89eccac: Pulling fs layer +2780920e5dbf: Pulling fs layer +7c12895b777b: Pulling fs layer +3214acf345c0: Pulling fs layer +52630fc75a18: Pulling fs layer +dd64bf2dd177: Pulling fs layer +b839dfae01f6: Pulling fs layer +ebddc55facdc: Pulling fs layer +a0921f9d8de9: Pulling fs layer +b96fe2995f90: Pulling fs layer +810e2b405a08: Pulling fs layer +bd8962e29291: Pulling fs layer +cac2ae0193cb: Pulling fs layer +eb8da9d09a5e: Pulling fs layer +52630fc75a18: Waiting +dd64bf2dd177: Waiting +b839dfae01f6: Waiting +ebddc55facdc: Waiting +a0921f9d8de9: Waiting +b96fe2995f90: Waiting +810e2b405a08: Waiting +bd8962e29291: Waiting +cac2ae0193cb: Waiting +eb8da9d09a5e: Waiting +d6b1b89eccac: Waiting +2780920e5dbf: Waiting +7c12895b777b: Waiting +3214acf345c0: Waiting +f6069939f718: Waiting +2cc7ee286bf3: Verifying Checksum +2cc7ee286bf3: Download complete +c172f21841df: Verifying Checksum +c172f21841df: Download complete +218cf840d0d9: Verifying Checksum +218cf840d0d9: Download complete +f6069939f718: Verifying Checksum +f6069939f718: Download complete +d6b1b89eccac: Download complete +2780920e5dbf: Verifying Checksum +2780920e5dbf: Download complete +7c12895b777b: Verifying Checksum +7c12895b777b: Download complete +3214acf345c0: Verifying Checksum +3214acf345c0: Download complete +52630fc75a18: Verifying Checksum +52630fc75a18: Download complete +2cc7ee286bf3: Pull complete +dd64bf2dd177: Verifying Checksum +dd64bf2dd177: Download complete +b839dfae01f6: Verifying Checksum +b839dfae01f6: Download complete +ebddc55facdc: Download complete +a0921f9d8de9: Download complete +810e2b405a08: Verifying Checksum +810e2b405a08: Download complete +b96fe2995f90: Verifying Checksum +b96fe2995f90: Download complete +bd8962e29291: Download complete +c172f21841df: Pull complete +cac2ae0193cb: Verifying Checksum +cac2ae0193cb: Download complete +eb8da9d09a5e: Verifying Checksum +eb8da9d09a5e: Download complete +218cf840d0d9: Pull complete +f6069939f718: Pull complete +d6b1b89eccac: Pull complete +2780920e5dbf: Pull complete +7c12895b777b: Pull complete +3214acf345c0: Pull complete +52630fc75a18: Pull complete +dd64bf2dd177: Pull complete +b839dfae01f6: Pull complete +ebddc55facdc: Pull complete +a0921f9d8de9: Pull complete +b96fe2995f90: Pull complete +810e2b405a08: Pull complete +bd8962e29291: Pull complete +cac2ae0193cb: Pull complete +eb8da9d09a5e: Pull complete +Digest: sha256:c6c7ae1d5dfdcb094ad344ed68ccffc26635c9c6be80965664c33baf064b0515 +Status: Downloaded newer image for cloudflare/cloudflared:2026.10.4 +docker.io/cloudflare/cloudflared:2026.10.4 +1.5.8-stable: Pulling from gtstef/filebrowser +e2de96513ba9: Pulling fs layer +93c39e448959: Pulling fs layer +b7597036c324: Pulling fs layer +c6591d351da9: Pulling fs layer +4f4fb700ef54: Pulling fs layer +74aec9809d97: Pulling fs layer +9f7b5accdf20: Pulling fs layer +19114688b75e: Pulling fs layer +2067d5df1f91: Pulling fs layer +5669e416ca05: Pulling fs layer +08c322a8a1d4: Pulling fs layer +19114688b75e: Waiting +2067d5df1f91: Waiting +5669e416ca05: Waiting +08c322a8a1d4: Waiting +4f4fb700ef54: Waiting +74aec9809d97: Waiting +c6591d351da9: Waiting +9f7b5accdf20: Waiting +e2de96513ba9: Verifying Checksum +e2de96513ba9: Download complete +c6591d351da9: Verifying Checksum +c6591d351da9: Download complete +4f4fb700ef54: Verifying Checksum +4f4fb700ef54: Download complete +b7597036c324: Verifying Checksum +b7597036c324: Download complete +9f7b5accdf20: Verifying Checksum +9f7b5accdf20: Download complete +e2de96513ba9: Pull complete +74aec9809d97: Verifying Checksum +74aec9809d97: Download complete +2067d5df1f91: Verifying Checksum +2067d5df1f91: Download complete +19114688b75e: Verifying Checksum +19114688b75e: Download complete +08c322a8a1d4: Verifying Checksum +08c322a8a1d4: Download complete +5669e416ca05: Verifying Checksum +5669e416ca05: Download complete +93c39e448959: Verifying Checksum +93c39e448959: Download complete +93c39e448959: Pull complete +b7597036c324: Pull complete +c6591d351da9: Pull complete +4f4fb700ef54: Pull complete +74aec9809d97: Pull complete +9f7b5accdf20: Pull complete +19114688b75e: Pull complete +2067d5df1f91: Pull complete +5669e416ca05: Pull complete +08c322a8a1d4: Pull complete +Digest: sha256:68455d4953bad8e984e1bc5d112ed1e9c918a0a04f104c13c9ece50790308894 +Status: Downloaded newer image for gtstef/filebrowser:1.5.8-stable +docker.io/gtstef/filebrowser:1.5.8-stable +1.2.0: Pulling from admin/felhom-samba +e2de96513ba9: Already exists +0ddbc9f3a6b6: Pulling fs layer +ce57a3f93416: Pulling fs layer +fb94eeec2fe1: Pulling fs layer +ce57a3f93416: Verifying Checksum +ce57a3f93416: Download complete +fb94eeec2fe1: Verifying Checksum +fb94eeec2fe1: Download complete +0ddbc9f3a6b6: Verifying Checksum +0ddbc9f3a6b6: Download complete +0ddbc9f3a6b6: Pull complete +ce57a3f93416: Pull complete +fb94eeec2fe1: Pull complete +Digest: sha256:da96f25033827fbb77d0ee398e86fe5b744e2f29be9de898994533340c97acf9 +Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.2.0 +gitea.dooplex.hu/admin/felhom-samba:1.2.0 +[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'. +[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'. +[golden] baking the first-boot SSH host-key regeneration unit (F3) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'. +[golden] identity-clean + minimize … +[golden] stop + archive … +INFO: including mount point rootfs ('/') in backup +INFO: including mount point mp0 ('/var/lib/felhom') in backup +INFO: archive file size: 609MB +INFO: Finished Backup of VM 9100 (00:00:28) +[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_11_14-11_06_32.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive) +[golden] publishing golden (638851360 bytes, sha256 4f411cbeb4b28a7d…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.352.1/golden.tar.zst +[golden] pre-delete existing: HTTP 404 (404/204 expected) +[golden] upload OK (HTTP 201) +GOLDEN_VERSION=0.352.1 +GOLDEN_SHA256=4f411cbeb4b28a7dd65d4121e34e6ebb98f985bd07a48051502ac8d25c01346d +[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.352.1 / 4f411cbeb4b28a7dd65d4121e34e6ebb98f985bd07a48051502ac8d25c01346d +[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge) diff --git a/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/app.txt b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/app.txt new file mode 100644 index 00000000..a2390ca3 --- /dev/null +++ b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/app.txt @@ -0,0 +1,23 @@ +session csrf_len 64 +controller version on the box: {"ok":true,"message":"felhom-controller is healthy"} +2026-11-14T10:22:07Z +{"ok":true,"message":"Telepítés elindítva – az állapot a kártyán követhető"} + +HTTP 202 +10:22:07 {'deployed': True, 'state': 'not_deployed', 'subdomain': None} +10:22:17 {'deployed': True, 'state': 'deploying', 'subdomain': None} +10:22:27 {'deployed': True, 'state': 'deploying', 'subdomain': None} +10:22:37 {'deployed': True, 'state': 'deploying', 'subdomain': None} +10:22:47 {'deployed': True, 'state': 'deploying', 'subdomain': None} +10:22:57 {'deployed': True, 'state': 'deploying', 'subdomain': None} +10:23:07 {'deployed': True, 'state': 'deploying', 'subdomain': None} +10:23:17 {'deployed': True, 'state': 'deploying', 'subdomain': None} +10:23:27 {'deployed': True, 'state': 'deploying', 'subdomain': None} +10:23:37 {'deployed': True, 'state': 'deploying', 'subdomain': None} +10:23:47 {'deployed': True, 'state': 'deploying', 'subdomain': None} +10:23:57 {'deployed': True, 'state': 'running', 'subdomain': None} +bentopdf name: pdf.tester4.test +route: 200 at 2026-11-14T10:24:08Z +negative control (no such app name): 404 +2026-11-14T10:24:20Z Tester4-46836f Tester4 0.173.0 floor: declared MinAgent 9.2.2 7.0.2-6-pve ONLINE 0/0 0% 18% 16% unknown (no running customer guest) 16% local demo-felhom-8363b5 Demo Ügyfél 0.173.0 floor: declared Min +Unclaimed appliances (booted from the generic ISO, awaiting a bind) A box that installed from the universal secret-free ISO and registered itself. Bind it to a customer to deliver its retrieval passphrase once; discard to ignore it. Appliance Pairing code MACs Hardware SSH host keys Seen Bind to cus diff --git a/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/box-state-and-install-log.txt b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/box-state-and-install-log.txt new file mode 100644 index 00000000..56f3f7ed --- /dev/null +++ b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/box-state-and-install-log.txt @@ -0,0 +1,157 @@ +ghcr.io/alam00000/bentopdf:v2.8.10 Up 21 seconds (healthy) +gtstef/filebrowser:1.5.8-stable Up 10 minutes (healthy) +gitea.dooplex.hu/admin/felhom-controller:0.352.1 Up 10 minutes (healthy) +traefik:v3.7.17 Up 10 minutes +2026/11/14 10:14:05 [INFO] felhom-controller 0.352.1 starting (customer: Tester4, domain: tester4.test) +0 +[INFO] auto-sized guest RAM: 5847 MiB (host 9943 MiB; clamp(host-4096, min 4096, max host-2048), ceiling host-1024) +[INFO] auto-sized guest cores: 5 (host 6 cores; host-1, min 2) + +[INFO] felhom-host-install v1.36.1 — mode=appliance customer=Tester4 vmid=9201 + +[STEP] 1/8 pre-flight +[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve) +[INFO] node: goldenbox (auto) +[INFO] agent config: /etc/felhom-agent/agent.json +[INFO] agent: not installed yet — will be fetched + installed in step 5/8 +[INFO] local-lvm free: ~19 GiB +[WARN] local-lvm free ~19 GiB is below the recommended 120 GiB — the install continues; proceed only if you sized the grows deliberately +[INFO] free RAM: ~8344 MiB +[INFO] existing guests on this host: 0 (pct+qm) +[INFO] archive storage 'local' present +[INFO] acl storage 'felhom-pbs' not present yet — expected: the PBS-DR tier creates it; the grant is pre-positioned deliberately +[INFO] dnsmasq: not present before Felhom — recorded; uninstall will remove it again if we install it +[INFO] hub reachable (https://hub.felhom.eu) +[OK] customer 'Tester4' exists + passphrase valid +[INFO] golden: none local — will fetch + verify from Gitea in step 7/8 +[OK] pre-flight passed +[STEP] 2/8 Proxmox API token +[OK] token minted (secret captured, not logged) +[OK] scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm felhom-pbs]) +[SKIP] old broad role FelhomAgent already absent +[STEP] 3/8 compute volume grows +[INFO] auto-computed from a 19 GiB thin pool (16 §7: 50/100/130 GiB) +[INFO] grows: rootfs +0G (->32G), data +26G (->50G, ONE volume) +[STEP] 4/8 host enrollment (POST /host-enroll) +[OK] host MINTED (first enroll) +[INFO] host_id: Tester4-46836f (api_key captured, not logged) +[STEP] 4b/8 break-glass credential (root@pam console password → hub vault) +[INFO] recovery password vaulted at the hub (stored) — now setting it on root@pam +[OK] root@pam password vaulted, set, and the PVE login takes it (retrieve via the operator /admin path; never logged here) +[WARN] NOTE: the root@pam password just CHANGED — the old one now fails at the PVE web GUI (:8006). +[WARN] Retrieve the new one at hub → host page (vaulted recovery credential). +[STEP] 5/8 agent install (fetch + verify + install) +[INFO] disabled pve-enterprise.sources (Enabled: no) +[INFO] disabled ceph.sources (Enabled: no) +[INFO] added pve-no-subscription.sources (suite=trixie) +[OK] apt repos aligned to no-subscription (changed; apt-get update OK) +[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged) +[INFO] manifest: agent v0.173.0 (sha 752b2727fb604341…), golden v0.352.1 +[INFO] fetching agent binary v0.173.0 from Gitea … +[OK] verified sha256 752b2727fb604341… matches the hub manifest +[OK] installed /usr/local/bin/felhom-agent (felhom-agent 0.173.0) +[INFO] installing the 'sudo' package (required for the non-root agent model) … +[OK] sudo installed (Sudo version 1.9.16p2) +[INFO] installing the 'age' package (escrow ceremony identity-wrap dependency) … +[OK] age installed (1.2.1) +[OK] created service user felhom-agent +[OK] added felhom-agent to systemd-journal (unprivileged journal read for NAS verify) +[OK] verified sha256 51ebc1f4c66d8791… matches the hub manifest + os-apply: BUNDLE START agent=0.173.0 sha=51ebc1f4c66d8791 authority=installer files=32 write=31 same=1 kept=0 skipped=0 + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-mkfs-guarded (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-selfupdate-guarded (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-pbs-apply (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-backup-target-apply (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-vzdump-partial-clean (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-thin-check (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-internal-place (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-crash-guard (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-priv-apply (new) + os-apply: BUNDLE WROTE /var/lib/vz/snippets/felhom-guest-hook.sh (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-shared-parent.sh (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-shared-parent.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard-check.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard-check.timer (new) + os-apply: BUNDLE WROTE /etc/felhom/crash-guard.conf (new) + os-apply: BUNDLE WROTE /etc/grub.d/01_felhom_oneshot (new) + os-apply: BUNDLE WROTE /etc/grub.d/42_felhom_oneshot (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent-rollback.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-mgmt-watchdog (new) + os-apply: BUNDLE WROTE /etc/tmpfiles.d/felhom-privsep.conf (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-mgmt-watchdog.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-mgmt-watchdog.timer (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-sshd.service (new) + os-apply: BUNDLE WROTE /etc/felhom-oob.nft (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-oob-nft.service (new) + os-apply: BUNDLE WROTE /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf (new) + os-apply: BUNDLE WROTE /etc/sudoers.d/felhom-op (new) + os-apply: BUNDLE WROTE /etc/sudoers.d/felhom-agent (new) + os-apply: BUNDLE DONE agent=0.173.0 written=31 same=1 self-check=ok signers-created=False +[OK] installed every root-owned file from config bundle v0.173.0 (sha 51ebc1f4c66d8791…; checked, previous copies kept) +[OK] OOB felhom-sshd instance + static belt from the bundle (agent renders config + fills sets once oob.enabled) +[STEP] 5b/8 internal data place (Belső tárhely) +[INFO] no internal place: only -79 GiB free for it (pool 19, reserve 16, promised 0, planned 82; needs 20) +[STEP] 6/8 agent config + service +[WARN] backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive. +[WARN] It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard. +[INFO] creating island bridge vmbr9 (portless, 169.254.253.1/30) +[OK] vmbr9 up: 169.254.253.1/30 +[INFO] R-50 island ON: local_api=169.254.253.1:8443 (vmbr9); guest net1=169.254.253.2/30; lan_resolver.host_ip=192.168.0.122 +[INFO] node=goldenbox local_api=169.254.253.1:8443 tls_fp=30:9D:EE:6C:AE:2D… +[OK] wrote /etc/felhom-agent/agent.json (0600 felhom-agent) +[OK] agent --selftest (read-only) passed +[OK] felhom-agent service active (non-root felhom-agent reads the config OK) +[OK] wrote /etc/felhom/os-trust.json + operator-signers (felhom-op-1) — the root-owned slow-lane anchors +[STEP] 7/8 golden archive +[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged) +[INFO] fetching golden v0.352.1 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_11_14-11_12_35.tar.zst +[OK] verified sha256 4f411cbeb4b28a7d… matches the hub manifest +[OK] golden imported + verified: local:backup/vzdump-lxc-9100-2026_11_14-11_12_35.tar.zst +[STEP] 8/8 provision guest 9201 +[SKIP] pool felhom already exists +=== felhom-agent 0.173.0 selftest=provision (vmid=9201 customer=Tester4 hostname=tester4) === + --- front half: bring-up (provision) local:backup/vzdump-lxc-9100-2026_11_14-11_12_35.tar.zst → vmid 9201 --- +time=2026-11-14T11:13:43.357+01:00 level=INFO msg="bring-up: pool membership re-asserted" vmid=9201 pool=felhom + [OK] front half: vmid 9201 up (boot+running) in 54s; MAC=BC:24:11:19:43:F5 + --- back half: mint per-guest token + populate bootstrap config mount --- +time=2026-11-14T11:13:48.482+01:00 level=INFO msg="provision: back-half complete" vmid=9201 mount=mp9 guest_path=/etc/felhom-bootstrap endpoint=169.254.253.1:8443 + [OK] back half: bootstrap mount mp9 → /etc/felhom-bootstrap on vmid 9201 (host dir /var/lib/felhom-agent/guests/9201/bootstrap) + local-api endpoint 169.254.253.1:8443 · leaf fp cb19319caff897de5a3eb551625b084f03b79f677342bfd0d6f15b0a3553ce73 · token: minted (not printed) +=== selftest=provision OK — guest 9201 provisioned + bootstrap-mounted (KEPT) === + next: reboot the guest → the golden's baked controller-bootstrap unit deploys the controller, + which PULLS its controller.yaml from the hub (retrieval passphrase) and merges in this local_api. +[OK] provision completed +[INFO] rebooting guest 9201 so the baked controller-bootstrap unit picks up the mount +[STEP] verify +[OK] pct status: running +[OK] onboot: 1 + mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=50G + mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives + rootfs: local-lvm:vm-9201-disk-0,size=32G +[OK] pool: guest 9201 is a member of felhom +[OK] acl: FelhomAgentBase@/ present (user+token) +[OK] acl: FelhomAgentGuest@/pool/felhom present (user+token) +[OK] acl: FelhomAgentStore@/storage/local present (user+token) +[OK] acl: FelhomAgentStore@/storage/local-lvm present (user+token) +[OK] acl: FelhomAgentStore@/storage/felhom-pbs present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990000 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990001 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990002 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990003 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990004 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990005 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990006 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990007 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990008 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990009 present (user+token) +[OK] authz signers: 2 (operator-signed self-update armed) +[OK] controller: Up 20 seconds (healthy) (after ~0s) +[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.352.1 +[WARN] cloudflared not visible yet +[INFO] (confirm in the hub UI that host Tester4-46836f reports guest 9201) + +[OK] Day-0 provision SUCCESS — vmid=9201 host_id=Tester4-46836f customer=Tester4 golden=local:backup/vzdump-lxc-9100-2026_11_14-11_12_35.tar.zst +[INFO] root@pam was rotated + vaulted at step 4b — retrieve at hub → host page (the old GUI password no longer works). diff --git a/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/install-pressed.txt b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/install-pressed.txt new file mode 100644 index 00000000..3507ccfb --- /dev/null +++ b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/install-pressed.txt @@ -0,0 +1 @@ +Install pressed 2026-11-14T10:07:53Z diff --git a/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/keep-02-eula.png b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/keep-02-eula.png new file mode 100644 index 00000000..934350e8 Binary files /dev/null and b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/keep-02-eula.png differ diff --git a/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/keep-03-disk.png b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/keep-03-disk.png new file mode 100644 index 00000000..8f3d1826 Binary files /dev/null and b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/keep-03-disk.png differ diff --git a/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/keep-04-location.png b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/keep-04-location.png new file mode 100644 index 00000000..70d3f4f4 Binary files /dev/null and b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/keep-04-location.png differ diff --git a/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/keep-05-password.png b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/keep-05-password.png new file mode 100644 index 00000000..5403739b Binary files /dev/null and b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/keep-05-password.png differ diff --git a/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/keep-06-network.png b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/keep-06-network.png new file mode 100644 index 00000000..84a0f3b5 Binary files /dev/null and b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/keep-06-network.png differ diff --git a/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/keep-07-summary.png b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/keep-07-summary.png new file mode 100644 index 00000000..efce1b09 Binary files /dev/null and b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/install/keep-07-summary.png differ diff --git a/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/installer-the-one-line.diff.txt b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/installer-the-one-line.diff.txt new file mode 100644 index 00000000..e92bf68c --- /dev/null +++ b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/installer-the-one-line.diff.txt @@ -0,0 +1,9 @@ +installer fetched on DooPlex from https://felhom.eu/scripts/felhom-host-install.sh and copied in (on this LAN the name answers 404 from the retired site, R-1389) +e18cdd17823911e7d87da12fcb8d48b6827f1706a84b3168114fe19fe30aa1f1 /root/hi-orig.sh +197:SCRIPT_VERSION="1.36.1" # the SINGLE version source (F-1): -h and the run banners follow it. +line: 479 +479a480 +> ART_GOLDEN_VER=0.352.1; ART_GOLDEN_SHA=4f411cbeb4b28a7dd65d4121e34e6ebb98f985bd07a48051502ac8d25c01346d +syntax rc=0 +2026-11-14T10:11:24Z +Running as unit: install1114.service; invocation ID: 23e509586d6d40608a427f1cecc16f40 diff --git a/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/teardown.txt b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/teardown.txt new file mode 100644 index 00000000..108d74d8 --- /dev/null +++ b/documentation/tests/golden-0.352.1-2026-11-14/fresh-install/teardown.txt @@ -0,0 +1,8 @@ +## teardown 2026-11-14T10:26:02Z +purging VM 9385 from related configurations.. +Configuration file 'nodes/demo-hp/qemu-server/9385.conf' does not exist +ls: cannot access '/mnt/hdd_1/images/9385': No such file or directory +0 + 341 night1004-tester1 running 8192 200.00 1950 +the installer ISO removed from demo-hp (it was not there before this run) +left for the operator: hub host Tester4-46836f; appliance with smbios uuid 23d48cf6-1984-46a7-9954-557faf333f10 (if it is on the Hosts page) diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 0598ec20..2a3a9ba5 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,3 +1,8 @@ +## Unreleased (after v0.181.0) + +- `bundlefiles`: the line „bundle X holds the same files as the vouched Y” is logged once per pair (INFO), not at + every view of the System page (it was 6 DEBUG lines per view, seen at the 0.181.0 deploy). No behaviour change. + ## v0.181.0 (2026-11-14) — R-1392: root files are „behind” only when the FILES differ; Go 1.27.4 (R-1391) (image `sha256:be2e9e5676eec7622622402a830d8f92913a93fe746e421db60f949138d1f429` built from `83316fc9`, CI run id 2636; the operator's yes 2026-11-14 ~09:25 UTC; the manifest bump is the commit that carries this line) - **The defect (R-1392, found 2026-11-14).** The agent's config bundle carries its `agent_version`, so its sha256 diff --git a/hub/internal/bundlefiles/bundlefiles.go b/hub/internal/bundlefiles/bundlefiles.go index 22cd4b5c..33fec15a 100644 --- a/hub/internal/bundlefiles/bundlefiles.go +++ b/hub/internal/bundlefiles/bundlefiles.go @@ -53,6 +53,20 @@ type Resolver struct { mu sync.Mutex cache map[string]entry + said map[string]bool +} + +func (r *Resolver) firstSight(pair string) bool { + r.mu.Lock() + defer r.mu.Unlock() + if r.said[pair] { + return false + } + if r.said == nil { + r.said = map[string]bool{} + } + r.said[pair] = true + return true } func (r *Resolver) now() time.Time { @@ -135,8 +149,8 @@ func (r *Resolver) Same(boxVersion, boxSHA, vouchedVersion, vouchedSHA string) b } b := r.digest(boxVersion, boxSHA) same := b != "" && a == b - if same { - r.logf("[DEBUG] [bundlefiles] bundle %s holds the same files as the vouched %s (only the label differs)", boxVersion, vouchedVersion) + if same && r.firstSight(boxSHA+"="+vouchedSHA) { // once per pair: the System page asks on every view + r.logf("[INFO] [bundlefiles] bundle %s holds the same files as the vouched %s (only the label differs)", boxVersion, vouchedVersion) } return same }