Release 1 (dates) closed out: controller 0.352.1, hub 0.181.0 live, golden 0.352.1 vouched; R-1392 closed, R-1393 filed, R-1391 narrowed; evidence and report
gates / gates (push) Successful in 2m11s

This commit is contained in:
admin committed 2026-11-14 11:29:50 +01:00
1 parent 63f69b99ce
commit a2173830c1
36 files changed
+1084 -11

No files matched your search

+93
View File
@@ -0,0 +1,93 @@
# REPORT — release 1 of four, the „dates” release (2026-11-14, daytime, attended at two STOPs)
## 1. Claims in the prompt that turned out wrong, named first
1. **„The hub's box facts hold no list of files” — true for the PARSED facts, not for what the box sends.** The box's
host report carries a sha256 per installed root file (`system.facts.host.config_bundle.live`); the hub's parser
(`sysfacts.ConfigBundle`) drops it. The fix did not need it.
2. **„The vouch holds only one checksum of the bundle” — true.** But the hub can get more at any time: it already has
a read-only client for the package registry, and the bundle is a published file. That made a hub-only fix small.
3. **„Go 1.27.4 and its images are published” — true** (go.dev; Docker Hub `golang:1.27.4-trixie` and
`-alpine3.24`, both pinned by digest).
4. **The baseline's „golden 0.352.0 baked 2026-11-11 … plus 7 days” is right**, and moot: a new golden is vouched.
5. **CC's own wrong claim, in the 30-minute hub watch:** the second read was taken 12 seconds before the 30 minutes
were over (10:28:03 against 10:28:15 UTC) and its heading said they had passed. A third read is in the file.
## 2. Five lines
1. **Every box that reports runs controller 0.352.1** (Tester 1, demo-hp, demo-felhom, scratch 9202), built with
**Go 1.27.4** (read from the running binary). The HTTPS header is **unchanged: `max-age=86400`** on all 26 public
names — the one-year setting is NOT in this release (built on 11-14; the words „one day early is OK” were not given).
2. **Hub 0.181.0 is live** since 09:58 UTC, on the operator's yes; Synced/Healthy; 0 ERROR lines.
3. **The false „root files behind” warning is stopped by the fix, not the stopgap.** The hub now compares the files
of the two bundles. Proof that no mail is due on 11-17: the three „behind since” marks in the hub's database are
empty since 09:59:20 UTC, the alarm check sent 0 alarms, and the System page reads „All boxes look fine”.
4. **Golden 0.352.1 is baked, proven by one fresh install, and vouched by the operator** (10:25:43 UTC).
`golden_currency_gate.py`: OK, rc 0.
5. **Not proven:** that a box's report never timed out in the hub's slow first 8 minutes (R-1393); the vouch through
a second channel (read from the hub's log only); that no https trouble happened since 10-15 (not searched, because
the date condition already failed); the agent on Go 1.27.4 (release 2).
## 3. Details
**Rulings recorded (Part 0):** `09-update-architecture.md` §3 decisions 303 (D80 = the root CA copy), 304 (D81 = yes,
with the stopgap word), 305 (the N100's time service: no hand repair; 2026-12-15), 306 (the build order). D80 and D81
are out of the decide table; dated lines on R-1380, R-1392, R-1301, R-1388.
| | Version | Digest | Commit | CI (run id, by commit) |
|---|---|---|---|---|
| Controller | 0.352.1 (rc.1 first) | `sha256:1a6e8945211a…13b6bb` | felhom-controller `8747744b` | 2637 success |
| Hub | 0.181.0 | `sha256:be2e9e5676ee…d1f429` | felhom.eu `83316fc9` (image), `63f69b99` (manifest) | 2636 success, 2638 success (read before the sync) |
| Golden | 0.352.1 | `4f411cbeb4b2…01346d` | build-golden.sh v3.2.0 | — |
| Agent | 0.173.0, not released | — | — | — |
**HSTS (R-1050).** Condition 1 (the date) failed, so the other two were not judged. The check date moved to
2026-11-21 with the reason in the row; `STATUS.md` says how to ask for it.
**Go, the 20-minute reads.** Controller: seven `MaxBytesReader` places, each reads the body to its end in the
handler's own goroutine before it answers; no `TimeoutHandler`. Hub: no `MaxBytesReader` at all. Neither matches the
advisory's crash pattern. The agent has one place; its turn (release 2) is hygiene. Written on R-1391.
**R-1392, the fix.** `hub/internal/bundlefiles` (new): fetch both bundles, accept each only by a sha256 already
trusted, compare everything but `agent_version`. Fail direction: anything unread → „behind”, as before. Seven tests;
the three the prompt asked for were seen red with the fix line switched off. The two real bundles (0.172.1, 0.173.0)
hold the same 32 files. Design text: `11-os-updates.md` §5.4.2 `[FACT]`. No `[DESIGN]` text was needed.
**Rows:** 250 before, 250 after. Opened R-1393 (hub slow after a restart). Closed R-1392. Narrowed R-1391 (the agent
is left). Dated lines: R-1050, R-468, R-1380, R-1301, R-1388. (R-1391 and R-1392 were opened by this morning's health
check, before this run.)
**Machines changed, and how each was left.**
- Tester 1, demo-hp, demo-felhom: controller 0.352.1 by the normal update path; per-box floors cleared.
- Scratch 9202: rc.1, then 0.352.1 by hand through its image file.
- demo-hp: VM 9385 made and destroyed with its disks; the installer ISO fetched and removed.
- DooPlex: two image builds, the bake in the drill VM (now off, disk back on `virgin`), the hub deploy, two copies
of the hub database for reads (shredded). Nothing else.
- Hub: host `Tester4-46836f` is left for the operator to delete (and its appliance entry, if listed). The hub
re-issued Tester4's off-site key and carried its sealed package by itself. No ep0 change by CC.
- The N100: took 0.352.1 like any box; no hand step.
**Architecture documents read:** `09-update-architecture.md` (§3 decisions 247, 249, 250, 251, 304), `11-os-updates.md`
§5.4.2, `05-hub-architecture.md` §17, `runbooks/publish-train-rules.md`, `runbooks/RUNBOOK-manual-build.md` §4,
`tests/golden-0.352.0-2026-11-11/README.md`, the `felhom-build-deploy` skill. **Not read:** `08-alarm-ladder.md`
(the alarm's wording and audience were not changed).
**Capability map:** no row changed — nothing a box newly proves. **System poster:** no listed fact changed; none of
the three acts was needed. **`unproven.py --summary`:** 32 walked, 15 partial, 4 built, 4 missing, 23 not walked —
the tool's numbers; no number was moved by this run.
**Instruction files:** none edited.
**Unreleased after this run:** felhom.eu hub source holds one commit after 0.181.0 — the „same files” log line is
written once per pair, not at every page view (`hub/CHANGELOG.md` „Unreleased”). It rides the next hub release.
## 4. Observations
- The hub is slow for 8 minutes after a restart (24 slow statements, the System page 12.7 s, an enrolment 11.4 s). `FILED: R-1393`.
- The hub logged the „same files” line 6 times per page view. Fixed in source, unreleased. `NOT-A-FINDING: small, fixed in the session (the size rule)`.
- Tester 1 pushed „household cleared its mail address” at the controller's start. `NOT-A-FINDING: the box's own stored setting says so (email_cleared=true) and the start only repeats it; demo-felhom kept its address through the same update. Not checked: when Tester 1's address was cleared.`
- demo-felhom's controller start reads 09:18:23Z, 24 s before the floor that caused it. `NOT-A-FINDING: the N100's clock, R-1301.`
- The hub keeps `bundle_behind_since` marks of two deleted Tester4 hosts. `NOT-A-FINDING: decision 293 (D73) already rules that a host delete removes such rows; not built yet.`
- `os_config_bundle_behind` fired 4 times before today (last 2026-11-06). `NOT-A-FINDING: the same defect, now closed (R-1392).`
- On the LAN the installer address answers 404. `NOT-A-FINDING: R-1389, already on the operator's list.`
- The claim's two status lines are not kept in a file (the script printed them to the session only). `NOT-A-FINDING: an evidence gap of this run, said in the golden's README.`
+2 -2
View File
@@ -2,7 +2,7 @@
**Ready for Tester 2? Yes — tell the friend go (`documentation/pilot/TESTER2-handover.md`).**
**Updated 2026-11-14 (morning).** Health check: the three test boxes are healthy; last night's backups and restore proofs passed. CI was red three nights for one due check; it is read now. The yellow „root files behind" is a false alarm that mails on 11-17 unless you answer D81.
**Updated 2026-11-14 (midday).** Release 1 („dates”) is out: controller 0.352.1 and hub 0.181.0 with the Go security fix; golden 0.352.1 is vouched. The false „root files behind” warning is fixed; no alarm mail is due on 11-17. The one-year HTTPS setting is NOT in (next controller release, from 11-15).
## What works
- **The Plex attack did not reach Felhom's secrets** (R-1241).
@@ -39,7 +39,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
| D7 | R-990, R-989 | Rule on the forum ideas and the selfh.st ideas now, or park them? | **Park them** (P4; nothing waits on them) | Rule now: about one hour of reading | nothing changes; the research gets old | — |
## Waiting on you — do
**2026-11-14.** Build order: four releases (dates, host, „hub tells you”, promises); no new gap-review list first. **2026-11-11.** 1. N100's time service: no hand step now; release 2 repairs it. If it is not live by 2026-12-15: `apparmor_parser -r -T -W /etc/apparmor.d/usr.sbin.chronyd && systemctl restart chrony` (R-1301). 2. Hub Hosts page: Discard appliance `3fb5fed3…` (11-11 test box). 3. Rotate login `kisfenyo`'s Gitea token into the store as `GITEA_GIT_TOKEN` (R-925); the push after `sec put` is in R-1390. 4. Home DNS: remove the `felhom.eu` override (R-1389). 5. Read the dates only you can read (R-1381). 6. Search Console read (R-1035, 11-18).
**2026-11-14.** Next builds: host, „hub tells you”, promises; no new gap list first. Hub Hosts page: delete host `Tester4-46836f` (after 12:15 CET; else it alarms). Say „HSTS now” from 11-15 (R-1050). **2026-11-11.** 1. N100's time service: no hand step now; release 2 repairs it. If not live by 2026-12-15: the hand step in R-1301. 2. Hosts page: discard `3fb5fed3…`. 3. Rotate login `kisfenyo`'s Gitea token into the store as `GITEA_GIT_TOKEN` (R-925); the push after `sec put` is in R-1390. 4. Home DNS: remove the `felhom.eu` override (R-1389). 5. Read the dates only you can read (R-1381). 6. Search Console read (R-1035, 11-18).
**Still needed? (yes/no)** — answer per line.
1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). (since 10-16)
@@ -0,0 +1,19 @@
# Release 1 of four — the „dates” release — evidence (2026-11-14)
Report: `REPORT-release-1-dates-2026-11-14.md` (repo root). Golden: `documentation/tests/golden-0.352.1-2026-11-14/`.
| File | What it shows |
|---|---|
| `controller/08-boxwalk-after.txt` | Tester 1, demo-hp, demo-felhom on 0.352.1 two minutes after the per-box floors: containers 10 / 26 / 9, none unhealthy, 0 ERROR lines, the running binary reads `go1.27.4`, image digest `1a6e8945…` |
| `controller/10-https-only.txt` | `scripts/https_only_check.py`, exit 0 |
| `controller/10b-hsts-header.txt` | every public name of the three boxes: `strict-transport-security: max-age=86400` and nothing more (26 names) — unchanged, HSTS one year is not in this release |
| `controller/11-global-floor.txt` | the floor save (303 `floor_set`), the System page's read-back, the hub's log lines |
| `controller/12-watch.txt` | 35 minutes after the floor: the three boxes, the hub's log with a positive control, the dashboard |
| `controller/13-per-box-floors-cleared.txt` | the three per-box floors cleared |
| `hub/01-before-system-page.txt` | hub 0.180.0: three boxes „root files behind”, behind since 2026-11-10 16:14 UTC |
| `hub/02-deploy.txt` | the sync of Deployment/hub at `63f69b99`, Synced/Healthy, image digest `be2e9e56…`, the start line „bundles are compared by their files” |
| `hub/03-after-system-page.txt` | hub 0.181.0: „All boxes look fine”; the cells say „the same files … only the label differs” |
| `hub/04-behind-since-marks.txt` | the hub database with its WAL (newest host report 30 s old): the three marks emptied at 09:59:20 UTC; the alarm check „0 alarm(s) sent” is in `02`/`05` |
| `hub/05-watch.txt` | the hub's log from the start: 0 ERROR, 24 slow-statement WARN lines in the first 8 minutes (R-1393), none after; the events |
Not kept as files: the rc.1 read on 9202 (healthy, 0 ERROR; in the session only), the CI answers (run ids in the report).
@@ -0,0 +1,33 @@
### root@192.168.0.154 2026-11-14T09:20:35Z
containers=10 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.1 Up About a minute (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.1
image digest: sha256:0d3c28fab7ae2d08e0c92c3c4318316d4920cd440c4c9e3fac4581896314543a repo: gitea.dooplex.hu/admin/felhom-controller@sha256:1a6e8945211a0f1995fd2bd0782053c7f33152fcb00ca465d0596d961a13b6bb
started=2026-11-14T09:18:58.892043895Z
go in the running binary: 2 go1.27.4
errors=0 warn=0 info=139 panics=0
2026/11/14 09:18:59 [INFO] felhom-controller 0.352.1 starting (customer: tester-1, domain: enkicsifelhom.hu)
2026/11/14 09:18:59 [INFO] [selfupdate] Post-update startup: update successful (0.352.0 → 0.352.1)
### demo-hp 2026-11-14T09:20:37Z
containers=26 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.1 Up About a minute (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.1
image digest: sha256:0d3c28fab7ae2d08e0c92c3c4318316d4920cd440c4c9e3fac4581896314543a repo: gitea.dooplex.hu/admin/felhom-controller@sha256:1a6e8945211a0f1995fd2bd0782053c7f33152fcb00ca465d0596d961a13b6bb
started=2026-11-14T09:18:59.079611972Z
go in the running binary: 2 go1.27.4
errors=0 warn=3 info=147 panics=0
2026/11/14 09:18:59 main.go:335: [INFO] felhom-controller 0.352.1 starting (customer: demo-hp, domain: enkisfelhom.hu)
2026/11/14 09:19:00 deploy.go:1470: [WARN] [stacks] Stack bookstack: new field ADMIN_PASSWORD (type=password) requires manual configuration
2026/11/14 09:19:00 deploy.go:1470: [WARN] [stacks] Stack calibre-web: new field ADMIN_PASSWORD (type=password) requires manual configuration
2026/11/14 09:19:00 deploy.go:1470: [WARN] [stacks] Stack plex: new field PLEX_CLAIM (type=text) requires manual configuration
2026/11/14 09:19:00 updater.go:855: [INFO] [selfupdate] Post-update startup: update successful (0.352.0 → 0.352.1)
### felhom-pve 2026-11-14T09:20:39Z
containers=9 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.1 Up About a minute (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.1
image digest: sha256:0d3c28fab7ae2d08e0c92c3c4318316d4920cd440c4c9e3fac4581896314543a repo: gitea.dooplex.hu/admin/felhom-controller@sha256:1a6e8945211a0f1995fd2bd0782053c7f33152fcb00ca465d0596d961a13b6bb
started=2026-11-14T09:18:23.620814926Z
go in the running binary: 2 go1.27.4
errors=0 warn=0 info=154 panics=0
2026/11/14 09:18:24 [INFO] felhom-controller 0.352.1 starting (customer: demo-felhom, domain: demo-felhom.eu)
2026/11/14 09:18:24 [INFO] [selfupdate] Post-update startup: update successful (0.352.0 → 0.352.1)
@@ -0,0 +1,36 @@
== 2026-11-14T09:20:49Z python3 scripts/https_only_check.py
control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -)
== tester-1 (enkicsifelhom.hu): 6 public name(s)
OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1
OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1
OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1
OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1
OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1
OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1
== demo-hp (enkisfelhom.hu): 15 public name(s)
OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1
OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1
OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1
OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1
OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1
OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1
OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1
OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1
OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1
OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1
OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1
OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1
OK http://time.enkisfelhom.hu/r1033-check?q=1 -> 301 https://time.enkisfelhom.hu/r1033-check?q=1
OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1
OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1
== demo-felhom (demo-felhom.eu): 8 public name(s)
OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1
OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1
OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1
OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1
OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1
OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1
OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1
OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
== every public name redirects http -> https
exit=0
@@ -0,0 +1,53 @@
== 2026-11-14T09:20:59Z Strict-Transport-Security of each public name (GET https://<name>/, header only; unchanged in 0.352.1 — HSTS one year is NOT in this release)
https://arcade.enkisfelhom.hu strict-transport-security: max-age=86400
https://books.enkisfelhom.hu strict-transport-security: max-age=86400
https://docs.enkisfelhom.hu strict-transport-security: max-age=86400
https://felhom.demo-felhom.eu strict-transport-security: max-age=86400
https://felhom.enkicsifelhom.hu strict-transport-security: max-age=86400
https://felhom.enkisfelhom.hu strict-transport-security: max-age=86400
https://files.demo-felhom.eu strict-transport-security: max-age=86400
https://files.enkicsifelhom.hu strict-transport-security: max-age=86400
https://files.enkisfelhom.hu strict-transport-security: max-age=86400
https://gist.demo-felhom.eu strict-transport-security: max-age=86400
https://gist.enkisfelhom.hu strict-transport-security: max-age=86400
https://ha.enkisfelhom.hu strict-transport-security: max-age=86400
https://paperless.enkicsifelhom.hu strict-transport-security: max-age=86400
https://papir.enkisfelhom.hu strict-transport-security: max-age=86400
https://paste.enkicsifelhom.hu strict-transport-security: max-age=86400
https://paste.enkisfelhom.hu strict-transport-security: max-age=86400
https://pdf.enkisfelhom.hu strict-transport-security: max-age=86400
https://plex.enkisfelhom.hu strict-transport-security: max-age=86400
https://prowlarr.demo-felhom.eu strict-transport-security: max-age=86400
https://radarr.demo-felhom.eu strict-transport-security: max-age=86400
https://sonarr.demo-felhom.eu strict-transport-security: max-age=86400
https://time.enkisfelhom.hu strict-transport-security: max-age=86400
https://torrent.demo-felhom.eu strict-transport-security: max-age=86400
https://travel.enkisfelhom.hu strict-transport-security: max-age=86400
https://wiki.enkicsifelhom.hu strict-transport-security: max-age=86400
https://wiki.enkisfelhom.hu strict-transport-security: max-age=86400
@@ -0,0 +1,10 @@
== 2026-11-14T09:21:15Z global floor 0.352.1, MinAgent 0.158.1 (CC's hub token)
== /configuration/global-floor
HTTP/1.1 303 See Other
Location: /configuration?flash=floor_set
Global controller floor: 0.352.1 · vouched agent: 0.173.0
2026/11/14 10:18:51 [INFO] managed floor SERVED for tester-1: floor 0.352.1, agent requirement "0.158.1" from declared (golden 0.352.0)
2026/11/14 10:18:51 [INFO] managed floor SERVED for demo-felhom: floor 0.352.1, agent requirement "0.158.1" from declared (golden 0.352.0)
2026/11/14 10:18:51 [INFO] managed floor SERVED for demo-hp: floor 0.352.1, agent requirement "0.158.1" from declared (golden 0.352.0)
2026/11/14 10:21:15 [INFO] Global controller-version floor set to "0.352.1" (declared MinAgent "0.158.1")
2026/11/14 10:21:15 [INFO] access token "cc-releases-2": POST /configuration/global-floor → 303
@@ -0,0 +1,52 @@
== controller watch: global floor 09:21:15Z; read 2026-11-14T09:56:21Z
-- the three boxes (read on the box, over SSH)
### root@192.168.0.154 2026-11-14T09:56:21Z
containers=10 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.1 Up 37 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.1
image digest: sha256:0d3c28fab7ae2d08e0c92c3c4318316d4920cd440c4c9e3fac4581896314543a repo: gitea.dooplex.hu/admin/felhom-controller@sha256:1a6e8945211a0f1995fd2bd0782053c7f33152fcb00ca465d0596d961a13b6bb
started=2026-11-14T09:18:58.892043895Z
go in the running binary: 2 go1.27.4
errors=0 warn=0 info=787 panics=0
2026/11/14 09:18:59 [INFO] felhom-controller 0.352.1 starting (customer: tester-1, domain: enkicsifelhom.hu)
2026/11/14 09:18:59 [INFO] [selfupdate] Post-update startup: update successful (0.352.0 → 0.352.1)
### demo-hp 2026-11-14T09:56:22Z
containers=26 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.1 Up 37 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.1
image digest: sha256:0d3c28fab7ae2d08e0c92c3c4318316d4920cd440c4c9e3fac4581896314543a repo: gitea.dooplex.hu/admin/felhom-controller@sha256:1a6e8945211a0f1995fd2bd0782053c7f33152fcb00ca465d0596d961a13b6bb
started=2026-11-14T09:18:59.079611972Z
go in the running binary: 2 go1.27.4
errors=0 warn=3 info=808 panics=0
2026/11/14 09:18:59 main.go:335: [INFO] felhom-controller 0.352.1 starting (customer: demo-hp, domain: enkisfelhom.hu)
2026/11/14 09:19:00 deploy.go:1470: [WARN] [stacks] Stack bookstack: new field ADMIN_PASSWORD (type=password) requires manual configuration
2026/11/14 09:19:00 deploy.go:1470: [WARN] [stacks] Stack calibre-web: new field ADMIN_PASSWORD (type=password) requires manual configuration
2026/11/14 09:19:00 deploy.go:1470: [WARN] [stacks] Stack plex: new field PLEX_CLAIM (type=text) requires manual configuration
2026/11/14 09:19:00 updater.go:855: [INFO] [selfupdate] Post-update startup: update successful (0.352.0 → 0.352.1)
### felhom-pve 2026-11-14T09:56:24Z
containers=9 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.352.1 Up 37 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.352.1
image digest: sha256:0d3c28fab7ae2d08e0c92c3c4318316d4920cd440c4c9e3fac4581896314543a repo: gitea.dooplex.hu/admin/felhom-controller@sha256:1a6e8945211a0f1995fd2bd0782053c7f33152fcb00ca465d0596d961a13b6bb
started=2026-11-14T09:18:23.620814926Z
go in the running binary: 2 go1.27.4
errors=0 warn=0 info=761 panics=0
2026/11/14 09:18:24 [INFO] felhom-controller 0.352.1 starting (customer: demo-felhom, domain: demo-felhom.eu)
2026/11/14 09:18:24 [INFO] [selfupdate] Post-update startup: update successful (0.352.0 → 0.352.1)
-- hub log since the floor: events, alarms, error lines
-- hub dashboard
Demo Ügyfél OK — 7 min ago | 0.352.1
Demo HP OK — 7 min ago | 0.352.1
Tester 1 OK — 7 min ago | 0.352.1
-- positive control for the empty list above: the hub log of the window has 169 lines; by kind:
126 [INFO]
43 [DEBUG]
-- lines that name an event or a mail:
2026/11/14 10:18:57 [INFO] Notification preferences updated for demo-felhom: address set=true, household=true, events=[app_stopped_unhealthy app_update_held app_update_undone backup_failed db_dump_fai
2026/11/14 10:19:00 [INFO] Event from tester-1: controller_updated (info) — Controller frissítve: 0.352.0 → 0.352.1
2026/11/14 10:19:00 [INFO] Notification prefs push for tester-1: household cleared its mail address — deleted
2026/11/14 10:19:00 [INFO] Notification preferences updated for tester-1: address set=false, household=true, events=[]
2026/11/14 10:19:00 [INFO] Event from demo-hp: controller_updated (info) — Controller frissítve: 0.352.0 → 0.352.1
2026/11/14 10:19:02 [INFO] Event from demo-felhom: controller_started (info) — Controller elindult (0.352.1)
2026/11/14 10:19:05 [INFO] Event from tester-1: controller_started (info) — Controller elindult (0.352.1)
2026/11/14 10:19:05 [INFO] Event from demo-hp: controller_started (info) — Controller elindult (0.352.1)
@@ -0,0 +1,10 @@
== 2026-11-14T09:56:38Z per-box floors cleared
== /customers/tester-1/floor
HTTP/1.1 303 See Other
Location: /customers/tester-1?flash=floor_set
== /customers/demo-hp/floor
HTTP/1.1 303 See Other
Location: /customers/demo-hp?flash=floor_set
== /customers/demo-felhom/floor
HTTP/1.1 303 See Other
Location: /customers/demo-felhom?flash=floor_set
@@ -0,0 +1,3 @@
== 2026-11-14T09:54:09Z BEFORE the deploy (hub 0.180.0): the System page's Root files cells and Needs-attention lines
9 title="behind the vouched agent 0.173.0&#39;s bundle — send it with a signed agent_config_update (behind since 2026-11-10 16:14 UTC)"
3
@@ -0,0 +1,13 @@
== 2026-11-14T09:58:11Z sync: Deployment/hub only, revision 63f69b99
application.argoproj.io/felhom patched
Waiting for deployment "hub" rollout to finish: 0 of 1 updated replicas are available...
deployment "hub" successfully rolled out
sync=Synced rev=63f69b99ceff84d83d44144c91b73e11bdc152d0 health=Healthy op=Succeeded
gitea.dooplex.hu/admin/felhom-hub:0.181.0
hub-dbfb69758-mzvtk gitea.dooplex.hu/admin/felhom-hub@sha256:be2e9e5676eec7622622402a830d8f92913a93fe746e421db60f949138d1f429 ready=true restarts=0
2026/11/14 10:58:15 [INFO] felhom-hub 0.181.0 starting
2026/11/14 10:58:17 [INFO] Root files: bundles are compared by their files (R-1392)
2026/11/14 10:58:21 [WARN] slow statement: 2570ms (over 1s): DELETE FROM events WHERE created_at < ?
2026/11/14 10:58:25 [WARN] slow statement: 3549ms (over 1s): DELETE FROM app_log_issues WHERE last_seen < ?
2026/11/14 10:58:28 [WARN] slow statement: 3051ms (over 1s): SELECT r.customer_id, r.received_at, r.report_json, r.health_status, r.cpu_percent, r.memory_percent, r.container_total, r.container_running, r.backup_last_sna
2026/11/14 10:58:32 [WARN] slow statement: 3816ms (over 1s): SELECT hr.host_id, hr.customer_id, hr.report_json FROM host_reports hr JOIN (SELECT host_id, MAX(id) AS mx FROM host_reports GROUP BY host_id) latest ON hr.id
@@ -0,0 +1,7 @@
== 2026-11-14T09:59:20Z AFTER the deploy (hub 0.181.0): the System page
lines 'root files behind the vouched': 0
cells titled 'behind the vouched agent': 0
6 title="the same files as the vouched agent 0.173.0&#39;s bundle (only the label differs)"
Needs attention
One line per box that is amber or red, and why. Amber: worth a look. Red: an operator alarm fires (`08` §6.3).
All boxes look fine.
@@ -0,0 +1,17 @@
== hub DB copy with its WAL, read 2026-11-14T10:00:21Z
freshness: quick_check ok | newest host report 2026-11-14 09:59:52 | db now 2026-11-14 10:00:22
hub_settings columns ['key', 'value', 'updated_at']
-- bundle_behind_since marks (empty value = not behind):
('bundle_behind_since:Tester4-16fdef', '2026-10-23T08:05:18Z', '2026-10-23 08:05:20')
('bundle_behind_since:Tester4-84bbe7', '2026-10-25T22:55:53Z', '2026-10-25 22:55:54')
('bundle_behind_since:demo-felhom-8363b5', '', '2026-11-14 09:59:20')
('bundle_behind_since:demo-hp-bb76ea', '', '2026-11-14 09:59:20')
('bundle_behind_since:tester-1-d70be4', '', '2026-11-14 09:59:20')
-- control, same table: agent_behind_since marks and the floor:
('agent_behind_since:Tester-2-be8404', '2026-10-05T09:15:29Z', '2026-10-05 09:15:30')
('agent_behind_since:Tester4-16fdef', '2026-10-23T08:05:18Z', '2026-10-23 08:05:20')
('agent_behind_since:demo-felhom-8363b5', '', '2026-10-14 12:43:27')
('agent_behind_since:tester-1-d70be4', '', '2026-10-14 12:43:27')
('min_controller_version', '0.352.1', '2026-11-14 09:21:15')
('min_controller_version_declared_min_agent', '0.352.1=0.158.1', '2026-11-14 09:21:15')
-- os_config_bundle_behind events ever: (4, '2026-11-06 10:53:20')
@@ -0,0 +1,55 @@
== hub watch: 0.181.0 live since 09:58:15Z; read 2026-11-14T10:26:37Z
sync=Synced health=Healthy
hub-dbfb69758-mzvtk ready=true restarts=0
log lines since the start: 244; by kind:
181 [INFO]
38 [DEBUG]
24 [WARN]
1 [offsite] re-issued shared credentials for Tester4 (subaccount 343429)
-- every [ERROR] and [WARN] line:
2026/11/14 10:58:21 [WARN] slow statement: 2570ms (over 1s): DELETE FROM events WHERE created_at < ?
2026/11/14 10:58:25 [WARN] slow statement: 3549ms (over 1s): DELETE FROM app_log_issues WHERE last_seen < ?
2026/11/14 10:58:28 [WARN] slow statement: 3051ms (over 1s): SELECT r.customer_id, r.received_at, r.report_json, r.health_status, r.cpu_percent, r.memory_percent, r.container_total, r.container_running, r.backup_last_snapshot, r.c
2026/11/14 10:58:32 [WARN] slow statement: 3816ms (over 1s): SELECT hr.host_id, hr.customer_id, hr.report_json FROM host_reports hr JOIN (SELECT host_id, MAX(id) AS mx FROM host_reports GROUP BY host_id) latest ON hr.id = latest.m
2026/11/14 10:59:11 [WARN] slow statement: 4053ms (over 1s): SELECT id, host_id, layer, received_at, trigger, mode, outcome, healthy, release_id, report_json FROM os_reports WHERE host_id = ? AND layer = ? ORDER BY id DESC LIMIT ?
2026/11/14 10:59:15 [WARN] slow statement: 3888ms (over 1s): SELECT id, host_id, layer, received_at, trigger, mode, outcome, healthy, release_id, report_json FROM os_reports WHERE host_id = ? AND layer = ? ORDER BY id DESC LIMIT ?
2026/11/14 10:59:19 [WARN] slow statement: 1152ms (over 1s): SELECT id, host_id, layer, received_at, trigger, mode, outcome, healthy, release_id, report_json FROM os_reports WHERE host_id = ? AND layer = ? ORDER BY id DESC LIMIT ?
2026/11/14 10:59:19 [WARN] slow statement: 3911ms (over 1s): SELECT id, host_id, layer, received_at, trigger, mode, outcome, healthy, release_id, report_json FROM os_reports WHERE host_id = ? AND layer = ? ORDER BY id DESC LIMIT ?
2026/11/14 10:59:20 [WARN] slow request: GET /system 200 12656ms (over 5s)
2026/11/14 11:00:05 [WARN] slow statement: 32362ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC
2026/11/14 11:00:18 [WARN] slow statement: 12036ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC
2026/11/14 11:01:39 [WARN] slow statement: 7245ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC
2026/11/14 11:01:43 [WARN] slow statement: 2425ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC
2026/11/14 11:01:47 [WARN] slow statement: 2490ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC
2026/11/14 11:02:37 [WARN] slow statement: 4590ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC
2026/11/14 11:02:39 [WARN] slow statement: 1045ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC
2026/11/14 11:02:42 [WARN] slow statement: 1355ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC
2026/11/14 11:03:02 [WARN] slow statement: 1857ms (over 1s): SELECT id, customer_id, event_type, severity, message, details_json, source, created_at FROM events WHERE customer_id = ? AND event_type IN ('wg_tunnel_stale', 'wg_tunne
2026/11/14 11:03:35 [WARN] slow statement: 2580ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC
2026/11/14 11:03:37 [WARN] slow statement: 1254ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC
2026/11/14 11:03:39 [WARN] slow statement: 1233ms (over 1s): SELECT received_at, report_json FROM host_reports WHERE customer_id = ? AND received_at >= ? ORDER BY received_at DESC
2026/11/14 11:06:33 [WARN] slow statement: 1407ms (over 1s): SELECT id, customer_id, event_type, severity, message, details_json, source, created_at FROM events WHERE customer_id = ? AND event_type IN ('wg_tunnel_stale', 'wg_tunne
2026/11/14 11:12:13 [WARN] slow request: POST /api/v1/host-enroll 201 11357ms (over 5s)
2026/11/14 11:12:34 [WARN] escrow CARRIED for Tester4: the sealed package of deleted host Tester4-d35551 is now the current package of Tester4-46836f (repo-key hash 5501e4bf2cda…); the box will ask for the recovery code
-- events and mails:
2026/11/14 10:58:18 [INFO] osupdates: alarms = no OS leg 168h0m0s, reboot needed 336h0m0s, ring 0 stalled 168h0m0s, not covered 336h0m0s
2026/11/14 10:58:28 [INFO] Host staleness checker initialized: 3 ok, 0 already alarmed (seeded silently, R-1001), 0 stale, 0 down (left unseeded → first Check emits; host_stale after 45m0s, host_dow
2026/11/14 10:59:20 [INFO] osupdates: alarm check done in 1.847s, 0 alarm(s) sent []
2026/11/14 11:12:02 [INFO] claim reenroll email sent to the registered address of Tester4
2026/11/14 11:13:51 [INFO] Event from Tester4: controller_started (info) — Controller elindult (0.352.1)
2026/11/14 11:14:11 [INFO] Event from Tester4: controller_started (info) — Controller elindult (0.352.1)
2026/11/14 11:15:37 [INFO] Golden vouch check: golden 0.352.1 is published, the hub vouches "0.352.0" — alarm in 2h0m0s if still so
2026/11/14 11:19:06 [INFO] Event from Tester4: backup_tier_skipped (warning) — Whole-guest backup tier felhom-pbs skipped: its storage does not exist on the host (never provisioned or removed). No a
2026/11/14 11:22:07 [INFO] Event from Tester4: app_deploy_started (info) — Alkalmazás telepítése elindult: BentoPDF
2026/11/14 11:23:49 [INFO] Event from Tester4: app_deployed (info) — Alkalmazás telepítve: BentoPDF
== second read 2026-11-14T10:28:03Z (12 seconds BEFORE the 30 minutes after 09:58:15Z were over — see the third read)
sync=Synced health=Healthy
hub-dbfb69758-mzvtk ready=true restarts=0
[ERROR] lines since the start: 0 panic: 0 slow statements in the last 20 min: 0
-- events of the three test boxes and operator mails since the deploy:
(end of list)
== third read 2026-11-14T10:29:35Z
sync=Synced health=Healthy
hub-dbfb69758-mzvtk ready=true restarts=0
lines 251 | [ERROR] 0 | panic 0 | slow statements after 10:10Z: 0
operator mailbox (label Monitoring, a second channel), read 10:28Z: no alarm mail since 09:18Z; the one new mail is Tester4's setup code of 10:12:03Z (this run's install)
+8
View File
@@ -26,6 +26,14 @@
---
## 2026-11-14 — R-1392 closed: root files are „behind” only when the files differ (full text of the row: `git show 561a4d98:documentation/backlog/OPEN-ITEMS.md`)
| Row | What | Closed | Evidence |
|---|---|---|---|
| **R-1392** | **Every agent release whose `configs/` did not change turned every box's „Root files” amber, and after 7 days the hub mailed `os_config_bundle_behind`, although the files on the box were the vouched ones.** (P3) | CLOSED 2026-11-14 — hub 0.181.0 (D81 = yes, `09` §3 decision 304): the hub fetches both bundles, accepts each only by a sha256 it already trusts, and compares everything but the label. Live since 09:58 UTC: the System page reads „All boxes look fine”; the three „behind since 2026-11-10 16:14” marks were cleared at 09:59:20 UTC (read from the hub database with its WAL), so no mail was due on 2026-11-17. The stopgap („send the bundle”) was not needed | `audits/release-1-dates-2026-11-14/hub/`; `11-os-updates.md` §5.4.2; hub `83316fc9` |
**Reasoning kept.** Anything the hub cannot read, verify or parse answers „not the same” — the old verdict. A registry that is away never makes a box look current.
## 2026-11-07 — R-1228 closed: DooPlex's copy of the website retired (full text of the row: `git show 7e56d166:documentation/backlog/OPEN-ITEMS.md`)
| Row | What | Closed | Evidence |
+7 -7
View File
@@ -268,12 +268,12 @@ stopping line that lies.
| **R-779** | Security & access | P4 | **[P3-LOW] Part A's "two outside addresses seen as two" is proven through the simulated tunnel only; on the REAL tunnel the second outside address (ep0, one request allowed) was refused by Cloudflare's edge with 403 and never reached the box.** Measured 2026-10-01 19:51 UTC (`audits/visitors-2026-10-01/A/L2-demo-hp-real-tunnel.txt`): no log line on demo-hp; demo-hp's box has no geo restriction in its settings, so a Cloudflare ZONE rule (country or bot, not read) refused a German datacenter address. DooPlex's own address on the real tunnel was seen as itself. **Needs:** one sign-in from a second Hungarian address (the operator's phone off wifi) while DooPlex is locked out — 2 minutes; and say which Cloudflare rule refused ep0. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (a phone), CC reads the logs** **Re-ranked 2026-10-03: P3→P4: a proof gap on the real tunnel; operator-only follow-up.** | — | — | CC + operator |
| **R-904** | Security & access | P4 | **Cloudflare can read every household's app traffic; replacing it with our own relay is a later item.** Facts (reviewer discussion 2026-10-08; `01`): app traffic and the dashboard reach the box through the Cloudflare Tunnel (`01` §5 trust table, rows end-user ↔ apps and customer ↔ controller UI; §7), and the tunnel's public end is Cloudflare's edge, where TLS ends — so Cloudflare can technically read that traffic (the FAQ says so since 2026-10-08, R-900; „TLS ends at the edge" is not written in `01` — add it there). What Cloudflare gives today, free: inbound reach with no router setup, the CGNAT answer (`01` §4, §7); certificates (`01` §7, the free tier covers one level below a zone); the geo-WAF the hub enforces (`01` §5 last row, §7); flood protection (not in `01`). The alternative named: our own EU relay over WireGuard with TLS passthrough by SNI, certificates on the box, the geo-block on the relay. Its costs: one more machine the operator keeps up, and a single point of reach for every box; weaker flood protection; about a week of work after a spike. Operator ruling 2026-10-08 09:07 (`09` §3 decision 184): a later item. | **DEFERRED** — **2026-10-10: the asked sentence is in `01` §7** („TLS ends at Cloudflare's edge …”); the spike stays deferred. **DEFERRED — after the first customers (operator ruling 2026-10-08 09:07)** | — | A spike after the first customers (the relay's reach, cost and flood behaviour, measured) | operator |
| **R-1040** | Security & access | P4 | **Until the household makes Sonarr's or Radarr's first admin, any container on the box can read the app's API key.** Measured on the bench 2026-10-14 with the R-1030 seed in place: `AuthenticationRequired=Enabled`, `AuthenticationMethod=None` — `/initialize.json` from another container answers 200 with `apiKey`. Outsiders are blocked by the box's setup gate in that window; a compromised other app is not. | **OPEN** — filed 2026-10-14 | — | Seed the API key at install (`SONARR__AUTH__APIKEY`, measured to work) so the box knows it, and close the window by telling the household in the first steps; or seed a Forms user | CC |
| **R-1050** | Security & access | P4 | **Raise HSTS to one year on or after 2026-11-15, if no https trouble was reported.** `09` §3 decision 249 (D2 = A, operator 2026-10-15): HSTS stays `max-age=86400` now; after a month with no https trouble it becomes one year (`31536000`); never preload, never `includeSubDomains`. The value is `HSTSMaxAgeSeconds` in `felhom-controller/controller/internal/infra/infra.go`, pinned by `TestRenderForwardedHeaders_HTTPSOnly`. „Trouble” = any report of a name that will not open, or a box answering http after the floor. | **WATCHING** — filed 2026-10-15 | the date | On or after 2026-11-15: read the register and the operator's mail for https trouble since 2026-10-15; none → a controller release with one year (test updated), bake → vouch → floor | CC |
| **R-1050** | Security & access | P4 | **Raise HSTS to one year on or after 2026-11-15, if no https trouble was reported.** `09` §3 decision 249 (D2 = A, operator 2026-10-15): HSTS stays `max-age=86400` now; after a month with no https trouble it becomes one year (`31536000`); never preload, never `includeSubDomains`. The value is `HSTSMaxAgeSeconds` in `felhom-controller/controller/internal/infra/infra.go`, pinned by `TestRenderForwardedHeaders_HTTPSOnly`. „Trouble” = any report of a name that will not open, or a box answering http after the floor. | WATCHING — **2026-11-14: NOT in controller 0.352.1.** That release was built on 2026-11-14, one day before the ruled date, and the operator's words „one day early is OK” were not given. Check date moved to 2026-11-21 (one week: the next day a controller release is likely; a one-day move would only turn the nightly CI red with nothing to release). The search for https trouble since 2026-10-15 is NOT done yet. Earlier: **WATCHING** — filed 2026-10-15 | the date | On or after 2026-11-15: read the register and the operator's mail for https trouble since 2026-10-15; none → a controller release with one year (test updated), bake → vouch → floor | CC |
| **R-1054** | Security & access | P3 | **The hub's operator/global API key is still one bearer that can do everything on `/api/v1`, including reading every box's break-glass root password (`GET /admin/hosts/{id}/recovery-credential`).** Hub v0.152.0 gave the web surface scoped tokens (`05` §17) but tokens are refused on `/api/v1` by design. Who uses the global key today is not listed in one place. Fix: list its consumers; move each operator use to a scoped token on the web side (or a token class on the API), and keep the global key only where a box path needs it — or retire it. | OPEN — filed 2026-10-15 by the token build | — | List the global key's consumers (pod env, scripts, runbooks) | CC |
| **R-1194** | Security & access | P3 | **On a fresh box the operator's break-glass SSH door was broken from the first minute: `oob_degraded` „felhom-sshd config invalid (sshd -t fails)".** Rehearsal 2026-10-20, Tester4-816e21, mail at 12:03 UTC (one minute after the host record was created): details `active:false, config_invalid:true, felhom_sshd_port:0, operator_key_configured:false, operator_key_reported:true, reachable:false`. Possibly the shape of a customer with no operator key set (Tester4 is a test customer) — then the alarm is a false one on every such box; or a real config defect. Not diagnosed (no shell). Checked the same day: the only `oob_degraded` in the operator mailbox in 30 days, and no recovery mail followed; the report says port 0. Two candidate mechanisms in the agent (`internal/felhomsshd/health.go:63-66`): `sshd -t -f <conf>` run by the NON-root agent fails on unreadable host keys, or a config rendered before the desired state arrived (port 0). **CAUSE FOUND AND FIXED 2026-10-21 night** (`audits/night-2026-10-21/P5-checks/`): reproduced on a fresh box from the public ISO — `oob_degraded` mailed 20:34 CEST while felhom-sshd was applied 0.4 s after the check and is healthy (port 8822 listening, `sshd -t` passes as the agent's user through its sudo grant — so the non-root theory is FALSE; without sudo it fails on the host key, which the agent never does). The first host report runs before the felhom-sshd loop's first pass and reads a config never rendered. Fix: agent `0da792e` — no OOB stanza until the first pass has run (the hub does not evaluate a missing one; a failed pass counts). `TestR1194_NoOOBStanzaBeforeTheFirstPass` red-proved. | WATCHING — released in agent v0.162.0 (2026-10-22, operator yes; on the 3 test boxes); the false alarm comes only on a FRESH box's first report | the next fresh install with agent ≥ 0.162.0 | At the next fresh install read the hub's events for `oob_degraded` in the first hour (none expected); then close | CC |
| **R-1237** | Security & access | P3 | **The nightly CC-secrets sweep found stored one-time codes in a session record dated 2026-10-23: 14 names in one file (bind tokens and setup codes of test boxes — NIGHT_2026_10_23_*, FIXES_2026_10_21_*, NIGHT_2026_10_21_*, REHEARSAL_2026_10_20_* — and R_DEMO_FELHOM in a file of 2026-10-20).** Mail „[DooPlex CC secrets] 2 secret name(s) found in old session records", 2026-10-23 01:57 UTC, read in the day session (names only). One file holding all of them suggests one command printed many stored values at once (a listing or a dump) in the night session. All but R_DEMO_FELHOM are one-time install codes of test boxes; NIGHT_2026_10_23_* die when Tester4-16fdef is deleted (the operator's do list). | **OPEN** — 2026-10-27 (operator yes): the dead names were REMOVED from the store with `sec rm` (46, one commit each; `audits/r1299-window-button-2026-10-27/sec-rm.txt`, names only): every one-time code this row names is gone; R_DEMO_FELHOM stays (unsure). Consequence: the nightly sweep no longer watches for the removed values — they open nothing (their boxes are destroyed). The session transcripts themselves are unchanged — that question stays here. **Earlier:** OPEN | — | CC: find the file and the command by the sweep's own report (names only), add a hook shape for that command if it is not one; R_DEMO_FELHOM (demo-felhom's recovery code) — the operator decides whether to rotate it | CC |
| **R-1248** | Security & access | P3 | **An app's own numbers can lie: an attacker with code inside an app's container can answer the app's API with whatever matches its flood, and the upload watch (R-1244) would then call it explained.** Measured remedy, not built: a per-container upload ceiling with `tc` from a helper container in the app's network namespace — on scratch 9202 (unprivileged) a flood was held at 19.9–20.1 Mbit/s and the app kept answering (`audits/SPIKE-upload-watch-2026-10-23.md` §5). Costs: a helper image with `tc` (a new external image, or `tc` in the controller image), `NET_ADMIN` for that helper, re-applied after each app restart. | OPEN | an operator decision (the new image / capability) | Decide; if yes, CC builds the ceiling for the download and media apps first | operator |
| **R-1391** | Security & access | P3 | **The hub, the controller and the agent are built on Go 1.27.3, which has GO-2026-6859 (an HTTP/1 server can crash with a runtime panic when a `MaxBytesReader` limit is hit while the handler reads the body and writes a header at the same time); fixed in Go 1.27.4.** Found 2026-11-14 (health-check session) from the security watch's 2026-11-11 mail, which names only the hub image (`golang 1.27.3-alpine3.24`); `go.mod` in all three repos reads `go 1.27.3`. Reach not measured: the crash needs a handler that reads the request body concurrently with writing the response header — whether any Felhom handler does is not read yet. A crash restarts the process (the controller's supervisor, the agent's unit, the hub's pod), so the effect is an outage of seconds, not data loss. Source: https://vuln.go.dev/ID/GO-2026-6859.json | OPEN — filed 2026-11-14 | — | build each component's next release on Go 1.27.4 (`go.mod` + the build image); until then, grep each repo for `MaxBytesReader`/`MaxBytesHandler` and a handler that writes headers while a goroutine reads the body | CC |
| **R-1391** | Security & access | P3 | **The hub, the controller and the agent are built on Go 1.27.3, which has GO-2026-6859 (an HTTP/1 server can crash with a runtime panic when a `MaxBytesReader` limit is hit while the handler reads the body and writes a header at the same time); fixed in Go 1.27.4.** Found 2026-11-14 (health-check session) from the security watch's 2026-11-11 mail, which names only the hub image (`golang 1.27.3-alpine3.24`); `go.mod` in all three repos reads `go 1.27.3`. Reach not measured: the crash needs a handler that reads the request body concurrently with writing the response header — whether any Felhom handler does is not read yet. A crash restarts the process (the controller's supervisor, the agent's unit, the hub's pod), so the effect is an outage of seconds, not data loss. Source: https://vuln.go.dev/ID/GO-2026-6859.json | NARROWED — **2026-11-14: the controller (0.352.1, global floor 09:21 UTC, on every box that reports) and the hub (0.181.0, live 09:58 UTC) are built on Go 1.27.4. LEFT: the agent (still 1.27.3), release 2.** Reach read the same day: no controller handler matches the crash pattern (seven `MaxBytesReader` places, each reads the body to its end before it answers; no `TimeoutHandler`); the hub has no `MaxBytesReader` at all; the agent has ONE (`internal/localapi/server.go`, a JSON decode in the handler's own goroutine — not read further). So the agent's turn is hygiene, not urgent. Evidence `audits/release-1-dates-2026-11-14/`. Earlier: OPEN — filed 2026-11-14 | — | build each component's next release on Go 1.27.4 (`go.mod` + the build image); until then, grep each repo for `MaxBytesReader`/`MaxBytesHandler` and a handler that writes headers while a goroutine reads the body | CC |
| **R-1309** | Security & access | P4 | **What no automated source shows: a fix announced only by mail, forum or a blog we do not read; a security fix nobody calls one (cloudflared 2026.10.2's „bump distroless image"); closed-source apps (Plex, Emby).** The watch (R-1303) shows these only as BEHIND (weekly). The vendor mails the operator receives stay a source. `audits/security-watch-2026-10-26/SPIKE.md` | OPEN — filed 2026-10-26 (a stated limit) | — | none until a vendor offers a feed; add a vendor's feed to `security_watch.FEEDS` when one exists | CC |
## Box system & updates — 21 rows (P3 16, P4 5)
@@ -292,7 +292,7 @@ stopping line that lies.
| **R-1214** | Box system & updates | P3 | **On the 8 GB box with the swap fix, the night's second-drive backup still pushes memory „full" pressure above 20 % for about 3 minutes; the box keeps answering.** Night soak 2026-10-21→22 (`audits/night-2026-10-21/P1-soak/03-rec.jsonl`, 10-s samples; fresh box from the public ISO, agent with swap = RAM, 8 apps, the rehearsal uploads for 2 h): at 01:30–01:34 UTC (the 03:30 tier2 job) host 39 % / guest 45 % over 10 s, over 60 s at most 20 % / 25 %; IO „full" 66 %; guest swap 1.35 → 2.1 GB; `/api/health` answered every probe (slowest 4.1 s, largest interval 16 s); 0 OOM kills. Once more in the upload window (Immich loading its models, 31 % / 35 % for one sample). The brief's soak pass line was „full" < 20 %, so the soak missed that one line; the freeze itself (96 % for an hour, R-1192) did not come back. Not diagnosed: which part of the tier2 job (the copy's page cache, the volume dumps) causes it. | WATCHING — **2026-10-24 02:4xZ dated check read (weekend-night session, read only):** Tester 1 (8 GB host, guest 4 GB, controller 0.341.0) ran its 03:30 copy at 01:30:00–01:30:13Z (3 apps, 157 MB + 86 MB + 16 KB; no „capped copy container did not start" WARN, no leftover `felhom-tier2-copy-*`); the guest cgroup's memory „full" stall counter on the host (`/sys/fs/cgroup/lxc/9201/memory.pressure`) read **total 444 µs** over the whole 11.1 h since the host started (15:32Z), so „full" stayed far under 20 % (20 % of 10 s = 2 s); 0 OOM kills. **Not shown:** that the copy ran in the capped container (the INFO log does not name it; Docker keeps only ~200 events, from 02:36Z). **Weak load:** 0.24 GB, not the soak's large copy — so the row stays WATCHING; close after a night whose copy moves ≥ 1 GB, read from the same counter. WATCHING — **RELEASED 2026-10-23 in controller 0.340.0** (floor 06:41 UTC; on all three test boxes, start-up „no leftover”, 0 copy containers; Tester 1's manual second-drive run: 3 apps, 0 containers left). **Open only for the consequence:** dated check 2026-10-24 — Tester 1's first night copy ran in the capped container and memory „full” pressure stayed under 20 % (the box's own night log). READY — rides controller 0.340.0. READY — waits for the controller 0.340.0 release. **2026-10-22 night — cause measured, fix built and proven on 9202, waits for the controller release.** Cause: the copy's page cache is charged to the guest and evicts the apps' working set. On 9202 (4 GB RAM, 1.8 GB held by a hog, a 6 GB copy): plain rsync full stall 5 702 / 2 464 ms, peak „full” 16.8 / 6.9 %; the same copy in a 512 MB memory-capped container 374 / 320 ms, 0.9 / 1.0 %. Controller main `03c0a69` + `ae39467` (unreleased, rides 0.340.0): the copy runs capped only for clean `/mnt/` paths with no symlink (security review); a docker error (exit 125) falls back to plain rsync. A real `POST /api/backup/tier2` on rc.4: 6 capped containers, each exit 0; Immich's copy equals its source (dry-run 0 differences, 44 = 44 files). Evidence `audits/night-2026-10-22/P4-R1214/`. **2026-10-23 night (stray-copy fix, night brief Part 1).** The capped copy's container outlived the copy (a timeout or a controller restart killed only the `docker` CLI; `--delete` went on outside the R-1180 guard). Controller main `66a69d4` + `912e592` (unreleased, rides 0.340.0): a fixed name + label per copy, removed by name on any error, every leftover removed at start before the scheduler and the web server, a restore refused while a copy runs or may run; `--mount` (a path the daemon does not see is refused, not created empty) and the image ID with `--pull never` (review C4, C7). Proven on 9202 with rc.5 (`audits/night-2026-10-23/P1-straycopy/`): a restart in the middle of a 16 GB copy → the new controller removed `felhom-tier2-copy-immich-948f2280` at start, the restore probe was refused during the copy, the redone copy equals the source; a 15 s test build → the timed-out copy's container removed, the partial copy stopped growing. A second review of that code (N2–N5) and the commit's own security review (two races) → `9d9e404`, `1babfe9`: a stray clears when a later removal works, the start-up check never removes this process's own copy and holds the destination's lock, one copy per destination at a time, `RunTier2` refuses during a restore for every caller and records itself in one step with that check, every leg asks again. Re-proven on 9202 with rc.8 (`audits/night-2026-10-23/P1-straycopy/1*`). 9202 back on 0.339.0. | — | Release 0.340.0; then re-measure the 03:30 job on the 8 GB soak box and close if „full” stays low | CC |
| **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a new install from the public download replaces the bootstrap and the off-site check; `pilot/TESTER2-handover.md`).** | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator |
| **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC |
| **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. **2026-11-11:** the End-dates list leans on this row (`documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB21); the newest golden passed its 7-day limit on 2026-11-10 and no waiver is in force, so the gate refuses code pushes to `felhom.eu`. **2026-11-11: golden 0.352.0 baked and published; one fresh box installed from it (`documentation/tests/golden-0.352.0-2026-11-11/README.md`); the gate is green (`python3 scripts/golden_currency_gate.py`, rc 0). The operator vouched it on 2026-11-11 at 14:48:11 CET.** | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement **2026-11-03: the Sunday 2026-11-01 bake was missed (it was in no brief): golden 0.344.0 is 8 days old, the floor 0.349.0; a CODE push to felhom.eu is refused until a bake and the vouch or a dated waiver. The night's close-out was documents only, which the gate exempts; no waiver written. First on the morning list.** | — | — | CC |
| **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. **2026-11-11:** the End-dates list leans on this row (`documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB21); the newest golden passed its 7-day limit on 2026-11-10 and no waiver is in force, so the gate refuses code pushes to `felhom.eu`. **2026-11-11: golden 0.352.0 baked and published; one fresh box installed from it (`documentation/tests/golden-0.352.0-2026-11-11/README.md`); the gate is green (`python3 scripts/golden_currency_gate.py`, rc 0). The operator vouched it on 2026-11-11 at 14:48:11 CET.** | WATCHING — **2026-11-14: golden 0.352.1 baked (sha256 `4f411cbe…346d`), one fresh install proven on a nested VM on demo-hp, vouched by the operator 10:25:43 UTC (`documentation/tests/golden-0.352.1-2026-11-14/`). The gate judges the bake's age: the next bake is due by 2026-11-21.** Earlier: **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement **2026-11-03: the Sunday 2026-11-01 bake was missed (it was in no brief): golden 0.344.0 is 8 days old, the floor 0.349.0; a CODE push to felhom.eu is refused until a bake and the vouch or a dated waiver. The night's close-out was documents only, which the gate exempts; no waiver written. First on the morning list.** | — | — | CC |
| **R-1358** | Box system & updates | P3 | **cloudflared 2026.10.4 misses two dependency fixes (VULN-183456 chi, 2026.10.5; VULN-184686 go-jose, 2026.10.6; the newest is 2026.10.7).** No public text on reach. Not pinned tonight: the tunnel is a box's only door, and 9202 has none to prove it on. | OPEN — filed 2026-11-03 | — | raise `CloudflaredImage` on the next controller release with a tunnel proof on Tester 1 | CC |
| **R-1380** | Box system & updates | P3 | **Proxmox renews its own certificate and the agent's pin does not follow: from that day the agent cannot control or back up a guest (5 answers).** Found 2026-11-11 (the End-dates list, class T2, `documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB02-3 to DB02-7). MEASURED: the agent trusts one exact Proxmox certificate (a pin), and the certificate on the three test boxes ends about two years after install — the N100's about 2028-07-16, then demo-hp and Tester 1 within three months. Proxmox makes a new certificate by itself about 14 days before the old one ends (so about 2028-07-02 on the N100). No code path gives the agent the new fingerprint, no alarm names the cause, no runbook holds the hand step, and nobody has tried it. Every box installed in the same season fails in the same fortnight. P3 only because the first date is in 2028; it is certain, not a risk. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 297). Not built.** **2026-11-11 (afternoon): tried on demo-hp (`documentation/audits/certpin-spike-2026-11-11/README.md`).** A forced new certificate (`pvecm updatecerts --force`, then a reload of the web service; Proxmox's own renewal does that reload by itself) blinds the agent at once: `TLS pin mismatch`, no reconcile, no host report to the hub, 502 to the controller; guests and apps stay up. One operator mail names a symptom (`agent_channel_unknown`), none names the cause; the hub's Hosts page still read ONLINE eleven minutes after the last report (when it turns offline was not measured). The hand repair works and is written down (`documentation/runbooks/RUNBOOK-agent-proxmox-pin-repair.md`, 2.1 s from restart to a working call). **A CA file in place of the fingerprint works with no agent code:** with `proxmox.tls.ca_file` on a readable copy of the box's root CA and the fingerprint empty, the agent went through a second new certificate with no hand step; a wrong CA file was refused. Two facts for the fix: the agent's user cannot read Proxmox's certificate files, and a set fingerprint wins over the CA file. The design with both ways is `03-host-agent.md` §12a; CC's pick differs from the ruled shape, so it is decision D80. | READY — **2026-11-14 09:55 — RULED (D80, `09` §3 decision 303): the root CA copy.** Built in release 2 (the host release). OPEN — filed 2026-11-11 (end-dates list) | — | the operator rules D80 (CA file, or follow the fingerprint); then the design's open parts in `03-host-agent.md` §12a (the root step that refreshes the copy; an alarm that names the cause), then build, then prove on a scratch host with a forced renewal | CC |
| **R-1382** | Box system & updates | P3 | **A box that was off past an end date: nobody knows if it comes back by itself (5 answers).** Found 2026-11-11 (the End-dates list, class T4, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; read in source and documents, never tried). After months without power: the web certificate has ended (DB01-6: does Traefik renew at the first start, and what does the household see until then), Debian's package keys may be too old to verify an update (DB16-6), the tunnel client may be below what Cloudflare accepts (DB19-6), a 30-day restore copy is past its date (DB24-6), and the 14-day countdown after a stopped off-site copy ran out while the box was off (DB04-6). Tester 2 has been off since 2026-10-04 and is the natural test. | OPEN — filed 2026-11-11 (end-dates list) | — | When Tester 2 comes back (or with a scratch host and a moved clock on Tier 0, as a task): record what renews by itself and what needs a hand; put the hand steps in a runbook. | CC |
@@ -304,7 +304,7 @@ stopping line that lies.
| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — 2026-10-27: agent v0.167.0 RELEASED; in the NORMAL signed delivery the pass now runs right after the commit on all three boxes („running the bundle policy pass after the self-update commit (R-1285)” → `BUNDLE POLICY DONE install_mode=appliance removed=0`; `audits/release-2026-10-27/agent-0167-tester1-demofelhom.txt`). Not live: a BYO box coming FROM 0.164.0 (none in the fleet). **Earlier:** **VERIFY** — 2026-10-26: agent `b1bc63d` (unreleased) — the new agent runs the bundle-policy pass once right after its self-update commits (`selfupdate.SetOnCommitted`); test in the real order (commit, then one pass), red-proved. **Not live:** only a box coming FROM 0.164.0 discriminates (0.165.0's wrapper already applies D43), which needs a signed downgrade chain on a BYO box — the live proof rides the next agent release. **Earlier:** **OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:** **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
## Monitoring & notifications — 29 rows (P2 3, P3 18, P4 8)
## Monitoring & notifications — 28 rows (P2 3, P3 17, P4 8)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -335,10 +335,9 @@ stopping line that lies.
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC |
| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC |
| **R-1392** | Monitoring & notifications | P3 | **Every agent release whose `configs/` did not change turns every box's „Root files” amber, and after 7 days the hub mails `os_config_bundle_behind` for each box, although the files on the box are the vouched ones.** Found 2026-11-14 (health-check session): the System page shows „root files behind the vouched agent's” on demo-felhom, demo-hp and Tester 1 (bundle 0.172.1, sha `3e7acaee…`); the vouched agent 0.173.0 carries bundle sha `51ebc1f4…`; `git diff v0.172.1 v0.173.0 -- configs/` is EMPTY, and the 0.173.0 release delivered no bundle job on purpose („`configs/` did not change”, felhom-agent CHANGELOG). The hub compares the WHOLE bundle's sha256 (`hub/internal/web/system.go` `bundleCell`, `hub/internal/osupdates/service.go` ~line 1182), and the bundle carries its version, so the sha moves with every release. Vouched 2026-11-10 16:29 CET → the alarm is due about 2026-11-17 on all three boxes. **Two ways out:** (A) the release sends a signed `agent_config_update` every time, even with no file change (cheap, one job per box, keeps the hub as is); (B) the hub compares the files (the box reports `config_bundle.live` per-file sha256s) with the vouched bundle's files, not the bundle's own sha. | READY — **2026-11-14 09:55 — RULED (D81, `09` §3 decision 304): yes, the hub must not call vouched files „behind”; the stopgap „send the bundle” is allowed for the three test boxes.** OPEN — filed 2026-11-14 | operator: A or B | today: send the 0.173.0 bundle to the three boxes (clears the amber and the 11-17 alarm); then A in `felhom-build-deploy` or B in the hub | CC |
| **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC |
## Hub & operator — 17 rows (P2 1, P3 5, P4 11)
## Hub & operator — 18 rows (P2 1, P3 5, P4 12)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -359,6 +358,7 @@ stopping line that lies.
| **R-1375** | Hub & operator | P2 | **The outside part of three endings is done by hand or not at all — the worst: an off-site account can stay at the provider with no record in the hub.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH06, EH01, EA21 and the ending "a tester ends"; read in source and documents; no account was called). (1) **Unticking the off-site tier** of a household takes nothing down and leaves no reminder (`hub/internal/web/configs.go:1737-1743`). A later RESET takes the account down only when the tier is ticked (`hub/internal/web/customer_reset.go:203-245`), so the sub-account and its data stay, are paid for, and the hub holds no record that they exist. (2) **Switching the second off-site tier off** takes nothing down: storage entry, token and copies stay; the source calls the teardown 'a deliberate future op' (agent `internal/pbsdr/manager.go:274`). (3) **A tester ends**: no handler and no runbook; done by hand once (2026-09-25); no list says what to collect. **Members (answers marked N):** EH06-4, EH01-4, EA21-4, and 'a tester ends' in the table's head. Related: R-810 (a household leaves), R-32. | OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D71 = yes, D74 = yes; `09` §3 decisions 291, 294).** Not built. | — | Hub (D71): keep the account record after an untick; RESET and delete take the account down, ticked or not. Docs (D74): the runbook 'a household or a tester leaves', from the five code doors, in a later documents-only session. Parts (2) and (3) of this row have no ruling of their own. | CC |
| **R-1376** | Hub & operator | P3 | **Records of a thing that is gone stay in six places.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH00, EH02, EH21, EA00, EA11, EA27; read in source, `documentation/audits/endings-list-2026-11-10/evidence/hub-delete-statements.txt`; no hub database was read). (1) **Hub:** thirteen tables have no delete statement anywhere. Deleting a box leaves its update settings, its update reports and its kernel notices; deleting a household leaves its off-site key rows, off-site window rows and abandon requests. Most are small; the update reports of each box grow with time. (2) **Host uninstall** leaves the mount units and host mounts of household drives (said when R-179 closed; no open row carried it). (3) **A drive wiped by a signed job** keeps a record and a mount unit that name the old filesystem (agent `internal/signedjobs/wipe.go:61-110`). (4) **The token of a destroyed guest** stays valid until a new guest takes its number: the token store has no remove (agent `internal/localapi/tokenstore.go:127-210`). **Members (answers marked N):** EH00-4, EH02-4, EH21-4, EA00-4, EA11-4, EA27-4. | OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D73 = yes, with one exception; `09` §3 decision 293).** Not built. | — | Hub (D73): add the tables with no delete to the two purges and to the daily clean-up; the delete journal (`host_deletions`, `customer_resets`) stays on purpose, and the privacy text says so. Agent: remove a guest's token when the guest is destroyed. Parts (2) and (3) have no ruling of their own. | CC |
| **R-1386** | Hub & operator | P3 | **Timed jobs with no time limit and no measured duration (25 answers).** Found 2026-11-11 (the Clock list, class W4, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source; the hub was not measured, by ruling). The hub's 60-second sweep runs its checks one after the other with no limit for each: one slow check delays all the others, the liveness check too (CH19-3, CH19-8, CH01-3, CH08-3, CH08-8, CH10-3, CH13-8, CH18-3, CH06-8, CH02-3); the 07:10 audit has 10 minutes, measured with three boxes only (CH11-3, CH11-8). On the box: the interval jobs have no limit of their own (CC11-3, CC12-3, CC13-3, CC14-3, CC15-3, CC16-3, CC20-3, CA06-3, CA10-3, CA10-8, CA13-8, CA15-3, CA19-3). Every measured duration is from a small test store. | OPEN — filed 2026-11-11 (clock list) | — | Hub: log each sweep step's duration and alarm when one sweep takes longer than its interval; then read a week of numbers and set limits. Box: the same for the controller's interval jobs. | CC |
| **R-1393** | Hub & operator | P4 | **For about 8 minutes after a hub restart its database statements take 1 to 32 seconds; the System page took 12.7 s and a box's enrolment 11.4 s.** Measured 2026-11-14 at the 0.181.0 deploy (`audits/release-1-dates-2026-11-14/hub/05-watch.txt`): 24 `slow statement` lines from 09:58 to 10:06 UTC, none in the 20 minutes after. The database file is 678 MB; the slow ones read `host_reports`, `os_reports` and `events` (whole report JSON per row). One 32-second statement ran while CC copied the database out of the pod for a read (CC's own load). No report was lost that CC could see; not measured: whether a box's report timed out in that window. | OPEN — filed 2026-11-14 | — | measure a restart with no copy running; then decide: an index or a shorter keep on `host_reports`/`os_reports`, or accept | CC |
## Business & legal — 14 rows (P2 4, P3 2, P4 8)
@@ -424,7 +424,7 @@ stopping line that lies.
| item | due (UTC) | what to measure |
|---|---|---|
| R-1293 | 2027-03-29 | the spring clock change: the hub's 02:00 snapshot and the 02:30 push on 2027-03-28 (02:00–03:00 does not exist) — which file each push took; detail in the R-1293 row |
| R-1050 | 2026-11-15 | HSTS one year if no https trouble since 2026-10-15; detail in the R-1050 row |
| R-1050 | 2026-11-21 | HSTS one year in the next controller release if no https trouble since 2026-10-15; detail in the R-1050 row |
| R-1035 | 2026-11-18 | „felhő tárhely” + „saját felhő tárhely” impressions and average position, and the new page's own impressions/clicks/position, 4 weeks after publishing; operator supplies the read; detail in the R-1035 row |
| R-528 | 2026-11-28 | no further failed oom_check after the one on 2026-11-11 (demo-felhom), no unexplained app_oom; detail in the R-528 row |
| R-1035 | 2026-12-09 | the same three numbers 8 weeks after publishing, compared with the 4-week read; a null at 4 weeks is not a verdict, both reads are needed; detail in the R-1035 row |
+4
View File
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
## Moved from STATUS on 2026-11-14 (midday)
**Updated 2026-11-14 (morning).** Health check: the three test boxes are healthy; last night's backups and restore proofs passed. CI was red three nights for one due check; it is read now. The yellow „root files behind" is a false alarm that mails on 11-17 unless you answer D81.
## Moved from STATUS on 2026-11-14 (morning)
**Updated 2026-11-11 (afternoon).** Golden 0.352.0 is vouched, so code pushes to `felhom.eu` work again. The N100 keeps no time: one hand step repairs it (below). A new Proxmox certificate blinds the agent; the hand repair is proven on demo-hp and one decision waits (D80).
@@ -0,0 +1,6 @@
## boot 2026-11-14T10:01Z
drill VM reverted to virgin and cold-booted (RUNBOOK-manual-build §4.0); pve-manager/9.2.2
pveam update: update successful; template debian-13-standard_13.6-1_amd64.tar.zst (checksum verified)
build-golden.sh v3.2.0 from felhom-agent 91a2924ad0, sha256 645b3b65… (the same file as the 0.352.0 bake)
token + registry user copied file->file from sec (GITEA_PKG_TOKEN, GITEA_PKG_USER); unit properties grep for the token value: 0
bake started 10:02:02Z, ended before 10:07:53Z
@@ -0,0 +1,6 @@
## round trip 2026-11-14T10:08:42Z (no login)
curl rc=0
bytes: 638851360
sha256: 4f411cbeb4b28a7dd65d4121e34e6ebb98f985bd07a48051502ac8d25c01346d
bake: 4f411cbeb4b28a7dd65d4121e34e6ebb98f985bd07a48051502ac8d25c01346d
registry login in the archive: tar listing for .docker/config.json = 0 (listing control: 8 other lines found)
@@ -0,0 +1,8 @@
## in the drill VM 2026-11-14T10:08:06Z
token in bake.log: 0 (control with the token appended: 1)
pass markers: docker OK=1 including-mount-point=2 excluding=0 FATAL=0 not-set=0 upload=1
purging CT 9100 from related configurations..
pct destroy rc=0
ls: cannot access '/root/.gitea-token': No such file or directory
ls: cannot access '/root/bake.log': No such file or directory
drill VM off; disk reverted to virgin 2026-11-14T10:08:29Z
@@ -0,0 +1,5 @@
## vouch read-back 2026-11-14T10:25:50Z (the hub's log; the pod's clock is CET)
2026/11/14 11:12:15 [INFO] Artifact manifest served for customer Tester4 (agent=0.173.0 golden=0.352.0)
2026/11/14 11:25:31 [INFO] artifact choices (felhom-agent): 1 version(s) without felhom-agent left out (R-1011): 0.163.0-step1
2026/11/14 11:25:43 [INFO] Artifact manifest set: agent=0.173.0 golden=0.352.1 min_agent="0.158.1" wrapper_sha=false bundle_sha="51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1"
2026/11/14 11:25:43 [INFO] artifact vouch timing (R-1346): registry probes 94ms, sha lookups 15ms, bundle sha 10ms, fleet check + save 239ms, total 359ms
@@ -0,0 +1,79 @@
# Golden 0.352.1 — bake, publish, one fresh install, vouch — 2026-11-14 (release 1 „dates”, Part 3)
Why: the weekly bake, done inside a brief so it is not missed (golden 0.352.0 was baked 2026-11-11; R-468).
Procedure: 0.352.0's README, the same recipe. Controller 0.352.1 is the global floor (MinAgent 0.158.1).
Documents read: `architecture/09-update-architecture.md` (§3 decisions 247, 250), `architecture/05-hub-architecture.md`
(Day-0 vouch, D50), `runbooks/publish-train-rules.md`, `runbooks/RUNBOOK-manual-build.md` §4, rows R-468, R-242.
| | Previous (`../golden-0.352.0-2026-11-11/`) | This bake |
|---|---|---|
| `build-golden.sh` | v3.2.0 | same file (sha256 `645b3b65…`) |
| Controller | `felhom-controller:0.352.0` | **`felhom-controller:0.352.1`** (Go 1.27.4; MinAgent **0.158.1**) |
| Traefik baked | v3.7.17 | same |
| Docker engine | the six approved pairs (29.9.2) | same |
| Template | `debian-13-standard_13.6-1_amd64.tar.zst` | same |
| Token | from `sec`, file → file | same |
## Pass markers (`bake.log`)
```
[golden] build-golden.sh v3.2.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.352.1
docker OK (overlay2; data-root /var/lib/docker)
[golden] upload OK (HTTP 201)
GOLDEN_VERSION=0.352.1
GOLDEN_SHA256=4f411cbeb4b28a7dd65d4121e34e6ebb98f985bd07a48051502ac8d25c01346d
```
`including mount point` 2, `excluding` 0, `FATAL` 0, `GOLDEN_DOCKER_PKGS not set` 0 (`03-teardown.txt`). The token is
not in `bake.log`: 0, with a control of 1, in the VM and in the committed copy.
Round trip: the registry's file, downloaded with no login and hashed, equals the bake's sha; 638,851,360 bytes; no
registry login in the archive (`02-round-trip.txt`). Build guest and teardown: `00-boot.txt`, `03-teardown.txt`.
**Golden gate:** `python3 scripts/golden_currency_gate.py` → OK, rc 0. No waiver was written.
## The fresh install (`fresh-install/`)
**The path, exactly.** The hub offered only the vouched golden (0.352.0) at that time. So the box ran the public
installer 1.36.1 (sha256 `e18cdd17…aa1f1`) with **one added line** after line 479, which names golden 0.352.1 and its
sha256 (`installer-the-one-line.diff.txt`), run by hand as root in the direct shape
(`--customer-id Tester4 --mode appliance --hub-url https://hub.felhom.eu --passphrase-file …`), not through the bind
page. **One more difference from a household's path:** the installer file was fetched on DooPlex and copied in,
because on this LAN the name `felhom.eu` answers 404 from the retired site (R-1389). All other steps are the
product's own: the hub's enrolment, the agent download, the golden's fetch and sha check, the guest build, the
setup-code mail, the claim page, the app page.
| Step | Result | Proof |
|---|---|---|
| VM 9385 `golden1114-box` on demo-hp, disks on `nvme-scratch` (never the thin pool), Proxmox from the Felhom ISO 1.30.0 (sha256 checked against the published file) | installed; Install pressed 10:07:53Z, SSH answered 10:10:46Z | `install/` |
| CC's SSH key on the host **before** the installer ran (the 2026-11-11 lesson) | done; the key still opened the host after the installer closed it | `box-state-and-install-log.txt` |
| Time service on the fresh host, before the installer | `chrony` active, 0.0005 s off (10:10:58Z); active after it too | — |
| Installer run, started 10:11:24Z | „Day-0 provision SUCCESS — vmid=9201 host_id=Tester4-46836f” before 10:14:29Z | `box-state-and-install-log.txt` |
| Golden on the box | sha256 `4f411cbe…346d` = the bake and the registry | read on the box |
| Agent | **0.173.0**, active | read on the box |
| Controller | **0.352.1** from the golden: one `starting` line, no `controller_updated`; 0 ERROR lines | `box-state-and-install-log.txt` |
| Hub | host `Tester4-46836f`, agent 0.173.0, ONLINE | `app.txt` (last lines) |
| Setup-code mail | arrived 10:12:03Z | the test mailbox |
| Claim page | `POST /claim` → 302 `/` at 10:15:16Z; the box then offers the recovery page for Tester4's sealed package (not entered) | this README (the request's two status lines are not kept in a file) |
| One app: BentoPDF, through the page's own deploy request | 202 at 10:22:07Z; `running` at 10:23:57Z; `pdf.tester4.test` answers 200; a name with no app answers 404 | `app.txt` |
**CC's two mistakes on the way, both in CC's own script:** the first two tries to install the app looked for the
session token in the wrong place and got `403 CSRF token missing or invalid` (nothing was installed; 6 minutes
lost). Not a product fault.
**The hub's own acts on this customer:** the fresh install made the hub re-issue Tester4's off-site key and carry the
sealed package of the deleted host to the new one. CC touched nothing on ep0.
## Vouch — done by the operator, 2026-11-14 10:25:43 UTC (`04-vouch-readback.txt`)
- golden **0.352.1**, sha256 `4f411cbeb4b28a7dd65d4121e34e6ebb98f985bd07a48051502ac8d25c01346d`
- agent **0.173.0**, bundle `51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1` (both unchanged)
- min_agent **0.158.1** (unchanged)
Read from the hub's log only (one channel). `pinned.day0_agent` in the agent's retention policy stays 0.173.0.
## Teardown
`fresh-install/teardown.txt`: VM 9385 stopped and destroyed with its two disks; the installer ISO removed from
demo-hp; the typed password, the session cookie and the setup-code file shredded on DooPlex. The drill VM is off and
its disk is back on `virgin`. The hub host `Tester4-46836f` (and its appliance entry, smbios uuid
`23d48cf6-1984-46a7-9954-557faf333f10`, if the Hosts page lists one) are the operator's to remove — the host from
about 45 minutes after its last report (R-599). Until then the hub mails „no report” for it.
@@ -0,0 +1,339 @@
[golden] build-golden.sh v3.2.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.352.1
[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) …
Logical volume "vm-9100-disk-0" created.
Logical volume pve/vm-9100-disk-0 changed.
Creating filesystem with 8388608 4k blocks and 2097152 inodes
Filesystem UUID: 6a11faf4-eafe-462d-bc6b-d3f3b54f2f77
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624
Logical volume "vm-9100-disk-1" created.
Logical volume pve/vm-9100-disk-1 changed.
Creating filesystem with 6291456 4k blocks and 1572864 inodes
Filesystem UUID: 63909107-2b40-4ed6-86fa-97320060f069
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst'
Total bytes read: 553512960 (528MiB, 135MiB/s)
Detected container architecture: amd64
Creating SSH host key 'ssh_host_rsa_key' - this may take some time ...
done: SHA256:OoKz/bAime/Y82Y6PAbV2mkqNSKqdJKKJ7IkVyDg0hc root@felhom-golden
Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ...
done: SHA256:mWiYmA+1ncXTPjQllxy0KkZfQL9rwOTgPfPLlK4Xvbo root@felhom-golden
Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ...
done: SHA256:p/oUB5Zf4PkGKBF3zwqh9ysgXyoB/8/KvelW7ZWg2JU root@felhom-golden
[golden] starting + installing Docker (official repo, trixie channel) …
[golden] Docker engine set PINNED to the approved release: containerd.io=2.4.2-3~debian.13~trixie docker-buildx-plugin=0.38.1-1~debian.13~trixie docker-ce=5:29.9.2-1~debian.13~trixie docker-ce-cli=5:29.9.2-1~debian.13~trixie docker-ce-rootless-extras=5:29.9.2-1~debian.13~trixie docker-compose-plugin=5.6.1-1~debian.13~trixie
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = (unset),
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = (unset),
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
installed: containerd.io 2.4.2-3~debian.13~trixie
installed: docker-buildx-plugin 0.38.1-1~debian.13~trixie
installed: docker-ce 5:29.9.2-1~debian.13~trixie
installed: docker-ce-cli 5:29.9.2-1~debian.13~trixie
installed: docker-ce-rootless-extras 5:29.9.2-1~debian.13~trixie
installed: docker-compose-plugin 5.6.1-1~debian.13~trixie
[golden] no approved guest release given - the template versions stay; first-night count vs an approved release: n/a
[golden] pending Debian upgrades in the baked guest (what a FUTURE approval may bring): 63
[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation …
[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds …
[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) …
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
4f55086f7dd0: Pulling fs layer
4f55086f7dd0: Download complete
4f55086f7dd0: Pull complete
Digest: sha256:cf1b20a0c89a666b6950fecda7ccbc1b1a86520e7c53f908f3a21e79d0d60458
Status: Downloaded newer image for hello-world:latest
docker OK (overlay2; data-root /var/lib/docker)
live-restore: on
/var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4
/mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4
both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576
[golden] baking the in-guest controller image gitea.dooplex.hu/admin/felhom-controller:0.352.1 (no registry cred at deploy) …
WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
Configure a credential helper to remove this warning. See
https://docs.docker.com/go/credential-store/
0.352.1: Pulling from admin/felhom-controller
ecc510c1e359: Pulling fs layer
c3f83389ce1b: Pulling fs layer
e81c1fce9826: Pulling fs layer
63a1406b1961: Pulling fs layer
d747c9b6922d: Pulling fs layer
eb49b63afe2b: Pulling fs layer
c6c5608e8d7b: Pulling fs layer
63a1406b1961: Waiting
d747c9b6922d: Waiting
eb49b63afe2b: Waiting
c6c5608e8d7b: Waiting
e81c1fce9826: Verifying Checksum
e81c1fce9826: Download complete
c3f83389ce1b: Verifying Checksum
c3f83389ce1b: Download complete
d747c9b6922d: Download complete
eb49b63afe2b: Verifying Checksum
eb49b63afe2b: Download complete
63a1406b1961: Verifying Checksum
63a1406b1961: Download complete
c6c5608e8d7b: Verifying Checksum
c6c5608e8d7b: Download complete
ecc510c1e359: Verifying Checksum
ecc510c1e359: Download complete
ecc510c1e359: Pull complete
c3f83389ce1b: Pull complete
e81c1fce9826: Pull complete
63a1406b1961: Pull complete
d747c9b6922d: Pull complete
eb49b63afe2b: Pull complete
c6c5608e8d7b: Pull complete
Digest: sha256:1a6e8945211a0f1995fd2bd0782053c7f33152fcb00ca465d0596d961a13b6bb
Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-controller:0.352.1
gitea.dooplex.hu/admin/felhom-controller:0.352.1
[golden] asking the controller which infra images it manages …
[golden] baking infra images (4): traefik:v3.7.17 cloudflare/cloudflared:2026.10.4 gtstef/filebrowser:1.5.8-stable gitea.dooplex.hu/admin/felhom-samba:1.2.0 …
v3.7.17: Pulling from library/traefik
881fc932939e: Pulling fs layer
c901ba6ff9e0: Pulling fs layer
4f376259b112: Pulling fs layer
7705a4c1b3f8: Pulling fs layer
7705a4c1b3f8: Waiting
881fc932939e: Verifying Checksum
881fc932939e: Download complete
c901ba6ff9e0: Verifying Checksum
c901ba6ff9e0: Download complete
7705a4c1b3f8: Verifying Checksum
7705a4c1b3f8: Download complete
4f376259b112: Verifying Checksum
4f376259b112: Download complete
881fc932939e: Pull complete
c901ba6ff9e0: Pull complete
4f376259b112: Pull complete
7705a4c1b3f8: Pull complete
Digest: sha256:f4bb143d1135a579b630ed491e17ec50f6f7c9c6a30585ff3d385471d2a86f5a
Status: Downloaded newer image for traefik:v3.7.17
docker.io/library/traefik:v3.7.17
2026.10.4: Pulling from cloudflare/cloudflared
2cc7ee286bf3: Pulling fs layer
c172f21841df: Pulling fs layer
218cf840d0d9: Pulling fs layer
f6069939f718: Pulling fs layer
d6b1b89eccac: Pulling fs layer
2780920e5dbf: Pulling fs layer
7c12895b777b: Pulling fs layer
3214acf345c0: Pulling fs layer
52630fc75a18: Pulling fs layer
dd64bf2dd177: Pulling fs layer
b839dfae01f6: Pulling fs layer
ebddc55facdc: Pulling fs layer
a0921f9d8de9: Pulling fs layer
b96fe2995f90: Pulling fs layer
810e2b405a08: Pulling fs layer
bd8962e29291: Pulling fs layer
cac2ae0193cb: Pulling fs layer
eb8da9d09a5e: Pulling fs layer
52630fc75a18: Waiting
dd64bf2dd177: Waiting
b839dfae01f6: Waiting
ebddc55facdc: Waiting
a0921f9d8de9: Waiting
b96fe2995f90: Waiting
810e2b405a08: Waiting
bd8962e29291: Waiting
cac2ae0193cb: Waiting
eb8da9d09a5e: Waiting
d6b1b89eccac: Waiting
2780920e5dbf: Waiting
7c12895b777b: Waiting
3214acf345c0: Waiting
f6069939f718: Waiting
2cc7ee286bf3: Verifying Checksum
2cc7ee286bf3: Download complete
c172f21841df: Verifying Checksum
c172f21841df: Download complete
218cf840d0d9: Verifying Checksum
218cf840d0d9: Download complete
f6069939f718: Verifying Checksum
f6069939f718: Download complete
d6b1b89eccac: Download complete
2780920e5dbf: Verifying Checksum
2780920e5dbf: Download complete
7c12895b777b: Verifying Checksum
7c12895b777b: Download complete
3214acf345c0: Verifying Checksum
3214acf345c0: Download complete
52630fc75a18: Verifying Checksum
52630fc75a18: Download complete
2cc7ee286bf3: Pull complete
dd64bf2dd177: Verifying Checksum
dd64bf2dd177: Download complete
b839dfae01f6: Verifying Checksum
b839dfae01f6: Download complete
ebddc55facdc: Download complete
a0921f9d8de9: Download complete
810e2b405a08: Verifying Checksum
810e2b405a08: Download complete
b96fe2995f90: Verifying Checksum
b96fe2995f90: Download complete
bd8962e29291: Download complete
c172f21841df: Pull complete
cac2ae0193cb: Verifying Checksum
cac2ae0193cb: Download complete
eb8da9d09a5e: Verifying Checksum
eb8da9d09a5e: Download complete
218cf840d0d9: Pull complete
f6069939f718: Pull complete
d6b1b89eccac: Pull complete
2780920e5dbf: Pull complete
7c12895b777b: Pull complete
3214acf345c0: Pull complete
52630fc75a18: Pull complete
dd64bf2dd177: Pull complete
b839dfae01f6: Pull complete
ebddc55facdc: Pull complete
a0921f9d8de9: Pull complete
b96fe2995f90: Pull complete
810e2b405a08: Pull complete
bd8962e29291: Pull complete
cac2ae0193cb: Pull complete
eb8da9d09a5e: Pull complete
Digest: sha256:c6c7ae1d5dfdcb094ad344ed68ccffc26635c9c6be80965664c33baf064b0515
Status: Downloaded newer image for cloudflare/cloudflared:2026.10.4
docker.io/cloudflare/cloudflared:2026.10.4
1.5.8-stable: Pulling from gtstef/filebrowser
e2de96513ba9: Pulling fs layer
93c39e448959: Pulling fs layer
b7597036c324: Pulling fs layer
c6591d351da9: Pulling fs layer
4f4fb700ef54: Pulling fs layer
74aec9809d97: Pulling fs layer
9f7b5accdf20: Pulling fs layer
19114688b75e: Pulling fs layer
2067d5df1f91: Pulling fs layer
5669e416ca05: Pulling fs layer
08c322a8a1d4: Pulling fs layer
19114688b75e: Waiting
2067d5df1f91: Waiting
5669e416ca05: Waiting
08c322a8a1d4: Waiting
4f4fb700ef54: Waiting
74aec9809d97: Waiting
c6591d351da9: Waiting
9f7b5accdf20: Waiting
e2de96513ba9: Verifying Checksum
e2de96513ba9: Download complete
c6591d351da9: Verifying Checksum
c6591d351da9: Download complete
4f4fb700ef54: Verifying Checksum
4f4fb700ef54: Download complete
b7597036c324: Verifying Checksum
b7597036c324: Download complete
9f7b5accdf20: Verifying Checksum
9f7b5accdf20: Download complete
e2de96513ba9: Pull complete
74aec9809d97: Verifying Checksum
74aec9809d97: Download complete
2067d5df1f91: Verifying Checksum
2067d5df1f91: Download complete
19114688b75e: Verifying Checksum
19114688b75e: Download complete
08c322a8a1d4: Verifying Checksum
08c322a8a1d4: Download complete
5669e416ca05: Verifying Checksum
5669e416ca05: Download complete
93c39e448959: Verifying Checksum
93c39e448959: Download complete
93c39e448959: Pull complete
b7597036c324: Pull complete
c6591d351da9: Pull complete
4f4fb700ef54: Pull complete
74aec9809d97: Pull complete
9f7b5accdf20: Pull complete
19114688b75e: Pull complete
2067d5df1f91: Pull complete
5669e416ca05: Pull complete
08c322a8a1d4: Pull complete
Digest: sha256:68455d4953bad8e984e1bc5d112ed1e9c918a0a04f104c13c9ece50790308894
Status: Downloaded newer image for gtstef/filebrowser:1.5.8-stable
docker.io/gtstef/filebrowser:1.5.8-stable
1.2.0: Pulling from admin/felhom-samba
e2de96513ba9: Already exists
0ddbc9f3a6b6: Pulling fs layer
ce57a3f93416: Pulling fs layer
fb94eeec2fe1: Pulling fs layer
ce57a3f93416: Verifying Checksum
ce57a3f93416: Download complete
fb94eeec2fe1: Verifying Checksum
fb94eeec2fe1: Download complete
0ddbc9f3a6b6: Verifying Checksum
0ddbc9f3a6b6: Download complete
0ddbc9f3a6b6: Pull complete
ce57a3f93416: Pull complete
fb94eeec2fe1: Pull complete
Digest: sha256:da96f25033827fbb77d0ee398e86fe5b744e2f29be9de898994533340c97acf9
Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.2.0
gitea.dooplex.hu/admin/felhom-samba:1.2.0
[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) …
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'.
[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) …
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'.
[golden] baking the first-boot SSH host-key regeneration unit (F3) …
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'.
[golden] identity-clean + minimize …
[golden] stop + archive …
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/var/lib/felhom') in backup
INFO: archive file size: 609MB
INFO: Finished Backup of VM 9100 (00:00:28)
[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_11_14-11_06_32.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive)
[golden] publishing golden (638851360 bytes, sha256 4f411cbeb4b28a7d…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.352.1/golden.tar.zst
[golden] pre-delete existing: HTTP 404 (404/204 expected)
[golden] upload OK (HTTP 201)
GOLDEN_VERSION=0.352.1
GOLDEN_SHA256=4f411cbeb4b28a7dd65d4121e34e6ebb98f985bd07a48051502ac8d25c01346d
[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.352.1 / 4f411cbeb4b28a7dd65d4121e34e6ebb98f985bd07a48051502ac8d25c01346d
[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge)
@@ -0,0 +1,23 @@
session csrf_len 64
controller version on the box: {"ok":true,"message":"felhom-controller is healthy"}
2026-11-14T10:22:07Z
{"ok":true,"message":"Telepítés elindítva – az állapot a kártyán követhető"}
HTTP 202
10:22:07 {'deployed': True, 'state': 'not_deployed', 'subdomain': None}
10:22:17 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:22:27 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:22:37 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:22:47 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:22:57 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:23:07 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:23:17 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:23:27 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:23:37 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:23:47 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:23:57 {'deployed': True, 'state': 'running', 'subdomain': None}
bentopdf name: pdf.tester4.test
route: 200 at 2026-11-14T10:24:08Z
negative control (no such app name): 404
2026-11-14T10:24:20Z Tester4-46836f Tester4 0.173.0 floor: declared MinAgent 9.2.2 7.0.2-6-pve ONLINE 0/0 0% 18% 16% unknown (no running customer guest) 16% local demo-felhom-8363b5 Demo Ügyfél 0.173.0 floor: declared Min
Unclaimed appliances (booted from the generic ISO, awaiting a bind) A box that installed from the universal secret-free ISO and registered itself. Bind it to a customer to deliver its retrieval passphrase once; discard to ignore it. Appliance Pairing code MACs Hardware SSH host keys Seen Bind to cus
@@ -0,0 +1,157 @@
ghcr.io/alam00000/bentopdf:v2.8.10 Up 21 seconds (healthy)
gtstef/filebrowser:1.5.8-stable Up 10 minutes (healthy)
gitea.dooplex.hu/admin/felhom-controller:0.352.1 Up 10 minutes (healthy)
traefik:v3.7.17 Up 10 minutes
2026/11/14 10:14:05 [INFO] felhom-controller 0.352.1 starting (customer: Tester4, domain: tester4.test)
0
[INFO] auto-sized guest RAM: 5847 MiB (host 9943 MiB; clamp(host-4096, min 4096, max host-2048), ceiling host-1024)
[INFO] auto-sized guest cores: 5 (host 6 cores; host-1, min 2)
[INFO] felhom-host-install v1.36.1 — mode=appliance customer=Tester4 vmid=9201
[STEP] 1/8 pre-flight
[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
[INFO] node: goldenbox (auto)
[INFO] agent config: /etc/felhom-agent/agent.json
[INFO] agent: not installed yet — will be fetched + installed in step 5/8
[INFO] local-lvm free: ~19 GiB
[WARN] local-lvm free ~19 GiB is below the recommended 120 GiB — the install continues; proceed only if you sized the grows deliberately
[INFO] free RAM: ~8344 MiB
[INFO] existing guests on this host: 0 (pct+qm)
[INFO] archive storage 'local' present
[INFO] acl storage 'felhom-pbs' not present yet — expected: the PBS-DR tier creates it; the grant is pre-positioned deliberately
[INFO] dnsmasq: not present before Felhom — recorded; uninstall will remove it again if we install it
[INFO] hub reachable (https://hub.felhom.eu)
[OK] customer 'Tester4' exists + passphrase valid
[INFO] golden: none local — will fetch + verify from Gitea in step 7/8
[OK] pre-flight passed
[STEP] 2/8 Proxmox API token
[OK] token minted (secret captured, not logged)
[OK] scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm felhom-pbs])
[SKIP] old broad role FelhomAgent already absent
[STEP] 3/8 compute volume grows
[INFO] auto-computed from a 19 GiB thin pool (16 §7: 50/100/130 GiB)
[INFO] grows: rootfs +0G (->32G), data +26G (->50G, ONE volume)
[STEP] 4/8 host enrollment (POST /host-enroll)
[OK] host MINTED (first enroll)
[INFO] host_id: Tester4-46836f (api_key captured, not logged)
[STEP] 4b/8 break-glass credential (root@pam console password → hub vault)
[INFO] recovery password vaulted at the hub (stored) — now setting it on root@pam
[OK] root@pam password vaulted, set, and the PVE login takes it (retrieve via the operator /admin path; never logged here)
[WARN] NOTE: the root@pam password just CHANGED — the old one now fails at the PVE web GUI (:8006).
[WARN] Retrieve the new one at hub → host page (vaulted recovery credential).
[STEP] 5/8 agent install (fetch + verify + install)
[INFO] disabled pve-enterprise.sources (Enabled: no)
[INFO] disabled ceph.sources (Enabled: no)
[INFO] added pve-no-subscription.sources (suite=trixie)
[OK] apt repos aligned to no-subscription (changed; apt-get update OK)
[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
[INFO] manifest: agent v0.173.0 (sha 752b2727fb604341…), golden v0.352.1
[INFO] fetching agent binary v0.173.0 from Gitea …
[OK] verified sha256 752b2727fb604341… matches the hub manifest
[OK] installed /usr/local/bin/felhom-agent (felhom-agent 0.173.0)
[INFO] installing the 'sudo' package (required for the non-root agent model) …
[OK] sudo installed (Sudo version 1.9.16p2)
[INFO] installing the 'age' package (escrow ceremony identity-wrap dependency) …
[OK] age installed (1.2.1)
[OK] created service user felhom-agent
[OK] added felhom-agent to systemd-journal (unprivileged journal read for NAS verify)
[OK] verified sha256 51ebc1f4c66d8791… matches the hub manifest
os-apply: BUNDLE START agent=0.173.0 sha=51ebc1f4c66d8791 authority=installer files=32 write=31 same=1 kept=0 skipped=0
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-mkfs-guarded (new)
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-selfupdate-guarded (new)
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-pbs-apply (new)
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-backup-target-apply (new)
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-vzdump-partial-clean (new)
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-thin-check (new)
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-internal-place (new)
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-crash-guard (new)
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-priv-apply (new)
os-apply: BUNDLE WROTE /var/lib/vz/snippets/felhom-guest-hook.sh (new)
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-shared-parent.sh (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-shared-parent.service (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard.service (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard-check.service (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard-check.timer (new)
os-apply: BUNDLE WROTE /etc/felhom/crash-guard.conf (new)
os-apply: BUNDLE WROTE /etc/grub.d/01_felhom_oneshot (new)
os-apply: BUNDLE WROTE /etc/grub.d/42_felhom_oneshot (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent.service (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent-rollback.service (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf (new)
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-mgmt-watchdog (new)
os-apply: BUNDLE WROTE /etc/tmpfiles.d/felhom-privsep.conf (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-mgmt-watchdog.service (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-mgmt-watchdog.timer (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-sshd.service (new)
os-apply: BUNDLE WROTE /etc/felhom-oob.nft (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-oob-nft.service (new)
os-apply: BUNDLE WROTE /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf (new)
os-apply: BUNDLE WROTE /etc/sudoers.d/felhom-op (new)
os-apply: BUNDLE WROTE /etc/sudoers.d/felhom-agent (new)
os-apply: BUNDLE DONE agent=0.173.0 written=31 same=1 self-check=ok signers-created=False
[OK] installed every root-owned file from config bundle v0.173.0 (sha 51ebc1f4c66d8791…; checked, previous copies kept)
[OK] OOB felhom-sshd instance + static belt from the bundle (agent renders config + fills sets once oob.enabled)
[STEP] 5b/8 internal data place (Belső tárhely)
[INFO] no internal place: only -79 GiB free for it (pool 19, reserve 16, promised 0, planned 82; needs 20)
[STEP] 6/8 agent config + service
[WARN] backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive.
[WARN] It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard.
[INFO] creating island bridge vmbr9 (portless, 169.254.253.1/30)
[OK] vmbr9 up: 169.254.253.1/30
[INFO] R-50 island ON: local_api=169.254.253.1:8443 (vmbr9); guest net1=169.254.253.2/30; lan_resolver.host_ip=192.168.0.122
[INFO] node=goldenbox local_api=169.254.253.1:8443 tls_fp=30:9D:EE:6C:AE:2D…
[OK] wrote /etc/felhom-agent/agent.json (0600 felhom-agent)
[OK] agent --selftest (read-only) passed
[OK] felhom-agent service active (non-root felhom-agent reads the config OK)
[OK] wrote /etc/felhom/os-trust.json + operator-signers (felhom-op-1) — the root-owned slow-lane anchors
[STEP] 7/8 golden archive
[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
[INFO] fetching golden v0.352.1 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_11_14-11_12_35.tar.zst
[OK] verified sha256 4f411cbeb4b28a7d… matches the hub manifest
[OK] golden imported + verified: local:backup/vzdump-lxc-9100-2026_11_14-11_12_35.tar.zst
[STEP] 8/8 provision guest 9201
[SKIP] pool felhom already exists
=== felhom-agent 0.173.0 selftest=provision (vmid=9201 customer=Tester4 hostname=tester4) ===
--- front half: bring-up (provision) local:backup/vzdump-lxc-9100-2026_11_14-11_12_35.tar.zst → vmid 9201 ---
time=2026-11-14T11:13:43.357+01:00 level=INFO msg="bring-up: pool membership re-asserted" vmid=9201 pool=felhom
[OK] front half: vmid 9201 up (boot+running) in 54s; MAC=BC:24:11:19:43:F5
--- back half: mint per-guest token + populate bootstrap config mount ---
time=2026-11-14T11:13:48.482+01:00 level=INFO msg="provision: back-half complete" vmid=9201 mount=mp9 guest_path=/etc/felhom-bootstrap endpoint=169.254.253.1:8443
[OK] back half: bootstrap mount mp9 → /etc/felhom-bootstrap on vmid 9201 (host dir /var/lib/felhom-agent/guests/9201/bootstrap)
local-api endpoint 169.254.253.1:8443 · leaf fp cb19319caff897de5a3eb551625b084f03b79f677342bfd0d6f15b0a3553ce73 · token: minted (not printed)
=== selftest=provision OK — guest 9201 provisioned + bootstrap-mounted (KEPT) ===
next: reboot the guest → the golden's baked controller-bootstrap unit deploys the controller,
which PULLS its controller.yaml from the hub (retrieval passphrase) and merges in this local_api.
[OK] provision completed
[INFO] rebooting guest 9201 so the baked controller-bootstrap unit picks up the mount
[STEP] verify
[OK] pct status: running
[OK] onboot: 1
mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=50G
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
rootfs: local-lvm:vm-9201-disk-0,size=32G
[OK] pool: guest 9201 is a member of felhom
[OK] acl: FelhomAgentBase@/ present (user+token)
[OK] acl: FelhomAgentGuest@/pool/felhom present (user+token)
[OK] acl: FelhomAgentStore@/storage/local present (user+token)
[OK] acl: FelhomAgentStore@/storage/local-lvm present (user+token)
[OK] acl: FelhomAgentStore@/storage/felhom-pbs present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990000 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990001 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990002 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990003 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990004 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990005 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990006 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990007 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990008 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990009 present (user+token)
[OK] authz signers: 2 (operator-signed self-update armed)
[OK] controller: Up 20 seconds (healthy) (after ~0s)
[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.352.1
[WARN] cloudflared not visible yet
[INFO] (confirm in the hub UI that host Tester4-46836f reports guest 9201)
[OK] Day-0 provision SUCCESS — vmid=9201 host_id=Tester4-46836f customer=Tester4 golden=local:backup/vzdump-lxc-9100-2026_11_14-11_12_35.tar.zst
[INFO] root@pam was rotated + vaulted at step 4b — retrieve at hub → host page (the old GUI password no longer works).
@@ -0,0 +1 @@
Install pressed 2026-11-14T10:07:53Z
Binary file not shown.

After

Width:  |  Height:  |  Size: 182 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 127 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 111 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 124 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 136 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 112 KiB

@@ -0,0 +1,9 @@
installer fetched on DooPlex from https://felhom.eu/scripts/felhom-host-install.sh and copied in (on this LAN the name answers 404 from the retired site, R-1389)
e18cdd17823911e7d87da12fcb8d48b6827f1706a84b3168114fe19fe30aa1f1 /root/hi-orig.sh
197:SCRIPT_VERSION="1.36.1" # the SINGLE version source (F-1): -h and the run banners follow it.
line: 479
479a480
> ART_GOLDEN_VER=0.352.1; ART_GOLDEN_SHA=4f411cbeb4b28a7dd65d4121e34e6ebb98f985bd07a48051502ac8d25c01346d
syntax rc=0
2026-11-14T10:11:24Z
Running as unit: install1114.service; invocation ID: 23e509586d6d40608a427f1cecc16f40
@@ -0,0 +1,8 @@
## teardown 2026-11-14T10:26:02Z
purging VM 9385 from related configurations..
Configuration file 'nodes/demo-hp/qemu-server/9385.conf' does not exist
ls: cannot access '/mnt/hdd_1/images/9385': No such file or directory
0
341 night1004-tester1 running 8192 200.00 1950
the installer ISO removed from demo-hp (it was not there before this run)
left for the operator: hub host Tester4-46836f; appliance with smbios uuid 23d48cf6-1984-46a7-9954-557faf333f10 (if it is on the Hosts page)
+5
View File
@@ -1,3 +1,8 @@
## Unreleased (after v0.181.0)
- `bundlefiles`: the line „bundle X holds the same files as the vouched Y” is logged once per pair (INFO), not at
every view of the System page (it was 6 DEBUG lines per view, seen at the 0.181.0 deploy). No behaviour change.
## v0.181.0 (2026-11-14) — R-1392: root files are „behind” only when the FILES differ; Go 1.27.4 (R-1391) (image `sha256:be2e9e5676eec7622622402a830d8f92913a93fe746e421db60f949138d1f429` built from `83316fc9`, CI run id 2636; the operator's yes 2026-11-14 ~09:25 UTC; the manifest bump is the commit that carries this line)
- **The defect (R-1392, found 2026-11-14).** The agent's config bundle carries its `agent_version`, so its sha256
+16 -2
View File
@@ -53,6 +53,20 @@ type Resolver struct {
mu sync.Mutex
cache map[string]entry
said map[string]bool
}
func (r *Resolver) firstSight(pair string) bool {
r.mu.Lock()
defer r.mu.Unlock()
if r.said[pair] {
return false
}
if r.said == nil {
r.said = map[string]bool{}
}
r.said[pair] = true
return true
}
func (r *Resolver) now() time.Time {
@@ -135,8 +149,8 @@ func (r *Resolver) Same(boxVersion, boxSHA, vouchedVersion, vouchedSHA string) b
}
b := r.digest(boxVersion, boxSHA)
same := b != "" && a == b
if same {
r.logf("[DEBUG] [bundlefiles] bundle %s holds the same files as the vouched %s (only the label differs)", boxVersion, vouchedVersion)
if same && r.firstSight(boxSHA+"="+vouchedSHA) { // once per pair: the System page asks on every view
r.logf("[INFO] [bundlefiles] bundle %s holds the same files as the vouched %s (only the label differs)", boxVersion, vouchedVersion)
}
return same
}