admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Register 187 -> 189: R-1230 (a held box reported stuck — measured on the test night, fixed in hub 0.167.0 source), R-1231 (the recovery page calls an unknown agent older); test-night provenance

Browse Source
This commit is contained in:
authentik Default Admin
2026-10-23 00:07:01 +02:00
parent e94d0256d2
commit f56f3328f7
2 changed files with 30 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/night-2026-10-23/P4-testnight/00-provenance.txt
+26
View File
@@ -0,0 +1,26 @@
Test night provenance (night 2026-10-22 → 23, Part 4)
- VM 9344 `night1023-soak` on demo-hp (4 vCPU, 8 GiB, 256 GB raw system disk + 200 GB raw data disk on nvme-scratch,
not the thin pool), created 18:46:17 UTC; installed from the public ISO felhom-installer-1.30.0-pve9.2-1.iso
(Install pressed 18:51:58 UTC; the ISO fetched the current installer at run time).
- Customer Tester4 (the kept fresh-install customer; no old host at the start — read from the hub DB). Bound 19:0x UTC
with the bind link minted 2026-10-22 07:37 UTC (unconsumed until tonight). Host enrolled as Tester4-16fdef (19:13 UTC);
guest 9201 (tester4), 192.168.0.151; host 192.168.0.128. Golden controller 0.339.0 → the box's first controller.
- Root: the day-0 install rotated the root password into the hub vault; read through the operator's break-glass path
(stored in sec as NIGHT_2026_10_23_VAULTED_ROOTPW, never printed); DooPlex's public key added to root.
- Fixtures (by hand, release candidates, not releases):
controller 0.340.0-rc.6 (= felhom-controller 943c517) 19:30:44 UTC → rc.8 (= 1babfe9) 22:03:31 UTC;
agent 0.163.0-night.1 (= felhom-agent ae69ad4, D30) 19:30:07 UTC → 0.162.99 (= b9b831f, D30 hardened) 21:51:40 UTC
(the first name is a pre-release string the controller cannot compare — the off-site unlock refused; see 07a);
the D4 drop-in by hand (the bundle's own steps: sshd -t -f, install, sshd -t, try-reload-or-restart, sshd -T).
- Drive: the 200 GB disk initialised through the product as `hdd_1` (the second-drive target); the apps on `belso`.
- Apps (the rehearsal's 8, install API): nextcloud, immich, paperless-ngx, vaultwarden, mealie, actualbudget, radicale,
bookstack — all running (05-installs.txt).
- D30 live proof ran on this box BEFORE the uploads (19:45–20:16 UTC; P2-D30/).
- Household data: gen_household.py with SEED 20261023 (3414 files, 3.5 GB, MANIFEST in the loader CT 9345).
- Load: soakrun.sh 120 on CT 9345 from 20:04:16 UTC (load.py + load2.py cycles); the Nextcloud tree stream restarted
20:09:21 UTC with a hosts entry. A first attempt at 19:55:59 UTC was VOID (the tool used the dashboard cookie for the
setup gate; the gate now hands out its own cookie) — stopped after 2 min, logs kept in the loader's /root/void-1955/.
- Off-site: the box held for the household's recovery code (offsite.state=awaiting_recovery_key); unlocked 21:52 UTC
with the household's recovery code (07-unlock.txt).
- Recorders: rec.py on the box host (10 s) from 19:47 UTC; outside.sh on DooPlex (10 s /api/health + qemu I/O) from
20:09 UTC.
documentation/backlog/OPEN-ITEMS.md
+4 -2
View File
@@ -153,7 +153,7 @@ stopping line that lies.
| **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC |
| **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. | **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC |
## Backup & restore — 45 rows (P2 10, P3 19, P4 16)
## Backup & restore — 46 rows (P2 10, P3 19, P4 17)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -202,6 +202,7 @@ stopping line that lies.
| **R-1227** | Backup & restore | P2 | **The container registry and the agent packages exist only on DooPlex — after its loss no box updates, reinstalls or installs.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G7): `/data/gitea/packages` is 38.6 GB, Longhorn on DooPlex only; the nightly off-site copy leaves it out on purpose (2026-10-09: „rebuilt from the code”), and a rebuild was never timed; a rebuilt image is not the same digest. | OPEN | the operator's yes (a DooPlex change) | Operator decides (STATUS): A) copy what a box can be offered (released controller/hub tags, agent packages, infra images) nightly to ep0 `operator` — CC's pick; B) a timed rebuild drill and accept the outage | operator |
| **R-1228** | Backup & restore | P3 | **The website, the installer script and the hub are all served from the home line — a DooPlex loss stops new installs and the public site.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G8): `felhom.eu`, `hub.felhom.eu` and `gitea.dooplex.hu` resolve to the home address; the installer fetches `felhom.eu/scripts/felhom-host-install.sh`; only the ISO (Cloudflare R2) survives. | OPEN | the operator's yes | Operator decides (STATUS): A) accept — Gitea and the hub first, the site after; B) the static site + the installer tag on Cloudflare like the ISO — CC's pick, after the release week | operator |
| **R-1229** | Backup & restore | P4 | **ArgoCD's per-app auto-sync settings live only on the cluster, not in git.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G9): the Applications differ (`enabled` true/false, prune, selfHeal); a rebuilt cluster from `homelab-manifests` would not have them. | OPEN | — | Write each Application's sync policy into `homelab-manifests` (a homelab change, the operator's yes for the commit that changes live behaviour) | operator |
| **R-1231** | Backup & restore | P4 | **The recovery page says the box's host service is „older” when the controller only cannot tell.** Test night 2026-10-23 (`P4-testnight/07a-unlock-refused-fixture-version.txt`): with the agent's capability `unknown` (a version string it could not compare), the page read „…a hozzá tartozó házon belüli szolgáltatás régebbi… A gép magától frissül”. The same page is reached by a merely-down agent, for which „older” and „updates itself” are not true. | OPEN | — | A separate sentence for `unknown` („a gép most nem éri el a házon belüli szolgáltatást — próbáld újra pár perc múlva”); a render test per state; rides a controller release | CC |
## Storage & devices — 11 rows (P3 4, P4 7)
@@ -261,7 +262,7 @@ stopping line that lies.
| **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC |
## Monitoring & notifications — 18 rows (P2 1, P3 12, P4 5)
## Monitoring & notifications — 19 rows (P2 1, P3 13, P4 5)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -283,6 +284,7 @@ stopping line that lies.
| **R-1222** | Monitoring & notifications | P3 | **A household that sets an EVENING backup time gets a false „database dump missed” error every morning.** REAL in code (night 2026-10-22; split from R-1189 (e)): the hub's 05:00 check wants a dump since Budapest midnight (`hub/internal/monitor/deadline.go:416-419, 555-565`), so an evening dump (the banner suggests evening times) is always „yesterday”; on 2026-10-25 the check also runs at 04:00 (R-1183). Evidence `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN | — | Fix: the check wants a dump within the last 26 h (or since the box's own scheduled time); a hub test with an evening schedule; rides a hub deploy | CC |
| **R-1223** | Monitoring & notifications | P3 | **After a reboot, a box whose tunnel ep0 refuses is invisible: the future stamp is gone, so neither the D30 re-key nor the hub's alarm sees it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, A8 + H1): the kernel forgets `latest-handshake` when the interface restarts (0 = never); the agent then sends neither `last_handshake_age_s` nor `handshake_future_s`, and `checkWGTunnel` (`hub/internal/api/host_tunnel.go`) does not judge a registered, active tunnel with no handshake at all. Shape: the clock ran ahead, was corrected, the box rebooted (some households switch off at night), ep0 still holds the future stamp and starts nothing. | OPEN | — | Agent: report `tunnel_up_s` (how long the unit has been active) with no handshake; hub: a registered, active tunnel with no handshake for > 30 min while reporting → `wg_tunnel_stale`; the agent re-keys the same way after 30 min with no handshake. Tests both sides; rides an agent release + a hub deploy | CC |
| **R-1224** | Monitoring & notifications | P4 | **The operator's 1-hour mail cooldown for `wg_tunnel_stale` is per customer, not per host, and `wg_tunnel_ok` does not end it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, H3): `notify/dispatcher.go:620` keys the cooldown `customerID:eventType`. Two hosts of one household going stale within an hour, or stale → ok → stale, store a second event row but mail once (the suppression is in `notification_log`). | OPEN | — | Key the cooldown by `host_id` from the details for host-scoped operator events, and let the matching ok event end it; a dispatcher test with two hosts; rides a hub deploy | CC |
| **R-1230** | Monitoring & notifications | P3 | **A reinstalled box that waits for the household's recovery code was reported „stuck” after an hour: a false operator mail, a needless re-issue of the off-site login, and a controller restart on the box.** Measured on the test night 2026-10-23 (Tester4-16fdef, `audits/night-2026-10-23/P4-testnight/`): the box declared `offsite.state=awaiting_recovery_key`; the hub's delivery check knew only `needs_credential`, read „consumed, never applied” and at 20:16 UTC mailed `offsite_delivery_stuck` and re-staged the password (the config bump restarted the controller — one 502 on the outside probe). The controller's own comment said this branch is skipped — false. | READY — hub 0.167.0 source `14388e21`: the checker stands down for that declared state; `TestR1230_AHeldBoxIsNotStuck`, red-proved. | the hub deploy (the operator's yes) | Deploy hub 0.167.0; close then | CC |
## Hub & operator — 12 rows (P3 5, P4 7)