Night power drill complete: 8 real cuts on demo-hp (all came back), the estimator checked on a second machine and on 8 GB; R-1003..R-1009 filed; STATUS to one screen (history moved byte for byte); G18 and F5 corrected
gates / gates (push) Successful in 6m39s
gates / gates (push) Successful in 6m39s
This commit is contained in:
@@ -112,10 +112,12 @@ match what the reader sees is how an instrument stops being believed (R-421). It
|
||||
**nothing was proven on the day this line was drawn** and a stopping line that moves a status is a
|
||||
stopping line that lies.
|
||||
|
||||
## Install & onboarding — 9 rows (P3 6, P4 3)
|
||||
## Install & onboarding — 11 rows (P1 1, P3 7, P4 3)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **R-1003** | Install & onboarding | P1 | **The FAQ promises that the BIOS is set at installation to switch the box on after a power cut — „a standard part of every installation" — but a self-install from the ISO has no such step, and the setting decides whether a box comes back.** Measured 2026-10-13/14 (`audits/night-power-2026-10-13/`): demo-hp stayed OFF after a real cut until the operator set „After Power Loss = Previous State" by hand (20:05); with it, demo-hp came back by itself from 6 cuts that night. The N100's setting is unknown (no monitor). In July both boxes stayed off 4.5 h (`AUDIT-power-outage-recovery-2026-07-22.md` F8). Same text in `website/gyik.html` and `website/en/faq.html`. | **OPEN** — filed 2026-10-14. P1 by the register's own definition (a promise we make is false today); the operator ranks it | operator: what to promise | Operator: choose — (a) the FAQ says the household (or the install guide) sets it, with the menu names; (b) the installer's first-boot check reads it where the firmware exposes it; the website is not changed until then | operator |
|
||||
| **R-1009** | Install & onboarding | P3 | **Operator: decide whether the machine estimator's figures move with the 2026-10-14 check on a second machine** (`audits/sizing-check-2026-10-14/CHECK.md`, „Proposal"). The three answers on an 8 GB box held (files + passwords fit well; Immich alone tight — 14 OOM kills in its ML container; the family set does not fit — real harm); the family set's real simultaneous peak was 97 % of the page's sum. Proposed (A): Immich peak 3,061 → 3,300, Jellyfin 373 → 450, Home Assistant 328 → 435, Nextcloud 525 → 300, Vaultwarden 99 → 45, Grimmory typical 346 → 450; no verdict changes (site rule: family 5,515 → 5,659). (B) a sentence for Immich on 8 GB. | **WAITING-ON-OPERATOR** — filed 2026-10-14 | operator | Operator: A, A + B, or neither; then CC edits the data and rebuilds the page's data file | operator |
|
||||
| **R-1000** | Install & onboarding | P4 | **The website does not say how to turn a Felhom machine off.** Since controller v0.310.0 (R-998) the dashboard has „A gép kikapcsolása" (Beállítások → Rendszer); the FAQ still has only the power-cut line. Text to add (HU/EN): turn it off there; switch it on again with the power button; the hub stays quiet for 14 days. The plug trick is not mentioned to households. | **READY** — filed 2026-10-13 | — | One website task: one FAQ entry per language, `site_gates.py` | CC |
|
||||
| **R-980** | Install & onboarding | P3 | **A person cannot check that the installer is ours: the ISO's first boot runs the install script from a movable tag with no hash or signature, and the ISO checksum sits on the same website as the download.** Fix: a signed manifest (or signature) checked by the ISO before it runs the script, and the public key printed in the guide. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN — design-proposal | — | Design note, then build | CC |
|
||||
| **R-250** | Install & onboarding | P3 | **A customer create can fail fail-closed because the host-key scan ladder is shorter than the DNS/AAAA settle time.** Found 2026-08-07 creating the fifth walk's venue. `POST /configs/new` with off-site enabled provisions the Storage Box sub-account and then scans its SSH host key to pin it — **fail-closed by design** (`offsite.go:111-121`, *"don't serve a descriptor the controller can't verify"*), with `defaultScanBackoff` = 2+4+8+16+30 ≈ **60 s**, sized by its own comment to *"the observed DNS propagation lag"*. **Measured, both halves:** the first create exhausted the ladder — five `no such host`, then `dial tcp [2a01:4f8:bacc:2:200::d30]:23: connect: network is unreachable` (the name had just begun resolving, **AAAA-first, into a pod with no IPv6 route**) — and the hub logged `[ERROR] offsite provision for walk5`. An **identical second POST succeeded** ~70 s later on its own final rung (`shared already provisioned … (subaccount 285351)`). **Total settle ≈ 100 s against a 60 s budget.** The endpoint was never the problem: verified afterwards from both the node and the hub pod, by name, `SSH-2.0-OpenSSH_9.6p1`. **Two distinct things are wrong and should not be merged:** (1) the budget is sized against DNS *existence*, but what actually bit is the **AAAA-before-A** window, a different and longer phenomenon — lengthen the ladder and/or prefer the A record for the scan dial; (2) **the operator is told nothing actionable** — the create fails with a generic error, the remedy is "press it again", and nothing says so. The retry is genuinely safe (idempotent on the label; confirmed afterwards — `one_time_secrets` 1, one sub-account, no double-provision) **but that safety is invisible to the person deciding whether pressing again will double-charge them.** **Severity LOW-MEDIUM:** self-clearing, no data at risk — but it is the first thing a new customer's provisioning does and it fails looking like an outage. | **VERIFY** — **2026-10-12: hub 0.147.0 live** (S29, decision 224): the host-key scan dials IPv4 first, waits ~2 min (was ~1), and when it still runs out says the store was created and it is safe to press Save again (`TestScannerDialsIPv4First`, `TestScanOutOfRetriesSaysSafeToPressAgain`, red-proved). Known limit: the hub Ingress reads for 60 s, so a very slow first try can show a gateway timeout while provisioning finishes in the background (pressing again is safe). **READY** — operator ruling 2026-10-12 (S29, `09` §3 decision 224): yes — the scan prefers IPv4, waits longer, and the error says „safe to press again". **READY** — owner Viktor | — | The next real off-site setup (a new customer): read the hub log for the scan; success or the new message → close | CC |
|
||||
@@ -125,7 +127,7 @@ stopping line that lies.
|
||||
| **R-494** | Install & onboarding | P4 | **NARROWED 2026-09-14 by operator ruling → [P3-LOW] the hub COULD create the tunnel at customer creation, for a domain already on Cloudflare. Not blocking: every customer has their own domain and the operator creates the tunnel per day-0 A.1 (`architecture/01-topology-and-trust.md`).** *Original finding, kept:* **[P1-HIGH] A new customer's dashboard has NO reachable address unless the operator hand-makes a Cloudflare tunnel — the link in the setup-code mail is dead.** MEASURED 2026-09-14 on a fresh install from the public ISO (drill intervention **I1**): the claim mail points at `https://felhom.drill0242.felhom.eu`; that name has **no A and no AAAA** record (`dig @1.1.1.1`, control `felhom.enkisfelhom.hu` resolves); the hub has **no tunnel- or DNS-creation code** (`hub/internal/cloudflare/` holds only geo-rule removal; `cf_tunnel_token` is a pasted, optional form field, `configs.go:1478`) — day-0 runbook A.1 makes it a manual Cloudflare-dashboard step that nothing on the customer-create page asks for; the box's own split-horizon resolver on the appliance LAN IP answered `google.com` but not the dashboard name at 13:27:39Z; the agent applied the record at **13:27:44Z** (`lanresolver: applied split-horizon record … ip=192.168.0.158`, 3 m 46 s after the controller started), so the box CAN answer the name — **but only to a device that uses the box as its DNS server, and no document, screen or mail tells a household to do that**; the router and the installer-offered DNS answer nothing. The page was reachable only at the guest's LAN address with the name forced (`curl --resolve …:443:192.168.0.158`). **A volunteer could not have done that.** **What it needs:** an operator ruling — the hub creates the tunnel and DNS at customer creation, or the product gives a household a LAN address that works with no DNS change. | **READY — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P3→P4: operator-side automation; the operator creates the tunnel by hand per the day-0 runbook.** | — | — | CC |
|
||||
| **R-504** | Install & onboarding | P4 | **[P3-LOW] `iso.felhom.eu` cannot show an index page on its own — its root returns 404, and the download page lives on the website instead.** MEASURED 2026-09-14: `https://iso.felhom.eu/` and `/index.html` → 404; only named objects answer. The host is an R2 bucket behind a custom domain; whether R2 would serve an uploaded `index.html` at `/` was **not measured** (uploading anything to the public bucket is a publication). The ISO v1.27.0 task puts the Hungarian download page at `felhom.eu/letoltes` (published with the ISO, after the operator's yes). **Remaining:** a redirect from `iso.felhom.eu/` to that page needs a Cloudflare rule the session has no credential for. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (Cloudflare rule)** **Re-ranked 2026-10-03: P3→P4: households are sent to the website's download page; the bare address is cosmetic.** | — | — | operator |
|
||||
|
||||
## Apps & catalog — 8 rows (P2 1, P3 3, P4 4)
|
||||
## Apps & catalog — 9 rows (P2 1, P3 4, P4 4)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -137,6 +139,8 @@ stopping line that lies.
|
||||
| **R-707** | Apps & catalog | P4 | **[P2] 37 apps still start with a login a stranger can take (`09` §3 decision 45).** Audit of all 53 apps: `app-catalog-felhom.eu/FIRST-ADMIN.md` (class, fix route, status, measured or read). Open: **3 hard-coded defaults** — calibre-web (`admin / admin123`, measured working on demo-hp and 9202; its own `cps.py -s` route needs a generated password WITH a special character — our generator is letters+digits, a controller change), mealie (`changeme@example.com / MyPassword`), wger (`admin / adminadmin`); **34 open first-run screens** (the first visitor creates the admin: actualbudget, adventurelog, audiobookshelf, calcom, docmost, emby, ghost, gitea, gramps-web, home-assistant, homebox, immich, jellyfin, komga, n8n, navidrome, opengist, outline, papra, plant-it, radarr, rallly, recipe-importer, romm, seerr, sonarr, sparkyfitness, tandoor, termix, uptime-kuma, vikunja, wanderer, wishlist, zipline). **Stale notes:** romm's `default_creds` `admin / admin` answers 401 on demo-hp (like a wrong password) — the page now warns with a login that does not exist; zipline's looks stale too. **Measured on demo-hp 2026-09-28 (read-only):** bookstack's default still logs in on the INSTALLED app (the fix is for new installs; the page now warns). Each fix: route (a) env or (b) the app's own CLI/API via `after_install:`, proven on 9202 with the default failing and the generated password working; route (c) a page sentence. Several sessions (operator, 2026-09-28). **2026-09-29 (controller v0.280.0, catalog `d0e7e2e`):** every class-3 app fixed — mealie, wger, calibre-web by `after_install` (calibre-web with the new `password:24:special`), proven on 9202 fresh installs (`audits/login-gate-2026-09-29/D/`); the setup gate (decision 46, spike PASSED) built and live on immich, n8n, audiobookshelf (probes measured) and uptime-kuma (button) (`…/C/`); romm's and zipline's stale notes removed. **Left: 30 class-4 apps** — gate each (probe measured on 9202 where one exists — 11 upstream candidates listed in `…/B/B-VERDICT.md` §3; the button otherwise). **2026-09-29 afternoon (controller v0.281.0, catalog `6faf432`):** 28 more class-4 apps gated — 32 of 34 — each proven on 9202 (`audits/gate-rollout-2026-09-29/`B): stranger → gate page / 401, household reached the first-setup screen, the gate opened (9 by a measured probe, the rest by the press), the app answered after. seerr, outline, rallly: gated, their opening needs a media server / e-mail (not proven). **Left:** wanderer (R-714); plant-it is not installable. | **NARROWED** (2026-10-03 triage: the row's verdict was finished, but it names open work no other row carries — seerr, outline and rallly are gated, but the gate OPENING is not proven (needs a media server / e-mail)) — **CLOSED — 2026-09-29 (the rest → R-714)** | — | — | CC |
|
||||
| **R-905** | Apps & catalog | P4 | **wger's collected style files (283 MB) go into every wger backup, though the app rebuilds them at every start.** Since R-762's fix (`DJANGO_DEBUG=False` + `collectstatic`, catalog `cf1ed43`) the static files sit in a named volume that the backup legs copy like data (measured 283 MB, `audits/design-build-2026-10-06/`); `collectstatic` regenerates them, so they are rebuildable bytes in every Tier-1 unit, Tier-2 copy and off-site snapshot. wger is `lifecycle: hidden` and no box runs it (hub read 2026-10-08). Options to weigh: a non-backed-up volume class for regenerable data, or an anonymous volume. Found by the R-762 helpers 2026-10-08. | **OPEN — needs a design before wger is shown** | — | Design the „regenerable volume" exclusion (catalog + controller backup legs) | CC |
|
||||
| **R-995** | Apps & catalog | P4 | **Removing an app through the product can leave unnamed Docker volumes and an empty import folder behind.** MEASURED 2026-10-13 on demo-felhom after 61 removals (`documentation/audits/sizing-2026-10-13/METHOD.md` „Teardown”, `evidence/leftover-volumes-after-bench.txt`): 8 unnamed volumes (4–72 KB; immich 1, romm 2, privatebin 3, onlyoffice 2 — volumes the images declare themselves) and `userdata/import/{paperless,calibre,grimmory}` (empty, created by the deploy). The removal answered 200 and its own leftover check was clean, because it looks only at labelled volumes. No data risk; disk and tidiness only. Removed by name on the bench. | **READY** — filed 2026-10-13 | — | Make the removal account for the containers' unnamed volumes (and an empty import folder it created), with a test that deploys an image that declares a VOLUME | CC |
|
||||
| **R-1007** | Apps & catalog | P3 | **After a power cut Immich shows photos as done that are broken, and nothing repairs them: of 101 photos uploaded in the last ~30 s before the cut, 81 kept their original but got an EMPTY thumbnail and preview (162 zero-byte files), and 1 — answered „201 Created" 0.3 s before the cut — has no original at all.** Immich's job queues are empty after the start, so it never redoes them (`audits/night-power-2026-10-13/` R2, `immich-check.txt`). Same ext4 effect on a plain 40-byte file. Other apps may do the same with files written just before a cut. | **READY** — filed 2026-10-14 | — | CC: on a scratch box, find whether Immich's own „missing thumbnails" job and its integrity report find and fix these; if so, decide (operator) whether Felhom runs them after an unclean boot | CC |
|
||||
|
||||
|
||||
## App updates — 4 rows (P3 4)
|
||||
|
||||
@@ -147,10 +151,11 @@ stopping line that lies.
|
||||
| **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC |
|
||||
| **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. | **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC |
|
||||
|
||||
## Backup & restore — 28 rows (P2 3, P3 15, P4 10)
|
||||
## Backup & restore — 29 rows (P2 4, P3 15, P4 10)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **R-1006** | Backup & restore | P2 | **A power cut during the night's whole-guest backup leaves the half-written archive on the host, and from then on every local whole-guest backup of that box is skipped „not enough space" — nothing removes it.** Measured 2026-10-14 on demo-hp (`audits/night-power-2026-10-13/` R3): the cut at 02:52 left `vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat` (12.2 GB) + `.tmp`; host root at 92 %; agent retry 03:08 „skipped: not enough space: local has 2.9 GiB free; the last archive … was 4.6 GiB". The operator got `whole_guest_backup_failed` (true). Second part: the space preflight sizes the new archive from the LAST one (7.2 GB planned) while the guest had grown (12.4 GB written when cut) — a backup that outgrows the plan can fill the host root. Third part, the same rule the other way: after one big archive (14.4 GB at 05:17, while bench data sat on the guest disk) the next preflight would need ~21 GB of 5.5 GB free — the box skips every later backup though the guest is small again; cleared by hand at 07:28 UTC (`night-power-2026-10-13/README.md`). | **READY** — filed 2026-10-14 | — | CC: on agent start and before each backup, remove a `vzdump-*.tar.dat`/`.tmp` left by a run that is not in flight (with a test that plants one); size the preflight from the guest's used space, not the last archive; prove on demo-hp with a cut | CC |
|
||||
| **R-978** | Backup & restore | P3 | **Starting the escrow ceremony again can make the customer's written-down recovery code useless: the new package replaces the old one before the new code is confirmed.** It can happen to an ordinary customer who starts and walks away. Fix: keep the previous package until the new code is confirmed. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Fix in the agent with a test | CC |
|
||||
| **R-973** | Backup & restore | P3 | **A compromised controller can make the whole-guest backup run again and again, and each run prunes older restore points: the local tier directly, and the off-site tier through the store's own nightly keep-last-2.** So `07` §8 row 10 (ransomware) is weaker than it reads. The second reader confirmed the local half; the off-site half is inferred. Fix: a cadence or daily cap on the local-API backup route, and off-site snapshots that a nightly prune cannot remove. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Add the cap with a test; decide the off-site retention shape | CC |
|
||||
| **R-926** | Backup & restore | P3 | **After a household removes an `after_install` app keeping its backups and then restores it, nobody has checked whether the app page still shows a password that works.** A `type: password` deploy field is NOT in `stacks.PortableSecretEnvVars` (that list is `type: secret` only, `deploy.go:1199`), so it does not travel in the backup unit — the same shape as R-765, where Radicale's restored login was replaced by a value no page showed and every phone lost its calendar. MEASURED on 9202 2026-10-10 for grocy: remove keeping the backups, restore, and **the household's own password still signs in and the seeded data reads back** — so the DATABASE half is right. What was NOT settled is the PAGE half: `app.yaml` holds the value encrypted at rest, so reading it proves nothing (a 76-character string that of course did not sign in), and `POST /apps/grocy/initial-credentials/reveal` answered 404 for this app. Seven apps are in the same class: bookstack, calibre-web, claper, dawarich, grocy, mealie, wger. | **OPEN — the question is unanswered, not the answer bad** | — | Ask the controller what the app page renders for a generated `type: password` after a restore (one endpoint read, one app), and if it differs from the one that works, decide: carry the value in the unit, or re-run `after_install` after a restore | CC |
|
||||
@@ -209,7 +214,7 @@ stopping line that lies.
|
||||
| **R-779** | Security & access | P4 | **[P3-LOW] Part A's "two outside addresses seen as two" is proven through the simulated tunnel only; on the REAL tunnel the second outside address (ep0, one request allowed) was refused by Cloudflare's edge with 403 and never reached the box.** Measured 2026-10-01 19:51 UTC (`audits/visitors-2026-10-01/A/L2-demo-hp-real-tunnel.txt`): no log line on demo-hp; demo-hp's box has no geo restriction in its settings, so a Cloudflare ZONE rule (country or bot, not read) refused a German datacenter address. DooPlex's own address on the real tunnel was seen as itself. **Needs:** one sign-in from a second Hungarian address (the operator's phone off wifi) while DooPlex is locked out — 2 minutes; and say which Cloudflare rule refused ep0. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (a phone), CC reads the logs** **Re-ranked 2026-10-03: P3→P4: a proof gap on the real tunnel; operator-only follow-up.** | — | — | CC + operator |
|
||||
| **R-904** | Security & access | P4 | **Cloudflare can read every household's app traffic; replacing it with our own relay is a later item.** Facts (reviewer discussion 2026-10-08; `01`): app traffic and the dashboard reach the box through the Cloudflare Tunnel (`01` §5 trust table, rows end-user ↔ apps and customer ↔ controller UI; §7), and the tunnel's public end is Cloudflare's edge, where TLS ends — so Cloudflare can technically read that traffic (the FAQ says so since 2026-10-08, R-900; „TLS ends at the edge" is not written in `01` — add it there). What Cloudflare gives today, free: inbound reach with no router setup, the CGNAT answer (`01` §4, §7); certificates (`01` §7, the free tier covers one level below a zone); the geo-WAF the hub enforces (`01` §5 last row, §7); flood protection (not in `01`). The alternative named: our own EU relay over WireGuard with TLS passthrough by SNI, certificates on the box, the geo-block on the relay. Its costs: one more machine the operator keeps up, and a single point of reach for every box; weaker flood protection; about a week of work after a spike. Operator ruling 2026-10-08 09:07 (`09` §3 decision 184): a later item. | **DEFERRED** — **2026-10-10: the asked sentence is in `01` §7** („TLS ends at Cloudflare's edge …”); the spike stays deferred. **DEFERRED — after the first customers (operator ruling 2026-10-08 09:07)** | — | A spike after the first customers (the relay's reach, cost and flood behaviour, measured) | operator |
|
||||
|
||||
## Box system & updates — 8 rows (P2 1, P3 6, P4 1)
|
||||
## Box system & updates — 10 rows (P2 1, P3 6, P4 3)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -221,8 +226,11 @@ stopping line that lies.
|
||||
| **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator |
|
||||
| **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC |
|
||||
| **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** | **WATCHING — rank P3-LOW; owner: CC (renew ≤ 14 days or bake); retire at the first external install** | — | — | CC |
|
||||
| **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC |
|
||||
| **R-1008** | Box system & updates | P4 | **After a power cut the agent's stale-lock check calls the dead backup „genuinely in-flight" and leaves `lock: backup` on the guest.** Harmless today only because Proxmox's own start-all removes it in the same second („removed left over backup lock from '9201'!", demo-hp 2026-10-14 02:52:56, `audits/night-power-2026-10-13/R3/lock-lines.txt`). Nothing can be in flight on a fresh boot. | **READY** — filed 2026-10-14 | — | CC: treat a vzdump task from before the current boot as not in flight (boot id / btime), with a test | CC |
|
||||
|
||||
## Monitoring & notifications — 7 rows (P2 1, P3 5, P4 1)
|
||||
|
||||
## Monitoring & notifications — 8 rows (P2 1, P3 5, P4 2)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -233,6 +241,8 @@ stopping line that lies.
|
||||
| **R-435** | Monitoring & notifications | P3 | **The snapshot-drop detector is blind to a single-app deletion — the exact shape the box can produce.** `snapshotDropFraction = 0.5` and `snapshotDropFloor = 5` (`hub/internal/monitor/offsite.go`) require a fall of MORE than half the previous count. demo-hp's baseline is **69** across 9 apps, so ~35 snapshots must go before it speaks; **one app's tag is ~9 and is invisible.** `offbox.go:1388` runs `forget --prune` **grouped by host,tags** — a per-tag wipe is precisely the shape a faulty retention or a targeted deletion produces. **This is deliberate, not accidental:** the constant's own comment argues the insensitivity, and *"a detector that cries wolf is switched off within a fortnight"* is a lesson this project paid for. **So this row is NOT a demand to lower the threshold.** It is a demand that the blind spot be written where the operator reads it, because "an unexplained fall is noticed within a day" (R-431, and `STATUS.md`) is true only of falls above half. **Discovered by arithmetic while planning the drill's Phase 2/3 pairing, which could not have worked:** Phase 2 deletes one app and Phase 3 expects the alarm to fire. **LIMITATION NOW WRITTEN INTO THE ALARM'S OWN DOCUMENTATION, hub v0.111.1** — the comment above `snapshotDropFraction` in `hub/internal/monitor/offsite.go` now states what the detector does NOT see, with the demo-hp arithmetic and the `forget --prune` grouping that makes the blind spot sit on the most likely single-app failure. **It also says explicitly that the numbers must NOT be lowered to "fix" this** and that per-app detection needs a SECOND signal keyed on the per-tag count. **The row stays OPEN because documenting a blind spot is not covering it** — and because `STATUS.md` and R-431 both still say "noticed within a day", which is true only of falls above half. **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-435.md` — the box's key can no longer delete (decisions 68–69); deletion through any other credential still needs >50 % to alarm. Question D7 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D7 (`09` §3 decision 191):** yes — option C, an error mail when even one snapshot disappears outside a hub-opened clean-up window (pinned tiers). | **VERIFY** — **2026-10-12 read: not provable today.** No mail for a snapshot drop reached the operator since 2026-10-11 (Gmail), but the window itself could not be read: no hub page shows `offsite_windows`, Tester 1's controller log restarted with 0.308.0 at 13:49Z, and the guest-agent path to its VM did not answer. Re-dated after the next weekly clean-up. **VERIFY — 2026-10-09: DELIVERED in hub 0.144.0. Live proof NOT run: a window is granted only when the box asks for one, and Tester 1's weekly clean-up is next due ~2026-10-12 (last window 2026-10-05); prove it then (no alarm inside the window; the hub's window row as control).** **VERIFY — built on hub main 2026-10-08 (D7, decision 191), ships tomorrow; closes after the 9202 proof (a hand-granted window removes N, no alarm; the hub's window row as the control).** **OPEN — documented in code v0.111.1; the coverage gap itself is unclosed** **2026-10-05 (burn-down night): NEEDS A DESIGN.** Covering one-app deletions needs per-app snapshot counts from the controller and a per-tag threshold measured on real counts — two repos, a mechanism nobody has measured. | — | 2026-10-20: read Tester 1's controller log (window opened/closed, count before/after) and the operator mailbox (no drop alarm) the morning after its weekly clean-up | CC |
|
||||
| **R-724** | Monitoring & notifications | P3 | **[P3-LOW] The status pages disagree with each other in small ways a household notices.** MEASURED 2026-09-29 on a fresh box: Beállítások reads „Mentés ütemezés 02:30 / 03:00" while Biztonsági mentés says 02:30 / 03:30 / 04:15 / 04:30–08:30; Beállítások shows a raw `2026-09-29T19:22:30Z`; its „Helyi cím (LAN)" and „Átjáró" read „nem állapítható meg" on the page the household is asked to read out for remote help; the backups overview still says „Következő mentés — 0 órája" (the age of the LAST run, under the word "next" — noted 2026-09-14, never filed); the dashboard shows the backup at 19:33 where the backup pages say 21:33 (R-500, still reproducing). **FIXED 2026-09-30 (controller v0.283.0):** Beállítások shows the real schedule and a local time; the dashboard's last backup is local time (R-500); „az előző N órája készült" replaces „0 órája". **NARROWED — remaining:** „Helyi cím (LAN)" / „Átjáró" read through the file-sharing container, so a box without it says „nem állapítható meg" — not a text fix (the read needs another path). | **NARROWED — the LAN/gateway read only; owner: CC** **2026-10-06 night: confirmed not a controller-only fix** — the only host-network container is samba (`infra/samba.go:123`) and the agent's local API has no guest-network route; the fix is a new agent route (e.g. the PVE lxc interfaces read) + a controller reader — a two-repo design. | — | — | CC |
|
||||
| **R-285** | Monitoring & notifications | P4 | **A planned, supervised reinstall pages the operator as if the machine had died — there is no notion of expected downtime anywhere.** During the 2026-08-09 rehearsal the hub sent, all `status: sent` to the operator channel: `host_stale` 08:58 UTC, `node_stale` 09:00, **`host_down` 09:28 (error)**, **`node_down` 09:30 (error)**, `host_leaf_changed` 09:31, `host_recovered` 09:31, `node_recovered` 09:34, `offsite_delivery_stuck` 09:34 — eight operator mails for work that was deliberate, attended and announced. **This is the OPPOSITE gap from the one R-281 filed:** the alarms are not missing, they are indiscriminate. `host_stale` at 30 min and `host_down` at 60 min (`monitor/host_staleness.go:22-23`, `downAfter = 2 * threshold`) cannot distinguish a wiped-on-purpose box from a dead one, and `host_leaf_changed` firing on a reinstall is correct-but-expected. **Note the interaction with the mute used on 2026-08-09 evening:** blocking a customer silences everything, so today the only two settings are *page me for planned work* and *tell me nothing at all*. **What is owed is a middle:** a maintenance window, or an operator-set expected-downtime flag, that suppresses staleness and leaf-change while leaving genuine faults audible | **READY (M) — NEW 2026-08-09** | — | The evidence is the operator's mailbox plus `events`/`notification_log` for 2026-08-09 | CC |
|
||||
| **R-1005** | Monitoring & notifications | P4 | **A recovery mail says how long the box was „stale", not how long it was gone: after a 51-minute power cut demo-hp's `host_recovered` read „was stale for 8m" and `node_recovered` „23m".** An operator reading only the recovery mail under-reads the outage (`audits/night-power-2026-10-13/` R6). Same kind, the other way: the off-site mail of 2026-10-14 02:18 was titled „3 of 3 apps failed to back up (offsite run)" while 11 other apps were backed up in that run — only the 3 flagged apps were counted. | **READY** — filed 2026-10-14 | — | CC: add the time since the last report before the gap to both recovery messages, and count every app of the run in the off-site subject, with a test that asserts each text; next hub release | CC |
|
||||
|
||||
|
||||
## Hub & operator — 11 rows (P3 6, P4 5)
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user