Tester 2: a new install, not an upgrade (operator ruling 2026-10-22 14:11) — hand-over list rewritten (host delete then RESET, what each removes, the friend's page in Hungarian); R-862, R-1139 close at the reinstall; R-32's first live purge noted; STATUS, CONTEXT
gates / gates (push) Successful in 16m5s

This commit is contained in:
2026-10-22 14:19:26 +02:00
parent ecfc09c3ef
commit 82ce9d72d6
4 changed files with 129 additions and 52 deletions
+8
View File
@@ -16,6 +16,14 @@
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
> **2026-10-22 14:11 — operator ruling: Tester 2 is a NEW INSTALL, not an upgrade.** The friend reinstalls the laptop
> from the public download (installer 1.30.0 or newer) with the new drive in before the install; the old box's data is
> not needed („he barely used the laptop, no useful data on it"). Read on the hub the same day (read only): no sealed
> key package; off-site provisioned (shared) but its key never deposited; one ep0 whole-box copy (2026-10-04); the mail
> language is English. CC's pick for the operator: host delete, then customer RESET (it empties the Storage Box folder
> and the ep0 namespace — the operator's press). R-862 and R-1139 close at the reinstall; Tester 2's RESET is R-32's
> first live purge. List: `documentation/pilot/TESTER2-handover.md`.
> **2026-10-22 day — operator rulings (11:42) and what was built.** Rulings: **D28 = A** (loosen the host-swap check);
> agent 0.162.0 **vouched** (read back: `artifact_agent_version` 0.162.0 at 07:57 UTC); Tester 2's laptop was installed
> with **our installer**; Tester 2 is off, no date. Built: **R-1199 reopened** — controller 0.338.0's start-up default
+3 -2
View File
@@ -1,6 +1,6 @@
# STATUS — what works, what's broken, what's next
**Ready for Tester 2? Yes — when the laptop is on: the operator sends agent 0.162.0 first (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
**Ready for Tester 2? Yes — a new install: you do the hub step first (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
**Updated 2026-10-22 (day).** A rebuilt box no longer overwrites the family's saved mail list (hub 0.165.0, controller 0.339.0); demo-felhom's own list, replaced this morning, is back. The swap warning now fires only when a machine has clearly too little swap (your D28 = A).
@@ -52,7 +52,8 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12.
**Today, 2026-10-22.**
1. When Tester 2's laptop is on: follow the hand-over list, step 1 first (tell CC; CC sends agent 0.162.0). If not: the box stays on old software without the swap fix.
1. Before Tester 2's friend reinstalls: delete the old host, then RESET the customer (hand-over list, section 1). The RESET deletes Tester 2's old copies on ep0 and the Storage Box. If not: the new box carries an old, unreadable off-site history and a stale whole-box copy.
2. Tester 2's mail language is English on the hub. If the friend reads Hungarian, change it first. If not: his mails and first screen are in English.
**The rest**
- Pull the failed disk Z6A07P2G out of DooPlex (R-934). If not: its alarms return on 11 November. Since 2026-10-12.
- Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). Since 2026-10-09.
+3 -3
View File
@@ -177,7 +177,7 @@ stopping line that lies.
| **R-816** | Backup & restore | P4 | **No off-site failure class has ever been seen live.** F-DIAG (controller v0.182.0, 2026-07-28) split off-site failures into six causes — quota, orphaned, no_repo, no_units, transport, unknown — each with its own Hungarian message. None of the six has been exercised by a real failure on a box; the recovery inventory records it only as a known limit (`documentation/architecture/_recovery-inventory-2026-07-28.md:955`). Filed 2026-10-03 from the F-DIAG row's residue when that row moved to `CLOSED-ITEMS.md`. | **READY — filed 2026-10-03 (triage); owner: CC.** Exercise each class once on a scratch guest (a full quota, a missing repository, a blocked transport) and read the message the household sees. | — | — | CC |
| **R-832** | Backup & restore | P4 | **ep0's copy in a place outside both Hetzner and the operator's home (roadmap).** Today DooPlex (the operator's home) holds it (decision 71). A Hetzner Storage Box would share a provider with ep0 and with every household's file backups, and cannot run PBS, so the copy could not be verified or restored from directly. | **DEFERRED — later, if the product grows** | — | — | operator |
| **R-878** | Backup & restore | P4 | **A catch-up (R-871) runs the database-dump leg in the DAY, and that leg stops an app with a volume for its copy — the household may notice the stop, and a large volume makes it longer.** MEASURED 2026-10-05 on demo-felhom: the catch-up at 08:25:02 stopped opengist, copied 182.5 KB, started it again — about 1 s, then a few seconds of `health: starting`; the night does exactly the same, unseen. Nothing measured for a large volume. Fix direction (if it matters): skip the volume copy of a running app in a DAYTIME catch-up and leave it to the next night, or warn. `audits/catchup-2026-10-05/partA/live-demo-felhom.txt` | **READY — owner: CC** **2026-10-05 (burn-down night): NEEDS A LIVE MEASUREMENT** — a timed catch-up on 9202 for an app with a multi-GB volume, then the skip-or-warn choice. | — | measure a large volume first | CC |
| **R-32** | Backup & restore | P4 | **[P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible.** The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's `"hetzner":"ok"` leg destroys the sub-account, but **a Hetzner sub-account is an access-control object, not a data object** — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **2026-10-06 night: design written** (`audits/night-burndown-2026-10-06/design-R-32.md`) — pick A: purge the repo + set-aside copies through the sub-account's own password login BEFORE deleting it (no main-account credential); the „repo data destroyed" comment at `hub/internal/offsite/offsite.go:276-280` is false for the shared tier; part (3) waits on a read-only measurement (`du` on port 23). Operator: agree to route A. **2026-10-07 07:58: `09` §3 decision 167 — option A yes, and delete the old RESET leftovers once.** **2026-10-07 (Part D step 2, read only):** the hub shows bx11 holding 3.3 GB of data while the four live customers account for ~0.6 GB → **~2.7 GB belongs to no live customer**; no live customer shows a set-aside folder, so the leftovers are in homes whose sub-account is gone, reachable only by the pool box's MAIN account. **The one-time clean-up is STOPPED:** no main-account login for bx11 is held on DooPlex (the API token in the credentials file sees no box). Needs: the main-account login, or the operator lists and deletes in the Hetzner console (`audits/day-2026-10-07/D/D2-listing-readonly.txt`). **2026-10-07: option A DELIVERED (hub 0.142.0):** RESET purges the repo and every `.orphaned-*` through the sub-account's own login before deleting it, and keeps the sub-account if the purge fails (red-proved, `audits/day-2026-10-07/D/`). Not yet seen live (no scratch RESET today). **The one-time clean-up of the ~2.7 GB of leftovers is STOPPED:** it needs the pool box's main-account login (see above). **2026-10-07 14:43 — `09` §3 decision 173: leave the ~2.7 GB of old leftovers for now; re-ranked P2 → P4.** Option A itself is delivered (hub 0.142.0). What stays open is ONLY the leftovers: they were computed (box total minus live customers), never listed, and deleting them needs the pool box's MAIN login, which reaches every customer's data. **Blocker: a safe main-account path.** | — | **Ruling from the run (three parts, deliberately separate):** (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable **BY DESIGN** → RESET gains a **main-account purge of the customer base dir** (the existing operator ack already covers it); (2) the **move-aside guard STAYS** for reinstall-*without*-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator **Restic tab shows per-customer directory bytes vs attributed snapshot bytes**, so dead data cannot hide. Measured on the pool box that night: **49 M attributed** (2 snapshots, 48.717 MiB) against **1.4 G + 3.0 M unattributed** across TWO `.orphaned-*` dirs. Evidence `restic-and-pool.txt` | CC |
| **R-32** | Backup & restore | P4 | **[P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible.** The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's `"hetzner":"ok"` leg destroys the sub-account, but **a Hetzner sub-account is an access-control object, not a data object** — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **2026-10-06 night: design written** (`audits/night-burndown-2026-10-06/design-R-32.md`) — pick A: purge the repo + set-aside copies through the sub-account's own password login BEFORE deleting it (no main-account credential); the „repo data destroyed" comment at `hub/internal/offsite/offsite.go:276-280` is false for the shared tier; part (3) waits on a read-only measurement (`du` on port 23). Operator: agree to route A. **2026-10-07 07:58: `09` §3 decision 167 — option A yes, and delete the old RESET leftovers once.** **2026-10-07 (Part D step 2, read only):** the hub shows bx11 holding 3.3 GB of data while the four live customers account for ~0.6 GB → **~2.7 GB belongs to no live customer**; no live customer shows a set-aside folder, so the leftovers are in homes whose sub-account is gone, reachable only by the pool box's MAIN account. **The one-time clean-up is STOPPED:** no main-account login for bx11 is held on DooPlex (the API token in the credentials file sees no box). Needs: the main-account login, or the operator lists and deletes in the Hetzner console (`audits/day-2026-10-07/D/D2-listing-readonly.txt`). **2026-10-07: option A DELIVERED (hub 0.142.0):** RESET purges the repo and every `.orphaned-*` through the sub-account's own login before deleting it, and keeps the sub-account if the purge fails (red-proved, `audits/day-2026-10-07/D/`). Not yet seen live (no scratch RESET today). **The one-time clean-up of the ~2.7 GB of leftovers is STOPPED:** it needs the pool box's main-account login (see above). **2026-10-07 14:43 — `09` §3 decision 173: leave the ~2.7 GB of old leftovers for now; re-ranked P2 → P4.** Option A itself is delivered (hub 0.142.0). What stays open is ONLY the leftovers: they were computed (box total minus live customers), never listed, and deleting them needs the pool box's MAIN login, which reaches every customer's data. **Blocker: a safe main-account path.** **2026-10-22: Tester 2's planned RESET (operator ruling 2026-10-22 14:11, `pilot/TESTER2-handover.md`) will be the purge's first live run — CC reads its leg journal after.** | — | **Ruling from the run (three parts, deliberately separate):** (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable **BY DESIGN** → RESET gains a **main-account purge of the customer base dir** (the existing operator ack already covers it); (2) the **move-aside guard STAYS** for reinstall-*without*-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator **Restic tab shows per-customer directory bytes vs attributed snapshot bytes**, so dead data cannot hide. Measured on the pool box that night: **49 M attributed** (2 snapshots, 48.717 MiB) against **1.4 G + 3.0 M unattributed** across TWO `.orphaned-*` dirs. Evidence `restic-and-pool.txt` | CC |
| **R-1087** | Backup & restore | P4 | **Whole-box copies left on the OLD local target are never pruned after the target moves.** MEASURED 2026-10-15 (storage-identity build B space check): the N100's host root (`local`) holds three copies from 2026-07-28 (5.5–5.7 GB each, ≈ 16.8 GB); its agent's `local_backup_target` is now `felhom-backup` (the USB disk, retention 3), and retention prunes only the current target. On a small-root box such leftovers would starve the local copies (R-548). Evidence: `audits/storage-identity-buildB-2026-10-15/space/felhom-pve-9201.txt`. | OPEN | — | Decide whether the agent prunes its own `vzdump-lxc-<vmid>-*` files on a former target (after a newer copy exists on the new one), or the target move asks; until then, the N100's three July files can be removed by hand with the operator's word. | CC |
| **R-1171** | Backup & restore | P3 | **demo-hp's LOCAL whole-guest copy is never restore-tested: the box keeps one copy (keep-daily 1) and the restore test wants a copy 24 h old, so the only copy turns 24 h old exactly when the next night replaces it.** Read 2026-10-20 on demo-hp (agent journal): every 6-hourly evaluation since 2026-10-18 says `local: no settled archive yet — nothing to prove`; the local storage holds one 9201 archive; the last local proof was 2026-10-14 03:42 (`settle=24h`, `keep_last=1`, `prune=keep-daily=1`). The hub's `restore_test_stale` (168 h) will alarm from ~2026-10-21 03:42. A real household with retention 1 would be the same. | OPEN — filed 2026-10-20 | demo-hp `journalctl -u felhom-agent` („restore-test evaluated"), the hub's host report `restore_tests` | Agent: a tier that keeps ONE copy proves its newest archive once it is older than the settle time measured from the BACKUP's end, not 24 h — or a settle below the cadence (e.g. 6 h) for a 1-copy tier; a test; then read the next evaluation | CC |
| **R-1172** | Backup & restore | P4 | **After option A (R-893): a failed off-site restore that was PUT BACK is logged at ERROR, and an empty `backups/restore-undo/` folder stays on disk.** Measured on 9202 2026-10-20 (rc.1): the web handler logs `[ERROR] [web] off-box reconstitute rallly (async): … Nem változott semmi…` — good news at ERROR; the put-back removes the per-app folder but not the empty `restore-undo/` parent. Small; 0.336.0 was already pushed (a version is pushed once). | OPEN — filed 2026-10-20 | `audits/night2-2026-10-20/P2/live/11-controller-log-window.txt`, `12-after-failed-restore.txt` | Next controller release: the put-back outcome logged at WARN (its own sentence is the household's), the empty parent removed; a test | CC |
@@ -216,7 +216,7 @@ stopping line that lies.
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| **R-1139** | Security & access | P2 | **Tester 2's break-glass root password was not rotated with the others (R-954, operator decision D1 = A, 2026-10-17): the box has been offline since 2026-10-04.** demo-hp, Tester 1 and demo-felhom were rotated and proven on 2026-10-17 (`audits/agent-allowlist-rotation-2026-10-17/P3/`). | OPEN — filed 2026-10-17 | Tester 2 reporting again | When it reports: the installer's step 4b alone (`P3/rotate-step4b.sh` shape), then the vaulted value logs in (PVE API 200) and the old one is refused | CC |
| **R-1139** | Security & access | P2 | **Tester 2's break-glass root password was not rotated with the others (R-954, operator decision D1 = A, 2026-10-17): the box has been offline since 2026-10-04.** demo-hp, Tester 1 and demo-felhom were rotated and proven on 2026-10-17 (`audits/agent-allowlist-rotation-2026-10-17/P3/`). | OPEN — filed 2026-10-17 **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a fresh install sets and vaults a new console password).** | Tester 2 reporting again | When it reports: the installer's step 4b alone (`P3/rotate-step4b.sh` shape), then the vaulted value logs in (PVE API 200) and the old one is refused | CC |
| **R-985** | Security & access | P3 | **Our zero-knowledge wording is stronger than the product: on a live box the operator can reach the off-site backup key (through the vaulted root credential and the operator tunnel) without the customer's recovery code.** Design-shaped; decision for the operator in `STATUS.md`. Fix options: change the wording, or remove operator root reach to the key. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | WAITING-ON-OPERATOR — design-proposal | operator decision | Decide the wording or the change | operator |
| **R-1042** | Security & access | P3 | **The boxes' Cloudflare keys cannot change zone settings, so Cloudflare's own „Always Use HTTPS” cannot be switched on.** Measured 2026-10-14 on all three zones (`audits/https-only-2026-10-14/``1A-1-zones-and-keys.txt`): each `cf_api_token` is valid and sees only its own zone, but GET/PATCH `settings/always_use_https`, `ssl`, `security_header` answer 403; the redirect and transform rule phases too. The https-only fix therefore lives on the box (decision 246). A second layer at Cloudflare needs „Zone Settings: Edit” (and, for headers, „Transform Rules”) added to each customer's key — the hub's zone-reach check stays. | **WAITING-ON-OPERATOR** — filed 2026-10-14 | a key permission only the operator can grant | Optional: widen each customer key, then switch „Always Use HTTPS” on per zone | operator |
| **R-1104** | Security & access | P3 | **Check https-only on Tester 2 when it reports again.** R-1033 (plain http on every public name) is closed on the three boxes that report (35 names → 301, 2026-10-16). Tester 2 (`sajatfelhom.hu`) has been off since 2026-10-04 on controller 0.292.0; its addresses answer Cloudflare 530 (tunnel down), so nothing is served in plain http while it is off. When it comes back it takes the floor (≥ 0.312.0 carries the redirect). | OPEN — filed 2026-10-16 | Tester 2 reporting again | Run `scripts/https_only_check.py` with Tester 2 in it; close | CC |
@@ -247,7 +247,7 @@ stopping line that lies.
| **R-972** | Box system & updates | P2 | **The shipped controller and hub programs are built with Go 1.24.13 and golang.org/x/crypto v0.31.0, and the agent with Go 1.26.0, so dozens of fixed Go security bugs (including html/template escaping fixes) are still in what runs on boxes and the hub.** Go 1.24 has had no security fixes since Go 1.26 was released (Go supports the two newest major releases: https://go.dev/doc/devel/release#policy); the scanners name fixes in Go 1.26.9 and 1.27.2. Also: the controller image's Debian 12 base has 12 fixable critical packages, and the Samba image has 48 fixable high ones. Fix: build all three on a supported, patched Go; update x/crypto; rebuild the base images. Scanner output: `audits/security-review-2026-10-12/scanners/`. | **NARROWED** 2026-10-16 (burndown 2): **controller v0.322.0 released** on Go 1.27.2 + x/crypto v0.58.0, runtime debian:12.13-slim + `apt-get upgrade` (Debian 13 deliberately not: restic 0.18, R-1105) — scanner: critical 16 → 4, fixable criticals and highs in the OS 0; govulncheck 62 → 1 (module-level, `openpgp` not linked). **Hub v0.156.0 deployed** 2026-10-16 19:26 on the operator's yes (Go 1.27.2, alpine 3.24.2: 0 critical, 0 high); Synced/Healthy, reports from all three boxes after the start, the floor reads 0.322.0. **Agent: Go 1.27.2 committed (`37b99d7`), tested, NOT released** — the next vouched agent release carries it. Open: the Samba image (48 fixable high, not rebuilt); docker-compose plugin's own Go (4 high, upstream). Evidence `audits/burndown2-2026-10-16/P4/` **2026-10-16 (evening): the Samba part is DONE in controller v0.323.0** — felhom-samba 1.2.0 on Alpine 3.24.2 (0 critical / 0 high, was 48 fixable high), proven on demo-hp with the same household password (smbclient, nmbd, mDNS, WS-Discovery); on the three test boxes now, on every box with the global floor (R-1105's night check). The controller's Debian 13 OS part: 1 critical / 70 high, none fixable. Left: docker-compose plugin's own Go (4 high, upstream); the agent release; restic's binary is R-1107. **2026-10-17 (evening): agent v0.159.1 (Go 1.27.2) RELEASED and running on demo-hp, Tester 1 and demo-felhom** (signed jobs on the operator's yes); waiting for the operator's vouch — then only the upstream compose plugin is left (close then, recorded). | the next vouched agent release | Close when the global floor carries v0.323.0 and the next vouched agent release carries Go 1.27.2 (compose plugin: upstream, recorded) | CC |
| **R-1207** | Box system & updates | P2 | **A host with no (or too little) swap gets nothing from the guest swap fix (R-1192), and the agent never adds host swap.** The fix lets a guest swap as much as its RAM; that swap lives on the HOST. Measured 2026-10-21 night: our installer gives an 8 GB / 256 GB machine 7.75 GB of host swap (LVM `pve/swap`), so the installer's boxes are covered; a BYO host (or one with swap turned off) may have none. Now reported and said (agent `0da792e`: `host_swap_mb` / `guest_swap_mb`, one WARN; hub 0.164.0 `host_swap_short` to the operator; the install page sentence) — but nothing adds swap. The no-swap test on the soak VM (`audits/night-2026-10-21/P2-noswap/`) says what happens without it. | WAITING-ON-OPERATOR **NO-SWAP TEST 2026-10-22 04:08–04:38 UTC (`audits/night-2026-10-21/P2-noswap/`):** the soak box, `swapoff -a` on the host, the same burst → the freeze CAME BACK within 20 s: `/api/health` dead for 289 s, then 172 s and 68 s (24 of 125 probes failed), memory „full" 36 % host / 47 % guest, the VM reading ~40 GB per 25 s from disk; Immich lost 207 of 306 photos and its stream broke; the box recovered only when the Immich stream stopped (04:23). After `swapon -a`: 0 failed probes, „full" ≤ 1.3 %. The agent logged its one WARN at the swapoff (host 938 MB < guests 4096 MB). **Operator answer 2026-10-22: Tester 2's laptop was installed with OUR installer** — so it has host swap (the installer gives ≈ RAM up to 8 GB) and the swap fix helps it once its agent is ≥ 0.162.0. The decision for BYO hosts stays open. | decision: should the agent (or the installer for BYO) create host swap when it is short, or should an 8 GB BYO host without swap be refused at install? | Operator decides; CC builds the pick | operator |
| **R-1214** | Box system & updates | P3 | **On the 8 GB box with the swap fix, the night's second-drive backup still pushes memory „full" pressure above 20 % for about 3 minutes; the box keeps answering.** Night soak 2026-10-21→22 (`audits/night-2026-10-21/P1-soak/03-rec.jsonl`, 10-s samples; fresh box from the public ISO, agent with swap = RAM, 8 apps, the rehearsal uploads for 2 h): at 01:30–01:34 UTC (the 03:30 tier2 job) host 39 % / guest 45 % over 10 s, over 60 s at most 20 % / 25 %; IO „full" 66 %; guest swap 1.35 → 2.1 GB; `/api/health` answered every probe (slowest 4.1 s, largest interval 16 s); 0 OOM kills. Once more in the upload window (Immich loading its models, 31 % / 35 % for one sample). The brief's soak pass line was „full" < 20 %, so the soak missed that one line; the freeze itself (96 % for an hour, R-1192) did not come back. Not diagnosed: which part of the tier2 job (the copy's page cache, the volume dumps) causes it. | OPEN | — | Measure the tier2 job alone on the 8 GB box (per-step pressure); decide whether 20 % over 10 s is the right line, or limit the copy (ionice / cgroup IO weight) | CC |
| **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator |
| **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a new install from the public download replaces the bootstrap and the off-site check; `pilot/TESTER2-handover.md`).** | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator |
| **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC |
| **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement | — | — | CC |
| **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC |
+115 -47
View File
@@ -1,61 +1,129 @@
# Tester 2 — the hand-over list (written 2026-10-22)
# Tester 2 — the hand-over list: a NEW INSTALL (rewritten 2026-10-22, operator ruling 14:11)
Tester 2's laptop (`Tester-2-be8404`, customer `Tester-2`) has been off since 2026-10-04. It was installed with **our
installer** (operator answer, 2026-10-22), so it has host swap and the swap fix will work on it. When it reports
again it runs old software: agent 0.142.0, controller 0.292.0. This page says what happens then, in order.
**The ruling:** Tester 2's friend reinstalls the laptop from the public download (installer 1.30.0 or newer) when he
puts the new drive in, with the drive in before the install. The old box's data is not needed („he barely used the
laptop, no useful data on it"). So nothing is upgraded and nothing is restored. (The earlier upgrade list is in git:
`git log -- documentation/pilot/TESTER2-handover.md`.)
## 1. The operator, when the laptop is on — in this order
## What the hub holds for Tester 2 today (read only, 2026-10-22)
- Host `Tester-2-be8404`, agent 0.142.0, last report **2026-10-04 18:05 UTC** (offline since). A 16 GB machine, guest
9201 with 12 GB.
- **No sealed key package** (`host_escrow`: none; retained packages: none). The hub mailed `offsite_escrow_pending` on
10-11 and 10-18: the box never deposited its off-site key.
- **Off-site was set up**: tier `shared` (the Storage Box), its descriptor provisioned on 2026-10-04 (`offsite_key_installed`
16:16 UTC). Whether the repository holds any snapshot is not known from the hub; without a key package it can never
be opened by anyone.
- **PBS whole-box tier (ep0)**: namespace provisioned (its secret used 2026-10-04 16:28 UTC); the box reported **one
whole-box copy**, 2026-10-04 16:31 UTC; its prune job `prune-Tester-2` was added by hand on 2026-10-19.
- DR recipe (2026-10-04), claim (claimed 2026-10-04 16:30 UTC), mail list (written at the claim, never saved by the
household — `household_saved` false).
- **The customer's mail language is English** (`language = en`). The box's first screen, the link mail and the
set-up mail follow it. If the friend reads Hungarian, change it on the customer page before step 1.
## Host delete or customer RESET? — for a fresh start with nothing to keep
A new install for the same customer needs the old host gone first (D19): the passphrase works only while the customer
has no host, and an install over a kept host skips the claim and off-site re-issues (R-1154). Both paths begin with the
host delete; RESET is a second press after it (RESET refuses while any host exists).
**Host delete alone** (Hosts → `Tester-2-be8404` → Delete; allowed because the box is offline; no key-package tick is
needed — there is none):
- removes the host record, its keys and its WireGuard peer; records the deletion; mails the friend a new link;
- **keeps** the off-site descriptor and whatever the Storage Box repository holds, the ep0 namespace with the
2026-10-04 whole-box copy and its prune job, the DR recipe, the claim;
- the new box gets new off-site credentials and mints a **new** off-site key; with no package to carry, it never asks
for an old recovery code. If the old repository holds snapshots, the new box sets them aside on its first night and
they stay on the Storage Box, unreadable, until the abandonment clean-up (R-1182) removes them. The ep0 copy stays
until its prune job ages it out.
**Customer RESET** (Configs → `Tester-2` → Reset, after the host delete; type the customer id; no key-package tick —
there is none). From the code (`hub/internal/web/customer_reset.go`, `store/customer_reset.go`) and A20:
- **destroys on the Storage Box:** Tester-2's off-site folder — the repository and any set-aside copy — through its own
sub-account login, then the sub-account (the Hetzner leg — only because off-site is on). This purge is hub 0.142.0's
(R-32) and **has never run live**: Tester 2's RESET is its first. If the purge fails, the sub-account is kept and the
leg reads `failed` — press Reset again; CC reads the journal after;
- **destroys on ep0:** the `Tester-2` PBS namespace **with the 2026-10-04 whole-box copy**, its token and its prune job
(`felhom-tenantsync` deprovision);
- **destroys on the hub:** the claim (a fresh setup code at the next install), the provisioned off-site descriptor (the tier stays
„on" but unprovisioned until the operator presses Save on the customer page — section 3), the DR recipe, the one-time secret, log bundles; pending operator actions are
cancelled;
- **keeps:** the customer, its address, domain, language, Cloudflare settings, the off-site tier CHOICE and its mail
list; every event (the audit trail);
- at the end it mails the friend a new link itself.
**Recommendation: host delete, then RESET.** „Fresh start, nothing to keep" is exactly what RESET is for: the new box
starts with an empty Storage Box path and an empty ep0 namespace, instead of carrying a set-aside history nobody can
open and a stale whole-box copy. **RESET deletes data on ep0 and on the Storage Box — that is the operator's press,
never CC's** (both are Tier 2). It is proven live twice (A20, 2026-07-18), each leg idempotent: a failed leg is
re-run from the top, and nothing on the hub is purged until both external legs are ok.
## 1. The operator, before the friend starts
| # | What | Where | Done when |
|---|---|---|---|
| 1 | **Tell CC „Tester 2 is on".** CC signs the agent update to the vouched **0.162.0** (one signed `agent_update` job, the same as for the three test boxes on 2026-10-22). | this chat | CC reports `update committed` within about 15 minutes |
| 2 | **Read the host swap.** No new `host_swap_short` alarm for Tester 2 within 15 minutes of the update means the swap is fine. Our installer gives about the machine's RAM, at most 8 GB. CC also reads the two numbers from the host report. | Hub → Hosts → `Tester-2-be8404`, and the operator mailbox | no alarm; CC confirms the numbers |
| 3 | **The bootstrap step (R-862)** — once, by hand, because this box is older than the config bundle. Connect your WireGuard operator tunnel; `ssh -p 8822 felhom-op@10.77.0.5`; Hub → Hosts → `Tester-2-be8404` → Console access → **Reveal**; `su -` with it; then the three commands in `runbooks/config-bundle.md` („A box from before agent v0.143.0"). | your tunnel | the script ends with `DONE. This box can now take signed config bundles.` Then tell CC „bootstrap done" — CC sends the bundles and reads them back |
| 4 | **The emergency root password (R-1139)** — in the same root shell: `bash felhom-host-install.sh --rotate-recovery-only` (`runbooks/break-glass.md`). It sets a new random console password and stores it in the hub. | the same root shell | it prints `root@pam password set + vaulted to the hub`; tell CC — CC proves the new one logs in and the old one is refused |
| 5 | **One off-site check (R-862).** You believed on 2026-10-07 that Tester 2 never set up the off-site backup. Look: Hub → Customers → `Tester-2` → Off-site. If it is not set up, decide with the tester whether to set it up now. | the hub | you tell CC what you saw |
| 6 | **Tell CC „Tester 2 hand-over done".** | this chat | CC runs section 3 |
| 1 | (If the friend reads Hungarian) set the mail language to Magyar | Configs → `Tester-2` | the page shows Magyar |
| 2 | **Delete the old host.** Tick the box-off confirmation if the page asks; type the host id | Hosts → `Tester-2-be8404` → Delete | the host is gone from the Hosts page. A `409` means the hub still sees the box online — it is not (last report 2026-10-04), so a 409 here is worth telling CC |
| 3 | **RESET the customer.** Read the preview: host count 0, off-site on (shared), no key-package line | Configs → `Tester-2` → Reset → type `Tester-2` | every leg reads `ok` (hetzner, pbs, claim, descriptor, db_purge); a failed leg: press Reset again (it resumes) |
| 4 | Check the Cloudflare tunnel still routes `*.<his domain>` (R-494) — RESET does not touch it | Cloudflare | the tunnel exists; its token is on the customer page |
| 5 | Give the friend the 5-word **owner passphrase** in person or by message (never by the link mail) and the page in section 2. Tell him „go". | — | — |
The controller needs nothing from you: once the agent is 0.162.0, the hub's floor moves the controller to the current
release at the next report (the floor waits for an agent ≥ 0.158.1).
## 2. The friend — one page, Hungarian (send it as it is)
**What happens if you do nothing:** the box keeps agent 0.142.0 and controller 0.292.0 (the floor holds it there),
without the swap fix — a big upload can freeze it for minutes — and with the old emergency password.
## 2. For the tester (Hungarian, one page — send it as it is)
> **Szia! A Felhom dobozod újra bekapcsolható.**
> **Szia! Így telepítsd újra a Felhom dobozodat.** Kb. egy óra. A gépen minden adat törlődik — a régi rendszerre nincs
> szükség.
>
> **Mi változott október 4. óta?**
> 1. A doboz nagy feltöltéskor (például sok fénykép egyszerre) nem akad meg többé percekre.
> 2. Ha egy visszaállítás félbeszakad (hiba vagy áramszünet miatt), az adataid visszakerülnek oda, ahol előtte voltak.
> 3. Áramszünet után a doboz magától rendbe teszi magát, és az alkalmazások újraindulnak.
> 4. Az értesítéseid listáját a doboz megőrzi; a levelek és a képernyők szövegei érthetőbbek lettek.
> 5. A doboz minden címe csak biztonságos (https) kapcsolaton érhető el.
> **Előtte:** építsd be az új meghajtót. Kell egy legalább 4 GB-os USB-kulcs, hálózati kábel a routerhez, a Felhomtól
> kapott **Tulajdonosi jelmondat** (5 szó) és az e-mail fiókod.
>
> **Mit próbálj ki az első órában?**
> - Nyisd meg a doboz kezdőlapját, és nézd meg, hogy minden alkalmazásod fut-e.
> - Tölts fel néhány fényképet vagy dokumentumot abba az alkalmazásba, amit a legtöbbet használsz.
> - Nézd meg a Beállítások → Értesítések oldalt: azok a levelek vannak-e bekapcsolva, amelyeket kérni szeretnél.
> - Nyomd meg a „Mentés most" gombot a Mentések oldalon, és várd meg, amíg kész lesz.
> 1. **Letöltés:** nyisd meg a https://felhom.eu/letoltes oldalt, és nyomd meg a **„Letöltés (1,6 GB)"** gombot.
> 2. **Ellenőrzés:** az oldal kiírja a fájl ellenőrző összegét (SHA-256), és azt is, hogyan számold ki a gépeden.
> A két érték legyen pontosan ugyanaz. Ha nem az, ne telepítsd — szólj.
> 3. **USB-kulcs:** írd ki a fájlt a kulcsra balenaEtcherrel (vagy Windows-on Rufusszal, „DD" módban).
> 4. **Telepítés:** indítsd a gépet a kulcsról. A képernyők angolul vannak — ez rendben van.
> - Felhom logó, két sor: **„Felhom telepítés"**. Utána: **I agree**.
> - **Target harddisk — ez a legfontosabb lépés.** A telepítő felsorolja a lemezeket méretükkel. Válaszd azt, amelyre
> a rendszer kerüljön (a gép belső rendszerlemeze). **Az új meghajtót ne válaszd** — azt a doboz később, a
> Tárhely oldalon kapja meg. Ha nem tudod, melyik melyik, állj meg és szólj.
> - Country / Timezone / Keyboard: hagyd így. Root password: adj meg bármilyen legalább 8 karakteres jelszót kétszer
> (a Felhom később lecseréli). Administrator email: a saját címed. Hostname: `felhom.<a-te-domained>`.
> - Summary: a jobbra nyíllal lépj az **Install** gombra, és nyomd meg. A gép a végén magától újraindul — a másolás
> alatt húzd ki az USB-kulcsot.
> 5. **Párosító kód:** a doboz képernyőjén 2–4 perc múlva megjelenik: „Párosító kód: XXX-XXX". Írd fel.
> 6. **Összekötés:** nyisd meg a **„Kösd össze a Felhom dobozodat"** e-mailt (ha kettő jött, a frissebbet), kattints a
> hivatkozásra, és add meg a Párosító kódot és a Tulajdonosi jelmondatot.
> 7. **Vezérlőpult jelszó:** kb. 3 perc múlva jön a **beállító kódos** e-mail. Kattints a benne lévő címre, add meg a
> Beállító kódot (3 szó), és válassz legalább 12 karakteres jelszót. Ez a vezérlőpult jelszava.
> 8. **Helyreállítási kód:** a vezérlőpult kéri, hogy hozd létre. Írd le **papírra**, és tartsd a gépen kívül. Amíg
> nincs meg, a távoli mentés nem indul el.
> 9. **Első két alkalmazás:** Alkalmazások → például **BookStack** és **PrivateBin** → Telepítés. Ha valamelyik
> jelszót kér, nyomd meg a „Generálás" gombot.
>
> **Ha valami nem működik:** írd le röviden, mit csináltál, mit vártál, és mi történt helyette. Ha tudsz, küldj
> képernyőképet is. Küldd el a Felhom üzemeltetőjének azon a címen, ahol eddig is levelezettetek.
> **Ha valami nem működik:** írd le röviden, mit csináltál, mit vártál, és mi történt helyette, és ha tudsz, küldj
> képernyőképet. Küldd el a Felhom üzemeltetőjének azon a címen, ahol eddig is levelezettetek.
>
> **Amit ne csinálj:**
> - Visszaállítás közben ne húzd ki a dobozt a konnektorból, és ne kapcsold ki.
> - A helyreállítási kódodat tartsd papíron, a dobozon kívül. A dobozon vagy a laptopon ne tárold.
> **Amit ne csinálj:** visszaállítás közben ne húzd ki a gépet a konnektorból, és ne kapcsold ki. A helyreállítási
> kódot ne a gépen vagy a laptopon tárold.
## 3. What CC checks after the hand-over
## 3. The operator, after the install
1. **A read-only box walk of Tester 2's box:** `scripts/https_only_check.py` with Tester 2 in it (R-1104); the
containers, unhealthy and restarting counts, and the controller's error lines (the 2026-10-22 release shape); the
agent's capability self-check (`ok=N total=N`) and the config bundle read back (R-862).
2. **The next night chain:** the database dump, the second-drive copy (if the tester has added a drive), the off-site
copy (if set up), the integrity check — each ran, read from the box's log the next morning.
3. **The hub alarms:** for 24 hours after it is back, every alarm Tester 2 raises, and whether the household's mail
list is the one the household saved (the hub keeps it; the box adopts it — R-1199).
4. **Tester 2's mail list:** its hub row was written at the claim and never saved by the household
(`household_saved` = false), so the box will push its default list once and keep it. If the tester had chosen a
different list on 2026-10-04, ask them to save it again on the box.
- **Off-site, if the friend wants it:** Configs → `Tester-2` → Edit → **Save** (off-site stays ticked). After the RESET
the tier is on but unprovisioned — the page says so — and the hub provisions a fresh Storage Box account only on
Save (about 25 s; do not click twice). Then the box's off-site starts once the friend has made the recovery code.
If he does not want it: untick off-site, Save. Nothing else.
Rows: R-862, R-1139, R-1104, R-1207, R-1199.
## 4. CC, after (tell CC „Tester 2 installed")
1. **A read-only box walk:** `scripts/https_only_check.py` with Tester 2 in it (R-1104); containers, unhealthy and
restarting counts; the controller's error lines; the agent's capability self-check and the config bundle (a fresh
install has both).
2. **The host swap (D28):** from the host report — our installer gives this 16 GB machine 8 GB; the new rule says
short only below 90 % of min(the guest's swap, 8 GB). No `host_swap_short` expected.
3. **The first night chain, the next morning:** database dump, second-drive copy (if the new drive was set up as the
backup target), off-site copy (if switched on), the whole-box copy to ep0 (CC confirms the new host's PBS namespace
was made again after the RESET), the integrity check — each ran.
4. **The hub alarms for 24 hours** after the install, read from the hub.
5. **The mail list:** the hub's row was never saved by a household, so the new box pushes its default list once and
keeps it; the hub and the box page must agree.
Rows: R-862 and R-1139 close when Tester 2 is reinstalled; R-1104 is checked by step 4.1.