From 82ce9d72d6b542aa5be67b6ea945362ae27107de Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 22 Oct 2026 14:19:26 +0200 Subject: [PATCH] =?UTF-8?q?Tester=202:=20a=20new=20install,=20not=20an=20u?= =?UTF-8?q?pgrade=20(operator=20ruling=202026-10-22=2014:11)=20=E2=80=94?= =?UTF-8?q?=20hand-over=20list=20rewritten=20(host=20delete=20then=20RESET?= =?UTF-8?q?,=20what=20each=20removes,=20the=20friend's=20page=20in=20Hunga?= =?UTF-8?q?rian);=20R-862,=20R-1139=20close=20at=20the=20reinstall;=20R-32?= =?UTF-8?q?'s=20first=20live=20purge=20noted;=20STATUS,=20CONTEXT?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CONTEXT.md | 8 ++ STATUS.md | 5 +- documentation/backlog/OPEN-ITEMS.md | 6 +- documentation/pilot/TESTER2-handover.md | 162 +++++++++++++++++------- 4 files changed, 129 insertions(+), 52 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index 1d3372fb..9d7ab018 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -16,6 +16,14 @@ > and holds nothing of its own; this file does hold its own content, namely the standing rulings below. +> **2026-10-22 14:11 — operator ruling: Tester 2 is a NEW INSTALL, not an upgrade.** The friend reinstalls the laptop +> from the public download (installer 1.30.0 or newer) with the new drive in before the install; the old box's data is +> not needed („he barely used the laptop, no useful data on it"). Read on the hub the same day (read only): no sealed +> key package; off-site provisioned (shared) but its key never deposited; one ep0 whole-box copy (2026-10-04); the mail +> language is English. CC's pick for the operator: host delete, then customer RESET (it empties the Storage Box folder +> and the ep0 namespace — the operator's press). R-862 and R-1139 close at the reinstall; Tester 2's RESET is R-32's +> first live purge. List: `documentation/pilot/TESTER2-handover.md`. + > **2026-10-22 day — operator rulings (11:42) and what was built.** Rulings: **D28 = A** (loosen the host-swap check); > agent 0.162.0 **vouched** (read back: `artifact_agent_version` 0.162.0 at 07:57 UTC); Tester 2's laptop was installed > with **our installer**; Tester 2 is off, no date. Built: **R-1199 reopened** — controller 0.338.0's start-up default diff --git a/STATUS.md b/STATUS.md index 9c2a38e6..27d1dbf0 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,6 +1,6 @@ # STATUS — what works, what's broken, what's next -**Ready for Tester 2? Yes — when the laptop is on: the operator sends agent 0.162.0 first (hand-over list: `documentation/pilot/TESTER2-handover.md`).** +**Ready for Tester 2? Yes — a new install: you do the hub step first (hand-over list: `documentation/pilot/TESTER2-handover.md`).** **Updated 2026-10-22 (day).** A rebuilt box no longer overwrites the family's saved mail list (hub 0.165.0, controller 0.339.0); demo-felhom's own list, replaced this morning, is back. The swap warning now fires only when a machine has clearly too little swap (your D28 = A). @@ -52,7 +52,8 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te 7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12. **Today, 2026-10-22.** -1. When Tester 2's laptop is on: follow the hand-over list, step 1 first (tell CC; CC sends agent 0.162.0). If not: the box stays on old software without the swap fix. +1. Before Tester 2's friend reinstalls: delete the old host, then RESET the customer (hand-over list, section 1). The RESET deletes Tester 2's old copies on ep0 and the Storage Box. If not: the new box carries an old, unreadable off-site history and a stale whole-box copy. +2. Tester 2's mail language is English on the hub. If the friend reads Hungarian, change it first. If not: his mails and first screen are in English. **The rest** - Pull the failed disk Z6A07P2G out of DooPlex (R-934). If not: its alarms return on 11 November. Since 2026-10-12. - Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). Since 2026-10-09. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index bff5e599..58fdca6a 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -177,7 +177,7 @@ stopping line that lies. | **R-816** | Backup & restore | P4 | **No off-site failure class has ever been seen live.** F-DIAG (controller v0.182.0, 2026-07-28) split off-site failures into six causes — quota, orphaned, no_repo, no_units, transport, unknown — each with its own Hungarian message. None of the six has been exercised by a real failure on a box; the recovery inventory records it only as a known limit (`documentation/architecture/_recovery-inventory-2026-07-28.md:955`). Filed 2026-10-03 from the F-DIAG row's residue when that row moved to `CLOSED-ITEMS.md`. | **READY — filed 2026-10-03 (triage); owner: CC.** Exercise each class once on a scratch guest (a full quota, a missing repository, a blocked transport) and read the message the household sees. | — | — | CC | | **R-832** | Backup & restore | P4 | **ep0's copy in a place outside both Hetzner and the operator's home (roadmap).** Today DooPlex (the operator's home) holds it (decision 71). A Hetzner Storage Box would share a provider with ep0 and with every household's file backups, and cannot run PBS, so the copy could not be verified or restored from directly. | **DEFERRED — later, if the product grows** | — | — | operator | | **R-878** | Backup & restore | P4 | **A catch-up (R-871) runs the database-dump leg in the DAY, and that leg stops an app with a volume for its copy — the household may notice the stop, and a large volume makes it longer.** MEASURED 2026-10-05 on demo-felhom: the catch-up at 08:25:02 stopped opengist, copied 182.5 KB, started it again — about 1 s, then a few seconds of `health: starting`; the night does exactly the same, unseen. Nothing measured for a large volume. Fix direction (if it matters): skip the volume copy of a running app in a DAYTIME catch-up and leave it to the next night, or warn. `audits/catchup-2026-10-05/partA/live-demo-felhom.txt` | **READY — owner: CC** **2026-10-05 (burn-down night): NEEDS A LIVE MEASUREMENT** — a timed catch-up on 9202 for an app with a multi-GB volume, then the skip-or-warn choice. | — | measure a large volume first | CC | -| **R-32** | Backup & restore | P4 | **[P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible.** The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's `"hetzner":"ok"` leg destroys the sub-account, but **a Hetzner sub-account is an access-control object, not a data object** — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **2026-10-06 night: design written** (`audits/night-burndown-2026-10-06/design-R-32.md`) — pick A: purge the repo + set-aside copies through the sub-account's own password login BEFORE deleting it (no main-account credential); the „repo data destroyed" comment at `hub/internal/offsite/offsite.go:276-280` is false for the shared tier; part (3) waits on a read-only measurement (`du` on port 23). Operator: agree to route A. **2026-10-07 07:58: `09` §3 decision 167 — option A yes, and delete the old RESET leftovers once.** **2026-10-07 (Part D step 2, read only):** the hub shows bx11 holding 3.3 GB of data while the four live customers account for ~0.6 GB → **~2.7 GB belongs to no live customer**; no live customer shows a set-aside folder, so the leftovers are in homes whose sub-account is gone, reachable only by the pool box's MAIN account. **The one-time clean-up is STOPPED:** no main-account login for bx11 is held on DooPlex (the API token in the credentials file sees no box). Needs: the main-account login, or the operator lists and deletes in the Hetzner console (`audits/day-2026-10-07/D/D2-listing-readonly.txt`). **2026-10-07: option A DELIVERED (hub 0.142.0):** RESET purges the repo and every `.orphaned-*` through the sub-account's own login before deleting it, and keeps the sub-account if the purge fails (red-proved, `audits/day-2026-10-07/D/`). Not yet seen live (no scratch RESET today). **The one-time clean-up of the ~2.7 GB of leftovers is STOPPED:** it needs the pool box's main-account login (see above). **2026-10-07 14:43 — `09` §3 decision 173: leave the ~2.7 GB of old leftovers for now; re-ranked P2 → P4.** Option A itself is delivered (hub 0.142.0). What stays open is ONLY the leftovers: they were computed (box total minus live customers), never listed, and deleting them needs the pool box's MAIN login, which reaches every customer's data. **Blocker: a safe main-account path.** | — | **Ruling from the run (three parts, deliberately separate):** (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable **BY DESIGN** → RESET gains a **main-account purge of the customer base dir** (the existing operator ack already covers it); (2) the **move-aside guard STAYS** for reinstall-*without*-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator **Restic tab shows per-customer directory bytes vs attributed snapshot bytes**, so dead data cannot hide. Measured on the pool box that night: **49 M attributed** (2 snapshots, 48.717 MiB) against **1.4 G + 3.0 M unattributed** across TWO `.orphaned-*` dirs. Evidence `restic-and-pool.txt` | CC | +| **R-32** | Backup & restore | P4 | **[P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible.** The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's `"hetzner":"ok"` leg destroys the sub-account, but **a Hetzner sub-account is an access-control object, not a data object** — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **2026-10-06 night: design written** (`audits/night-burndown-2026-10-06/design-R-32.md`) — pick A: purge the repo + set-aside copies through the sub-account's own password login BEFORE deleting it (no main-account credential); the „repo data destroyed" comment at `hub/internal/offsite/offsite.go:276-280` is false for the shared tier; part (3) waits on a read-only measurement (`du` on port 23). Operator: agree to route A. **2026-10-07 07:58: `09` §3 decision 167 — option A yes, and delete the old RESET leftovers once.** **2026-10-07 (Part D step 2, read only):** the hub shows bx11 holding 3.3 GB of data while the four live customers account for ~0.6 GB → **~2.7 GB belongs to no live customer**; no live customer shows a set-aside folder, so the leftovers are in homes whose sub-account is gone, reachable only by the pool box's MAIN account. **The one-time clean-up is STOPPED:** no main-account login for bx11 is held on DooPlex (the API token in the credentials file sees no box). Needs: the main-account login, or the operator lists and deletes in the Hetzner console (`audits/day-2026-10-07/D/D2-listing-readonly.txt`). **2026-10-07: option A DELIVERED (hub 0.142.0):** RESET purges the repo and every `.orphaned-*` through the sub-account's own login before deleting it, and keeps the sub-account if the purge fails (red-proved, `audits/day-2026-10-07/D/`). Not yet seen live (no scratch RESET today). **The one-time clean-up of the ~2.7 GB of leftovers is STOPPED:** it needs the pool box's main-account login (see above). **2026-10-07 14:43 — `09` §3 decision 173: leave the ~2.7 GB of old leftovers for now; re-ranked P2 → P4.** Option A itself is delivered (hub 0.142.0). What stays open is ONLY the leftovers: they were computed (box total minus live customers), never listed, and deleting them needs the pool box's MAIN login, which reaches every customer's data. **Blocker: a safe main-account path.** **2026-10-22: Tester 2's planned RESET (operator ruling 2026-10-22 14:11, `pilot/TESTER2-handover.md`) will be the purge's first live run — CC reads its leg journal after.** | — | **Ruling from the run (three parts, deliberately separate):** (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable **BY DESIGN** → RESET gains a **main-account purge of the customer base dir** (the existing operator ack already covers it); (2) the **move-aside guard STAYS** for reinstall-*without*-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator **Restic tab shows per-customer directory bytes vs attributed snapshot bytes**, so dead data cannot hide. Measured on the pool box that night: **49 M attributed** (2 snapshots, 48.717 MiB) against **1.4 G + 3.0 M unattributed** across TWO `.orphaned-*` dirs. Evidence `restic-and-pool.txt` | CC | | **R-1087** | Backup & restore | P4 | **Whole-box copies left on the OLD local target are never pruned after the target moves.** MEASURED 2026-10-15 (storage-identity build B space check): the N100's host root (`local`) holds three copies from 2026-07-28 (5.5–5.7 GB each, ≈ 16.8 GB); its agent's `local_backup_target` is now `felhom-backup` (the USB disk, retention 3), and retention prunes only the current target. On a small-root box such leftovers would starve the local copies (R-548). Evidence: `audits/storage-identity-buildB-2026-10-15/space/felhom-pve-9201.txt`. | OPEN | — | Decide whether the agent prunes its own `vzdump-lxc--*` files on a former target (after a newer copy exists on the new one), or the target move asks; until then, the N100's three July files can be removed by hand with the operator's word. | CC | | **R-1171** | Backup & restore | P3 | **demo-hp's LOCAL whole-guest copy is never restore-tested: the box keeps one copy (keep-daily 1) and the restore test wants a copy 24 h old, so the only copy turns 24 h old exactly when the next night replaces it.** Read 2026-10-20 on demo-hp (agent journal): every 6-hourly evaluation since 2026-10-18 says `local: no settled archive yet — nothing to prove`; the local storage holds one 9201 archive; the last local proof was 2026-10-14 03:42 (`settle=24h`, `keep_last=1`, `prune=keep-daily=1`). The hub's `restore_test_stale` (168 h) will alarm from ~2026-10-21 03:42. A real household with retention 1 would be the same. | OPEN — filed 2026-10-20 | demo-hp `journalctl -u felhom-agent` („restore-test evaluated"), the hub's host report `restore_tests` | Agent: a tier that keeps ONE copy proves its newest archive once it is older than the settle time measured from the BACKUP's end, not 24 h — or a settle below the cadence (e.g. 6 h) for a 1-copy tier; a test; then read the next evaluation | CC | | **R-1172** | Backup & restore | P4 | **After option A (R-893): a failed off-site restore that was PUT BACK is logged at ERROR, and an empty `backups/restore-undo/` folder stays on disk.** Measured on 9202 2026-10-20 (rc.1): the web handler logs `[ERROR] [web] off-box reconstitute rallly (async): … Nem változott semmi…` — good news at ERROR; the put-back removes the per-app folder but not the empty `restore-undo/` parent. Small; 0.336.0 was already pushed (a version is pushed once). | OPEN — filed 2026-10-20 | `audits/night2-2026-10-20/P2/live/11-controller-log-window.txt`, `12-after-failed-restore.txt` | Next controller release: the put-back outcome logged at WARN (its own sentence is the household's), the empty parent removed; a test | CC | @@ -216,7 +216,7 @@ stopping line that lies. | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| -| **R-1139** | Security & access | P2 | **Tester 2's break-glass root password was not rotated with the others (R-954, operator decision D1 = A, 2026-10-17): the box has been offline since 2026-10-04.** demo-hp, Tester 1 and demo-felhom were rotated and proven on 2026-10-17 (`audits/agent-allowlist-rotation-2026-10-17/P3/`). | OPEN — filed 2026-10-17 | Tester 2 reporting again | When it reports: the installer's step 4b alone (`P3/rotate-step4b.sh` shape), then the vaulted value logs in (PVE API 200) and the old one is refused | CC | +| **R-1139** | Security & access | P2 | **Tester 2's break-glass root password was not rotated with the others (R-954, operator decision D1 = A, 2026-10-17): the box has been offline since 2026-10-04.** demo-hp, Tester 1 and demo-felhom were rotated and proven on 2026-10-17 (`audits/agent-allowlist-rotation-2026-10-17/P3/`). | OPEN — filed 2026-10-17 **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a fresh install sets and vaults a new console password).** | Tester 2 reporting again | When it reports: the installer's step 4b alone (`P3/rotate-step4b.sh` shape), then the vaulted value logs in (PVE API 200) and the old one is refused | CC | | **R-985** | Security & access | P3 | **Our zero-knowledge wording is stronger than the product: on a live box the operator can reach the off-site backup key (through the vaulted root credential and the operator tunnel) without the customer's recovery code.** Design-shaped; decision for the operator in `STATUS.md`. Fix options: change the wording, or remove operator root reach to the key. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | WAITING-ON-OPERATOR — design-proposal | operator decision | Decide the wording or the change | operator | | **R-1042** | Security & access | P3 | **The boxes' Cloudflare keys cannot change zone settings, so Cloudflare's own „Always Use HTTPS” cannot be switched on.** Measured 2026-10-14 on all three zones (`audits/https-only-2026-10-14/``1A-1-zones-and-keys.txt`): each `cf_api_token` is valid and sees only its own zone, but GET/PATCH `settings/always_use_https`, `ssl`, `security_header` answer 403; the redirect and transform rule phases too. The https-only fix therefore lives on the box (decision 246). A second layer at Cloudflare needs „Zone Settings: Edit” (and, for headers, „Transform Rules”) added to each customer's key — the hub's zone-reach check stays. | **WAITING-ON-OPERATOR** — filed 2026-10-14 | a key permission only the operator can grant | Optional: widen each customer key, then switch „Always Use HTTPS” on per zone | operator | | **R-1104** | Security & access | P3 | **Check https-only on Tester 2 when it reports again.** R-1033 (plain http on every public name) is closed on the three boxes that report (35 names → 301, 2026-10-16). Tester 2 (`sajatfelhom.hu`) has been off since 2026-10-04 on controller 0.292.0; its addresses answer Cloudflare 530 (tunnel down), so nothing is served in plain http while it is off. When it comes back it takes the floor (≥ 0.312.0 carries the redirect). | OPEN — filed 2026-10-16 | Tester 2 reporting again | Run `scripts/https_only_check.py` with Tester 2 in it; close | CC | @@ -247,7 +247,7 @@ stopping line that lies. | **R-972** | Box system & updates | P2 | **The shipped controller and hub programs are built with Go 1.24.13 and golang.org/x/crypto v0.31.0, and the agent with Go 1.26.0, so dozens of fixed Go security bugs (including html/template escaping fixes) are still in what runs on boxes and the hub.** Go 1.24 has had no security fixes since Go 1.26 was released (Go supports the two newest major releases: https://go.dev/doc/devel/release#policy); the scanners name fixes in Go 1.26.9 and 1.27.2. Also: the controller image's Debian 12 base has 12 fixable critical packages, and the Samba image has 48 fixable high ones. Fix: build all three on a supported, patched Go; update x/crypto; rebuild the base images. Scanner output: `audits/security-review-2026-10-12/scanners/`. | **NARROWED** 2026-10-16 (burndown 2): **controller v0.322.0 released** on Go 1.27.2 + x/crypto v0.58.0, runtime debian:12.13-slim + `apt-get upgrade` (Debian 13 deliberately not: restic 0.18, R-1105) — scanner: critical 16 → 4, fixable criticals and highs in the OS 0; govulncheck 62 → 1 (module-level, `openpgp` not linked). **Hub v0.156.0 deployed** 2026-10-16 19:26 on the operator's yes (Go 1.27.2, alpine 3.24.2: 0 critical, 0 high); Synced/Healthy, reports from all three boxes after the start, the floor reads 0.322.0. **Agent: Go 1.27.2 committed (`37b99d7`), tested, NOT released** — the next vouched agent release carries it. Open: the Samba image (48 fixable high, not rebuilt); docker-compose plugin's own Go (4 high, upstream). Evidence `audits/burndown2-2026-10-16/P4/` **2026-10-16 (evening): the Samba part is DONE in controller v0.323.0** — felhom-samba 1.2.0 on Alpine 3.24.2 (0 critical / 0 high, was 48 fixable high), proven on demo-hp with the same household password (smbclient, nmbd, mDNS, WS-Discovery); on the three test boxes now, on every box with the global floor (R-1105's night check). The controller's Debian 13 OS part: 1 critical / 70 high, none fixable. Left: docker-compose plugin's own Go (4 high, upstream); the agent release; restic's binary is R-1107. **2026-10-17 (evening): agent v0.159.1 (Go 1.27.2) RELEASED and running on demo-hp, Tester 1 and demo-felhom** (signed jobs on the operator's yes); waiting for the operator's vouch — then only the upstream compose plugin is left (close then, recorded). | the next vouched agent release | Close when the global floor carries v0.323.0 and the next vouched agent release carries Go 1.27.2 (compose plugin: upstream, recorded) | CC | | **R-1207** | Box system & updates | P2 | **A host with no (or too little) swap gets nothing from the guest swap fix (R-1192), and the agent never adds host swap.** The fix lets a guest swap as much as its RAM; that swap lives on the HOST. Measured 2026-10-21 night: our installer gives an 8 GB / 256 GB machine 7.75 GB of host swap (LVM `pve/swap`), so the installer's boxes are covered; a BYO host (or one with swap turned off) may have none. Now reported and said (agent `0da792e`: `host_swap_mb` / `guest_swap_mb`, one WARN; hub 0.164.0 `host_swap_short` to the operator; the install page sentence) — but nothing adds swap. The no-swap test on the soak VM (`audits/night-2026-10-21/P2-noswap/`) says what happens without it. | WAITING-ON-OPERATOR **NO-SWAP TEST 2026-10-22 04:08–04:38 UTC (`audits/night-2026-10-21/P2-noswap/`):** the soak box, `swapoff -a` on the host, the same burst → the freeze CAME BACK within 20 s: `/api/health` dead for 289 s, then 172 s and 68 s (24 of 125 probes failed), memory „full" 36 % host / 47 % guest, the VM reading ~40 GB per 25 s from disk; Immich lost 207 of 306 photos and its stream broke; the box recovered only when the Immich stream stopped (04:23). After `swapon -a`: 0 failed probes, „full" ≤ 1.3 %. The agent logged its one WARN at the swapoff (host 938 MB < guests 4096 MB). **Operator answer 2026-10-22: Tester 2's laptop was installed with OUR installer** — so it has host swap (the installer gives ≈ RAM up to 8 GB) and the swap fix helps it once its agent is ≥ 0.162.0. The decision for BYO hosts stays open. | decision: should the agent (or the installer for BYO) create host swap when it is short, or should an 8 GB BYO host without swap be refused at install? | Operator decides; CC builds the pick | operator | | **R-1214** | Box system & updates | P3 | **On the 8 GB box with the swap fix, the night's second-drive backup still pushes memory „full" pressure above 20 % for about 3 minutes; the box keeps answering.** Night soak 2026-10-21→22 (`audits/night-2026-10-21/P1-soak/03-rec.jsonl`, 10-s samples; fresh box from the public ISO, agent with swap = RAM, 8 apps, the rehearsal uploads for 2 h): at 01:30–01:34 UTC (the 03:30 tier2 job) host 39 % / guest 45 % over 10 s, over 60 s at most 20 % / 25 %; IO „full" 66 %; guest swap 1.35 → 2.1 GB; `/api/health` answered every probe (slowest 4.1 s, largest interval 16 s); 0 OOM kills. Once more in the upload window (Immich loading its models, 31 % / 35 % for one sample). The brief's soak pass line was „full" < 20 %, so the soak missed that one line; the freeze itself (96 % for an hour, R-1192) did not come back. Not diagnosed: which part of the tier2 job (the copy's page cache, the volume dumps) causes it. | OPEN | — | Measure the tier2 job alone on the 8 GB box (per-step pressure); decide whether 20 % over 10 s is the right line, or limit the copy (ionice / cgroup IO weight) | CC | -| **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator | +| **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a new install from the public download replaces the bootstrap and the off-site check; `pilot/TESTER2-handover.md`).** | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator | | **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC | | **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement | — | — | CC | | **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC | diff --git a/documentation/pilot/TESTER2-handover.md b/documentation/pilot/TESTER2-handover.md index 70b9c93f..f618270f 100644 --- a/documentation/pilot/TESTER2-handover.md +++ b/documentation/pilot/TESTER2-handover.md @@ -1,61 +1,129 @@ -# Tester 2 — the hand-over list (written 2026-10-22) +# Tester 2 — the hand-over list: a NEW INSTALL (rewritten 2026-10-22, operator ruling 14:11) -Tester 2's laptop (`Tester-2-be8404`, customer `Tester-2`) has been off since 2026-10-04. It was installed with **our -installer** (operator answer, 2026-10-22), so it has host swap and the swap fix will work on it. When it reports -again it runs old software: agent 0.142.0, controller 0.292.0. This page says what happens then, in order. +**The ruling:** Tester 2's friend reinstalls the laptop from the public download (installer 1.30.0 or newer) when he +puts the new drive in, with the drive in before the install. The old box's data is not needed („he barely used the +laptop, no useful data on it"). So nothing is upgraded and nothing is restored. (The earlier upgrade list is in git: +`git log -- documentation/pilot/TESTER2-handover.md`.) -## 1. The operator, when the laptop is on — in this order +## What the hub holds for Tester 2 today (read only, 2026-10-22) + +- Host `Tester-2-be8404`, agent 0.142.0, last report **2026-10-04 18:05 UTC** (offline since). A 16 GB machine, guest + 9201 with 12 GB. +- **No sealed key package** (`host_escrow`: none; retained packages: none). The hub mailed `offsite_escrow_pending` on + 10-11 and 10-18: the box never deposited its off-site key. +- **Off-site was set up**: tier `shared` (the Storage Box), its descriptor provisioned on 2026-10-04 (`offsite_key_installed` + 16:16 UTC). Whether the repository holds any snapshot is not known from the hub; without a key package it can never + be opened by anyone. +- **PBS whole-box tier (ep0)**: namespace provisioned (its secret used 2026-10-04 16:28 UTC); the box reported **one + whole-box copy**, 2026-10-04 16:31 UTC; its prune job `prune-Tester-2` was added by hand on 2026-10-19. +- DR recipe (2026-10-04), claim (claimed 2026-10-04 16:30 UTC), mail list (written at the claim, never saved by the + household — `household_saved` false). +- **The customer's mail language is English** (`language = en`). The box's first screen, the link mail and the + set-up mail follow it. If the friend reads Hungarian, change it on the customer page before step 1. + +## Host delete or customer RESET? — for a fresh start with nothing to keep + +A new install for the same customer needs the old host gone first (D19): the passphrase works only while the customer +has no host, and an install over a kept host skips the claim and off-site re-issues (R-1154). Both paths begin with the +host delete; RESET is a second press after it (RESET refuses while any host exists). + +**Host delete alone** (Hosts → `Tester-2-be8404` → Delete; allowed because the box is offline; no key-package tick is +needed — there is none): +- removes the host record, its keys and its WireGuard peer; records the deletion; mails the friend a new link; +- **keeps** the off-site descriptor and whatever the Storage Box repository holds, the ep0 namespace with the + 2026-10-04 whole-box copy and its prune job, the DR recipe, the claim; +- the new box gets new off-site credentials and mints a **new** off-site key; with no package to carry, it never asks + for an old recovery code. If the old repository holds snapshots, the new box sets them aside on its first night and + they stay on the Storage Box, unreadable, until the abandonment clean-up (R-1182) removes them. The ep0 copy stays + until its prune job ages it out. + +**Customer RESET** (Configs → `Tester-2` → Reset, after the host delete; type the customer id; no key-package tick — +there is none). From the code (`hub/internal/web/customer_reset.go`, `store/customer_reset.go`) and A20: +- **destroys on the Storage Box:** Tester-2's off-site folder — the repository and any set-aside copy — through its own + sub-account login, then the sub-account (the Hetzner leg — only because off-site is on). This purge is hub 0.142.0's + (R-32) and **has never run live**: Tester 2's RESET is its first. If the purge fails, the sub-account is kept and the + leg reads `failed` — press Reset again; CC reads the journal after; +- **destroys on ep0:** the `Tester-2` PBS namespace **with the 2026-10-04 whole-box copy**, its token and its prune job + (`felhom-tenantsync` deprovision); +- **destroys on the hub:** the claim (a fresh setup code at the next install), the provisioned off-site descriptor (the tier stays + „on" but unprovisioned until the operator presses Save on the customer page — section 3), the DR recipe, the one-time secret, log bundles; pending operator actions are + cancelled; +- **keeps:** the customer, its address, domain, language, Cloudflare settings, the off-site tier CHOICE and its mail + list; every event (the audit trail); +- at the end it mails the friend a new link itself. + +**Recommendation: host delete, then RESET.** „Fresh start, nothing to keep" is exactly what RESET is for: the new box +starts with an empty Storage Box path and an empty ep0 namespace, instead of carrying a set-aside history nobody can +open and a stale whole-box copy. **RESET deletes data on ep0 and on the Storage Box — that is the operator's press, +never CC's** (both are Tier 2). It is proven live twice (A20, 2026-07-18), each leg idempotent: a failed leg is +re-run from the top, and nothing on the hub is purged until both external legs are ok. + +## 1. The operator, before the friend starts | # | What | Where | Done when | |---|---|---|---| -| 1 | **Tell CC „Tester 2 is on".** CC signs the agent update to the vouched **0.162.0** (one signed `agent_update` job, the same as for the three test boxes on 2026-10-22). | this chat | CC reports `update committed` within about 15 minutes | -| 2 | **Read the host swap.** No new `host_swap_short` alarm for Tester 2 within 15 minutes of the update means the swap is fine. Our installer gives about the machine's RAM, at most 8 GB. CC also reads the two numbers from the host report. | Hub → Hosts → `Tester-2-be8404`, and the operator mailbox | no alarm; CC confirms the numbers | -| 3 | **The bootstrap step (R-862)** — once, by hand, because this box is older than the config bundle. Connect your WireGuard operator tunnel; `ssh -p 8822 felhom-op@10.77.0.5`; Hub → Hosts → `Tester-2-be8404` → Console access → **Reveal**; `su -` with it; then the three commands in `runbooks/config-bundle.md` („A box from before agent v0.143.0"). | your tunnel | the script ends with `DONE. This box can now take signed config bundles.` Then tell CC „bootstrap done" — CC sends the bundles and reads them back | -| 4 | **The emergency root password (R-1139)** — in the same root shell: `bash felhom-host-install.sh --rotate-recovery-only` (`runbooks/break-glass.md`). It sets a new random console password and stores it in the hub. | the same root shell | it prints `root@pam password set + vaulted to the hub`; tell CC — CC proves the new one logs in and the old one is refused | -| 5 | **One off-site check (R-862).** You believed on 2026-10-07 that Tester 2 never set up the off-site backup. Look: Hub → Customers → `Tester-2` → Off-site. If it is not set up, decide with the tester whether to set it up now. | the hub | you tell CC what you saw | -| 6 | **Tell CC „Tester 2 hand-over done".** | this chat | CC runs section 3 | +| 1 | (If the friend reads Hungarian) set the mail language to Magyar | Configs → `Tester-2` | the page shows Magyar | +| 2 | **Delete the old host.** Tick the box-off confirmation if the page asks; type the host id | Hosts → `Tester-2-be8404` → Delete | the host is gone from the Hosts page. A `409` means the hub still sees the box online — it is not (last report 2026-10-04), so a 409 here is worth telling CC | +| 3 | **RESET the customer.** Read the preview: host count 0, off-site on (shared), no key-package line | Configs → `Tester-2` → Reset → type `Tester-2` | every leg reads `ok` (hetzner, pbs, claim, descriptor, db_purge); a failed leg: press Reset again (it resumes) | +| 4 | Check the Cloudflare tunnel still routes `*.` (R-494) — RESET does not touch it | Cloudflare | the tunnel exists; its token is on the customer page | +| 5 | Give the friend the 5-word **owner passphrase** in person or by message (never by the link mail) and the page in section 2. Tell him „go". | — | — | -The controller needs nothing from you: once the agent is 0.162.0, the hub's floor moves the controller to the current -release at the next report (the floor waits for an agent ≥ 0.158.1). +## 2. The friend — one page, Hungarian (send it as it is) -**What happens if you do nothing:** the box keeps agent 0.142.0 and controller 0.292.0 (the floor holds it there), -without the swap fix — a big upload can freeze it for minutes — and with the old emergency password. - -## 2. For the tester (Hungarian, one page — send it as it is) - -> **Szia! A Felhom dobozod újra bekapcsolható.** +> **Szia! Így telepítsd újra a Felhom dobozodat.** Kb. egy óra. A gépen minden adat törlődik — a régi rendszerre nincs +> szükség. > -> **Mi változott október 4. óta?** -> 1. A doboz nagy feltöltéskor (például sok fénykép egyszerre) nem akad meg többé percekre. -> 2. Ha egy visszaállítás félbeszakad (hiba vagy áramszünet miatt), az adataid visszakerülnek oda, ahol előtte voltak. -> 3. Áramszünet után a doboz magától rendbe teszi magát, és az alkalmazások újraindulnak. -> 4. Az értesítéseid listáját a doboz megőrzi; a levelek és a képernyők szövegei érthetőbbek lettek. -> 5. A doboz minden címe csak biztonságos (https) kapcsolaton érhető el. +> **Előtte:** építsd be az új meghajtót. Kell egy legalább 4 GB-os USB-kulcs, hálózati kábel a routerhez, a Felhomtól +> kapott **Tulajdonosi jelmondat** (5 szó) és az e-mail fiókod. > -> **Mit próbálj ki az első órában?** -> - Nyisd meg a doboz kezdőlapját, és nézd meg, hogy minden alkalmazásod fut-e. -> - Tölts fel néhány fényképet vagy dokumentumot abba az alkalmazásba, amit a legtöbbet használsz. -> - Nézd meg a Beállítások → Értesítések oldalt: azok a levelek vannak-e bekapcsolva, amelyeket kérni szeretnél. -> - Nyomd meg a „Mentés most" gombot a Mentések oldalon, és várd meg, amíg kész lesz. +> 1. **Letöltés:** nyisd meg a https://felhom.eu/letoltes oldalt, és nyomd meg a **„Letöltés (1,6 GB)"** gombot. +> 2. **Ellenőrzés:** az oldal kiírja a fájl ellenőrző összegét (SHA-256), és azt is, hogyan számold ki a gépeden. +> A két érték legyen pontosan ugyanaz. Ha nem az, ne telepítsd — szólj. +> 3. **USB-kulcs:** írd ki a fájlt a kulcsra balenaEtcherrel (vagy Windows-on Rufusszal, „DD" módban). +> 4. **Telepítés:** indítsd a gépet a kulcsról. A képernyők angolul vannak — ez rendben van. +> - Felhom logó, két sor: **„Felhom telepítés"**. Utána: **I agree**. +> - **Target harddisk — ez a legfontosabb lépés.** A telepítő felsorolja a lemezeket méretükkel. Válaszd azt, amelyre +> a rendszer kerüljön (a gép belső rendszerlemeze). **Az új meghajtót ne válaszd** — azt a doboz később, a +> Tárhely oldalon kapja meg. Ha nem tudod, melyik melyik, állj meg és szólj. +> - Country / Timezone / Keyboard: hagyd így. Root password: adj meg bármilyen legalább 8 karakteres jelszót kétszer +> (a Felhom később lecseréli). Administrator email: a saját címed. Hostname: `felhom.`. +> - Summary: a jobbra nyíllal lépj az **Install** gombra, és nyomd meg. A gép a végén magától újraindul — a másolás +> alatt húzd ki az USB-kulcsot. +> 5. **Párosító kód:** a doboz képernyőjén 2–4 perc múlva megjelenik: „Párosító kód: XXX-XXX". Írd fel. +> 6. **Összekötés:** nyisd meg a **„Kösd össze a Felhom dobozodat"** e-mailt (ha kettő jött, a frissebbet), kattints a +> hivatkozásra, és add meg a Párosító kódot és a Tulajdonosi jelmondatot. +> 7. **Vezérlőpult jelszó:** kb. 3 perc múlva jön a **beállító kódos** e-mail. Kattints a benne lévő címre, add meg a +> Beállító kódot (3 szó), és válassz legalább 12 karakteres jelszót. Ez a vezérlőpult jelszava. +> 8. **Helyreállítási kód:** a vezérlőpult kéri, hogy hozd létre. Írd le **papírra**, és tartsd a gépen kívül. Amíg +> nincs meg, a távoli mentés nem indul el. +> 9. **Első két alkalmazás:** Alkalmazások → például **BookStack** és **PrivateBin** → Telepítés. Ha valamelyik +> jelszót kér, nyomd meg a „Generálás" gombot. > -> **Ha valami nem működik:** írd le röviden, mit csináltál, mit vártál, és mi történt helyette. Ha tudsz, küldj -> képernyőképet is. Küldd el a Felhom üzemeltetőjének azon a címen, ahol eddig is levelezettetek. +> **Ha valami nem működik:** írd le röviden, mit csináltál, mit vártál, és mi történt helyette, és ha tudsz, küldj +> képernyőképet. Küldd el a Felhom üzemeltetőjének azon a címen, ahol eddig is levelezettetek. > -> **Amit ne csinálj:** -> - Visszaállítás közben ne húzd ki a dobozt a konnektorból, és ne kapcsold ki. -> - A helyreállítási kódodat tartsd papíron, a dobozon kívül. A dobozon vagy a laptopon ne tárold. +> **Amit ne csinálj:** visszaállítás közben ne húzd ki a gépet a konnektorból, és ne kapcsold ki. A helyreállítási +> kódot ne a gépen vagy a laptopon tárold. -## 3. What CC checks after the hand-over +## 3. The operator, after the install -1. **A read-only box walk of Tester 2's box:** `scripts/https_only_check.py` with Tester 2 in it (R-1104); the - containers, unhealthy and restarting counts, and the controller's error lines (the 2026-10-22 release shape); the - agent's capability self-check (`ok=N total=N`) and the config bundle read back (R-862). -2. **The next night chain:** the database dump, the second-drive copy (if the tester has added a drive), the off-site - copy (if set up), the integrity check — each ran, read from the box's log the next morning. -3. **The hub alarms:** for 24 hours after it is back, every alarm Tester 2 raises, and whether the household's mail - list is the one the household saved (the hub keeps it; the box adopts it — R-1199). -4. **Tester 2's mail list:** its hub row was written at the claim and never saved by the household - (`household_saved` = false), so the box will push its default list once and keep it. If the tester had chosen a - different list on 2026-10-04, ask them to save it again on the box. +- **Off-site, if the friend wants it:** Configs → `Tester-2` → Edit → **Save** (off-site stays ticked). After the RESET + the tier is on but unprovisioned — the page says so — and the hub provisions a fresh Storage Box account only on + Save (about 25 s; do not click twice). Then the box's off-site starts once the friend has made the recovery code. + If he does not want it: untick off-site, Save. Nothing else. -Rows: R-862, R-1139, R-1104, R-1207, R-1199. +## 4. CC, after (tell CC „Tester 2 installed") + +1. **A read-only box walk:** `scripts/https_only_check.py` with Tester 2 in it (R-1104); containers, unhealthy and + restarting counts; the controller's error lines; the agent's capability self-check and the config bundle (a fresh + install has both). +2. **The host swap (D28):** from the host report — our installer gives this 16 GB machine 8 GB; the new rule says + short only below 90 % of min(the guest's swap, 8 GB). No `host_swap_short` expected. +3. **The first night chain, the next morning:** database dump, second-drive copy (if the new drive was set up as the + backup target), off-site copy (if switched on), the whole-box copy to ep0 (CC confirms the new host's PBS namespace + was made again after the RESET), the integrity check — each ran. +4. **The hub alarms for 24 hours** after the install, read from the hub. +5. **The mail list:** the hub's row was never saved by a household, so the new box pushes its default list once and + keeps it; the hub and the box page must agree. + +Rows: R-862 and R-1139 close when Tester 2 is reinstalled; R-1104 is checked by step 4.1.