From 1b442bcc2ea8cf37793ebbbe403935352e398be7 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 22 Oct 2026 15:04:07 +0200 Subject: [PATCH] Tester 2's RESET checked read only: every leg ok, the first live R-32 purge (folder held no repository; sub-account gone), ep0 namespace+copy+prune job+token gone, no other customer touched (Storage Box 303/62/43 and ep0 2/2/2/1/27 before = after); hand-over: tell the friend go; A20 third firing; R-32 result + dated check for Tester4 --- CONTEXT.md | 8 +++ STATUS.md | 5 +- .../architecture/00-capability-map.md | 2 +- .../architecture/capability-map-detail/A.md | 9 +++ .../01-reset-record.txt | 28 ++++++++ .../02-storagebox-hub-listings.txt | 11 +++ .../03-storagebox-after-reset.txt | 11 +++ .../tester2-reset-2026-10-22/04-ep0.txt | 69 +++++++++++++++++++ .../tester2-reset-2026-10-22/05-hub-after.txt | 14 ++++ documentation/backlog/OPEN-ITEMS.md | 3 +- documentation/pilot/TESTER2-handover.md | 17 ++++- 11 files changed, 169 insertions(+), 8 deletions(-) create mode 100644 documentation/audits/tester2-reset-2026-10-22/01-reset-record.txt create mode 100644 documentation/audits/tester2-reset-2026-10-22/02-storagebox-hub-listings.txt create mode 100644 documentation/audits/tester2-reset-2026-10-22/03-storagebox-after-reset.txt create mode 100644 documentation/audits/tester2-reset-2026-10-22/04-ep0.txt create mode 100644 documentation/audits/tester2-reset-2026-10-22/05-hub-after.txt diff --git a/CONTEXT.md b/CONTEXT.md index 9d7ab018..9cdd5fea 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -16,6 +16,14 @@ > and holds nothing of its own; this file does hold its own content, namely the standing rulings below. +> **2026-10-22 ~14:50 — operator acts, checked by CC (read only).** Host `Tester-2-be8404` deleted (12:54:47 UTC); +> customer `Tester-2` language set to Magyar; customer `Tester-2` RESET (12:55:56–12:56:04 UTC, journal #25, every leg +> ok). The first live run of R-32's purge-first leg: logged in with the sub-account's own password, found no repository +> (0 paths — the box never backed up off-site), deleted sub-account 322460; ep0 namespace + its 2026-10-04 copy + prune +> job + token gone. No other customer touched: Storage Box counts after = the hub's 05:10 listing (demo-hp 303, +> tester-1 62, demo-felhom 43; Tester4 has no box — compared by the hub on 10-23 05:10), ep0 after = the 03:30 prune +> keep-lists (2/2/2/1, operator 27). No false alarm. `audits/tester2-reset-2026-10-22/`. + > **2026-10-22 14:11 — operator ruling: Tester 2 is a NEW INSTALL, not an upgrade.** The friend reinstalls the laptop > from the public download (installer 1.30.0 or newer) with the new drive in before the install; the old box's data is > not needed („he barely used the laptop, no useful data on it"). Read on the hub the same day (read only): no sealed diff --git a/STATUS.md b/STATUS.md index 27d1dbf0..bbcc4b0e 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,6 +1,6 @@ # STATUS — what works, what's broken, what's next -**Ready for Tester 2? Yes — a new install: you do the hub step first (hand-over list: `documentation/pilot/TESTER2-handover.md`).** +**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).** **Updated 2026-10-22 (day).** A rebuilt box no longer overwrites the family's saved mail list (hub 0.165.0, controller 0.339.0); demo-felhom's own list, replaced this morning, is back. The swap warning now fires only when a machine has clearly too little swap (your D28 = A). @@ -52,8 +52,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te 7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12. **Today, 2026-10-22.** -1. Before Tester 2's friend reinstalls: delete the old host, then RESET the customer (hand-over list, section 1). The RESET deletes Tester 2's old copies on ep0 and the Storage Box. If not: the new box carries an old, unreadable off-site history and a stale whole-box copy. -2. Tester 2's mail language is English on the hub. If the friend reads Hungarian, change it first. If not: his mails and first screen are in English. +1. Give Tester 2's friend the owner passphrase and the one-page guide, and tell him „go". The reset worked: his old copies are gone, nobody else's changed. If not: nothing happens; the link mail expires in 7 days (he can ask for a new one). **The rest** - Pull the failed disk Z6A07P2G out of DooPlex (R-934). If not: its alarms return on 11 November. Since 2026-10-12. - Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). Since 2026-10-09. diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 5442c8dd..05d13d7c 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -93,7 +93,7 @@ | [A17](capability-map-detail/A.md#a17) | Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE** | demo-hp | `audits/SPIKE-r82-phase0-2026-07-26.md` (2026-07-26) | | [A18](capability-map-detail/A.md#a18) | Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard | agent v0.104.0 → v0.121.0, hub v0.77.0 → v0.91.0 | **PROVEN-LIVE** | — | *(evidence not on file — not re-verifiable)*: the row cites per-repo CHANGELOGs and the spec `backlog/SPEC-r85-phase4-5-2026-07-26.md` — a spec, not a run record | | [A19](capability-map-detail/A.md#a19) | A restore-test can never fill the box's disk: it sizes the restore first (uncompressed), keeps off the tested guest's pool when it can, refuses what does not fit, and retries a failed clean-up on a timer | agent v0.133.0 (RELEASED, NOT DELIVERED), hub v0.124.0 | **IMPLEMENTED** | written BUILT: the refusal proven on demo-hp 2026-09-24; the full-test path not live | `audits/r672-2026-09-24/` (2026-09-24); R-672 | -| [A20](capability-map-detail/A.md#a20) | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE** | two real firings | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) | +| [A20](capability-map-detail/A.md#a20) | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0; purge-first hub v0.142.0 | **PROVEN-LIVE** | two real firings (2026-07-18); third 2026-10-22 on a real customer (Tester-2) with the R-32 purge-first leg — all legs ok, others untouched | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18); `audits/tester2-reset-2026-10-22/` | | [A21](capability-map-detail/A.md#a21) | Customer DELETE cascade (top lifecycle tier): one guided operator action → `hosts → RESET → residue → purge`; host rows deleted (custody DEMOTED), full external teardown (Hetzner repo destroyed, PBS namespace/token revoked, tunnel + zone removed), then the customer record and ALL escrow ciphertext purged | hub v0.69.0 | **IMPLEMENTED** | written UNIT-PROVEN: live leg pending | hub unit tests `hub/internal/web/customer_delete_test.go` (outside `documentation/`) | | [A22](capability-map-detail/A.md#a22) | Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | GL6-F2 open | `pilot/DRILL-GL6-2026-07-08.md` (2026-07-08); R-9 | | [A23](capability-map-detail/A.md#a23) | Felhom in a virtual machine on OpenMediaVault 8 (KVM plugin): the whole NVMe passed to the VM, installed from the public ISO, bound, claimed, apps, backup, and back by itself after an OMV restart | ISO 1.29.0, agent 0.156.1, controller 0.314.0, OMV 8.5.11-1 + openmediavault-kvm 8.3.4 | **PARTIAL** | live on an exact OMV copy (VM on demo-hp), not yet on the pilot's real hardware; one-disk Jellyfin not proven (R-1068); NAS films not usable by Jellyfin (R-1071); no SMART inside the VM (R-1073) | `audits/omv-vm-2026-10-15/README.md` (2026-10-15); guide `pilot/GUIDE-omv-kvm-hu.md` | diff --git a/documentation/architecture/capability-map-detail/A.md b/documentation/architecture/capability-map-detail/A.md index 3225af6f..7cd20ce4 100644 --- a/documentation/architecture/capability-map-detail/A.md +++ b/documentation/architecture/capability-map-detail/A.md @@ -407,6 +407,15 @@ the ten places a household gets stuck are in `audits/rehearsal-2026-10-20/B10-st **Evidence (as written):** **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests +**Third live firing — 2026-10-22, a real customer (Tester-2), the first with the R-32 purge-first off-site leg (hub +0.165.0):** host delete 12:54:47 UTC, RESET 12:55:56–12:56:04, every leg `ok`. The off-site leg logged in with the +sub-account's own password, found no repository (the box never ran an off-site backup — so the removal of a folder +WITH data is still not seen live), re-listed and deleted the sub-account; ep0's namespace with its one whole-box copy +(2026-10-04), its prune job and its token are gone; the customer, address, domain, language, tier choice and quota +stayed. **No other customer touched:** the three live boxes' repositories re-listed with their own keys equal the hub's +05:10 listing (303 / 62 / 43); every other ep0 namespace equals today's 03:30 prune keep-list (2 / 2 / 2 / 1; operator +27). Evidence `audits/tester2-reset-2026-10-22/`. + **Gap / roadmap (as written):** External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **R-25b CLOSED (hub v0.69.0, 2026-07-21):** the Danger-zone DELETE is now the guided full-teardown cascade that runs this very sequence as its middle leg — see the row below --- diff --git a/documentation/audits/tester2-reset-2026-10-22/01-reset-record.txt b/documentation/audits/tester2-reset-2026-10-22/01-reset-record.txt new file mode 100644 index 00000000..114e1926 --- /dev/null +++ b/documentation/audits/tester2-reset-2026-10-22/01-reset-record.txt @@ -0,0 +1,28 @@ +# Tester-2 host delete + RESET — the hub's own record (customer_resets #25, events, the hub log 14:54–14:58 CEST = 12:54–12:58 UTC; log times are CEST) +customer_resets #25: started 2026-10-22 12:55:56 UTC, completed 12:56:04 UTC, escrow_acked 0, legs {claim: ok, db_purge: ok, descriptor: ok, hetzner: ok, pbs: ok} +host_deletions: Tester-2-be8404 deleted 2026-10-22 12:54:47 UTC, escrow_acked 0 +events: 12:54:48 selfbind_link_sent (host delete) · 12:56:04 customer_reset (warning) · 12:56:05 selfbind_link_sent (RESET completion) +--- hub log +2026/10/22 14:54:47 [INFO] host deleted: Tester-2-be8404 (no key escrow held; WG peer removal push requested (asynchronous — the endpoint drops the peer on that push, check `wg show` before calling the box gone)) +2026/10/22 14:54:48 [INFO] self-bind link emailed to the registered address of Tester-2 +2026/10/22 14:54:48 [INFO] self-bind link (hash 25c9e7ff…, valid 7 days) emailed to the registered address of Tester-2 +2026/10/22 14:54:48 [INFO] self-bind link auto-minted for Tester-2 on host delete (the console banner's promised email now exists) +2026/10/22 14:54:48 [INFO] wgsync: pushed 4 peers to 167.233.158.164:22 +2026/10/22 14:55:10 [DEBUG] Offsite escrow pending: Tester-2 — no anchor (no successful run, no host report) — not judged +2026/10/22 14:55:18 [INFO] osupdates: guest candidate 654f62937e9293c8 (272 packages, first seen 2026-10-22T03:23:44Z): waiting — healthy for 9h32m0s of 24h0m0s +2026/10/22 14:55:18 [INFO] osupdates: host candidate 69c797991f7b72f5 (607 packages, first seen 2026-10-22T03:23:44Z): waiting — healthy for 9h32m0s of 24h0m0s +2026/10/22 14:55:20 [offsite] shared already provisioned for Tester-2 (subaccount 322460) +2026/10/22 14:55:21 [INFO] pbsdr: DR tier ON for Tester-2, no host enrolled yet — the descriptor applies once the cascade is ready (host → WG peer → apply) +2026/10/22 14:55:21 [INFO] Customer config updated: Tester-2 +2026/10/22 14:55:56 [INFO] customer RESET started for Tester-2 (journal #25, escrow_ack=false) +2026/10/22 14:55:57 [WARN] offsitekeys: RESET purged Tester-2's off-site folder through its own login: 0 path(s) [] +2026/10/22 14:56:04 [offsite] deprovisioned shared sub-account 322460 for Tester-2 (its folder was purged first) +2026/10/22 14:56:04 [INFO] reset Tester-2: offsite deprovisioned (repo data destroyed) +2026/10/22 14:56:04 [INFO] tenantsync: deprovision ok for Tester-2 (ns=Tester-2, existed=true) +2026/10/22 14:56:04 [INFO] reset Tester-2: PBS tenancy deprovisioned +2026/10/22 14:56:04 [INFO] [claim] reset to unclaimed for Tester-2 (customer RESET) — next onboarding mints a fresh code +2026/10/22 14:56:04 [INFO] customer RESET complete for Tester-2 (journal #25) — identity + basic config retained +2026/10/22 14:56:05 [INFO] self-bind link emailed to the registered address of Tester-2 +2026/10/22 14:56:05 [INFO] self-bind link (hash cea0be7a…, valid 7 days) emailed to the registered address of Tester-2 +2026/10/22 14:56:05 [INFO] self-bind link auto-minted for Tester-2 on RESET completion (the console banner's promised email now exists) +2026/10/22 14:56:10 [DEBUG] Offsite escrow pending: Tester-2 — no anchor (no successful run, no host report) — not judged diff --git a/documentation/audits/tester2-reset-2026-10-22/02-storagebox-hub-listings.txt b/documentation/audits/tester2-reset-2026-10-22/02-storagebox-hub-listings.txt new file mode 100644 index 00000000..a2c64568 --- /dev/null +++ b/documentation/audits/tester2-reset-2026-10-22/02-storagebox-hub-listings.txt @@ -0,0 +1,11 @@ +# the hub's own daily off-site listing per customer (hub_settings offsite_snapshot_listing:*) — before the reset (05:10 UTC today) and yesterday; Tester-2 has none in either +2026-10-21 (00:00Z snapshot) Tester4 listed_at 2026-10-21T05:10:01.734603053Z entries 44 +2026-10-21 (00:00Z snapshot) demo-felhom listed_at 2026-10-21T05:10:03.612595011Z entries 38 +2026-10-21 (00:00Z snapshot) demo-hp listed_at 2026-10-21T05:10:06.382113986Z entries 289 +2026-10-21 (00:00Z snapshot) tester-1 listed_at 2026-10-21T05:10:08.570986682Z entries 58 +2026-10-21 (00:00Z snapshot) Tester-2 listing rows: 0 +2026-10-22 05:10 UTC (live) Tester4 listed_at 2026-10-22T05:10:01.733186835Z entries 72 +2026-10-22 05:10 UTC (live) demo-felhom listed_at 2026-10-22T05:10:05.213622074Z entries 43 +2026-10-22 05:10 UTC (live) demo-hp listed_at 2026-10-22T05:10:08.606069689Z entries 303 +2026-10-22 05:10 UTC (live) tester-1 listed_at 2026-10-22T05:10:10.755277879Z entries 62 +2026-10-22 05:10 UTC (live) Tester-2 listing rows: 0 diff --git a/documentation/audits/tester2-reset-2026-10-22/03-storagebox-after-reset.txt b/documentation/audits/tester2-reset-2026-10-22/03-storagebox-after-reset.txt new file mode 100644 index 00000000..54eb07aa --- /dev/null +++ b/documentation/audits/tester2-reset-2026-10-22/03-storagebox-after-reset.txt @@ -0,0 +1,11 @@ +# Storage Box AFTER Tester-2's RESET (12:56:04 UTC), listed 2026-10-22 13:00 UTC — read only: each box's own +# append-only key, `restic snapshots --json --no-lock` from inside its controller container (the controller's own +# rclone-pinned transport); only counts printed. Compared with the hub's own 05:10 UTC listing (02-...). +demo-hp user=u629488-sub3 repo=/home/felhom-repo snapshots 303 newest 2026-10-22T02:18:13 hub 05:10 listing 303 SAME +tester-1 user=u629488-sub4 repo=/home/felhom-repo snapshots 62 newest 2026-10-22T02:15:58 hub 05:10 listing 62 SAME +demo-felhom user=u629488-sub1 repo=/home/felhom-repo snapshots 43 newest 2026-10-22T02:15:29 hub 05:10 listing 43 SAME +Tester4 no box (its VM was destroyed 2026-10-22 night) — not re-listable read-only by CC; hub 05:10 listing 72. + The hub's own 05:10 UTC listing on 2026-10-23 compares it with today's and raises + offsite_snapshots_vanished if any file is gone (R-895) — CC reads it then. +# Structure: each customer is a separate sub-account whose login sees only its own home; the purge ran through +# Tester-2's own login (sub-account 322460) and removed 0 paths (the hub log, 01-...). It could not reach the others. diff --git a/documentation/audits/tester2-reset-2026-10-22/04-ep0.txt b/documentation/audits/tester2-reset-2026-10-22/04-ep0.txt new file mode 100644 index 00000000..d0069d14 --- /dev/null +++ b/documentation/audits/tester2-reset-2026-10-22/04-ep0.txt @@ -0,0 +1,69 @@ +# ep0 (felhom-hetzner), read only (list commands), 2026-10-22 13:01 UTC +2026-10-22T13:01:05Z +== namespaces +Tester4 +demo-felhom +demo-hp +operator +tester-1 +== snapshot dirs +ns/Tester4/ct/9201/2026-10-20T12:17:26Z +ns/demo-felhom/ct/9201/2026-10-13T04:22:51Z +ns/demo-felhom/ct/9201/2026-10-20T04:27:45Z +ns/demo-hp/ct/9201/2026-10-08T20:34:27Z +ns/demo-hp/ct/9201/2026-10-15T20:38:47Z +ns/tester-1/ct/9201/2026-10-11T20:02:13Z +ns/tester-1/ct/9201/2026-10-18T20:04:51Z +== prune jobs +prune-demo-felhom demo-felhom 03:30 {'keep-weekly': 2} +prune-demo-hp demo-hp 03:30 {'keep-weekly': 2} +prune-Tester4 Tester4 03:30 {'keep-weekly': 2} +prune-tester-1 tester-1 03:30 {'keep-weekly': 2} +prune-operator-hubdb operator 03:45 {'keep-daily': 14, 'keep-weekly': 8} +== users +dooplex-hub@pbs None [] +felhom@pbs None [] +root@pam None [] +== tokens +felhom@pbs ['felhom@pbs!Tester4', 'felhom@pbs!demo-felhom', 'felhom@pbs!demo-hp', 'felhom@pbs!tester-1'] +dooplex-hub@pbs ['dooplex-hub@pbs!push', 'dooplex-hub@pbs!restore'] +== acl for Tester-2 +acl entries: 14 mentioning Tester-2: [] +== today prune tasks +UPID:felhom-hetzner:00086AE7:07B20B30:00000573:6AD986BC:prunejob:felhom\x2doffsite\x3aoperator:root@pam: OK +UPID:felhom-hetzner:00086AE7:07B20B30:00000572:6AD98338:prunejob:felhom\x2doffsite\x3atester\x2d1:root@pam: OK +UPID:felhom-hetzner:00086AE7:07B20B30:00000571:6AD98338:prunejob:felhom\x2doffsite\x3ademo\x2dhp:root@pam: OK +UPID:felhom-hetzner:00086AE7:07B20B30:00000570:6AD98338:prunejob:felhom\x2doffsite\x3aTester\x2d2:root@pam: OK +UPID:felhom-hetzner:00086AE7:07B20B30:0000056F:6AD98338:prunejob:felhom\x2doffsite\x3ademo\x2dfelhom:root@pam: OK +UPID:felhom-hetzner:00086AE7:07B20B30:0000056E:6AD98338:prunejob:felhom\x2doffsite\x3aTester4:root@pam: OK +operator kept at 03:45: 27 +operator now: 27 +Oct 22 12:56:04 felhom-hetzner sudo[2948694]: felhom-peersync : PWD=/home/felhom-peersync ; USER=root ; COMMAND=/usr/local/bin/felhom-tenantsync +Oct 22 12:56:04 felhom-hetzner proxmox-backup-proxy[551655]: removing whole namespace recursively below felhom-offsite:/Tester-2 +Oct 22 12:56:04 felhom-hetzner proxmox-backup-proxy[551655]: removing groups in namespace felhom-offsite:/Tester-2 +Oct 22 12:56:04 felhom-hetzner proxmox-backup-proxy[551655]: removing all groups in namespace felhom-offsite:/Tester-2 +Oct 22 12:56:04 felhom-hetzner proxmox-backup-proxy[551655]: removing backup group "/mnt/pbs-datastore/ns/Tester-2/ct/9201" +Oct 22 12:56:04 felhom-hetzner proxmox-backup-proxy[551655]: removing backup snapshot "/mnt/pbs-datastore/ns/Tester-2/ct/9201/2026-10-04T16:31:13Z" +(ns/operator: 27 snapshot folders now = 27 kept by today's 03:45 prune) + +# BEFORE: today's 03:30 UTC prune tasks (one per household namespace) — their keep lists +Starting datastore prune on datastore 'felhom-offsite', namespace 'tester-1', non-recursive +keep ct/9201/2026-10-11T20:02:13Z +keep ct/9201/2026-10-18T20:04:51Z +TASK OK +Starting datastore prune on datastore 'felhom-offsite', namespace 'demo-hp', non-recursive +keep ct/9201/2026-10-08T20:34:27Z +keep ct/9201/2026-10-15T20:38:47Z +TASK OK +Starting datastore prune on datastore 'felhom-offsite', namespace 'Tester-2', non-recursive +keep ct/9201/2026-10-04T16:31:13Z +TASK OK +Starting datastore prune on datastore 'felhom-offsite', namespace 'demo-felhom', non-recursive +keep ct/9201/2026-10-13T04:22:51Z +keep ct/9201/2026-10-20T04:27:45Z +TASK OK +Starting datastore prune on datastore 'felhom-offsite', namespace 'Tester4', non-recursive +keep ct/9201/2026-10-20T12:17:26Z +TASK OK + +# comparison: tester-1 2 -> 2, demo-hp 2 -> 2, demo-felhom 2 -> 2, Tester4 1 -> 1 (same names); operator 27 -> 27; Tester-2 1 -> namespace gone diff --git a/documentation/audits/tester2-reset-2026-10-22/05-hub-after.txt b/documentation/audits/tester2-reset-2026-10-22/05-hub-after.txt new file mode 100644 index 00000000..77edd988 --- /dev/null +++ b/documentation/audits/tester2-reset-2026-10-22/05-hub-after.txt @@ -0,0 +1,14 @@ +# the hub after the RESET (live, with WAL), read 2026-10-22 13:02:10 UTC — SELECT on named non-secret columns +customer: Tester-2 | name set: True | domain: sajatfelhom.hu | address set: True | language: hu | status: active | dr_tier: 1 +offsite: enabled True type shared | provisioned fields still set: ['quota_gb'] +hosts: 0 +claim rows: 0 +escrow (live/retained via deletions): 0 0 +dr_recipe rows: 0 | one_time_secrets: 0 +mail list: [(1, 0)] +offsite listing row: 0 +# events since 12:50 UTC, every customer, warning or worse + ('2026-10-22 12:56:04', 'Tester-2', 'customer_reset', 'warning', 'Ügyfél-visszaállítás (RESET): minden működési állapot törölve (offsite tároló, PBS névtér, DR-recept, azonosítási állapot). Az azonosság és ') +# any event of the R-1165 / key-audit kinds since 12:50: 0 +felhom.sajatfelhom.hu 530 server=server: cloudflare +files.sajatfelhom.hu 530 server=server: cloudflare diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 58fdca6a..c997e300 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -177,7 +177,7 @@ stopping line that lies. | **R-816** | Backup & restore | P4 | **No off-site failure class has ever been seen live.** F-DIAG (controller v0.182.0, 2026-07-28) split off-site failures into six causes — quota, orphaned, no_repo, no_units, transport, unknown — each with its own Hungarian message. None of the six has been exercised by a real failure on a box; the recovery inventory records it only as a known limit (`documentation/architecture/_recovery-inventory-2026-07-28.md:955`). Filed 2026-10-03 from the F-DIAG row's residue when that row moved to `CLOSED-ITEMS.md`. | **READY — filed 2026-10-03 (triage); owner: CC.** Exercise each class once on a scratch guest (a full quota, a missing repository, a blocked transport) and read the message the household sees. | — | — | CC | | **R-832** | Backup & restore | P4 | **ep0's copy in a place outside both Hetzner and the operator's home (roadmap).** Today DooPlex (the operator's home) holds it (decision 71). A Hetzner Storage Box would share a provider with ep0 and with every household's file backups, and cannot run PBS, so the copy could not be verified or restored from directly. | **DEFERRED — later, if the product grows** | — | — | operator | | **R-878** | Backup & restore | P4 | **A catch-up (R-871) runs the database-dump leg in the DAY, and that leg stops an app with a volume for its copy — the household may notice the stop, and a large volume makes it longer.** MEASURED 2026-10-05 on demo-felhom: the catch-up at 08:25:02 stopped opengist, copied 182.5 KB, started it again — about 1 s, then a few seconds of `health: starting`; the night does exactly the same, unseen. Nothing measured for a large volume. Fix direction (if it matters): skip the volume copy of a running app in a DAYTIME catch-up and leave it to the next night, or warn. `audits/catchup-2026-10-05/partA/live-demo-felhom.txt` | **READY — owner: CC** **2026-10-05 (burn-down night): NEEDS A LIVE MEASUREMENT** — a timed catch-up on 9202 for an app with a multi-GB volume, then the skip-or-warn choice. | — | measure a large volume first | CC | -| **R-32** | Backup & restore | P4 | **[P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible.** The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's `"hetzner":"ok"` leg destroys the sub-account, but **a Hetzner sub-account is an access-control object, not a data object** — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **2026-10-06 night: design written** (`audits/night-burndown-2026-10-06/design-R-32.md`) — pick A: purge the repo + set-aside copies through the sub-account's own password login BEFORE deleting it (no main-account credential); the „repo data destroyed" comment at `hub/internal/offsite/offsite.go:276-280` is false for the shared tier; part (3) waits on a read-only measurement (`du` on port 23). Operator: agree to route A. **2026-10-07 07:58: `09` §3 decision 167 — option A yes, and delete the old RESET leftovers once.** **2026-10-07 (Part D step 2, read only):** the hub shows bx11 holding 3.3 GB of data while the four live customers account for ~0.6 GB → **~2.7 GB belongs to no live customer**; no live customer shows a set-aside folder, so the leftovers are in homes whose sub-account is gone, reachable only by the pool box's MAIN account. **The one-time clean-up is STOPPED:** no main-account login for bx11 is held on DooPlex (the API token in the credentials file sees no box). Needs: the main-account login, or the operator lists and deletes in the Hetzner console (`audits/day-2026-10-07/D/D2-listing-readonly.txt`). **2026-10-07: option A DELIVERED (hub 0.142.0):** RESET purges the repo and every `.orphaned-*` through the sub-account's own login before deleting it, and keeps the sub-account if the purge fails (red-proved, `audits/day-2026-10-07/D/`). Not yet seen live (no scratch RESET today). **The one-time clean-up of the ~2.7 GB of leftovers is STOPPED:** it needs the pool box's main-account login (see above). **2026-10-07 14:43 — `09` §3 decision 173: leave the ~2.7 GB of old leftovers for now; re-ranked P2 → P4.** Option A itself is delivered (hub 0.142.0). What stays open is ONLY the leftovers: they were computed (box total minus live customers), never listed, and deleting them needs the pool box's MAIN login, which reaches every customer's data. **Blocker: a safe main-account path.** **2026-10-22: Tester 2's planned RESET (operator ruling 2026-10-22 14:11, `pilot/TESTER2-handover.md`) will be the purge's first live run — CC reads its leg journal after.** | — | **Ruling from the run (three parts, deliberately separate):** (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable **BY DESIGN** → RESET gains a **main-account purge of the customer base dir** (the existing operator ack already covers it); (2) the **move-aside guard STAYS** for reinstall-*without*-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator **Restic tab shows per-customer directory bytes vs attributed snapshot bytes**, so dead data cannot hide. Measured on the pool box that night: **49 M attributed** (2 snapshots, 48.717 MiB) against **1.4 G + 3.0 M unattributed** across TWO `.orphaned-*` dirs. Evidence `restic-and-pool.txt` | CC | +| **R-32** | Backup & restore | P4 | **[P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible.** The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's `"hetzner":"ok"` leg destroys the sub-account, but **a Hetzner sub-account is an access-control object, not a data object** — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **2026-10-06 night: design written** (`audits/night-burndown-2026-10-06/design-R-32.md`) — pick A: purge the repo + set-aside copies through the sub-account's own password login BEFORE deleting it (no main-account credential); the „repo data destroyed" comment at `hub/internal/offsite/offsite.go:276-280` is false for the shared tier; part (3) waits on a read-only measurement (`du` on port 23). Operator: agree to route A. **2026-10-07 07:58: `09` §3 decision 167 — option A yes, and delete the old RESET leftovers once.** **2026-10-07 (Part D step 2, read only):** the hub shows bx11 holding 3.3 GB of data while the four live customers account for ~0.6 GB → **~2.7 GB belongs to no live customer**; no live customer shows a set-aside folder, so the leftovers are in homes whose sub-account is gone, reachable only by the pool box's MAIN account. **The one-time clean-up is STOPPED:** no main-account login for bx11 is held on DooPlex (the API token in the credentials file sees no box). Needs: the main-account login, or the operator lists and deletes in the Hetzner console (`audits/day-2026-10-07/D/D2-listing-readonly.txt`). **2026-10-07: option A DELIVERED (hub 0.142.0):** RESET purges the repo and every `.orphaned-*` through the sub-account's own login before deleting it, and keeps the sub-account if the purge fails (red-proved, `audits/day-2026-10-07/D/`). Not yet seen live (no scratch RESET today). **The one-time clean-up of the ~2.7 GB of leftovers is STOPPED:** it needs the pool box's main-account login (see above). **2026-10-07 14:43 — `09` §3 decision 173: leave the ~2.7 GB of old leftovers for now; re-ranked P2 → P4.** Option A itself is delivered (hub 0.142.0). What stays open is ONLY the leftovers: they were computed (box total minus live customers), never listed, and deleting them needs the pool box's MAIN login, which reaches every customer's data. **Blocker: a safe main-account path.** **2026-10-22: Tester 2's planned RESET (operator ruling 2026-10-22 14:11, `pilot/TESTER2-handover.md`) will be the purge's first live run — CC reads its leg journal after.** **2026-10-22 12:56 UTC — the first live run (Tester 2's RESET, `audits/tester2-reset-2026-10-22/`): PASS for what it could show.** The hetzner leg logged in with the sub-account's own password, listed the folder, found NO repository (`purged … 0 path(s) []` — the box never ran an off-site backup; the hub never held a listing for it), re-listed, then deleted sub-account 322460; leg `ok`. The other customers' repositories are unchanged (each listed after the reset with its box's own key: demo-hp 303, tester-1 62, demo-felhom 43 = the hub's 05:10 listing; Tester4 has no box — the hub's own listing on 2026-10-23 05:10 compares it). **Not yet seen live: the removal of a folder that holds data** (the `rm -rf` branch) — the next RESET of a customer with off-site history shows it. Open as before: only the old leftovers. **Dated check 2026-10-23 (DUE-CHECKS):** read the hub's 05:10 UTC listing of Tester4 (72 entries on 2026-10-22) and its `offsite_snapshots_vanished` events — the one other customer whose folder CC could not re-list. | — | **Ruling from the run (three parts, deliberately separate):** (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable **BY DESIGN** → RESET gains a **main-account purge of the customer base dir** (the existing operator ack already covers it); (2) the **move-aside guard STAYS** for reinstall-*without*-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator **Restic tab shows per-customer directory bytes vs attributed snapshot bytes**, so dead data cannot hide. Measured on the pool box that night: **49 M attributed** (2 snapshots, 48.717 MiB) against **1.4 G + 3.0 M unattributed** across TWO `.orphaned-*` dirs. Evidence `restic-and-pool.txt` | CC | | **R-1087** | Backup & restore | P4 | **Whole-box copies left on the OLD local target are never pruned after the target moves.** MEASURED 2026-10-15 (storage-identity build B space check): the N100's host root (`local`) holds three copies from 2026-07-28 (5.5–5.7 GB each, ≈ 16.8 GB); its agent's `local_backup_target` is now `felhom-backup` (the USB disk, retention 3), and retention prunes only the current target. On a small-root box such leftovers would starve the local copies (R-548). Evidence: `audits/storage-identity-buildB-2026-10-15/space/felhom-pve-9201.txt`. | OPEN | — | Decide whether the agent prunes its own `vzdump-lxc--*` files on a former target (after a newer copy exists on the new one), or the target move asks; until then, the N100's three July files can be removed by hand with the operator's word. | CC | | **R-1171** | Backup & restore | P3 | **demo-hp's LOCAL whole-guest copy is never restore-tested: the box keeps one copy (keep-daily 1) and the restore test wants a copy 24 h old, so the only copy turns 24 h old exactly when the next night replaces it.** Read 2026-10-20 on demo-hp (agent journal): every 6-hourly evaluation since 2026-10-18 says `local: no settled archive yet — nothing to prove`; the local storage holds one 9201 archive; the last local proof was 2026-10-14 03:42 (`settle=24h`, `keep_last=1`, `prune=keep-daily=1`). The hub's `restore_test_stale` (168 h) will alarm from ~2026-10-21 03:42. A real household with retention 1 would be the same. | OPEN — filed 2026-10-20 | demo-hp `journalctl -u felhom-agent` („restore-test evaluated"), the hub's host report `restore_tests` | Agent: a tier that keeps ONE copy proves its newest archive once it is older than the settle time measured from the BACKUP's end, not 24 h — or a settle below the cadence (e.g. 6 h) for a 1-copy tier; a test; then read the next evaluation | CC | | **R-1172** | Backup & restore | P4 | **After option A (R-893): a failed off-site restore that was PUT BACK is logged at ERROR, and an empty `backups/restore-undo/` folder stays on disk.** Measured on 9202 2026-10-20 (rc.1): the web handler logs `[ERROR] [web] off-box reconstitute rallly (async): … Nem változott semmi…` — good news at ERROR; the put-back removes the per-app folder but not the empty `restore-undo/` parent. Small; 0.336.0 was already pushed (a version is pushed once). | OPEN — filed 2026-10-20 | `audits/night2-2026-10-20/P2/live/11-controller-log-window.txt`, `12-after-failed-restore.txt` | Next controller release: the put-back outcome logged at WARN (its own sentence is the household's), the empty parent removed; a test | CC | @@ -359,6 +359,7 @@ stopping line that lies. |---|---|---| | R-435 | 2026-10-27 | Tester 1's weekly off-site clean-up ran inside a hub window with no drop alarm (box log + mailbox); detail in the R-435 row | | R-1183 | 2026-10-26 | every hub daily job ran ONCE on 2026-10-25 (hub log + one snapshot file); detail in the R-1183 row | +| R-32 | 2026-10-23 | the hub's 05:10 off-site listing of Tester4 is unchanged after Tester 2's RESET (72 entries, no offsite_snapshots_vanished); detail in the R-32 row | | R-1173 | 2026-10-23 | demo-hp's night with the off-site tier due: the R-921 pre-check line or a start with nothing in flight; detail in the R-1173 row | | R-1174 | 2026-10-27 | the status gate refuses from today: one conviction seen exiting 1 (a hand run on a copy of the page is enough); detail in the R-1174 row | | R-1050 | 2026-11-15 | HSTS one year if no https trouble since 2026-10-15; detail in the R-1050 row | diff --git a/documentation/pilot/TESTER2-handover.md b/documentation/pilot/TESTER2-handover.md index f618270f..479a233a 100644 --- a/documentation/pilot/TESTER2-handover.md +++ b/documentation/pilot/TESTER2-handover.md @@ -5,7 +5,7 @@ puts the new drive in, with the drive in before the install. The old box's data laptop, no useful data on it"). So nothing is upgraded and nothing is restored. (The earlier upgrade list is in git: `git log -- documentation/pilot/TESTER2-handover.md`.) -## What the hub holds for Tester 2 today (read only, 2026-10-22) +## What the hub held for Tester 2 before the RESET (read only, 2026-10-22 morning) - Host `Tester-2-be8404`, agent 0.142.0, last report **2026-10-04 18:05 UTC** (offline since). A 16 GB machine, guest 9201 with 12 GB. @@ -52,13 +52,24 @@ there is none). From the code (`hub/internal/web/customer_reset.go`, `store/cust list; every event (the audit trail); - at the end it mails the friend a new link itself. -**Recommendation: host delete, then RESET.** „Fresh start, nothing to keep" is exactly what RESET is for: the new box +**Recommendation (taken 2026-10-22): host delete, then RESET.** „Fresh start, nothing to keep" is exactly what RESET is for: the new box starts with an empty Storage Box path and an empty ep0 namespace, instead of carrying a set-aside history nobody can open and a stale whole-box copy. **RESET deletes data on ep0 and on the Storage Box — that is the operator's press, never CC's** (both are Tier 2). It is proven live twice (A20, 2026-07-18), each leg idempotent: a failed leg is re-run from the top, and nothing on the hub is purged until both external legs are ok. -## 1. The operator, before the friend starts +## 1. The operator, before the friend starts — **DONE 2026-10-22** (checked by CC, read only) + +**Status: steps 1–4 done on 2026-10-22; next: step 5 — give the passphrase and tell the friend „go".** +- Step 1: the mail language is now **Magyar** (`hu`). +- Step 2: host `Tester-2-be8404` deleted 12:54:47 UTC. +- Step 3: RESET 12:55:56–12:56:04 UTC, every leg `ok`. The off-site folder held no repository (nothing to delete) and + its sub-account is gone; ep0's `Tester-2` namespace, its 2026-10-04 copy, its prune job and its token are gone; no + other customer's data changed (Storage Box and ep0 counts before = after). The friend got the newest link mail at + 12:56:05 UTC (the one sent at 12:54:48 no longer works). +- Step 4, seen from outside: `felhom.sajatfelhom.hu` and `files.sajatfelhom.hu` answer Cloudflare 530 — the route to his + tunnel exists and waits for a box. Evidence `audits/tester2-reset-2026-10-22/`. + | # | What | Where | Done when | |---|---|---|---|