rehearsal 2026-10-20: B7 — re-linked after the operator's press; the household cannot restore from off-site (R-1176 proven live)

documentation/audits/rehearsal-2026-10-20/B7-died/05-bind.txt

@@ -0,0 +1,3 @@
+GET 200 Felhom — Doboz összekötése
+Felhom doboz összekötése Kösd össze a most telepített Felhom dobozodat a fiókoddal. Add meg a doboz képernyőjén látható párosító kódot és a tulajdonosi jelmondatodat. Párosító kód A doboz monitorán jelenik meg, a telepítés után. Tulajdonosi jelmondat Az öt szóból álló kifejezés, amelyet a Felhom üzemeltetőjétől kaptál. Ez igazolja, hogy a fiók a tiéd. Összekötés Biztonsági okból 5 sikertelen próbálkozás után a hivatkozás zárolódik. Ilyenkor vedd fel a kapcsolatot az ügyfélszolgálattal. Felhom.eu
+POST answer: Felhom doboz összekötése Sikeres összekötés. A doboz kb. egy percen belül folytatja a telepítést. Ezt az oldalt bezárhatod — a beállítás a háttérben befejeződik, és a vezérlőpultod hamarosan elérhető lesz. Felhom.eu

documentation/audits/rehearsal-2026-10-20/B7-died/06-claim.txt

@@ -0,0 +1,3 @@
+GET /claim HTTP/2 200 
+csrf_len 64 cookie_present yes
+POST /claim HTTP/2 302  location: /

documentation/audits/rehearsal-2026-10-20/B7-died/07-dashboard-after-claim.txt

@@ -0,0 +1,10 @@
+== /launcher 778
+s ↗ A távoli mentés szünetel, amíg nem hozod létre a helyreállítási kódot. Helyreállítási kód létrehozása → Most nem Indítópult tester4.test Indítópult megosztása F Filebrowser Indítópult megosztása Egy megosztható link, amely csak megmutatja az alkalmazásokat és új lapon megnyitja őket. Fiók nélkül, felügyeleti hozzáférés nélkül — minden alkalmazás a saját bejelentkezése mögött marad. A megosztás jelenleg ki van kapcsolva. Bezárás Megosztás bekapcsolása 
+== /backups/remote 1748
+s ↗ A távoli mentés szünetel, amíg nem hozod létre a helyreállítási kódot. Helyreállítási kód létrehozása → Most nem Biztonsági mentés — Távoli mentés tester4.test Aktív — nincs kijelölt alkalmazás Távoli mentés (3. mentés) — titkosított, offsite Az alkalmazás-mentések titkosított másolata egy távoli tárolóra — saját NAS vagy Felhom offsite tárhely — restic + SFTP kapcsolaton. A tároló csak titkosított adatot lát. Ez a 3-2-1 szabály „1 off-site" lába — független a helyi másodpéldánytól és a teljes rendszermentéstől. – Utolsó távoli mentés – Tároló méret · a pillanatképek száma még ismeretlen u629488-sub5@u629488-sub5.your-storagebox.de /home/felhom-repo Helyreállítási kód szükséges A távoli mentések csak akkor állíthatók vissza egy teljes meghibásodás után, ha létrehozod a helyreállítási kódot. Helyreállítási kód létrehozása Távoli mentés most Mely alkalmazások mentődnek a távoli tárolóra? Nincs telepített alkalmazás. Távoli mentési cél beállítása Cél címe (IP vagy hosztnév — NAS vagy SFTP-kiszolgáló) SSH port Felhasználó Tároló útvonala a célgépen Napi automatikus t
+== /recovery 9
+ Found . 
+== /backups/restore
+s ↗ A távoli mentés szünetel, amíg nem hozod létre a helyreállítási kódot. Helyreállítási kód létrehozása → Most nem Biztonsági mentés — Visszaállítás tester4.test Visszaállítás a távoli tárolóból Válaszd ki az alkalmazást, és a következő oldalon döntsd el, mit szeretnél: ellenőrzést külön mappába, csak a hiányzó fájlok visszahozását, vagy teljes visszaállítást. Egyik sem indul el kérdés nélkül. Nem tudjuk elolvasni a távoli tárolót, ezért nem tudjuk, mi van benne . Ez nem azt jelenti, hogy üres — próbáld újra később, vagy jelezd az üzemeltetőnek. Meglévő ellenőrző másolatok Nincs ellenőrző másolat a meghajtón. Hordozható mentéscsomag (.fab) Hordozható pillanatfelvétel — bárhol tárolhatod, és bármikor visszatöltheted egy meghajtóról. A folyamatos védelmet az 1–3. szintű mentés adja. Nincs telepített alkalmazás. 
+== /backups
+s ↗ A távoli mentés szünetel, amíg nem hozod létre a helyreállítási kódot. Helyreállítási kód létrehozása → Most nem Biztonsági mentés tester4.test Csak egy helyi másolat készül: nincs második lemez, vagy ugyanazon a lemezen van, mint az adatok. Ez a véletlen törlés ellen véd, lemezhiba ellen nem. Lemezhiba ellen egy másik lemez vagy a távoli mentés véd. A rendszermentés jelenleg ugyanazon a lemezen van, mint a rendszer — így hibás fájlok ellen véd, lemezhiba ellen nem. Csatlakoztass egy második meghajtót a teljes védelemhez. Tárhely áttekintés Rendszer 0,93 GB / 49,0 GB (2%) Belső tárhely 0,00 GB / 54,8 GB (0%) DB mentések – Rendszermentés (teljes mentés) A teljes szerver — alkalmazások, beállítások és adatbázisok együtt — időszakos mentése, amelyből az egész készülék visszaállítható. Ezt a host-ügynök készíti és kezeli. ✓ Utolsó teljes mentés 2026. 10. 21. 11:20 (15 perce) 634,5 MB Helyi tároló (local) Naprakész Következő mentés az előző 0 órája készült — a mentési időn belül – Visszaállítás ellenőrizve Még nem futott Helyi tároló (local) ✓ Utolsó sikeres mentés: 2026. 10. 21. 11:20 (15 perce) · 634,5 MB Naprakész Biztonsági szerver – külön hardver (PBS) ✓ Utolsó sikeres mentés: 2026. 10. 20. 14:17 (21 órája) Naprakész Mentés most Mentés közben minden alkalmazás rövid időre leáll: kb. 1–1,5 percre, több alkalmazásnál tovább. Utána újra fu

documentation/audits/rehearsal-2026-10-20/B7-died/README.md

@@ -0,0 +1,33 @@
+# B7 — „the box died": destroy, reinstall, re-link, restore from off-site (2026-10-21 08:44–09:40 UTC)
+
+Plan written before: `00-plan.md`. Before-record: the 05:36Z household snapshot (`01-before/`; identical to the evening
+one; only backups ran after it). Method: the household's front doors — the public ISO, the bind page in Chromium, the
+claim page and the dashboard's own requests; the mailbox for links (one-time link and code read once, stored in sec,
+used within minutes).
+
+| UTC | step | observed |
+|---|---|---|
+| 08:44:37 | the box dies | hard stop + `qm destroy 9320 --purge` (`02-kill.txt`) |
+| 08:44:55–08:53 | reinstall from the same ISO | pairing screen after ~5 min (`screens/b7-install/`, code blacked out) |
+| ~08:54 | **the forgotten press** | spent link → „Új linket kérek" → „Kész … néhány percen belül új e-mailt kapsz" — **no mail came** (control: older mails found). Nothing tells the household to ask Felhom to delete the old box → **R-1203 (P2)** |
+| 08:59 → ~09:11 | ask the operator | asked in the terminal; „Host deleted" ~09:11 (intervention 3, ~12 min) |
+| 09:11:21 | new link mail | sent by the hub by itself when the old host was deleted |
+| 09:12:10 | bind | „Sikeres összekötés" (`05-bind.txt`) |
+| 09:13:02 | setup-code mail | „Új beállító kód — újratelepült a szervered" |
+| 09:33 | guest created | host Tester4-648b80 ONLINE, guest 1/1 (~21 min after the bind) |
+| 09:35 | claim | 302 → dashboard (`06-claim.txt`) |
+| 09:36–09:40 | **restore from off-site** | **NOT POSSIBLE for the household** (`07-dashboard-after-claim.txt`) |
+
+**What the rebuilt box shows** (live proof of **R-1176**): the same bar as a brand-new box — „A távoli mentés szünetel,
+amíg nem hozod létre a helyreállítási kódot. Helyreállítási kód létrehozása" — it offers to CREATE a new code, never to
+ENTER the existing one; the product's own sentence for this case („A korábbi házon kívüli mentéseid visszaszerezhetők…")
+is not shown; the restore page says „Nem tudjuk elolvasni a távoli tárolót, ezért nem tudjuk, mi van benne … jelezd
+az üzemeltetőnek". The new code was NOT created: by R-1176 that sets the old history aside on the first night. The box
+is torn down before any night, so the Storage Box repository keeps the household's history untouched.
+
+**Also seen:** the rebuilt box's backups page shows the DEAD box's PBS whole-box copy (2026-10-20 14:17) as its own
+„Utolsó sikeres mentés … Naprakész" — the operator's DR copy, not a household path.
+
+**Verdict: the household cannot bring its data back after a box death, even with the operator's press.** The data is not
+gone (off-site repository and the PBS copy remain), but no front door reaches it. The sha256 compare against the manifest
+could not run. → R-1176 (P1) confirmed live; R-1203 (P2) new.

documentation/audits/rehearsal-2026-10-20/journal.md

@@ -90,3 +90,5 @@
 | 08:44:55–08:53 | B7 | reinstall from the same public ISO, the guide as written; pairing screen after ~5 min. The pairing code was OCR'd into sec (two reads agreed) and blacked out in the screenshot |
 | ~08:54 | B7 | the forgotten press: the spent link → „Új linket kérek" → „Kész … néhány percen belül új e-mailt kapsz"; NO mail in 4 min (control: older mails found) → R-1203 (P2) |
 | 08:59 | B7 | the operator asked (in the terminal) to delete host Tester4-816e21 |
+| ~09:11 | B7 | the operator: „Host deleted" (intervention 3; asked 08:59). The hub mailed the household a new bind link at 09:11:21 by itself |
+| 09:12–09:40 | B7 | bind 09:12, setup mail 09:13, guest 09:33, claim 09:35; the rebuilt dashboard offers only to CREATE a recovery code and cannot read the off-site store → the household cannot restore (R-1176 proven live); the new code NOT created |

documentation/backlog/OPEN-ITEMS.md

@@ -158,7 +158,7 @@ stopping line that lies.
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
 | **R-1175** | Backup & restore | P1 | **Shared folders on two drives that use the same second drive: every night one drive's shares lose their second copy, and the night a drive dies its share copy is deleted.** The shares second-drive job runs one group per source drive; each group's clean-up walks EVERY drive key on the target and removes what is not its own (`internal/backup/tier2_shares.go:201-230`, `:266-267` → `tier2.go:299-329`). PROVEN against production code (fake copy seam): shares on hdd_2 and hdd_3 both target hdd_1 — run 1 logs `removed stale dest dir hdd_2/b_share`, so after EVERY run only the last group's shares have a copy while the status says ok; then hdd_3 goes away — run 2 logs `removed stale dest dir hdd_3/csaladi_fotok`: the dead drive's only second copy is deleted that night. Data-loss hunt 2026-10-20 (C1 reviewer 2, C2, C3: `audits/dataloss-hunt-2026-10-20/C3-D1-shares-tier2/`). | OPEN | — | Fix: reconcile only the group's OWN key dir (and keep a key whose drive is away/decommissioned until the household releases it); test with the C3 cases as red-proofs; release by the normal train | CC |
-| **R-1176** | Backup & restore | P1 | **A box rebuilt after „delete the old host first" cannot reach the household's off-site history: the recovery code is never asked for, a new key is minted, and the old history is set aside on night one.** D19 (`hub/internal/api/passphrase_scope.go:58`) makes the reinstall stop until the operator deletes the old host; the delete demotes the escrow under the OLD host id and deletes the host row (`hub/internal/store/store.go:3164-3229`); the new box's status query joins on live hosts only (`store.go:3499-3527`), so it reports no package and no retained package (both flags false) → the controller mints a fresh repo password (`offbox.go:542-555`), offers no recovery (`:1788`) and on its first night sets the old repository aside (`:905-906`); the retained-package API 404s for a deleted host (`handler.go:1691`). The reset and purge paths DO read `host_deletions` (`customer_reset.go:48,155`; `store.go:4189`) — the status path does not. PROVEN at the hub store against production code (`audits/dataloss-hunt-2026-10-20/C3-K1-escrow-after-host-delete/`: control = present; after delete + new host = nil). 03 §8a and 07 §7.0 promise this route; D19 + R-1154 removed it without a recorded choice. Data-loss hunt 2026-10-20 (C1 reviewer 3). | OPEN | the rehearsal's B7 is the live proof (needs Tester4 off-site ON) | Decide the shape (status query includes `host_deletions` for the same customer, and the retained GET answers for a deleted host of the caller's customer); red-proof with the C3 test; hub + controller release by the normal train | CC |
+| **R-1176** | Backup & restore | P1 | **A box rebuilt after „delete the old host first" cannot reach the household's off-site history: the recovery code is never asked for, a new key is minted, and the old history is set aside on night one.** D19 (`hub/internal/api/passphrase_scope.go:58`) makes the reinstall stop until the operator deletes the old host; the delete demotes the escrow under the OLD host id and deletes the host row (`hub/internal/store/store.go:3164-3229`); the new box's status query joins on live hosts only (`store.go:3499-3527`), so it reports no package and no retained package (both flags false) → the controller mints a fresh repo password (`offbox.go:542-555`), offers no recovery (`:1788`) and on its first night sets the old repository aside (`:905-906`); the retained-package API 404s for a deleted host (`handler.go:1691`). The reset and purge paths DO read `host_deletions` (`customer_reset.go:48,155`; `store.go:4189`) — the status path does not. PROVEN at the hub store against production code (`audits/dataloss-hunt-2026-10-20/C3-K1-escrow-after-host-delete/`: control = present; after delete + new host = nil). 03 §8a and 07 §7.0 promise this route; D19 + R-1154 removed it without a recorded choice. Data-loss hunt 2026-10-20 (C1 reviewer 3). | OPEN **PROVEN LIVE 2026-10-21 (rehearsal B7, `audits/rehearsal-2026-10-20/B7-died/README.md`):** Tester4 destroyed, reinstalled from the public ISO, the operator deleted the old host, bind + claim OK; the rebuilt dashboard shows only „A távoli mentés szünetel, amíg nem hozod létre a helyreállítási kódot. Helyreállítási kód létrehozása" (create, never enter), not the product's „A korábbi házon kívüli mentéseid visszaszerezhetők" sentence; the restore page: „Nem tudjuk elolvasni a távoli tárolót". The new code was not created and the box was torn down before its first night, so the history stays on the Storage Box. | the rehearsal's B7 is the live proof (needs Tester4 off-site ON) | Decide the shape (status query includes `host_deletions` for the same customer, and the retained GET answers for a deleted host of the caller's customer); red-proof with the C3 test; hub + controller release by the normal train | CC |
 | **R-232** | Backup & restore | P2 | **DooPlex's backup makes every copy inside the same box — and nothing tells anyone when it fails.** Surveyed read-only 2026-08-06 (`audits/RECON-dooplex-backup-2026-08-06.md`). **What works:** five sets, 14/14 successful runs in 14 days; a file was restored from the `data` repo and matched the live original **byte for byte**; every set except two is cross-disk; k3s is integrity-checked on every run. **What the matrix exposes, ranked:** (a) **`notify_failure` is a no-op** — `NOTIFY_ON_FAILURE=true` but `NOTIFY_WEBHOOK_URL` is commented out, so a failed backup notifies **nobody**; the project already has a working Resend path that CI uses. Cheapest item, and it makes every other failure visible. (b) **Nothing leaves the box** — no rclone, no remote repo, no off-site target anywhere; Longhorn's target is `nfs://192.168.0.180:` pointing at DooPlex itself, and the only outbound-looking cron pulls *inbound* from Hetzner for a different project. The machine that runs the hub managing the customers' off-site chain has no off-site copy of its own. (c) **The backup tree is a single writable path** and the restic repos are not append-only — one bad script or ransomware destroys every copy at once. (d) **Two same-disk sets**: `.claude-memory` and the PostgreSQL dumps, whose source directory sits *inside* the backup tree. (e) **Longhorn `retain=1`** — one generation per volume, so a corruption noticed a day late has no earlier copy. (f) **`/opt/backup/docs/BACKUP-RESTORE.md` does not exist** though the systemd unit advertises it. (g) **`secrets/restic-repo` has never held a snapshot** — `backup-secrets.sh` contains no `restic` call; the secrets are GPG files on `sda1` only. (h) **No restore has ever been run** beyond today's single-file probe — the matrix's "ever demonstrated?" column is otherwise entirely empty. **Not a finding:** the restic passphrase. The on-box copy is on `sdb1`, a different disk from the backups, and the **operator holds an offline copy out of band** — so a disk loss is recoverable. The narrow residual is that it is operator-held rather than system-held, unlike the customer case's hub-vaulted escrow, so it should be confirmed current and findable by someone else. **Nothing was changed by the recon.** | **NARROWED 2026-10-08 — owner Viktor.** (a) DONE with the operator's yes in chat: `notify_failure` now mails admin@felhom.eu through Resend; proven by one test mail that reached the inbox (`audits/day-2026-10-08/r232/`; no backup was started). (b) partly: the hub database leaves DooPlex nightly to ep0 (R-173); everything else stays on the box. (c)–(h) unchanged. **2026-10-09 (operator yes in chat): (b) and (h) DONE for Gitea and the secrets.** Nightly 00:20 `felhom-dooplex-offsite` pushes Gitea (614 MB of repositories, the `gitea` database dump taken first, `app.ini`) and the nightly GPG secrets export, encrypted with a NEW key, to ep0 `operator` (`host/dooplex-gitea`) on the hub-DB write-only token — no change on ep0; first push 57 s; restore test weekly (manifest, `git fsck` every repo, `pg_restore --list`), run once: 27 805 files, 10 repos OK; `DooplexGiteaOffsiteStale`/`DooplexGiteaRestoreTestStale` in force (`absent()` included, promtool + 2 red-proofs); every backup unit now mails on failure (`OnFailure=`, dry run reached the inbox). **First real Gitea restore:** into a throwaway on the bench, 10/10 repos, product `main` = live, a file byte for byte, a login (`runbooks/gitea-restore.md`). The hub came back too (R-173, closed). **The registry is left out on purpose:** its images rebuild from the code. **Correction to the recon §7:** the API mirror holds ONE repo (`homelab-manifests`), not all — Gitea had a single copy, on DooPlex. `audits/dooplex-survival-2026-10-09/` **LEFT:** (c) append-only for the on-box tree, (d)–(g) as written; the paper copy of the new key (operator). **READY** for the rest | — | — | operator |
 | **R-895** | Backup & restore | P2 | **The hub's clean-up-window check trusts the snapshot counts the box sends, so a broken-into box (or past-dated fakes added through the add-only key) can shrink the real off-site history without an alarm.** READ 2026-10-06 night in source (R-822's design): the before/after comparison uses counts the box itself reports (`hub/internal/offsitekeys/service.go:284`, `:343`); new fakes keep the count level. Decision 68 already accepts a box-trusted count. | **NARROWED 2026-10-18 — slice 2 built, not deployed** (hub v0.158.0): the 07:10 audit compares the `snapshots/` listing with yesterday's by name — files gone that no clean-up window explains → `offsite_snapshots_vanished` (error), files no reported box run explains → `offsite_snapshots_unexplained` (warning); operator-only, hub-origin (not in `allowedEventTypes`). First listing = baseline; a move-aside or RESET drops it; vanishing judged on a pinned tier only; files younger than 1 h judged the next day. Red-proofed twice (`snapshot_watch_test.go`). **Not built:** the window's before/after on the hub's own count (design §3.1) and refusing a window while unexplained files exist (§3.2). Unmeasured: the provider shell's `ls -l` time zone (± 3 h slack until then). **Deployed 2026-10-18 12:23 (hub v0.158.0).** The first listing (tomorrow 07:10) is a baseline; the first verdict is 2026-10-20 07:10. **Known limit (independent review 2026-10-18, F2):** `_unexplained` is explained by the box's OWN reported `offsite.last_run`, so whoever holds a box's key can silence it; `_vanished` is judged on the hub's own listing and is not affected. **2026-10-20 03:5x UTC: due date moved 10-20 → 10-21** — the first verdict lands at 07:10 hub time (05:10 UTC), after the night session that reached it ended; that brief did not include this read. Nothing is lost: the hub keeps its listing; read the 2026-10-20 07:10 lines on 10-21 (or later on 10-20). **FIRST VERDICT READ 2026-10-21 (rehearsal session, read-only):** the 07:10 CEST run judged three tiers against yesterday — demo-felhom 5, demo-hp 14, tester-1 4 new snapshot files, „each inside a reported box run"; nothing vanished, nothing unexplained; Tester4 stored its first baseline (44 files); Tester-2's own listing failed (no `snapshots/` dir — the box has never pushed). No `offsite_snapshots_vanished` / `_unexplained` mail in the operator mailbox (second channel). Evidence `audits/due-checks-2026-10-21/R-895-hub-0710.txt`. The positive alarm path is still proven only by the unit red-proofs. | the operator's yes to deploy hub v0.158.0 | Deploy; read the first 07:10 lines (newest upload vs the box's last run = the time-zone calibration), shrink the slack; then decide §3.1/§3.2 | CC |
 | **R-923** | Backup & restore | P2 | **The password manager (Vaultwarden) runs on DooPlex and holds the keys that open DooPlex's own off-site copies — a loss of DooPlex loses the keys with the copies.** Operator finding, 2026-10-09 11:19: "the password manager also runs on DooPlex (Vaultwarden), so I probably need to back up its DB somewhere too." Measured: `vaultwarden-system/vaultwarden` (1.37.3), PVC `vaultwarden-data` 7.4 MB (`db.sqlite3` + WAL, `rsa_key.pem`, no attachments); its only backups were Longhorn on DooPlex. **A key in a password manager that runs on DooPlex is not off DooPlex** — R-173's "both keys are off DooPlex" (2026-10-05) and the hub-DB runbook Step 0 were wrong in that sense. Also seen: Vaultwarden warns its `ADMIN_TOKEN` is plain text (an Argon2 hash is its guidance). | **NARROWED 2026-10-09 (operator yes in chat):** Vaultwarden's own `backup` copy + `rsa_key.pem` ride the nightly encrypted DooPlex→ep0 job (`scripts/dooplex-offsite/`), checked on every push and every Sunday (integrity, users, items — rows only). That copy opens only with the DooPlex off-site key, and the vault only with the master password — so the circle is broken only once both are on paper outside DooPlex: `runbooks/break-glass-sheet.md`, `runbooks/total-loss-of-dooplex.md`. **LEFT:** the operator prints the sheet and stores it away from home; the `ADMIN_TOKEN` hash (a DooPlex change, operator's word). | operator: print the sheet | Operator: print + store the sheet; then close | operator |