Data-loss hunt 2026-10-20: four reviewers' candidates, C2 verdicts, proofs (D1 shares second copy pruned, K1 escrow unreachable after host delete, R5 whole restore leaves app stopped while saying untouched, T9 hub jobs run twice on 25 Oct); rehearsal A5 manifest, B1 notes, B9 words, tools

2026-10-20 12:13:19 +02:00
parent c701b2530b
commit 43e5a5496d
17 changed files with 3973 additions and 0 deletions
@@ -0,0 +1,14 @@
+# C1 reviewer 2 — delete and retention (fresh, read-only; clones at controller 7ca59e2)
+
+| # | Sev guess | harm | mechanism |
+|---|---|---|---|
+| D1 | P1 (mech high, reach medium) | Shares second-drive copy: one drive group's reconcile deletes ANOTHER drive's share mirrors (on the night a drive is decommissioned/away; or every night when two groups target one drive, then a full re-copy) | `internal/backup/tier2_shares.go:135` groups by source; `:201-202` own destBase/legRels; `:230` `tier2ReconcileRoots(destBase, sharesTier2DestRoots(destBase), legRels)`; `:266-267` ReadDir lists every drive key on disk; `tier2.go:299-329` removes any child not in this group's legs; `shares_payload.go:233-251` away/decommissioned drives drop out |
+| D2 | P1 optional legs / P2 mandatory; design question with a defect corner | A missing or emptied app folder on the drive deletes its second-drive copy within one night; optional legs (audiobooks, comics, ROMs, Immich external library) are in no other tier | `tier2_capture.go:55-61` missing leg skipped (optional silently) → `tier2.go:455` reconcile deletes its mirror; run recorded success `:465`; emptied → `rsync -a --delete` `tier2.go:751` via `:443`; catalog optional legs `audiobookshelf/.felhom.yml:24-26`, komga, romm, immich `:32-34`; 07 §8.2 „data legs are NOT guarded" covers a leg leaving the class set, not a declared-but-missing leg |
+| D3 | P2 high (mech) | A host clock ahead during one whole-guest backup silences that tier until real time passes it; the hub reads the same clock and says OK | agent `internal/localapi/server.go:1178-1190` negative age never due; `internal/backup/runner.go:607-629` newest ctime unbounded; hub `monitor/deadline_tiers.go:155`, `deadline.go:243`; R-973 cap counts future entries `backup_cap.go:61`; restic: `offbox_window.go:133-136` pinned guard refuses every window → quota fills; NAS tier no guard `:209-216`; `offbox_abandon.go:248,287` set-aside rm on box clock |
+| D4 | P2 | A failed LOCAL unit restore leaves the app running on mixed data, no undo copy (same as R1 of reviewer 1 — independent agreement) | `restore_unit.go:430-489`; `writeSafetyDump` only in `offbox_reconstitute.go`; 07 §6.3 ladder not scoped to off-site |
+| D5 | P2 order defect / depth design | A dump that fails its own content check has ALREADY replaced last night's good dump; Tier-2 mirrors it an hour later | `appbackup/dbdump.go:388-397` size-only check, `:399` rename, `:419` `ValidateDump` after; `backup.go:688-690`; volume tars promoted with no content check `backup.go:902` |
+| D6 | P3 med | „Megőrzött adatok" can list a RUNNING app's live folder as deletable leftover if its app.yaml is unreadable | `stacks/manager.go:672`; `kept.go:220-235` skips `!Deployed || AppConfig==nil`; `ListKept :330-358`; `DeleteKept :385-402` RemoveAll; `deploy.go:1180-1190` write+rename without fsync |
+| D7 | P3 low-med | A unit restore can point the app back at an OLD drive; F5 then deletes the current drive's unit | `restore_unit.go:90-111, 507`; `cmd/controller/main.go:3252`; `deploy.go:820`; `backup.go:1517-1574` `pruneStalePrimaryDirs` even when this run's capture failed (`:726-733`) |
+| D8 | P3/P4 design | local whole-guest keep=1 prunes the only good archive before the new one is proven | agent `runner.go:229`, `:139-161`, restore-test settle 24 h `:462` |
+
+**Checked SAFE (reviewer):** hub off-site deletions by exact identity (`offsitekeys.go:366-515`); pinned off-site retention guard refuses future/too-new snapshots and big plans (`offbox_window.go:132-161`); dump and tar writes crash-safe (tmp+fsync+rename); R-403 hollow-unit guard; drive migration deletes source only after verify (`stacks/migrate.go:614-667`); app removal refusals and protected folders (`delete.go:136-153, 241-301, 839-846`); mirror/undo clean-ups allow-listed/prefix-guarded; agent prune never on PBS unless opted in; restore-test skips unsettled/future/foreign-key archives; `ForgetRetiredDrives` mount units only; SMB share removal deletes no folder. Not reviewed: agent `vzpartial` janitor, tenantsync/ep0 prune config, hub RESET beyond `PurgeRepos`.
@@ -0,0 +1,40 @@
+# C1 reviewer 4 — time and concurrency (fresh, read-only; controller 7ca59e2, agent 4e1621c, felhom.eu 5c48731)
+
+## (A) Schedulers and what each does on Sunday 2026-10-25 (03:00 CEST → 02:00 CET; 02:00–02:59 happens twice)
+
+| owner | job | schedule | zone | file:line | 2026-10-25 | after 3 days off |
+|---|---|---|---|---|---|---|
+| controller | db-dump | daily W (default 02:30) | Budapest (`time.Date`, day+1) | main.go:1262, scheduler.go:499-518 | once, at the SECOND 02:30 (CET = 01:30Z) — Go normalises the ambiguous time to the later offset | not replayed by the scheduler; the catch-up runs it once 15 min after start (nightchain.go:277-345) |
+| controller | tier2-backup | W+60m | Budapest | main.go:1352 | once, 03:30 CET | once, by the catch-up |
+| controller | offbox-backup + app-update leg | W+105m | Budapest | main.go:1519 | once, 04:15 CET | catch-up runs the off-site leg only |
+| controller | offsite-abandon-sweep 05:10, offsite-proof 05:30, offsite-integrity 06:00 | fixed | Budapest | main.go:1561, 1613, 1591 | once | **no catch-up** (finding T6) |
+| controller | metrics-prune 04:00, fill-watch 03:30, asset-sync 05:00, selfupdate-auto 04:30 (off by default) | fixed | Budapest | main.go:1621,1671,1768,1741 | once | floor update runs on any hub reply |
+| controller | whole-guest quiesce loop | poll 5 min; gate [W+2h, W+6h) by wall minute; valve when age > cadence+24h | Budapest wall | quiesce.go:150, 191-205, 790-818 | gate 04:30–08:30 CET (4 h real) | valve fires ~5 min after start, any time of day |
+| controller | ~20 `Every` tickers | monotonic | n/a | main.go:898-2146 | unaffected | restart at boot |
+| agent | whole-guest due | age ≥ cadence | UTC | localapi/server.go:1182-1192 | unaffected | due at once |
+| agent | restore-test | 30 min after start, then 6 h | UTC | backup/schedule.go:148-177 | unaffected | after 30 min |
+| agent | OS leg (+ kernel reboot if hub „Tonight") | after any non-manual primary backup +90 s, 20 h min gap | no wall clock | server.go:982-986, main.go:1272-1280, leg.go:655-664, kernel.go:101-114 | follows the backup | follows the valve backup = daytime (T8) |
+| agent | fstrim | hourly; due Wed 10:00, only 10:00–20:59 | host local | fstrim/fstrim.go:39-50,110 | n/a | next eligible hour |
+| hub | scheduleDaily: db-snapshot 02:00 (keep 2), prune 04:30, deadline-check 05:00, offsite-key-audit 07:10 | `next.Add(24*time.Hour)` | Budapest | hub cmd/hub/main.go:1049-1072 | **every job runs TWICE** (T9) | n/a |
+| host timers | mgmt-watchdog, crash-guard-check | monotonic | n/a | felhom-agent/configs/*.timer | unaffected | after boot |
+| DooPlex timers | hub-db-backup 02:30; dooplex-offsite 00:20; hub-db-restore-test Sun 04:30; offsite-restore-test Sun 05:30; ep0-copy-gc 08:00 | OnCalendar local | DooPlex | felhom.eu/scripts/*/*.timer | Oct 25 is a Sunday: both weekly restore tests run on the DST day; 02:30 is in the repeated hour (systemd behaviour not verified) | Persistent=true |
+
+## (B) Candidates
+
+| # | Sev guess | harm | mechanism |
+|---|---|---|---|
+| T1 | P2 med-high | The nightly second-drive mirror (`rsync --delete`) can run DURING a restore from that same mirror and delete the files not yet copied back | `tier2.go:479-520` RunAllTier2 checks held/version/disconnected only; `:443` mirror, `:744-751` `--delete`; restore `tier2_whole.go:301` takes only `acquireRunning`; Tier-2 leg takes only `LegLock` (main.go:1348-1352); `update_guard.go:595-601` isHeld ignores restores |
+| T2 | P2 if true; low-med | The Debian 13 controller image may lack tzdata → every schedule on UTC silently | `Dockerfile:77-100` debian:13.7-slim, no tzdata, no `time/tzdata` import; fallbacks scheduler.go:19-23 (logs), quiesce.go:749-751 (silent), funcmap.go:62-66 |
+| T3 | P3 med | Whole-guest start creeps ~5 min/night until a night is skipped silently | agent `server.go:1184` `age >= Cadence` no slack; age from archive start; controller poll 5 min `quiesce.go:150`; gate refusal `:807-818` |
+| T4 | P3 med-high | The whole-guest quiesce stops apps in the middle of a restore or an update | `quiesce.go:213-275` no `IsRunning()`/restore check; `RunningAppStacks` (stacks/manager.go:620-636); `TriggerNow` (:419-445) same; `DumpAppVolumesSafe` (backup.go:1001-1045) can restart an app quiesce stopped |
+| T5 | P3 high | The nightly off-site copy is skipped SILENTLY when anything else holds the backup flag, and the catch-up never retries | `offbox.go:981-996` INFO „skipped — another backup is running" return nil; `withLeg` MarkEnded unconditionally (main.go:1246); proof 05:30 / integrity 06:00 take the same flag |
+| T6 | P3 high | Off-site check and proof never run on a box off at 05:30–06:00; the code comment says the opposite | main.go:1575-1577 comment vs 1591-1619; `catchUpLegs` nightchain.go:17-18; no hub alarm on integrity/proof age |
+| T7 | P3 high | Hub raises a false „no DB dump" error every morning for W after ~04:45 or evening W | hub `monitor/deadline.go:416-419` midnight Budapest; `:556-565` → `expected_dbdump_missed`; hub has no idea of W |
+| T8 | P3 design | The OS leg and kernel reboot follow ANY scheduled backup, including the daytime valve after power-on | agent server.go:982-986; main.go:1279 labels „night"; kernel.go:101-114; hub osupdates/kernel.go:46-48,181-185 |
+| T9 | P4 high | Hub daily jobs run TWICE on 25 October (db-snapshot keep-2 prunes a day early; duplicate deadline alarms); once an hour late in spring | hub cmd/hub/main.go:1058-1062 `next.Add(24*time.Hour)`; the controller fixed this (scheduler.go:511-516) |
+| T10 | P4 high | Spring forward 2027-03-28: db-dump and tier2 fire at the same instant; tier2 may mirror yesterday's unit | scheduler normalisation; `LegLock` main.go:1237 plain mutex |
+| T11 | P4 design | A clock ahead stops whole-guest backups silently (same as D3) | server.go:1182-1192; quiesce.go:816-817; deadline.go:244; backup_cap.go:61 |
+| T12 | P4 high | self-update busy check before the image pull, swap after it with no re-check | updater.go:532-545, :764 |
+| T13 | P4 med | double-clicking a restore shows it as failed while it runs | handlers.go:1853, :1874 `BeginRestoreOp` after the check; second POST's `EndRestoreOp(false)` |
+
+**Checked SAFE (reviewer):** controller DST day+1 arithmetic and late-fire skip (scheduler.go:342-348, 511-516); night ledger persisted, seeded, one catch-up for many nights; quiesce marker persisted and recovered; in-memory backup flag (no stale lock); flock-based OS-leg locks; restic stale locks self-heal; two restores blocked by `m.running`; Remove refused during restore/quiesce; catch-up and whole-guest wait for each other; agent due/cap UTC durations; off-site prune guard refuses future snapshots; host power-off re-checks; no cron/OnCalendar in the installer; hub staleness uses the hub's own receive clock.
@@ -0,0 +1,22 @@
+# C2 register search, 2026-10-20T10:11:13Z, OPEN-ITEMS.md + CLOSED-ITEMS.md at c701b253; grep -ciE
+1 open / 2 closed : R-893
+0 open / 0 closed : tier2_shares|_shares.*tier|shares.*second.drive|Megoszt.*2\. ment
+0 open / 2 closed : host_deletions|retained.*delet|deleted host|escrow.*reinstall
+2 open / 7 closed : scheduleDaily|DST|daylight|oraallit|time.?zone
+1 open / 8 closed : restore-undo|undo copy|safety dump|unit restore.*fail
+1 open / 0 closed : ValidateDump|dump.*replac.*valid
+0 open / 0 closed : SharesScratchReady|shares.*scratch|shares restore
+0 open / 4 closed : tier2_whole|whole restore|Teljes vissza
+0 open / 3 closed : optional leg|missing on disk|--delete
+0 open / 0 closed : PurgeSuperseded|abandon.*purge
+0 open / 1 closed : future.dated|clock ahead|clock.*future|negative age
+0 open / 0 closed : another backup is running
+1 open / 2 closed : expected_dbdump_missed
+2 open / 1 closed : symlink
+0 open / 1 closed : family.gate.*fab|fab.*family|\.fab
+0 open / 0 closed : felhom-restoring
+0 open / 0 closed : quiesce.*restore|restore.*quiesce
+1 open / 4 closed : offsite-proof|integrity.*catch|proof.*catch
+0 open / 0 closed : double.click|dupla
+0 open / 1 closed : disconnected.*target|away.*target
+Every non-zero hit was read: none describes the candidate (positive control R-893 found in both).
@@ -0,0 +1,51 @@
+# C2 — the main session's verdict on every candidate (2026-10-20)
+
+Method: each candidate's `file:line` read in LIVE source (controller `7ca59e2` = v0.336.0, agent `4e1621c`, felhom.eu
+`5c48731`+); then (a) the register searched, open AND closed, with ASCII fragments — positive control `R-893` found in
+both; (b) the design test against `07-backup-architecture.md` and `CONTEXT.md`. Verdict words:
+
+- **CONFIRMED** — the mechanism read true in live source AND a proof ran (unit test against production code, or live).
+- **CONFIRMED-CODE** — the mechanism read true in live source; no proof ran yet.
+- **PLAUSIBLE** — the reviewer's reading, spot-checked only.
+- **NOT-A-FINDING** — with the reason.
+
+Register: none of the candidates below is in OPEN-ITEMS or CLOSED-ITEMS (searches in `C2-register-search.txt`).
+
+| id | verdict | design or defect | note |
+|---|---|---|---|
+| **D1** shares second-drive copy pruned by another drive's group | **CONFIRMED** (unit test, production code, fake mirror seam — `C3-D1-shares-tier2/`) | defect | **Worse than reported:** with shares on two drives that pick the same target, EVERY night the second group's reconcile deletes the first group's copy (`removed stale dest dir hdd_2/b_share`) — one drive's shares never have a second copy while the run says ok. And the night a drive goes away, its copy is deleted (`removed stale dest dir hdd_3/csaladi_fotok`). |
+| D1b an AWAY drive is still chosen as the second-drive target | CONFIRMED-CODE (+ the same unit test: `hdd_3/iskola => hdd_1/...` after hdd_1 was marked disconnected) | defect | `tier2.go:151-180` loops `GetSchedulableStoragePaths()` (`settings.go:1593-1603` excludes only Decommissioned); no Disconnected check. On a real box the away drive's empty mount point is on the guest's root disk |
+| **K1** reinstall after „delete the old host" loses the route to the old escrow | **CONFIRMED** at the hub store (unit test against production code, `C3-K1-escrow-after-host-delete/`); the controller half (new key minted, history set aside on night one) CONFIRMED-CODE | defect (03 §8a / 07 §7.0 promise the route; D19+R-1154 removed it unrecorded) | hub `store.go:3164-3229` demotes into `host_escrow_superseded` under the OLD host_id and deletes the `hosts` row; `GetEscrowStatusForCustomer` `store.go:3499-3527` joins on `hosts` → nil for the new box; `customer_reset.go:48,155` and `store.go:4189` DO include `host_deletions`. Live proof needs Tester4 off-site ON (B7) |
+| K2 abandoning one set-aside copy purges every retained package | PLAUSIBLE | defect (breaks `store.go:4176-4179`'s own invariant) | `store.go:4185-4190` customer-wide delete read true |
+| **R1/D4** a failed LOCAL unit restore: no copy of the live state, app started anyway | CONFIRMED-CODE → live proof running (`C3-R1-local-restore-9202/`) | **design question** for the operator: decision 154 (R-638 A) chose „start anyway"; D20 (R-893 A) gave „put back exactly" to the OFF-SITE path only | `restore_unit.go:428-489` |
+| R2 a crash mid-restore starts the app on a mixed state; the undo copy is never read at start | PLAUSIBLE | defect | `appstop_marker.go:307`, `offbox_restore_undo.go:137` |
+| R3 a cut-off shares restore can be placed and reported restored | CONFIRMED-CODE | defect (R-358 class, fixed for apps, not shares) | `shares_restore.go:131-140` readiness = non-empty dir |
+| R4 `.fab` export drops a drive folder holding a symlink | PLAUSIBLE | defect | `appexport/fabplan.go:292`, `export.go:714` |
+| R5 second-drive whole restore: files swapped, then the unit half refuses → app left STOPPED, text says „untouched" | **CONFIRMED** (unit test, production code: `C3-R5-whole-restore/` — 1 file restored + 1 replaced, app stopped, no start, message „az alkalmazás és az adatai érintetlenek") | defect | `tier2_whole.go:301-340`: stop + merge, then `RestoreFromRecoveryUnitAtWith` whose early refusals return before its own stop/start; `hu.json` `err.backup.unit_versions_mixed` „Az alkalmazás érintetlen" |
+| R6 `.fab` import into a stopped app merges volumes | PLAUSIBLE | defect | `appexport/restore.go:361-374` |
+| R7 `.fab` import has no guard / undo / record | PLAUSIBLE | defect | |
+| R8 `.fab` import drops the family gate | PLAUSIBLE | defect | `cmd/controller/main.go:3517-3530`, `stacks/family_gate.go:173-190` |
+| R9–R11, R13, R14 | PLAUSIBLE | defect / design (R13) | low |
+| R12 interrupted whole restore leaves `*.felhom-restoring`, every retry fails | CONFIRMED-CODE | defect | `tier2_whole.go:86-87` O_EXCL; removed only on an in-process error |
+| **D2** a missing / emptied app folder deletes its second-drive copy within one night | CONFIRMED-CODE | defect-shaped corner of a choice: 07 §8.2 „data legs are NOT guarded" is about a leg LEAVING the class set; this is a declared leg that is MISSING, which the code itself logs as a gap (`tier2_capture.go:55-61`) and then reconciles away (`tier2.go:454`) | P1 for optional legs (no other tier holds them) |
+| D3 / T11 a host clock ahead silences the whole-guest tier; the hub reads the same clock | PLAUSIBLE (B6 #8 on the rehearsal VM is its live test) | defect | agent `localapi/server.go:1178-1190` |
+| D5 a dump that fails its own check has already replaced the good one | CONFIRMED-CODE | defect for the ORDER | `appbackup/dbdump.go:388-422`: size check, rename (`:399`), then `ValidateDump` |
+| D6 kept-data page can list a running app's folder | PLAUSIBLE | defect | needs an unreadable app.yaml |
+| D7 unit restore can point the app at an old drive | PLAUSIBLE (low) | defect | |
+| D8 local whole-guest keep=1 | design (07 §6.1 R-684 chose it) | design | **NOT-A-FINDING: the operator chose keep=1 for demo-hp; recorded in 07** |
+| T1 the nightly mirror (`rsync --delete`) can run during a restore from that mirror | CONFIRMED-CODE | defect | `RunAllTier2` (`tier2.go:479-520`) takes no backup flag; the whole restore holds only `acquireRunning` (`tier2_whole.go:301`) |
+| T2 controller image without tzdata → UTC schedules | **NOT-A-FINDING:** measured on 9202 (controller 0.331.0, Debian 13 base): `/usr/share/zoneinfo/Europe/Budapest` present, scheduler logs „scheduled for … CEST" | — | |
+| T3 whole-guest start creeps ~5 min/night | PLAUSIBLE | defect | needs the agent journal over weeks |
+| T4 whole-guest quiesce stops apps mid-restore | PLAUSIBLE | defect | `quiesce.go:213-275` |
+| T5 nightly off-site copy skipped SILENTLY when the flag is held; catch-up never retries | CONFIRMED-CODE | defect (the code chose nil for the nightly path; the catch-up gap is not chosen) | `offbox.go:981-996`, `main.go:1246` |
+| T6 proof / integrity never run on a box off at 05:30–06:00 | PLAUSIBLE | defect (comment `main.go:1575-1577` says the opposite) | |
+| T7 hub false „no DB dump" for late or evening W | PLAUSIBLE | defect | hub `monitor/deadline.go:416-419, 556-565` |
+| T8 the OS leg follows a daytime valve backup | design question | design | 07 §6.1 / 11 §8.1 set no time bound |
+| **T9** hub daily jobs run TWICE on 2026-10-25 | **CONFIRMED** (replay of the hub's exact arithmetic, `C3-T9-hub-dst/replay.txt`) | defect | deadline-check fires 04:00 CET AND 05:00 CET; db-snapshot twice (keep-2 → the previous day's copy pruned a day early). **In five days, on the live hub.** |
+| T10 spring forward collision | PLAUSIBLE | defect (P4) | |
+| T12, T13 | PLAUSIBLE | defect (P4) | |
+| K3–K8 | PLAUSIBLE | K6 design | |
+
+**NFC/NFD (the brief's question for A.1.4):** reviewer 1 found no normalisation anywhere — rsync, restic and the Go
+merge compare names byte for byte, so the missing-only compare sees **two names** for the NFC/NFD pair. The household
+data carries the pair; B8 measures it live.
@@ -0,0 +1,26 @@
+# go test at controller main 7ca59e2 (v0.336.0), scratch clone, fake mirror seam (no rsync/docker); 2026-10-20T10:04:31Z
+=== RUN   TestRehearsalD1_DeadDriveShareCopyIsPruned
+    zz_rehearsal_d1_test.go:46: drive keys: map[hdd_1:hdd_1 hdd_2:hdd_2 hdd_3:hdd_3]
+    zz_rehearsal_d1_test.go:47: POSITIVE CONTROL — copies after run 1 (both drives present): [hdd_1: hdd_3/iskola hdd_2: hdd_1/csaladi_fotok]
+    zz_rehearsal_d1_test.go:48: mirror calls run 1: [hdd_1/csaladi_fotok=>hdd_2/backups/secondary/_shares/hdd_1/csaladi_fotok data/shares-payload=>hdd_2/backups/secondary/_shares/_payload hdd_3/iskola=>hdd_1/backups/secondary/_shares/hdd_3/iskola data/shares-payload=>hdd_1/backups/secondary/_shares/_payload]
+    zz_rehearsal_d1_test.go:56: FAULT — copies after run 2 (hdd_1 away): [hdd_1: hdd_3/iskola hdd_2: hdd_1/csaladi_fotok]
+    zz_rehearsal_d1_test.go:57: mirror calls run 2: [hdd_3/iskola=>hdd_1/backups/secondary/_shares/hdd_3/iskola data/shares-payload=>hdd_1/backups/secondary/_shares/_payload]
+--- PASS: TestRehearsalD1_DeadDriveShareCopyIsPruned (0.02s)
+=== RUN   TestRehearsalD1_TwoGroupsOneTarget
+    zz_rehearsal_d1_test.go:73: copies after two runs: [hdd_1: hdd_3/c_share hdd_2: hdd_1/a_share]
+    zz_rehearsal_d1_test.go:75: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_1 → hdd_2/backups/secondary/_shares (18 B)
+    zz_rehearsal_d1_test.go:75: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_3 → hdd_1/backups/secondary/_shares (36 B)
+    zz_rehearsal_d1_test.go:75: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_1 → hdd_2/backups/secondary/_shares (18 B)
+    zz_rehearsal_d1_test.go:75: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_3 → hdd_1/backups/secondary/_shares (36 B)
+--- PASS: TestRehearsalD1_TwoGroupsOneTarget (0.01s)
+=== RUN   TestRehearsalD1_SameTargetThenDriveDies
+    zz_rehearsal_d1_test.go:89: POSITIVE CONTROL run 1: [hdd_1: hdd_3/csaladi_fotok]
+    zz_rehearsal_d1_test.go:91:   LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_2 → hdd_1/backups/secondary/_shares (18 B)
+    zz_rehearsal_d1_test.go:91:   LOG [INFO] [backup] Tier 2 reconcile: removed stale dest dir hdd_2/b_share
+    zz_rehearsal_d1_test.go:91:   LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_3 → hdd_1/backups/secondary/_shares (42 B)
+    zz_rehearsal_d1_test.go:100: FAULT run 2 (hdd_3 away): [hdd_1: hdd_2/b_share]
+    zz_rehearsal_d1_test.go:102:   LOG [INFO] [backup] Tier 2 reconcile: removed stale dest dir hdd_3/csaladi_fotok
+    zz_rehearsal_d1_test.go:102:   LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_2 → hdd_1/backups/secondary/_shares (18 B)
+--- PASS: TestRehearsalD1_SameTargetThenDriveDies (0.01s)
+PASS
+ok  	gitea.dooplex.hu/admin/felhom-controller/internal/backup	0.049s
@@ -0,0 +1,104 @@
+package backup
+
+// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate D1). Never committed.
+import (
+	"os"
+	"path/filepath"
+	"sort"
+	"testing"
+)
+
+func listShareCopies(env *sharesEnv) []string {
+	var out []string
+	for name, d := range env.drives {
+		base := filepath.Join(NamespaceRoot(d, true), "backups", "secondary", SharesPseudoStack)
+		keys, _ := os.ReadDir(base)
+		for _, k := range keys {
+			if !k.IsDir() || k.Name() == filepath.Base(sharesPayloadDestRel) {
+				continue
+			}
+			subs, _ := os.ReadDir(filepath.Join(base, k.Name()))
+			for _, s := range subs {
+				out = append(out, name+": "+k.Name()+"/"+s.Name())
+			}
+		}
+	}
+	sort.Strings(out)
+	return out
+}
+
+func keyOwner(env *sharesEnv) map[string]string {
+	m := map[string]string{}
+	for name, d := range env.drives {
+		m[sharesDriveKey(d)] = name
+	}
+	return m
+}
+
+// A drive whose shares were mirrored goes away; another share group's run lands on the same target.
+func TestRehearsalD1_DeadDriveShareCopyIsPruned(t *testing.T) {
+	env := newSharesEnv(t, "hdd_1", "hdd_2", "hdd_3")
+	env.addShare(t, "hdd_1", "csaladi_fotok", true)
+	env.addShare(t, "hdd_3", "iskola", true)
+	if err := env.m.RunSharesTier2(); err != nil {
+		t.Fatal(err)
+	}
+	t.Logf("drive keys: %v", keyOwner(env))
+	t.Logf("POSITIVE CONTROL — copies after run 1 (both drives present): %v", listShareCopies(env))
+	t.Logf("mirror calls run 1: %v", env.mirrored)
+	if err := env.sett.SetDisconnected(env.drives["hdd_1"], true, nil); err != nil {
+		t.Fatal(err)
+	}
+	env.mirrored = nil
+	if err := env.m.RunSharesTier2(); err != nil {
+		t.Fatal(err)
+	}
+	t.Logf("FAULT — copies after run 2 (hdd_1 away): %v", listShareCopies(env))
+	t.Logf("mirror calls run 2: %v", env.mirrored)
+}
+
+// Shares on two drives that both pick the SAME target: does group A's reconcile remove group B's copy?
+func TestRehearsalD1_TwoGroupsOneTarget(t *testing.T) {
+	env := newSharesEnv(t, "hdd_1", "hdd_2", "hdd_3")
+	env.addShare(t, "hdd_1", "a_share", true)
+	env.addShare(t, "hdd_3", "c_share", true)
+	var removed []string
+	env.m.logger = newCaptureLogger(&removed)
+	if err := env.m.RunSharesTier2(); err != nil {
+		t.Fatal(err)
+	}
+	if err := env.m.RunSharesTier2(); err != nil {
+		t.Fatal(err)
+	}
+	t.Logf("copies after two runs: %v", listShareCopies(env))
+	for _, l := range removed {
+		t.Logf("LOG %s", l)
+	}
+}
+
+// Shares on hdd_2 and hdd_3 (both pick hdd_1 as target). Then hdd_3 dies.
+func TestRehearsalD1_SameTargetThenDriveDies(t *testing.T) {
+	env := newSharesEnv(t, "hdd_1", "hdd_2", "hdd_3")
+	env.addShare(t, "hdd_2", "b_share", true)
+	env.addShare(t, "hdd_3", "csaladi_fotok", true)
+	var logs []string
+	env.m.logger = newCaptureLogger(&logs)
+	if err := env.m.RunSharesTier2(); err != nil {
+		t.Fatal(err)
+	}
+	t.Logf("POSITIVE CONTROL run 1: %v", listShareCopies(env))
+	for _, l := range logs {
+		t.Logf("  LOG %s", l)
+	}
+	logs = nil
+	if err := env.sett.SetDisconnected(env.drives["hdd_3"], true, nil); err != nil {
+		t.Fatal(err)
+	}
+	if err := env.m.RunSharesTier2(); err != nil {
+		t.Fatal(err)
+	}
+	t.Logf("FAULT run 2 (hdd_3 away): %v", listShareCopies(env))
+	for _, l := range logs {
+		t.Logf("  LOG %s", l)
+	}
+}
@@ -0,0 +1,18 @@
+package backup
+
+import (
+	"log"
+	"strings"
+)
+
+type capW struct{ dst *[]string }
+
+func (c capW) Write(p []byte) (int, error) {
+	s := string(p)
+	if strings.Contains(s, "reconcile") || strings.Contains(s, "tier-2 copied") {
+		*c.dst = append(*c.dst, strings.TrimSpace(s))
+	}
+	return len(p), nil
+}
+
+func newCaptureLogger(dst *[]string) *log.Logger { return log.New(capW{dst}, "", 0) }
@@ -0,0 +1,10 @@
+# go test, hub store at felhom.eu 5c48731 (scratch clone), in-memory SQLite; 2026-10-20T10:12:12Z
+=== RUN   TestRehearsalK1_NewBoxAfterHostDeleteSeesNoEscrow
+    zz_rehearsal_k1_test.go:20: POSITIVE CONTROL (old host live): status=&{IdentityBlobPresent:true ResticPwSHA256:SHA_OLD CreatedAt:2026-10-18T00:00:00Z SupersededPresent:false SupersededAt:} err=<nil>
+    zz_rehearsal_k1_test.go:27: retained rows under the DELETED host_id: 1 (identity blob present: true)
+    zz_rehearsal_k1_test.go:34: FAULT (new box, what its ACK carries): status=<nil> err=<nil>
+    zz_rehearsal_k1_test.go:36: => GetEscrowStatusForCustomer returns nil: identity_blob_present=false AND superseded_present=false for the new box
+    zz_rehearsal_k1_test.go:39: retained rows reachable under the NEW host_id: 0
+--- PASS: TestRehearsalK1_NewBoxAfterHostDeleteSeesNoEscrow (0.04s)
+PASS
+ok  	gitea.dooplex.hu/admin/felhom-hub/internal/store	0.048s
@@ -0,0 +1,40 @@
+package store
+
+// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate K1). Never committed.
+import "testing"
+
+func TestRehearsalK1_NewBoxAfterHostDeleteSeesNoEscrow(t *testing.T) {
+	s := newTestStore(t)
+	const cust = "Tester4"
+	// The household's first box, with an escrowed package (K blob + the identity blob that opens the off-site repo).
+	if err := s.UpsertHost(&Host{HostID: "Tester4-old", CustomerID: cust, APIKey: "k1"}); err != nil {
+		t.Fatal(err)
+	}
+	if _, _, err := s.SaveHostEscrow("Tester4-old", []byte("k-blob"), "fp", "zk", "2026-10-18T00:00:00Z", "SHA_OLD"); err != nil {
+		t.Fatal(err)
+	}
+	if err := s.SaveHostDRBundle("Tester4-old", []byte("identity-opening-the-old-repo")); err != nil {
+		t.Fatal(err)
+	}
+	st, err := s.GetEscrowStatusForCustomer(cust)
+	t.Logf("POSITIVE CONTROL (old host live): status=%+v err=%v", st, err)
+
+	// The operator deletes the old host (the installer's 409 tells them to), escrow acknowledged → demoted.
+	if err := s.DeleteHost("Tester4-old", true); err != nil {
+		t.Fatal(err)
+	}
+	ret, _ := s.ListSupersededEscrow("Tester4-old")
+	t.Logf("retained rows under the DELETED host_id: %d (identity blob present: %v)", len(ret), len(ret) == 1 && len(ret[0].IdentityBlob) > 0)
+
+	// The reinstalled box enrolls under a NEW host_id for the same customer.
+	if err := s.UpsertHost(&Host{HostID: "Tester4-new", CustomerID: cust, APIKey: "k2"}); err != nil {
+		t.Fatal(err)
+	}
+	st, err = s.GetEscrowStatusForCustomer(cust)
+	t.Logf("FAULT (new box, what its ACK carries): status=%+v err=%v", st, err)
+	if st == nil {
+		t.Logf("=> GetEscrowStatusForCustomer returns nil: identity_blob_present=false AND superseded_present=false for the new box")
+	}
+	ret2, _ := s.ListSupersededEscrow("Tester4-new")
+	t.Logf("retained rows reachable under the NEW host_id: %d", len(ret2))
+}
@@ -0,0 +1,10 @@
+# go test at controller 7ca59e2 (v0.336.0), scratch clone, fake provider seams; 2026-10-20T10:13:05Z
+=== RUN   TestRehearsalR5_WholeRestoreUnitRefusalAfterFileHalf
+    zz_rehearsal_r5_test.go:22: live files BEFORE: map[p2.jpg:P2 live older]
+    zz_rehearsal_r5_test.go:25: RestoreTier2Whole err = A(z) app adatbázis-másolata csonka: nem ér véget, ahogy egy teljes másolat. A visszaállítás biztonsági okból nem indult el — az alkalmazás és az adatai érintetlenek.
+    zz_rehearsal_r5_test.go:26: file counts: {Restored:1 Replaced:1 KeptNewer:0 Unchanged:0 BytesTotal:17}
+    zz_rehearsal_r5_test.go:27: live files AFTER: map[p1.jpg:P1 p2.jpg:P2 mirror NEWER p2.jpg.felhom-20261020T101306Z:P2 live older]
+    zz_rehearsal_r5_test.go:28: app stopped=true, full start happened=false, provider calls=[stop]
+--- PASS: TestRehearsalR5_WholeRestoreUnitRefusalAfterFileHalf (0.01s)
+PASS
+ok  	gitea.dooplex.hu/admin/felhom-controller/internal/backup	0.019s
@@ -0,0 +1,29 @@
+package backup
+
+// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate R5). Never committed.
+import (
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func TestRehearsalR5_WholeRestoreUnitRefusalAfterFileHalf(t *testing.T) {
+	cut := strings.Replace(pgDump(1), "PostgreSQL database dump complete", "", 1) // a cut-off copy (R-640)
+	f := r102Tier2Fixture(t, []string{"app_data.tar"}, cut)
+	f.m.stackProvider = fileAppProvider{f.fake}
+	mid := time.Date(2026, 9, 22, 10, 0, 0, 0, time.UTC)
+	mirrorLeg := filepath.Join(f.destBase, "hdd", "appdata", "app")
+	liveLeg := filepath.Join(f.liveDrive, "appdata", "app")
+	writeAt(t, filepath.Join(mirrorLeg, "p1.jpg"), "P1", mid)
+	writeAt(t, filepath.Join(mirrorLeg, "p2.jpg"), "P2 mirror NEWER", mid.Add(72*time.Hour))
+	writeAt(t, filepath.Join(liveLeg, "p2.jpg"), "P2 live older", mid.Add(48*time.Hour))
+	before := fingerprint(t, liveLeg)
+	t.Logf("live files BEFORE: %v", before)
+
+	res, err := f.m.RestoreTier2Whole("app")
+	t.Logf("RestoreTier2Whole err = %v", err)
+	t.Logf("file counts: %+v", res.Files)
+	t.Logf("live files AFTER: %v", fingerprint(t, liveLeg))
+	t.Logf("app stopped=%v, full start happened=%v, provider calls=%v", f.fake.stopped, f.fake.fullStarted, f.fake.calls)
+}
@@ -0,0 +1,57 @@
+// Replays hub cmd/hub/main.go scheduleDaily's arithmetic (felhom.eu 5c48731, lines 1057-1062) over
+// 2026-10-23..27 with a simulated clock, for each hub daily job. Prints every fire time.
+package main
+
+import (
+	"fmt"
+	"time"
+)
+
+func main() {
+	bud, _ := time.LoadLocation("Europe/Budapest")
+	jobs := []struct{ name string; h, m int }{{"db-snapshot", 2, 0}, {"prune", 4, 30}, {"deadline-check", 5, 0}, {"offsite-key-audit", 7, 10}}
+	for _, j := range jobs {
+		now := time.Date(2026, 10, 23, 12, 0, 0, 0, bud)
+		end := time.Date(2026, 10, 27, 12, 0, 0, 0, bud)
+		fmt.Printf("%s (%02d:%02d):\n", j.name, j.h, j.m)
+		for now.Before(end) {
+			n := now.In(bud)
+			next := time.Date(n.Year(), n.Month(), n.Day(), j.h, j.m, 0, 0, bud)
+			if !next.After(n) {
+				next = next.Add(24 * time.Hour)
+			}
+			fmt.Printf("   fires %s  (%s)\n", next.In(bud).Format("2006-01-02 15:04 MST"), next.UTC().Format("15:04Z"))
+			now = next // fn() runs instantly; loop re-reads the clock at the fire instant
+		}
+	}
+}
+
+--- go run, go version go1.26.0 linux/amd64
+db-snapshot (02:00):
+   fires 2026-10-24 02:00 CEST  (00:00Z)
+   fires 2026-10-25 02:00 CEST  (00:00Z)
+   fires 2026-10-25 02:00 CET  (01:00Z)
+   fires 2026-10-26 02:00 CET  (01:00Z)
+   fires 2026-10-27 02:00 CET  (01:00Z)
+   fires 2026-10-28 02:00 CET  (01:00Z)
+prune (04:30):
+   fires 2026-10-24 04:30 CEST  (02:30Z)
+   fires 2026-10-25 03:30 CET  (02:30Z)
+   fires 2026-10-25 04:30 CET  (03:30Z)
+   fires 2026-10-26 04:30 CET  (03:30Z)
+   fires 2026-10-27 04:30 CET  (03:30Z)
+   fires 2026-10-28 04:30 CET  (03:30Z)
+deadline-check (05:00):
+   fires 2026-10-24 05:00 CEST  (03:00Z)
+   fires 2026-10-25 04:00 CET  (03:00Z)
+   fires 2026-10-25 05:00 CET  (04:00Z)
+   fires 2026-10-26 05:00 CET  (04:00Z)
+   fires 2026-10-27 05:00 CET  (04:00Z)
+   fires 2026-10-28 05:00 CET  (04:00Z)
+offsite-key-audit (07:10):
+   fires 2026-10-24 07:10 CEST  (05:10Z)
+   fires 2026-10-25 06:10 CET  (05:10Z)
+   fires 2026-10-25 07:10 CET  (06:10Z)
+   fires 2026-10-26 07:10 CET  (06:10Z)
+   fires 2026-10-27 07:10 CET  (06:10Z)
+   fires 2026-10-28 07:10 CET  (06:10Z)
@@ -0,0 +1,23 @@
+# B1 — download and install (2026-10-20, 09:40–09:51 UTC)
+
+Method: the guide's URL `felhom.eu/letoltes` (no public page links to it — by design, the guide carries it); the ISO
+downloaded onto demo-hp; VM 9320 (4 vCPU, 8 GiB, 256 GB system disk + 200 GB data disk, raw on `nvme-scratch` =
+`/mnt/hdd_1` root); the graphical installer driven with `qm sendkey` (`tools/vmctl.sh`), the **Hungarian keyboard
+kept**; screenshots by QEMU screendump (`screens/install/`).
+
+| step | result | vs the guide (`runbooks/VOLUNTEER-first-hour.md`) |
+|---|---|---|
+| download | 1 705 324 544 B in 62 s; sha256 `dceacae5…4829` = the page = the `.sha256` file (`01-iso-download.txt`) | as written. **But the download link is not visible as a link** (white bold file name; no button, no colour, no underline — phone and computer, `screens/b1-site/`) |
+| boot menu | auto-boots „Felhom telepítés / Install Felhom" | as written; the boot text „válassza ki a lemezt" is FORMAL (Ön) — the guide and dashboard say „te" |
+| EULA → disk | `/dev/sda (256 GiB)` pre-selected = the right disk | as written. Proxmox's own text „Final configuration will be done on the graphical user interface, via a web browser" is not true for a household |
+| location | Hungary / Europe/Budapest / Hungarian, pre-filled | as written. **Enter in the Country field does not advance** — a keyboard user must use Next |
+| password + e-mail | ≥ 8 chars; the e-mail field pre-filled `mail@example.invalid` | as written |
+| network | DHCP address frozen as a STATIC one (192.168.0.134/24) | as written („hagyd, ahogy van"); the static address is the R-1166 shape (a later DHCP lease can clash) |
+| hostname | `felhom.tester4.test` | as written |
+| summary | auto-reboot ticked; Install pressed 09:46:50 | as written |
+| first boot | installer reboots by itself; the console: „Felhom otthoni szerver / Ezen a gépen most nincs dolgod, és bejelentkezni sem kell. A beállításhoz kövesd a Felhomtól kapott útmutatót." | **wording differs from the guide** (guide: „Ezen a képernyőn nincs teendőd…") — harmless |
+| pairing code | shown ≤ 09:51 (≤ 4 min after Install), HU + EN, code blacked out in the screenshot | as written |
+
+Harness substitutions (not interventions): H3 the VM boots its disk first (= the stick pulled in time); H5 a nested
+VM with virtual disks; the public domain `tester4.test` never resolves (Tester4 is a test customer with no tunnel) —
+the dashboard is reached on its LAN address with the `Host:` header, which a household cannot do.
@@ -0,0 +1,10 @@
+# B9 — every Hungarian text the household met (built as the walk goes)
+
+| # | where | the text | the problem | the fix |
+|---|---|---|---|---|
+| W1 | installer boot text (ISO 1.29.0) | „A Felhom telepítése indul — válassza ki a lemezt a telepítőben…" | formal „Ön" form; everywhere else Felhom says „te" | „…válaszd ki a lemezt…" (ISO rebuild — not in this run) |
+| W2 | felhom.eu/letoltes | the ISO link is the bold white file name | not recognisable as a link — a stranger looks for a button | a real „Letöltés (1,6 GB)" button above the checksum (website change = a publish — not in this run) |
+| W3 | bind mail sent when the old host was deleted (2026-10-18 19:41 UTC) | „Elkészült a Felhom dobozod, és készen áll az összekötésre." | sent when NO box exists (the operator deleted the old one); a promise the product cannot keep at that moment | „Ha új Felhom dobozt telepítesz, ezzel a hivatkozással kötheted össze…" |
+| W4 | bind page after a wrong try | „A megadott adatok nem megfelelőek. Ellenőrizd a párosító kódot és a tulajdonosi jelmondatot, majd próbáld újra." | does not say how many of the 5 tries are left; the 5th locks the link and only support can help | add „Még N próbálkozásod van." |
+| W5 | an old (replaced) bind link | „Ez a hivatkozás érvénytelen vagy lejárt." + „Új linket kérek" | a household with two mails may press „Új linket kérek" on the OLD one and invalidate the newer link they were about to use | say „Ha van újabb leveled, azt használd." above the button |
+| W6 | console after first boot vs the guide | console „Ezen a gépen most nincs dolgod…"; guide „Ezen a képernyőn nincs teendőd…" | the guide quotes a sentence the box does not show | guide edit (allowed, §3.7) |
@@ -0,0 +1,28 @@
+#!/bin/bash
+# claim.sh BASE DOMAIN CODE_ENV PW_ENV [WRONG]  — the household's „A szerver beállítása" page, headless.
+# BASE = https://<box ip>; DOMAIN = the customer domain (Host: felhom.<domain>). CODE_ENV / PW_ENV are the NAMES
+# of env vars holding the setup code and the new dashboard password (run under `sec run`). With WRONG=1 one letter
+# of the code is changed (a household's typo). Prints status codes, the Location and the page's visible message —
+# never a secret. Endpoint-level: the same POST the page's form sends.
+set -euo pipefail
+BASE=$1; DOM=$2; CODE=${!3}; PW=${!4}; WRONG=${5:-0}
+H="Host: felhom.$DOM"; T=$(mktemp -d); chmod 700 "$T"; trap 'rm -rf "$T"' EXIT
+if [ "$WRONG" = 1 ]; then CODE="${CODE:0:2}$( [ "${CODE:2:1}" = q ] && echo w || echo q )${CODE:3}"; fi
+curl -sk -H "$H" -D "$T/h1" -o "$T/b1" "$BASE/claim"
+echo "GET /claim $(head -1 "$T/h1" | tr -d '\r')"
+CK=$(grep -oiE 'felhom_claim_csrf=[^;]+' "$T/h1" | head -1)
+CS=$(grep -oE 'name="_csrf" value="[^"]+"' "$T/b1" | head -1 | sed 's/.*value="//;s/"$//')
+echo "csrf_len ${#CS} cookie_present $([ -n "$CK" ] && echo yes || echo no)"
+curl -sk -H "$H" -H "Cookie: $CK" -D "$T/h2" -o "$T/b2" -X POST \
+  --data-urlencode "_csrf=$CS" --data-urlencode "code=$CODE" \
+  --data-urlencode "new_password=$PW" --data-urlencode "confirm_password=$PW" "$BASE/claim"
+echo "POST /claim $(head -1 "$T/h2" | tr -d '\r') $(grep -i '^location:' "$T/h2" | tr -d '\r')"
+python3 - "$T/b2" <<'EOF'
+import re, sys, html
+t = open(sys.argv[1], encoding="utf-8", errors="replace").read()
+for cls in ("error", "alert", "flash", "banner", "notice"):
+    for m in re.finditer(r'class="[^"]*%s[^"]*"[^>]*>(.*?)</' % cls, t, re.S):
+        s = re.sub(r"\s+", " ", html.unescape(re.sub(r"<[^>]+>", " ", m.group(1)))).strip()
+        if s:
+            print("message:", s[:300])
+EOF
@@ -0,0 +1,76 @@
+#!/usr/bin/env python3
+"""ctl.py — a minimal headless client for a Felhom controller dashboard (rehearsal 2026-10-20).
+
+The same requests the dashboard's pages send (endpoint-level validation): POST /login, then JSON calls with the
+session cookie and the X-CSRF-Token header. Configuration by env: CTL_BASE (https://<ip>), CTL_DOMAIN (Host:
+felhom.<domain>), CTL_PW_ENV (the NAME of the env var holding the dashboard password — run under `sec run`).
+Session files live in a 0700 temp dir per process; nothing secret is printed.
+
+usage as a CLI:  ctl.py GET /api/stacks
+                 ctl.py POST /api/stacks/x/stop '{"a":1}'
+"""
+import json, os, re, subprocess, sys, tempfile
+
+BASE = os.environ["CTL_BASE"]
+HOST = "Host: felhom." + os.environ["CTL_DOMAIN"]
+_T = tempfile.mkdtemp(prefix="ctl-")
+os.chmod(_T, 0o700)
+_SESS = _CSRF = None
+
+
+def _curl(args, inp=None, timeout=600):
+    r = subprocess.run(["curl", "-sk", "-H", HOST] + args, capture_output=True, text=True, timeout=timeout, input=inp)
+    return r.stdout
+
+
+def login():
+    global _SESS, _CSRF
+    pw = os.environ[os.environ["CTL_PW_ENV"]]
+    hdr = os.path.join(_T, "h")
+    # the password goes on stdin (curl --data-urlencode password@-), never on a command line
+    _curl(["-D", hdr, "-o", "/dev/null", "-X", "POST", "--data-urlencode", "password@-", BASE + "/login"], inp=pw)
+    m = re.search(r"felhom_session=[A-Za-z0-9._-]+", open(hdr).read(), re.I)
+    if not m:
+        sys.exit("ctl: login failed (no session cookie)")
+    _SESS = m.group(0)
+    body = _curl(["-L", "-H", "Cookie: " + _SESS, BASE + "/"])
+    c = re.search(r'<meta name="csrf-token" content="([^"]+)"', body)
+    if not c:
+        sys.exit("ctl: login failed (no csrf token)")
+    _CSRF = c.group(1)
+
+
+def call(method, path, data=None, raw=False, timeout=600):
+    if _SESS is None:
+        login()
+    for attempt in (0, 1):
+        args = ["-H", "Cookie: " + _SESS, "-w", "\n%{http_code}"]
+        if method != "GET":
+            args += ["-H", "X-CSRF-Token: " + _CSRF, "-H", "Content-Type: application/json", "-X", method]
+            if data is not None:
+                args += ["--data", json.dumps(data)]
+        out = _curl(args + [BASE + path], timeout=timeout)
+        body, _, code = out.rpartition("\n")
+        if code in ("302", "401") and attempt == 0:
+            login()
+            continue
+        if raw:
+            return code, body
+        try:
+            return code, json.loads(body)
+        except ValueError:
+            return code, {"_raw": body[:800]}
+
+
+def page(path):
+    if _SESS is None:
+        login()
+    return _curl(["-H", "Cookie: " + _SESS, BASE + path])
+
+
+if __name__ == "__main__":
+    m, p = sys.argv[1], sys.argv[2]
+    d = json.loads(sys.argv[3]) if len(sys.argv) > 3 else None
+    code, body = call(m, p, d)
+    print(code)
+    print(json.dumps(body, ensure_ascii=False, indent=1)[:6000])