From 7ad17be6bd2f0ccaf0055613175beed3ab2b8e6d Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Wed, 21 Oct 2026 11:47:40 +0200 Subject: [PATCH] =?UTF-8?q?rehearsal=202026-10-20:=20final=20report,=20STA?= =?UTF-8?q?TUS=20(morning=20list,=20D24),=20capability=20map=20A12/A13=20?= =?UTF-8?q?=E2=86=92=20PARTIAL=20(B7),=20stuck=20point=2010?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- REPORT-rehearsal-2026-10-20.md | 59 +++++++++++++------ STATUS.md | 11 ++-- .../architecture/00-capability-map.md | 4 +- .../architecture/capability-map-detail/A.md | 4 ++ .../rehearsal-2026-10-20/B10-stuck/README.md | 4 +- documentation/backlog/STATUS-HISTORY.md | 4 ++ 6 files changed, 60 insertions(+), 26 deletions(-) diff --git a/REPORT-rehearsal-2026-10-20.md b/REPORT-rehearsal-2026-10-20.md index 8c68dfee..610eecd3 100644 --- a/REPORT-rehearsal-2026-10-20.md +++ b/REPORT-rehearsal-2026-10-20.md @@ -1,13 +1,21 @@ # REPORT — Tester-2 rehearsal with a pretend household + an independent data-loss hunt (2026-10-20 → 21) -> DRAFT written 2026-10-20 ~21:30 UTC, before the night. Sections marked **PENDING** are filled in the morning. +> Run 2026-10-20 09:30 UTC → 2026-10-21 ~09:50 UTC. Drafted before the night, finished in the morning. This repo's +> shared `REPORT.md` was not touched: a parallel session works in the same clone, and the repo rule says the second +> session writes `REPORT-.md` only (the brief asked for both; this line is why one was skipped). > Evidence: `documentation/audits/rehearsal-2026-10-20/` (journal, A0–B10, screens) and > `documentation/audits/dataloss-hunt-2026-10-20/` (C1–C3). Architecture read for the claims: `07-backup-architecture.md`, > `10-localisation.md`, `00-capability-map.md` (A15). ## Five sentences -PENDING (morning). +A pretend household installed a box from the public ISO, linked it, claimed it, put eight apps and about 2 GB of family +data on it, used it for a day, and every customer restore path brought its data back. The first night's backups, the +clock change, three days off and power cuts in the middle of installs and catch-ups all worked, and the morning data was +byte-identical to the evening's. But the 8 GB minimum box froze twice on its first afternoon and only a plug pull helped +(R-1192), and the household's warning mails went to the operator instead (R-1199). When the box died, the household could +not even start again without the operator, and after the operator's press the rebuilt box could not reach its off-site +history at all — proven live (R-1176, P1). Nothing was released; 29 new register rows carry the rest. ## The ten places a household gets stuck @@ -22,14 +30,15 @@ PENDING (morning). 7. A new drive needs a „mount name" nobody knows; the button silently does nothing (W13). 8. Sharing waits for ever for a password the page never asks for (W21). 9. Errors arrive as raw technical English (W22, W25). -10. Three apps' „first steps" are wrong for this household (wrong address, wrong user name, wrong sign-up state; W17–W19). +10. After the box died, the household was promised a mail that never came, and after the operator's press the rebuilt box + could not reach the family's off-site data (R-1203, R-1176). (Close behind: three apps' wrong „first steps", W17–W19.) ## Data-loss findings, ranked | row | Sev | harm | reproduced | design / defect | |---|---|---|---|---| | R-1175 | P1 | shares on two drives sharing one second drive: every night one drive's share copy is pruned; the night a drive dies its copy is deleted | unit test on production code (`C3-D1-shares-tier2/`) | defect | -| R-1176 | P1 | a box rebuilt after „delete the old host" cannot reach its off-site history: no recovery code asked, a new key, the old history set aside on night one | hub store unit test (`C3-K1-…`); controller half read in code; live test in B7 — PENDING | defect | +| R-1176 | P1 | a box rebuilt after „delete the old host" cannot reach its off-site history: no recovery code asked, a new key, the old history set aside on night one | hub store unit test (`C3-K1-…`) + **LIVE in B7** (2026-10-21) | defect | | R-1191 | P1 | a `.fab` import makes a family-only app open to anyone | LIVE on scratch 9202 | defect (security) | | R-1192 | P1 | the 8 GB box froze under first-day use; two plug pulls | LIVE on Tester4 (twice) | defect — cause not diagnosed (no shell on the box) | | R-1177 | P2 | a failed local restore destroys the live data, keeps no copy | LIVE on 9202 | design question (D22 in STATUS) | @@ -40,11 +49,15 @@ PENDING (morning). | R-1179 | P2 | a drive marked „away" can still be the second-drive target | code + unit test | defect | | R-1182 | P2 | abandoning one set-aside off-site copy deletes every retained key package | code | defect | | R-1199 | P2 | warning mails reach the operator, not the household | LIVE (two mailboxes) | defect | +| R-1204 | P2 | a box that ran once with its clock ahead loses its WireGuard tunnel to ep0 (PBS copies + operator door) until that date | LIVE (B6 #8, ep0 `wg show`) | defect | +| R-1203 | P2 | after a box death, the household is promised a link mail that never comes until the operator deletes the old host | LIVE (B7) | defect | | R-1185 | P3 | a dump that fails its own check has already replaced the good one | unit test (docker stub) | defect (order) | | R-1186 | P3 | the nightly off-site copy is skipped silently when another backup holds the flag | code | defect | | R-1184 | P3 | a cut-off shares restore can be reported restored with files missing | code | defect | | R-1196 | P3 | uploads confirmed just before a drive pull are 0 bytes or gone | LIVE (B6 #4) | defect | -| R-1183 | P3 | on 2026-10-25 every hub daily job runs twice (DB snapshot keep-2 pruned a day early) | replay of the hub's arithmetic | defect | +| R-1183 | P3 | on 2026-10-25 every hub daily job runs twice (DB snapshot keep-2 pruned a day early) | replay of the hub's arithmetic | defect (D24 in STATUS) | +| R-1200 | P3 | an afternoon install gets no whole-box backup on night one | LIVE (B5) | design question | +| R-1202 | P3 | a box with its clock behind stops an app silently and only says „Hub nem elérhető" | LIVE (B6 #8) | defect | | R-1187 … R-1190 | P3 | the unproven candidates of the four reviewers, grouped | code reading | mixed | ## Failure scenarios (B6) @@ -57,11 +70,11 @@ PENDING (morning). | ★4 | data drive pulled, put back | apps stopped in 42/34 s, back by themselves; last uploads 0 B/missing → R-1196 | `s4-…` | | ★5 | slow lossy line, first big off-site | run failed under 1 Mbit + 5 % loss, raw English; second press OK 3 m 38 s | `s5-…` | | ★6 | off-site blocked | failed in 2.5 min, raw English; catch-up OK 2 m 37 s (shortened, not a night) | `s6-…` | -| ★7 | DST | PENDING (morning) | | -| ★8 | wrong clock at boot ±2 y | PENDING (morning) | | +| ★7 | DST | PASS — the database copy ran once, at the second 02:30; W30 (no summer/winter mark in the repeated hour) | `s7-dst/` | +| ★8 | wrong clock at boot ±2 y | FINDINGS — −2 y became −6 months (the OS floor): „Hub nem elérhető" only, Paperless stopped silently (R-1202); +2 y: a whole-box archive dated 2028 (R-1189 a), then the ep0 tunnel dead (R-1204) | `s8-wrong-clock/` | | ★9 | two actions at once | PASS — second action refused with a clear sentence | `s9-…` | -| ★10 | box off 3 days (+#15 restart in the catch-up) | PENDING (morning) | | -| 11 | hard reset mid app update | PENDING — expected NOT RUN: no update is offered (nothing released during the run) | | +| ★10 | box off 3 days (+#15 restart in the catch-up) | PASS — ONE catch-up, a clear note to the household, the interrupted catch-up recovered | `s10-three-days-off/` | +| 11 | hard reset mid app update | NOT RUN — no update was offered to the box (nothing was released during the run) | — | | 12 | DNS dead 1 h | PASS for data; operator told after ~37 min, household never | `s12-…` | | 13 | second drive added later | not offered as the backup place → R-1198; safe removal OK; blank texts (W29) | `s13-…` | | 14 | remove keeping data → reinstall „use kept data" | PASS — 47/47 documents, 47/47 originals | `s14-…` | @@ -84,10 +97,7 @@ demo-hp's own kernel update rebooted the test host and shut the rehearsal VM dow host skips that step by design). Health 200 all night before that; memory steady. **The morning data snapshot is identical to the evening one, byte for byte** (`B5-night/README.md`). -**DST (#7):** PENDING — running on the box now. So far: at the first 02:30 (summer time) the database copy did NOT run, -as the code replay said. Code replay: the controller runs each daily job once on 2026-10-25 (a 02:xx job at the second, -CET occurrence); the hub runs every daily job twice (R-1183); spring 2027 puts the 02:30 dump on the 03:30 second-drive -copy (`B6-failures/dst-replay/`). +**DST (#7), live:** the box was started with its clock at 2026-10-25 02:20 CEST and left alone: no database copy at the first 02:30, one at the second (01:30:00Z, 1 m 55 s). The code replay agrees for the box; the hub runs every daily job twice on that day (R-1183, D24) and spring 2027 puts the 02:30 dump on the 03:30 copy (`B6-failures/dst-replay/`). ## Interventions (operator-only acts) @@ -95,7 +105,7 @@ copy (`B6-failures/dst-replay/`). |---|---|---|---| | 1 | 09:57 → 12:00:57 | the owner passphrase handed over (by design, out of band) | 124 | | 2 | asked 09:35, done 12:01–12:15 | off-site switched on for Tester4 | 0 (done before the household needed it) | -| 3 | PENDING | delete host Tester4-816e21 before the reinstall (B7) | PENDING | +| 3 | asked 08:59, done ~09:11 (2026-10-21) | delete host Tester4-816e21 before the reinstall (B7) | ~12 (and 17 min of a dead end before it, R-1203) | Not counted as interventions: the two plug pulls (14:11, 16:13) — a household can do that, and did. @@ -112,7 +122,10 @@ Works: Chromium 155 + Playwright 1.52 in a throwaway LXC on demo-hp, phone and c ## Teardown -PENDING (Part D). +Machine: VM 9320, CT 9330 and the ISO removed from demo-hp by their own IDs; no drill firewall rules left; memory back +to the start. DooPlex: three watchers stopped by PID. Hub: **host Tester4-648b80 is left for the operator to delete** +(the customer Tester4 stays). ep0: one PBS snapshot `ns/Tester4/ct/9201/2026-10-20T12:17:26Z` and the WireGuard peer +10.77.0.6 remain; the Storage Box keeps the household's restic history — listed, nothing removed (`D-teardown/`). ## Secrets (names only) @@ -124,10 +137,22 @@ All in `sec`, prefix `REHEARSAL_2026_10_20_`: `9202_PW`, `9202_RESETCODE`, `APP_ ## Register -Before 153 open rows; now 178 — opened R-1175 … R-1199 (25), closed 0. PENDING final count. +Before 153 open rows; now 182. Opened by this run: 29 (R-1175 … R-1200, R-1202, R-1203, R-1204). Closed: R-1019 (a dated +check read on demo-felhom, read-only). One row (R-1201) came from a parallel session. Updated: R-1176 (proven live), +R-1189 (a half-live), R-895 (first verdict), R-435 (moved). Capability map: A15 still PARTIAL with new evidence; A12 and +A13 PROVEN-LIVE → PARTIAL (B7). `unproven.py --summary`: unchanged (35 of 55 not walked). ## Observations FILED: everything in the tables above. NOT-A-FINDING: the dashboard ignores an English browser (by design: the household's setting wins); Nextcloud folding NFD into NFC (upstream behaviour; both forms survive on the share); the -films' off-site size (CBR filler compresses; nothing missing); PENDING more. +films' off-site size (CBR filler compresses; nothing missing); demo-hp's own kernel reboot at 02:46 (a harness effect: a +BYO host skips that step by design, `osupdate/leg.go:674`). FILED: everything in the tables above. + +**Harness slips (mine), each recorded in the journal:** a screenshot step blanked a form before submit (15 min lost); the +two watcher logs froze at 20:24 when `git pull --autostash` replaced their files (recovered in full from the open files); +my first #10 start used a date the #7 records had made meaningless (restarted, recorded before any result). **Secrets:** +the nightly DooPlex sweep found three rehearsal secret values in my session record (the old and new bind tokens and a +setup code — I had read their mails in full and typed one into a `sec put`); in B7 a second bind link and setup code +entered the record the same way. All are one-time values, spent within minutes; nothing secret is in any committed file +(323 files scanned, 0 hits; pictures checked by eye; the B7 pairing code was blacked out). diff --git a/STATUS.md b/STATUS.md index 8a2714e5..75bf3894 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for the first real tester (Tester 2)? Yes.** Tester 2 (a laptop) is off; nothing was sent to it. -**Updated 2026-10-20 (day).** Hub 0.162.0, agent 0.161.0 (you vouched it) and installer 1.35.0 are live, and a failed off-site restore now puts the app back exactly as it was (your D20 = A; controller 0.336.0 is the floor). This page is one screen again: older news moved to the history file (`documentation/backlog/STATUS-HISTORY.md`), and nothing was deleted. A long rehearsal on a fresh box runs today and tonight. +**Updated 2026-10-21 (morning).** The 24-hour rehearsal with a pretend household finished: the first night, the clock change, a 3-day power-off and every restore path worked, but the 8 GB box froze twice on day one and, after a box death, the household could not get its off-site data back (R-1176, proven live). Report: `REPORT-rehearsal-2026-10-20.md`; ten stuck points, 29 new rows, nothing released. ## What works - A box installs from nothing with the public installer, links to the hub, and moves to the floor by itself (map A1, A3, A4). @@ -15,7 +15,7 @@ - The hub mails alarms and recoveries, and keeps one whole-box copy per week on ep0; CC uses its own limited token, not your password (map F1, F11, G15). ## What is broken -- **P1 (the data-loss hunt, 2026-10-20 — still running):** shared folders on two drives lose their second copy every night, and a dead drive's copy is deleted the same night (R-1175); a box rebuilt after „delete the old host" cannot reach the household's off-site history (R-1176); importing an app bundle makes a family-only app public (R-1191); on the 8 GB minimum box the family's first afternoon froze the box for an hour, twice; the household was told nothing (R-1192, R-1193). +- **P1 (rehearsal + data-loss hunt, 2026-10-20/21):** shared folders on two drives lose their second copy every night (R-1175); after a box death the household cannot reach its off-site history — proven live on 2026-10-21 (R-1176); importing an app bundle makes a family-only app public (R-1191); the 8 GB box froze twice on its first afternoon (R-1192, R-1193). - **P1:** about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours). - Two root-level weaknesses wait on your decisions D3 and D4 (R-985, R-982). - Tester 2 is off since 2026-10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862). @@ -32,6 +32,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked | |---|---|---|---|---|---|---| +| D24 | R-1183 | On Sunday 25 Oct (clocks go back) the hub runs every daily job twice. Fix it before Sunday with a hub deploy you attend? | **Yes, Friday** (a small change; the box side was proven right live today) | Leave it: the alarm check runs twice and the hub keeps one day less of its database copies, once | the double run happens on Sunday; nothing is lost on the boxes | — | | D23 | R-1181 | A missing app folder deletes its second-drive copy within one night. Keep the old copy until the household lets it go? | **Yes, keep it** (a deleted folder is the reason a backup exists; audiobooks, comics and ROMs have no other copy) | Keep as now: a deleted folder is gone from every local copy by the next morning | an accidental folder delete is unrecoverable after one night | — | | D22 | R-1177 | A LOCAL restore that fails destroys the live data and keeps no copy. Give it the off-site restore's „put back exactly" (your D20)? | **Yes** (proven live today: a failed restore left the app dead and the newest data gone) | No: a failed local restore can lose the household's newest data | the main restore button can lose data when a backup copy is damaged | — | | D21 | R-49 | Immich's off-site copy: leave out only the AI model files now (B), or also the raw database folder and Immich's own backups (A)? | **B now** (−62 %, no new risk); A's other parts after their proof | A: −92 %, but it needs a restore path we have never run | every Immich box sends about 12 times its real data off-site each night | — | @@ -54,9 +55,9 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te 6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. Since 2026-10-14. 7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12. -**Morning list, 2026-10-20 (from the night session, part 2).** Item 1 (vouch agent 0.161.0) is done. Item 2 is D21 above. -- MeTube's downloaded videos are now out of the second-drive copy, like the torrent and usenet downloads. If you want them kept, say so. If not: nothing to do. -- The update test for more apps needs your word: start the test bench (guest 9401 on demo-hp) and let CC commit to the drill catalog (R-462). Five apps wait for a tested step. If not: they stay on their tested version, which is safe. +**Morning list, 2026-10-21 (rehearsal).** No decision was taken by CC. +- Delete host **Tester4-648b80** on the Hosts page (the rebuilt test box, now gone; if not: stale/down alarm mails). Keep or remove the 24 `REHEARSAL_2026_10_20_*` secrets in sec (7 are spent one-time values). +- From 2026-10-20: MeTube downloads are out of the second-drive copy (say if you want them kept); the update test for five apps waits for your word to start bench 9401 (R-462). **The rest** - Pull the failed disk Z6A07P2G out of DooPlex (R-934); it is still in. If not: its alarms return on 11 November. Since 2026-10-12. diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 392b89a9..d774a151 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -85,8 +85,8 @@ | [A9](capability-map-detail/A.md#a9) | One name per secret on every surface — „Beállító kód" (the setup code) and „Tulajdonosi jelmondat" (the owner passphrase) — and the re-enrol mail points at a page a rebuilt box shows | controller v0.211.0, hub v0.104.0 + v0.105.0 (R-295, R-323) | **IMPLEMENTED** | written BUILT: source-read; no customer has typed the passphrase | R-295; R-323 | | [A10](capability-map-detail/A.md#a10) | The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE** | demo-felhom, real endpoints | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04); `audits/RECON-offsite-dr-chain-2026-08-04.md` (2026-08-04) | | [A11](capability-map-detail/A.md#a11) | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** | demo/drill fleet only | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) | -| [A12](capability-map-detail/A.md#a12) | A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PROVEN-LIVE** | the success path; the failure path is not claimed (R-379, R-380) | `audits/DRILL-r201-night-run-2026-08-04.md` (2026-08-04); `audits/DRILL-r356-hot-only-restore-2026-08-22/` (2026-08-22); `audits/DRILL-r356b-driveless-db-restore-2026-08-22/` (2026-08-22) | -| [A13](capability-map-detail/A.md#a13) | The customer's own UNAIDED recovery journey, end to end | controller v0.206.0, hub v0.98.0, agent v0.127.0 | **PROVEN-LIVE** | fifth walk, scoped — read what it does not claim | `tests/walk5-r201-2026-08-07/journal.md` (2026-08-07); `audits/CAMPAIGN-11-recovery-journey-2026-08-05.md` (2026-08-05) | +| [A12](capability-map-detail/A.md#a12) | A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PARTIAL** | 2026-10-21: FAILS on the current versions — after a box death and the operator's delete, the rebuilt box offers only a NEW recovery code and cannot read the off-site store (R-1176); before: the success path; the failure path is not claimed (R-379, R-380) | `audits/rehearsal-2026-10-20/B7-died/README.md` (2026-10-21); `audits/DRILL-r201-night-run-2026-08-04.md` (2026-08-04); `audits/DRILL-r356-hot-only-restore-2026-08-22/` (2026-08-22); `audits/DRILL-r356b-driveless-db-restore-2026-08-22/` (2026-08-22) | +| [A13](capability-map-detail/A.md#a13) | The customer's own UNAIDED recovery journey, end to end | controller v0.206.0, hub v0.98.0, agent v0.127.0 | **PARTIAL** | 2026-10-21: FAILS on the current versions — the unaided journey stops at the rebuilt dashboard (R-1176, R-1203); before: fifth walk, scoped — read what it does not claim | `audits/rehearsal-2026-10-20/B7-died/README.md` (2026-10-21); `tests/walk5-r201-2026-08-07/journal.md` (2026-08-07); `audits/CAMPAIGN-11-recovery-journey-2026-08-05.md` (2026-08-05) | | [A14](capability-map-detail/A.md#a14) | A random night of household actions while random things go wrong — twelve rounds, unattended | controller v0.245.0, agent v0.131.0, hub v0.116.0, golden 0.245.0, ISO 1.28.0 | **PROVEN-LIVE** | nested box, twelve chaos rounds | `audits/DRILL-chaos-night-2026-09-17.md` (2026-09-17) | | [A15](capability-map-detail/A.md#a15) | A stranger's FIRST HOUR on a fresh install — download → install → claim → two apps → use → backup → remove → restore, plus a power cut and a mistyped code | controller v0.336.0, agent v0.161.0, hub v0.162.0, golden 0.331.0, installer 1.35.0, ISO 1.29.0 | **PARTIAL** | 2026-10-20: download → install → bind → claim → 8 apps → use → backup → remove → every restore path → power cut → mistyped codes all worked; the 8 GB box FROZE twice in the first afternoon and only a plug pull brought it back (R-1192), and the owner passphrase needed the operator | `audits/rehearsal-2026-10-20/journal.md` (2026-10-20); `audits/DRILL-fresh-install-0242-2026-09-14.md` (2026-09-14); `audits/DRILL-new-household-2026-09-30.md` (2026-09-30) | | [A16](capability-map-detail/A.md#a16) | Off-site app-data capture covers the paths an app declares MANDATORY — on BOTH drive layouts | controller v0.197.0 | **PROVEN-LIVE** | both drive layouts | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04) | diff --git a/documentation/architecture/capability-map-detail/A.md b/documentation/architecture/capability-map-detail/A.md index 26f8106a..ac4c38c8 100644 --- a/documentation/architecture/capability-map-detail/A.md +++ b/documentation/architecture/capability-map-detail/A.md @@ -241,6 +241,8 @@ likewise silent. Evidence: `audits/DRILL-r361-2026-08-22/evidence/06-part3-decis **NARROWED 2026-08-21 by the backup-truth drill — kept as history; both halves have since closed, see the entry immediately above.** Proven that night on `demo-hp` with planted, hash-recorded files: the **declared-userdata** leg of a **drive-declaring** app does come back byte-identical (`calibre-web`, 5/5 including two Hungarian accented filenames). **Two legs of the same story do NOT:** (a) the off-site restore has **no named-volume leg at all**, so an app's volume tar sits in the unit, in the snapshot and in the checking folder and is never replayed (**R-354**); (b) for the **40 of 53** apps that declare no data drive the off-site restore **refuses outright**, saying a running app „nincs telepítve" (**R-356**). Since the 40-class keeps ALL its data in named volumes, the end-to-end story is **unproven for that class and disproven for the volume leg generally**. The escrow/key half of this row is untouched by that and still stands. Evidence: `audits/DRILL-backup-truth-2026-08-21/evidence/` and `REPORT.md` (2026-08-21). + +**Update 2026-10-21 (rehearsal B7; controller 0.336.0, agent 0.161.0, hub 0.162.0): status PARTIAL.** [FACT] Tester4 was destroyed, reinstalled from the public ISO, re-linked after the operator deleted the old host, and claimed. The rebuilt dashboard showed only „A távoli mentés szünetel, amíg nem hozod létre a helyreállítási kódot" with „Helyreállítási kód létrehozása" (create, never enter); the restore page said „Nem tudjuk elolvasni a távoli tárolót". No front door reached the household's off-site history (R-1176, P1; the forgotten-press dead end R-1203). The August evidence stands for the versions it names; it does not hold today. `audits/rehearsal-2026-10-20/B7-died/README.md`. --- @@ -258,6 +260,8 @@ likewise silent. Evidence: `audits/DRILL-r361-2026-08-22/evidence/06-part3-decis **Gap / roadmap (as written):** **WHAT IT DOES NOT CLAIM, and what the v0.201.0/v0.97.1 fixes do and do not change.** Six of the nine findings are fixed and each is red-proofed (R-216, R-218, R-219, R-217, R-222, R-215) — but **fixes are not a journey.** This row goes green only when the walk is repeated end to end and completes with no operator intervention. **Three findings are deliberately still open and each is a live blocker for some flow:** **R-214** (the console never stops showing a stale pairing code), **R-220** (after a rebuild the customer's drives cannot be re-enrolled — the deploy refuses and the wizard's list is empty, so no app can be redeployed onto its own data drive), **R-221** (a rebuilt box cannot run the escrow ceremony at all, because the pbsdr convergence marker survives on the host while the seeded `escrow.pbs_storage_id` is rewritten out of `agent.json`). **And R-216's fix does not by itself make a NEW box work:** with the hub guard corrected but the Day-0 manifest still vouching agent 0.120.0, a new box is **HELD, not served** — it stops being lied to, which is better, but the feature does not work for it until agent 0.125.0 is vouched. **⚠ CAMPAIGN-11 PHASES 2 + 4 RAN OVERNIGHT 2026-08-05/06 AND THIS ROW STAYS **FAIL**.** **What they add:** eleven injected faults against the recovery journey, each with a positive control, and a full unattended scheduled cycle. **The backup promise strengthened** — the off-site tier ran itself at 04:15 on a box rebuilt twice and set aside hours earlier (`snapshot_count` 1 → 2), all five daily jobs fired exactly once, and nothing on the must-not list fired. **R-217's and R-215's fixes were proven live under exactly their faults**, the set-aside proved it does not delete (12 535 KB byte-exact at the far end), and a wrong code was refused three times with nothing written and no lockout. **What they do NOT add: any progress on the JOURNEY.** These faults are **not a re-walk** — no customer route was walked end to end — and four new findings say the journey got no better where it matters: **R-224** (a hub outage and a stopped agent are BOTH reported to the customer as a bad recovery code, in 0.056 s and 0.030 s — no unseal attempted; the agent's own `err` distinguishes them and it is discarded at the HTTP boundary, and R-216's gate answers `source=version` so it cannot see reachability — **Phase 1's headline defect relocated from the version channel to the transport**), **R-226** (M1, the only message telling a customer to check their typing, is unreachable on any box that has re-escrowed), **R-225** (an unread store renders `0 pillanatkép · 0 GB` above a card saying it holds backups — 1 snapshot and 12 535 KB were really there), **R-228** (the set-aside history is recorded in `orphaned_renamed_to` and shown by nobody — `OrphanedRenamedTo` has zero references in any template or handler). **§4.1 is now MEASURED rather than deduced** — the floor IS served, from the box's own rendered `GetFloor()` and a cold-started controller's settle-gate line. **§4.2's positive half is still NOT measured** and needs a rebuild. Evidence: `audits/CAMPAIGN-11-recovery-journey-2026-08-05.md`, `tests/campaign11-evidence-2026-08-05/journal-phase24.md` **2026-08-06 — THE FIVE PHASE-2 FINDINGS ARE FIXED AND THIS ROW STILL SAYS FAIL.** controller **v0.202.0** + agent **v0.126.0** close R-224, R-226, R-225, R-227, R-228. **What changed:** the unlock path now classifies WHY it failed, from the value and never the text, and the message that mentions typing is reachable from exactly one class — a real refusal — with **unknown defaulting to neutral**. Proven live on the venue with the same wrong code and only the hub's reachability changed: `400 → 502 → 400`. **What it does NOT change: this row.** Fixes are not a journey; nothing here walked a customer end to end, and three of the blockers that made Phase 1 fail are still open (**R-220** in particular is worked around BY HAND on the venue — without that unmount no app can be deployed on a rebuilt box). **What still needs proving:** a re-walk with no operator intervention, and specifically the customer-facing messages end-to-end — they were NOT re-driven, because `/recovery` correctly retires itself once the old data is set aside and restoring that state would be the reconfiguration the fix task forbade. Evidence: `felhom-controller/REPORT.md` **⚠ THE RE-WALK RAN 2026-08-06 (R-201, attended) AND THIS ROW STAYS FAIL — but it is a nearer no.** A brand-new appliance was built from the published ISO, given three sentinels, escrowed, backed up off-site, destroyed (guest purged AND both drives wiped), reinstalled and walked. **THE DATA: PASS** — all three sentinels byte-identical, including a 12 MB binary and an accented Hungarian filename **whose NAME BYTES are also identical**, restored in **23 s** out of the pre-destruction snapshot through the customer's own flow. **THE JOURNEY: FAIL — TWO dead ends against Phase 1's four.** (1) **R-218's CONSUME half**: the hub re-staged the credential and said "the box re-consumes on its next cycle"; a full cycle ran (positive control) and it did not — no customer-reachable action fetches it, and the only lever is a command line INSIDE THE GUEST. (2) **R-220**: the drives are still unenrollable after a rebuild, needing a Proxmox-host unmount, without which no app can be redeployed and the restore page stays empty. **The unaided RTO is therefore STILL UNDEFINED**; the attended figure was 30 m 13 s and must not be quoted as the customer number. **What passed and is new:** the recovery screen appeared WITHOUT being sought, answered all three of its questions with a seal date matching the hub exactly, the emailed reset code worked first try, and the unlock was a real 1.528 s unseal that placed the key. **⚠ AND IT DOES NOT CLAIM DELIVERY:** the fresh install landed on controller 0.201.0 / agent 0.125.0 — the vouched versions, neither carrying the fixes — which were installed BY HAND. Fleet delivery needs a golden carrying the fixed controller and a vouched agent, and nothing was vouched. Evidence: `tests/rewalk-r201-2026-08-06/journal.md`   **⚠ 2026-08-06 (later the same day) — THE TWO DEAD ENDS ARE CLOSED AND VOUCHED, AND THIS ROW STILL SAYS FAIL.** controller **v0.203.0** + agent **v0.127.0** close **R-218's consume half** and **R-220**, and golden **0.203.0** / agent **0.127.0** / `min_agent` **0.127.0** are now VOUCHED — so the delivery gap the re-walk recorded (“the fresh install landed on versions neither carrying the fixes”) is gone. **Both were then proven live on a genuinely rebuilt box** (Part 4 venue, VM 323): the reinstall landed on agent 0.127.0 with **no downgrade and no hand upgrade** (the previous re-walk's reinstall downgraded 0.126.0→0.125.0), `/disks/candidates` returned **both drives** after the guest was purged with the raw mounts still on the surviving host — before the fix, two empty lists — and both **re-attached through the customer endpoint**; the off-site credential was re-staged by `offsiteheal` at 13:24:57Z and **collected by the box on a tick, unaided**. **Why the row is still FAIL:** the walk did not finish. It stopped at the restore surface — **R-237**, the list was keyed on apps that are currently INSTALLED and currently TOGGLED ON for future backups, so a rebuilt box was shown nothing to restore while the repository held its snapshots. Fixed in **controller v0.204.0** (the store is now the source of the list) and proven live on that same venue with the toggle switched OFF, but **no sentinel was restored in that walk**, so the data half is unproven in EITHER direction for this venue. **R-238 was RECLASSIFIED, not fixed as filed:** `mode=full` without `confirm=1` is step 1 of a deliberate two-step that starts no job by design; the endpoint driver did not carry `?full_prep=` forward. The operator drove the same restore to completion in a browser. Its real residue — that step logging nothing at all, including when the headroom gate refuses — is fixed. **R-236 was WITHDRAWN: not a defect.** **This row goes green only when a walk completes end to end with no operator intervention, restoring a byte-identical sentinel.** Evidence: `tests/part4-rewalk-2026-08-06/journal.md`, `tests/teardown-2026-08-06.md`, `felhom-controller/REPORT.md` **⚠ THE FINAL WALK RAN OVERNIGHT 2026-08-06/07 AND THIS ROW STAYS FAIL — the data half passed again, the journey half failed further from the line than before.** A brand-new appliance was built from the published ISO, fixtured, soaked through a real scheduled cycle, destroyed and rebuilt. **THE DATA: PASS** — all three sentinels byte-identical out of snapshot `f5c53b03`, including a 12 MB binary and an accented Hungarian filename **whose name BYTES are identical too**. **THE JOURNEY: FAIL, and this time the customer has NO route at all** — `/` lands on the launcher with no recovery pointer, `/recovery` **302s away**, and the remote-backup page offers to **CREATE a new recovery code**, which would orphan the very history the customer's code protects. **There is no field anywhere to enter the code they hold**, and the operator's documented remedy refuses too. Cause (**R-241**): the credential self-heal — proven working unaided in the same walk, six hours earlier, as its best result — writes a FRESH repository password, which moves the box out of `OffsiteRecoveryOffer()`'s (a) pristine case while (b) orphan-detected is unreachable because runs are blocked by `escrow_state: pending`. **R-218's shape one level up.** **What DID pass, and is new:** the whole credential chain ran end to end with zero human action on an unclaimed box — declare, `offsiteheal` re-stages after two reports, the box's 5-minute retry collects it, tier applied — the **first live sighting** of that success line, settling R-218's consume half and R-236's withdrawal. Also new: **R-239** — a fresh install lands on controller **0.203.0** while 0.205.0 is released, so R-234 and R-237 are written but **not delivered**, proven from the customer's side (T2, T3). **This row goes green only when a walk completes with no operator intervention AND a byte-identical sentinel.** Evidence: `tests/finalwalk-r201-2026-08-07/journal.md` **⚠ 2026-08-07 — R-241 WAS DIAGNOSED BY A READ-ONLY SPIKE ON THE STANDING VENUE, AND THE DIAGNOSIS REVERSES THE FIX. This row is unchanged: still FAIL, and no code was written.** The question was whether R-241 is a screen-predicate defect or a minting defect. **It is a MINTING defect.** The screen was telling the truth — there genuinely was nothing recoverable under the key the box held, because the box minted that key itself over the top of a sealed package it already knew the hub was holding. **Three measurements, from the venue rather than from the earlier report:** (1) `WriteOffboxSecrets` (`offbox.go:411`) mints on ONE input — does the file exist — while `OffsiteRecoveryOffer()` and `needsOffsiteCredential()`, both in the same file, consult `GetHubEscrowIdentityPresent()`; the same fact is available on three paths and used on two. (2) That flag was the **precondition of the chain that reached the minting**: the retry job logs only when the declaration is live, and the venue logged `credential retry: … (the box still declares a need; retrying)` at **02:48:03Z** — thirty minutes and six ticks before the mint at 03:18:06Z. (3) **The box computed the answer and discarded it**: at **03:28:03Z**, thirty-five minutes before the customer looked, `EscrowAutoConfirmer.Reconcile` (`escrow_confirm.go:154`) logged `the hub's escrow blob does not cover the CURRENT repo password (hub hash 30ef574fe492… != local 9b4a9a9dcec7…)`. It is recomputed every report cycle and never persisted. **And the hub explicitly disclaims doing this** — `offsiteheal`'s package doc: *"credential automatic, key customer-present — the ruling this session implements and must not quietly widen."* **Shape (b) is structurally unreachable on this box** (the escrow gate at `offbox.go:743` sits upstream of `ensureOffboxRepo`, the only producer of `RepoState="orphaned"`; positive control: the scheduler was alive, 241 `agent-channel-health` ticks, and `offbox-backup` is a `sched.Daily` leg whose slot fell before the destruction). **Two by-products:** **R-243** — a box in this state silently stops backing up off-site and **no alarm fires** (`isStale` requires `escrowed`, `offsite_delivery_stuck` skips the `applied` shape, `backup_failed` needs a run that never happens); and the trap in the obvious fix — `ResetOrphanedRepo` clears the orphan flag without a ceremony, so a hash-mismatch discriminator alone would re-offer the screen forever to a customer who declined the old data. **Q7:** the „Helyreállítási kód létrehozása" button does NOT destroy the data — R-198's retention copies the `identity_blob` — but it converts a self-service recovery into one needing an unbuilt read path (R-199), and it re-enables the screen while invalidating the code that screen accepts. Evidence: `audits/SPIKE-r241-recovery-offer-2026-08-07.md` **⚠ 2026-08-07 (later) — R-241 IS FIXED (controller v0.206.0 + hub v0.98.0) AND THIS ROW STILL SAYS FAIL.** Three changes, following the spike's ruling rather than the obvious reading: the box **no longer mints a repository key while the hub holds a sealed package** (a conjunction, so a first-time box is untouched; the refusal is a HOLDING state that still writes the transport, declared as `offsite.state=awaiting_recovery_key` and shown inert to every existing hub reader); the **hub-vs-local key comparison that was computed every cycle and discarded is now persisted** and drives the offer as **shape (c)**; and **abandoning starts a 14-day countdown** whose terminal step removes the set-aside store and the sealed package **together**, so the offer ends because the state is right rather than because a flag suppresses it. Surface: the full page appears **once per ENTRY into the offered state**, three dismissal levers with three scopes, and **none removes the entry point**. Q7's trap is closed — „Helyreállítási kód létrehozása" is **unavailable** while a recovery is outstanding. **WHY THE ROW STAYS FAIL: these are fixes, not a walk.** Nothing here walked a customer end to end, and this row goes green only when one completes with **no operator intervention AND a byte-identical sentinel**. Two of Phase 1's blockers are also still open (**R-214**, **R-202**), and **R-240** is untouched. Evidence: `felhom-controller/CHANGELOG.md` v0.206.0, `hub/CHANGELOG.md` v0.98.0. **DELIVERY, separately: R-239 is CLOSED 2026-08-07** — golden **0.205.0** baked, published, round-trip verified (`./etc/felhom-controller-image` read OUT of the downloaded archive) and **VOUCHED**, so a fresh install now lands on 0.205.0 carrying R-234 and R-237. In the event it was a ONE-field change: `agent_version` and `min_agent` both stayed 0.127.0. Evidence: `tests/golden-0.205.0-2026-08-07/`. **The row still says FAIL** — delivery is not a journey, and R-241 is diagnosed, not fixed. **✅ 2026-08-07 — THE FIFTH WALK PASSED, BOTH HALVES, AND THIS ROW TURNS.** A brand-new appliance (VM **325**, customer `walk5`) was installed from the published ISO on `demo-hp`, claimed, given three sentinels, escrowed, backed up off-site, then **destroyed on purpose** — guest purged, both data drives wiped — and rebuilt through the documented day-0 path. **THE DATA: PASS** — all three sentinels byte-identical out of snapshot `5b0f20f7` (`11eb7fb2…`, `6b504d1e…`, `0baaf402…`), including a 12 MB binary and `WALK5-őrszem-ékezetes-árvíztűrő.txt` **whose name BYTES are identical too**, read back with `os.listdir` on a bytes path so no decode round trip could launder a `U+FFFD`. **THE JOURNEY: PASS — ZERO guest command lines were needed to progress**, against three on the previous walk. **RTO 71.7 s** from login to an open store (12.44 s of it the unseal; ~22 s a harness retry). The recovery screen **appeared without being sought** (`/` → `/launcher` → `/recovery`), answered all three of its questions, and its sealed-at timestamp matched `host_escrow.created_at` exactly. **WHAT MADE THE DIFFERENCE — R-241's mint guard, exercised live for the first time:** at **14:58:52Z**, unaided and before anyone logged in, the rebuilt box collected its re-staged credential, configured the transport and **refused to mint a repository password** over the sealed package — `NOT minting a repository password: the hub holds a sealed recovery package for this box`. Sampled every 20 s from T0: **no key at any moment**, with the scheduler proven alive throughout. At the equivalent moment the previous walk minted one and lost the journey silently. **AND DELIVERY IS PART OF THE PASS:** the fresh install landed on **controller 0.206.0 + agent 0.127.0 — the vouched set, no hand upgrade**, both times, so the box under test is the box a customer receives. **WHAT THIS ROW STILL DOES NOT CLAIM.** (1) **Putting files back in place is built and worked here** — `reconstitute` placed 6 files — but **only after two obstacles the customer must guess past**: **R-252** (the restore refuses with „nincs elérhető adatmeghajtó" because a rebuild loses the drive *registration*, and nothing on the recovery path says to re-attach) and **R-253** (the restore refuses because the app is not installed, on a page that says three lines above that the restore reinstalls it). Both were cleared **from the dashboard with no shell** — which is why the journey passes — but neither is signposted, so *unaided* here means *possible without a shell*, not *obvious*. (2) **Shape (c) did NOT fire positively.** With the mint guard holding there is no local key, so the offer comes from **shape (a)**; shape (c) was measured in Phase A in its **negative** half (hub hash == local hash, correctly silent). The mint guard is proven positively; the discriminator only negatively. (3) **R-214, R-202 and R-240 are untouched.** (4) The venue is **torn down** (2026-08-08) — full-schema census 168 rows → 67, of which 37 are audit-by-design and **30 are R-244**, predicted before the run rather than found after; every layer verified absent against a surviving positive control, and **16.64 GiB** returned. `tests/walk5-r201-2026-08-07/teardown-walk5-2026-08-08.md`. Evidence: `tests/walk5-r201-2026-08-07/journal.md` **SCOPE NOTE 2026-09-14 — not re-walked.** The first-hour drill on 0.242.0 (`audits/DRILL-fresh-install-0242-2026-09-14.md`) walked install → first use → restore of a deleted page on a fresh box, NOT a rebuild with off-site recovery (DR tier and off-site were off). This row is therefore neither re-proven nor contradicted on 0.242.0; its PROVEN-LIVE stands on 0.206.0 only. The first hour has its own row below. **SCOPE NOTE 2026-09-17 — not re-walked, and this time it could not be.** Chaos night (`audits/DRILL-chaos-night-2026-09-17.md`) ran twelve rounds of household actions under injected accidents on a fresh 0.245.0 box and **did not walk the recovery journey at all**. It could not have: the box was a REBUILD for an existing customer, so its restic repository was orphaned by design — zero readable snapshots, confirmed independently by `restic` (`Fatal: wrong password or no key found`, exit 1) and by the product's own status (`orphaned:true, snapshots:0, status:"error"`). The product surfaced that honestly as a true alarm within seconds of the first off-site run. This row is therefore **neither re-proven nor contradicted on 0.245.0**; its PROVEN-LIVE still stands on 0.206.0 only. + +**Update 2026-10-21 (rehearsal B7; controller 0.336.0, agent 0.161.0, hub 0.162.0): status PARTIAL.** [FACT] Tester4 was destroyed, reinstalled from the public ISO, re-linked after the operator deleted the old host, and claimed. The rebuilt dashboard showed only „A távoli mentés szünetel, amíg nem hozod létre a helyreállítási kódot" with „Helyreállítási kód létrehozása" (create, never enter); the restore page said „Nem tudjuk elolvasni a távoli tárolót". No front door reached the household's off-site history (R-1176, P1; the forgotten-press dead end R-1203). The August evidence stands for the versions it names; it does not hold today. `audits/rehearsal-2026-10-20/B7-died/README.md`. --- diff --git a/documentation/audits/rehearsal-2026-10-20/B10-stuck/README.md b/documentation/audits/rehearsal-2026-10-20/B10-stuck/README.md index ba0ea8de..d59329b7 100644 --- a/documentation/audits/rehearsal-2026-10-20/B10-stuck/README.md +++ b/documentation/audits/rehearsal-2026-10-20/B10-stuck/README.md @@ -14,8 +14,8 @@ Each point names what the family SAW, what it would do next, and the record (reg | 7 | new drive (storage init) | a required „Csatlakoztatási név * /mnt/…" field, empty; the button silently did nothing | press again; give up | the 200 GB disk unused until someone helps | W13 | | 8 | sharing („Megosztás") | „A megosztási szolgáltatás előkészítése folyamatban…" for ever — it waits for a share password the page never asks for | wait; ask for help | no shared folder until found | W21 | | 9 | errors in English | a failed restore, a failed off-site run, a failed install all show raw technical English („offbox backup immich: exit status 1…", „starting stack rallly: exit code 1…") | cannot tell what broke or what to do | a call to support each time | W22, W25, s1/s6/s5 READMEs | -| 10 | wrong „Első lépések" | Mealie's steps name the wrong address (mealie.… while the app is at recipes.…); Paperless says log in as „admin" while the family chose its own name; Vaultwarden says sign-up is closed while the family opened it | follow the steps, fail, think the app is broken | minutes per app; trust lost | W17–W19 | +| 10 | after the box died (B7) | the old link page: „new mail in a few minutes" — no mail until the operator deleted the old host; then the rebuilt box offered only a NEW recovery code and could not read the off-site store | wait; phone Felhom; start over with an empty box | the family's data out of reach, though it still exists | R-1203 (P2), R-1176 (P1) | -Close behind (not in the ten): the download link that does not look like a link (W2); the second disk never offered as +Close behind (not in the ten): three apps' wrong „first steps" (W17–W19); the download link that does not look like a link (W2); the second disk never offered as the backup place (R-1198); Jellyfin refused for memory with no help choosing what to remove (W20); the recovery-code page full of operator words (W10); Actual Budget wholly in English (W23). diff --git a/documentation/backlog/STATUS-HISTORY.md b/documentation/backlog/STATUS-HISTORY.md index 36f3de82..d4a53cfa 100644 --- a/documentation/backlog/STATUS-HISTORY.md +++ b/documentation/backlog/STATUS-HISTORY.md @@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S **Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.** +## Moved from STATUS on 2026-10-21 + +**Updated 2026-10-20 (day).** Hub 0.162.0, agent 0.161.0 (you vouched it) and installer 1.35.0 are live, and a failed off-site restore now puts the app back exactly as it was (your D20 = A; controller 0.336.0 is the floor). This page is one screen again: older news moved to the history file (`documentation/backlog/STATUS-HISTORY.md`), and nothing was deleted. A long rehearsal on a fresh box runs today and tonight. + ## Moved from STATUS on 2026-10-20 `STATUS.md` was rebuilt from the register on 2026-10-20 (107 lines and 31,695 bytes before; one screen after). Nothing