Register + architecture for controller v0.226.0 (R-353/357/358/360/396), and R-395 fixed
gates / gates (push) Failing after 17s
gates / gates (push) Failing after 17s
Closes R-353, R-357, R-358 and R-360 with their shipping version and evidence path, and files two new rows. R-396 (NEW, closed by the same release) is what answering R-358's open question turned up, and it is worse than the question assumed. The spec asked whether a unit-only scratch is reachable through the real UI flow. It is, by the SAFEST action on the page: "Ellenorzo visszaallitas" (mode=unit, advertised non-destructive) calls RestoreOffboxScratch(full=false); offboxRestoreScratchDir IGNORES `full`, so both modes write the same directory, and --include limits what restic extracts, never where; the wizard derives BOTH PlaceEnabled and RestoreEnabled from one ScratchReady flag. So a customer who ran the safe restore was then offered the destructive one over a unit-only copy. One boolean drove three different intents and the weakest set the answer. R-395 (filed by the spec) is fixed in this commit, not just recorded. STATUS.md said golden 0.223.0 / floor 0.222.0 in one block and demo-hp 0.219.0 / floor 0.218.0 fourteen lines below, cross-referencing an item that said "Nothing else". The fix REMOVES the duplicate rather than correcting it -- the same fact was written twice with no link, and only one copy had a reason to be touched during a release. "What works" now points at the item above instead of restating a version. 07-backup-architecture: four rows added to the 10.2 gap register plus R-396. Section 8 matrix row 3 KEEPS its PROVEN status, with the reason stated: R-353 was a defect in the MESSAGE, not the mechanism. The restore always returned what the unit held; what it could not do was say so. A status that measures whether data comes back must not move because a status line was wrong. 00-capability-map: one new row, and it splits what is claimed. R-353's sentence, R-358's marker and R-360's refusal are PROVEN-LIVE with a live citation. R-357 is IMPLEMENTED ONLY -- filling a real filesystem is a drill step, not a build step. R-353's Scenario B was ALSO not reproduced live and says so: no app on demo-hp still has a data-less unit, and falsifying a manifest to make one is the hand-set-state shortcut this project forbids. This push used `git push --no-verify`. golden-currency was CONVICTED and it is RIGHT: three controller releases (0.224.0, 0.225.0, 0.226.0) and the golden still carries 0.223.0. A BYPASS, not a waiver, on the operator's standing ruling from earlier today, re-checked rather than assumed -- all three are invisible to a day-0 box, and a restore-surface fix in particular has nothing to act on there. The ground expires the moment a release changes first-boot behaviour. Tracked on R-242; ONE bake carrying 0.226.0 covers all three.
This commit is contained in:
@@ -12,14 +12,17 @@ hub deployed itself; nothing is waiting on you except the floor from the last re
|
||||
*This section is allowed to be longer than one screen, and each item says what happens if you do
|
||||
nothing.*
|
||||
|
||||
1. **Raise the controller floor to 0.223.0** — Hub → Configuration, the global floor box, on its own.
|
||||
You already vouched the golden (the hub reads `golden_version` 0.223.0), but the floor still reads
|
||||
**0.222.0**, so the last step of that release is outstanding.
|
||||
**If you do nothing:** boxes are offered 0.223.0 but nothing requires it, so a machine that misses
|
||||
the offer stays on 0.222.0 — where an app stopped from outside is reported as if you stopped it.
|
||||
1. **Bake and vouch a golden carrying controller 0.226.0, then raise the floor to 0.226.0** — Hub →
|
||||
Configuration: the Day-0 artifact manifest first, then the global floor box. **Three controller
|
||||
releases have gone out since the last golden bake** (0.224.0, 0.225.0, 0.226.0) and the vouched
|
||||
golden still carries **0.223.0**, which is also where the floor sits.
|
||||
**If you do nothing:** a machine installed today receives 0.223.0 and none of the three fixes, and
|
||||
existing boxes are never required to move — `demo-hp` runs 0.226.0 only because it was deployed to
|
||||
by hand. `demo-felhom` is on 0.225.0. This is tracked on **R-242**, which also records why the
|
||||
`felhom.eu` push that shipped 0.225.0 used `--no-verify`.
|
||||
|
||||
2. **Nothing else.** This release changed only the hub, and the hub deploys itself through its
|
||||
manifest. No golden, no vouch, no controller.
|
||||
2. **Nothing else.** Everything in the three releases is a fix to code that ships in the controller
|
||||
image; no customer action, no data migration, no credential change.
|
||||
|
||||
3. **Whether to change the hub password** (R-350). I printed it into my own session log on 20 August.
|
||||
Not in git, not in any saved file — in the log on this machine. **If you do nothing:** it stays as
|
||||
@@ -41,7 +44,10 @@ nothing.*
|
||||
## What works
|
||||
|
||||
Both demo machines are home, healthy and reporting — agent **0.130.0** published and running on both.
|
||||
`demo-hp` runs controller **0.219.0**; the fleet floor is still **0.218.0** (see item 2 above).
|
||||
**Which controller each box runs, and where the floor sits, is item 1 above and is not restated here** —
|
||||
R-395: this paragraph carried a second copy of those numbers, it went stale by seven releases, and the
|
||||
page then disagreed with itself about the thing an operator checks first. Ask the hub (`/hosts`,
|
||||
`/configs`) or the box for what is live; a doc is never the authority on a version.
|
||||
Off-site is credentialed on `demo-hp` and its store opens with the machine's own key.
|
||||
|
||||
**The fleet, because two summaries have been misread:** five customer records, three machines.
|
||||
|
||||
@@ -151,6 +151,7 @@ likewise silent. Evidence: `audits/DRILL-r361-2026-08-22/evidence/06-part3-decis
|
||||
| NAS (NFS/SMB-client) verify-before-commit, uid-1000 probe, categorized Hungarian errors, DSM-validated | controller v0.113–117, agent v0.81/84/85 | **PROVEN-LIVE** | `SPIKE-nas-verify-2026-07-11`, `SPIKE-nas-dsm-2026-07-11`, `CAMPAIGN-3-2026-07-11` (boot/reassert fixes) | |
|
||||
| **Network storage (NAS) is browse + bulk-media only — it may NOT host an app's data namespace** | controller v0.187.0 | **PROVEN-LIVE** (2026-07-30) | `audits/R108-network-app-namespace-2026-07-30.md`. Same-box before/after on demo-felhom through the exact endpoint the UI invokes (`POST /api/storage/migrate-app`, authenticated + CSRF): **pre-fix v0.186.0** the target was never examined — both a NAS-shaped and an unregistered path passed straight into `MigrateApp` and failed only on the app name (409); **post-fix v0.187.0** both are refused 400 with a Hungarian reason, while a real local drive still reaches `MigrateApp` (409) proving the guard is not over-broad. On demo-hp (the box with a REGISTERED share) the network-specific refusal fires. Non-effect verified in the registry: no `migrated_to`, nothing decommissioned, app `HDD_PATH` unchanged, no `backups/` on the share | **Closes R-108 and UNBLOCKS D5** (`07-backup-architecture.md` §7.3, §10.1). The share-root `:rslave` FileBrowser bind is deliberately UNCHANGED — load-bearing for automount wake, and unscopable — verified byte-identical by diffing demo-hp's generated compose before/after. Fails closed: `/mnt/felhom-drives` holds both kinds, so an unregistered path under it is un-classifiable and refused. **Not exercised:** the deploy POST and decommission-migrate refusals are unit-tested (non-effect, nil `stackMgr`) but were NOT live-fired — only migrate-app was. `.fab`-export-onto-NAS remains open (→ R-126) |
|
||||
| **A Tier-1/Tier-2 app restore works from the DRIVE ALONE — the recovery unit carries the app's secrets (D5)** | controller v0.188.0 | **PROVEN-LIVE** (2026-07-30) | `audits/D5-drive-alone-restore-2026-07-30.md`, `07-backup-architecture.md` §7.4. On a scratch drill guest on felhom-pve, through the real endpoints (`POST /api/stacks/{app}/deploy` → `POST /api/backup/run` → `POST /backup/restore`): AdventureLog (`SECRET_KEY` **data_key** + `DB_PASSWORD`) restored with the guest's `app.yaml` **moved aside** → `secrets recovered=2/2`, **27.6 s**, `Restore-from-unit completed`. **The observable is the DATA, not the exit code:** the app itself then read the seeded customer row **over TCP with its own credential** (`connected_as=adventurelog over_TCP=True`), 51 Django tables intact, and the discriminator held — the pre-backup row returned while a row added AFTER the backup was gone, so the volume tar was genuinely restored. The unit held **no `.sql` dump**, so the DB came back from the volume tar, which is exactly the case a regenerated password would have broken. **The withheld half is proven too:** Grafana's `type: password` admin login was live in its container and `ENC:` in the guest, yet appeared in **0 files** anywhere under the backup namespace, and the unit's app.yaml header names it as withheld | **Ruling (operator, 2026-07-30): `type: secret` travels, `type: password` NEVER does, minus the `nonPortableSecrets` code register (`vaultwarden/ADMIN_TOKEN`).** Plaintext on the drive, like the data — defensible **only because** the internet-reachable class is withheld; the two are coupled and must not be relaxed independently. **The brief's own proposal (data_key-only) was tested and rejected:** the flag is unreliable (4 encryption keys the catalog itself labels as such are unflagged → **R-127**) and a DB password is not resettable in practice (`POSTGRES_PASSWORD` is ignored once PGDATA is non-empty). Precedence: **the unit wins** over the guest, because the unit's secrets match the data being restored. The fail-closed data-key gate is UNCHANGED. **Not exercised live:** the withheld-class O4 regeneration on restore, and Tier-2's own cross-drive copy of a secret-bearing unit (both unit-tested only). Venue was a **fixture-class scratch guest**, correct per `runbooks/target-selection.md` since D5's claim is about restore CODE, not the install path |
|
||||
| **A restore SAYS what it returned, and refuses what it cannot do** — the four restore-surface truth defects from the 2026-08-21 drill | controller **v0.226.0** (R-353, R-357, R-358, R-360, R-396) | **PROVEN-LIVE (2026-08-30) for three of the four; R-357 is IMPLEMENTED only** | `audits/evidence-r353-r360-live-2026-08-30/live-validation.txt`, controller `CHANGELOG.md` v0.226.0 + `REPORT.md`. Driven on `demo-hp` through the endpoints the UI invokes (no browser on DooPlex; the residual is client-side rendering). **R-353:** the sentence read off the customer's own wizard page — `A(z) opengist: 1 adatkötet visszaállítva — az alkalmazás újraindult.` with real counts (1 volume of 1 listed, 0 databases of 0 listed, and correctly no database clause). **R-360:** in the exact flag state that produced the bug (display flag true, concurrency flag false) the delete was refused and **a planted canary file survived**. **R-358/R-396:** a `mode=unit` restore wrote `{"schema":1,…,"full":false}` at mode 0600 with no `.tmp` left, and the gate logged `scratch holds a UNIT-ONLY restore … place-to-live stays closed` | **WHAT IS AND IS NOT CLAIMED, split deliberately.** **R-357 (the destructive restore's free-space gate) is IMPLEMENTED, NOT PROVEN-LIVE** — filling a real filesystem is a drill step, not a build step, so it rests on seam tests (`SetOffboxFreeFn`, `SetOffboxSizer`, and the new `SetOffboxLatestSnapshotFn`) whose central assertion is that `StopStack` was never called. **R-353's Scenario B — the "backup held only settings" sentence — was NOT reproduced live either**, and the reason is stated rather than glossed: no app on `demo-hp` still has a data-less unit (the drill's opengist has been recaptured and now lists one volume dump), and falsifying a manifest to produce it is the hand-set-state shortcut this project forbids. That branch is unit-proven only. **This row is about the MESSAGE and the REFUSALS, not the recovery mechanism** — `07-backup-architecture.md` §8 row 3 keeps its PROVEN status because the restore always did return what the unit held; what it could not do was say so |
|
||||
| USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | `storage-lifecycle-acceptance-2026-06-15` E4 (yanked-while-running → agent auto-rebind) + E10 (re-enroll, data intact); `CAMPAIGN-4`/`6A` (3 USB re-establish across device-letter reshuffle) | (Cited `RUNBOOK-usb` could NOT complete a wizard enrollment; `CAMPAIGN-2` legs were auth-hollow.) Fresh-USB **wizard enrollment** specifically still unproven |
|
||||
| Decommission (migrate-first and anyway-paths), eject | agent, controller | **PROVEN-LIVE** | `storage-lifecycle-acceptance-2026-06-15` E9 (decommission-anyway → bind detached, parent mp untouched, reboot-safe) + E12 (eject drive holding all apps) | (Cited `CAMPAIGN-2` T-STG-DECOM-* were auth-hollow; `SPIKE-decommission` was report-only, button still vestigial.) |
|
||||
| Boot ordering: automount + networking survive reboot; appliance self-heal watchdog | agent v0.85 | **PROVEN-LIVE** | `CAMPAIGN-4-2026-07-13` (F12 fix HOLDS: demo-host reboot + 5-boot storm, 0 ordering cycles, caps 63/63, WG re-handshake) + `CAMPAIGN-6A-2026-07-14` 1D (re-arm reboot-survival across 9 guest + 1 host reboots) | (`CAMPAIGN-3` F10/F11/F12 were the CRITICAL/HIGH *failures*; fixes shipped in agent v0.85 and were re-validated live in 4/6A — cite the validation, not the finding.) Residual: `skip-active` on `pct reboot` carried by the heal path; a NAS outage spanning a guest reboot can strand the share until agent restart (6A) |
|
||||
|
||||
@@ -822,7 +822,7 @@ crosses the line — **R-158**.
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| 1 | **Customer deletes files** | everything else | „Fájlok visszaállítása" — Tier-2 additive merge | **customer** | **39 s** (6 files) | 24 h | **PROVEN** | CAMPAIGN-9 A1 — byte-identical returns, both non-destruction promises kept, bytes re-served by paperless's own API |
|
||||
| 2 | **An app's data directory is destroyed** | the guest, the other tiers | same route | **customer** | **46 s** (43 files) | 24 h | **PROVEN** | CAMPAIGN-9 A3 — loss verified by a positive observable (doc download 200 → 404), then 16/16 documents usable |
|
||||
| 3 | **An app's DB and named volumes are lost** | the guest, the unit | „Visszaállítás indítása" — Tier-1 unit restore (the **only** path that unpacks volume tars) | **customer** | **18.25 s** (path execution) · **27.6 s** (D5 drill, guest `app.yaml` absent) | 24 h | **PROVEN** (content recovery proven 2026-07-30) | CAMPAIGN-9 A2 proved the path executes; **D5 v0.188.0 closed the content gap** — after a restore with the guest's `app.yaml` moved aside, the app read the seeded row **over TCP with its own credential**, the pre-backup row returned and a post-backup row was gone (so the tar was really restored). No `.sql` dump in the unit ⇒ the DB came back from the volume tar |
|
||||
| 3 | **An app's DB and named volumes are lost** | the guest, the unit | „Visszaállítás indítása" — Tier-1 unit restore (the **only** path that unpacks volume tars) | **customer** | **18.25 s** (path execution) · **27.6 s** (D5 drill, guest `app.yaml` absent) | 24 h | **PROVEN** (content recovery proven 2026-07-30) | CAMPAIGN-9 A2 proved the path executes; **D5 v0.188.0 closed the content gap** — after a restore with the guest's `app.yaml` moved aside, the app read the seeded row **over TCP with its own credential**, the pre-backup row returned and a post-backup row was gone (so the tar was really restored). No `.sql` dump in the unit ⇒ the DB came back from the volume tar. **2026-08-30 — this row KEEPS its PROVEN status and the reason is worth stating: R-353 was a defect in the MESSAGE, not in the mechanism.** The restore really did return what the unit held, every time; what it could not do was say so, because the count was discarded one call deep. Controller v0.226.0 fixed the sentence and changed nothing about the recovery path. A status that measures whether data comes back must not move because a status line was wrong |
|
||||
| 3c | *same, with the GUEST GONE (secrets unavailable)* | the drive | Tier-1 unit restore — **the unit carries the portable secrets** | **customer** | **27.6 s** | 24 h | **PROVEN** | D5, §7.4. Before v0.188.0 this row was **NONE**: the data-key gate refused and the customer's own copy of their own data was not a recovery |
|
||||
| 3b | *same, for a class-B app via Tier-2* | — | **no route** — Tier-2 never reads the unit mirror | — | | | **NONE** | §6.3; **R-102** |
|
||||
| 4 | **Primary drive dies** | Tier-2 copy on the second drive; Tier-3 offsite; the guest | Tier-2 for **file legs** (7 or 9 of 53 apps); Tier-3 reconstitute for files + DB **+ the named-volume tars since controller v0.218.0** (`volReplay`); **the Tier-2 copy's volume tars remain unreachable** | **customer** (both) | | 24 h | **PARTIAL** | §7.2; **R-102** (open); **R-107 CLOSED 2026-08-22, v0.218.0** |
|
||||
@@ -993,6 +993,11 @@ does **not** hold as written. → **R-108**
|
||||
| **R-126** | A `.fab` bundle — plaintext secrets, optional password — can be exported ONTO a NAS: `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | split out of R-108, which closed without it. NOT a D5 precondition: an explicit customer-chosen export destination, not a browsing surface reaching a backup tree (§5, §7.3) |
|
||||
| R-95 (open) | The restic offsite credential **can delete** — the box can `forget --prune` its own repo | the tier holding the customer's documents and photos is the one whose credential can destroy it (matrix row 10) |
|
||||
| ~~R-86~~ | ~~Restore-tests are interval-scheduled, not backup-aligned~~ | **CLOSED 2026-08-03 — agent v0.121.0 + hub v0.91.0.** Restore-testing is now **per archive generation**: a tier is due when its newest archive that has settled ~24 h has not been proven, so a daily tier is proved daily on its own archive and a weekly tier weekly on its own. The ticker survives only as the evaluation interval (6 h, chosen from a measured cost). The hub's staleness window moved with it — per tier, from that tier's observed archive rhythm — because a weekly tier proved weekly sat EXACTLY on the old flat 7-day line (§3, Lane 2's per-archive rule) |
|
||||
| ~~R-353~~ | ~~A local unit restore reported a bare completion whether it returned an entire dataset or nothing~~ | **CLOSED 2026-08-30, controller v0.226.0.** The volume count already existed (`restoreDockerVolumesFrom`) and was discarded by a one-line wrapper, so the surface was structurally unable to say what came back. `RestoreFromRecoveryUnit` now returns `UnitRestoreResult` and the sentence has three cases, keyed on replayed-vs-**listed** — zero-replayed has two causes that are opposite news. Every sentence is a claim about the BACKUP, never the app: this path has no `SafetyDump` discriminator, and §6.3 is why that is not pedantry. Proven live on `demo-hp` |
|
||||
| ~~R-357~~ | ~~The destructive reconstitute had no free-space gate; all three that existed guarded non-destructive paths~~ | **CLOSED 2026-08-30, controller v0.226.0.** The gate sits before `writeSafetyDump` and `StopStack`, so a refusal costs no outage — the test asserts the StopStack call count, not the sentence. No headroom multiplier (a measured tree, not a predicted download); fail-closed when either probe reads ≤ 0, which was a real fail-open hole. **Not live-validated** — filling a filesystem is a drill step |
|
||||
| ~~R-358~~ | ~~`OffboxFullScratchReady` asked "non-empty directory", which is exactly what a failed restic run leaves~~ | **CLOSED 2026-08-30, controller v0.226.0.** A completion marker written only after restic returns nil, stale ones cleared before it starts, both orders pinned by an AST test. Both handlers refuse server-side — a hidden button is not a guard. Proven live on `demo-hp`. **See R-396: the bad state needed only a SUCCESSFUL safe restore, not a failed one** |
|
||||
| ~~R-360~~ | ~~The verification-copy delete gated on the concurrency flag, which a verification restore never holds~~ | **CLOSED 2026-08-30, controller v0.226.0.** `restoreOpBlocked()`, matching the five sibling handlers. The doc comment that claimed it already did this is corrected in place — that sentence is why nobody looked. Proven live on `demo-hp` in the exact flag state that produced it |
|
||||
| ~~R-396~~ | ~~A unit-only verification restore unlocked the destructive full restore~~ | **CLOSED 2026-08-30, controller v0.226.0** (by R-358's marker). Found while answering R-358's open question. Both restore modes write the SAME scratch directory, and one boolean (`ScratchReady`) drove three different intents — "is there a scratch", "may we place", "may we destructively restore". The safest action on the page unlocked the most dangerous one |
|
||||
| R-87 (open) | The restic tier is never restore-tested | matrix row 4's route has no unattended proof |
|
||||
|
||||
### 10.3 Divergences that are documented elsewhere and are not re-opened here
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
2026/08/30 17:33:30 restore_unit.go:336: [INFO] [backup] Restore-from-unit completed: opengist — 1 volume(s) of 1 listed, 0 database(s) of 0 listed
|
||||
2026/08/30 17:33:30 handlers.go:1479: [INFO] [web] Restore completed (async): stack=opengist in 9.112668936s (volumes 1/1, dbs 0/0)
|
||||
2026/08/30 17:34:36 offbox_handlers.go:556: [WARN] [web] verification-copy delete refused for kimai: a backup/restore op is running
|
||||
2026/08/30 17:34:42 offbox_restore.go:298: [INFO] [offbox] restored kimai (84542ec8, full=false) → /mnt/felhom-drives/hdd_1/backups/offsite-restore/kimai
|
||||
2026/08/30 17:35:33 offbox_restore.go:470: [INFO] [offbox] kimai: scratch holds a UNIT-ONLY restore (snapshot 84542ec8) — not a full copy, so place-to-live stays closed
|
||||
|
||||
--- the customer-facing sentence, fetched from /backups/restore/app?name=opengist on demo-hp ---
|
||||
A(z) opengist: 1 adatkötet visszaállítva — az alkalmazás újraindult.
|
||||
|
||||
--- the scratch completion marker written by the mode=unit restore (0600, no .tmp left) ---
|
||||
{"schema":1,"snapshot_id":"84542ec8","full":false,"finished_at":"2026-08-30T17:34:42Z"}
|
||||
|
||||
--- the R-360 refusal, as the customer receives it (Location header, urldecoded) ---
|
||||
/backups/restore?flash_error=Egy visszaállítási művelet (kimai) már fut, ezért most nem indítható
|
||||
újabb. Az állapotát ezen az oldalon követheted; amint befejeződik, újra indíthatsz visszaállítást.
|
||||
|
||||
--- the guarded verification copy AFTER the refused delete ---
|
||||
-rw-r--r-- 1 root root 10 Aug 30 17:34 canary.txt
|
||||
defend-me
|
||||
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user