R-212 CLOSED: the three orphaned stores deleted after a corrected list (~1.45 GB)
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
The register said 'two set-aside stores, ~1.2 GB'. Measured before touching anything: THREE set-aside stores totalling ~1.45 GB, and the thing that was exactly 1.2 GB was demo-felhom's LIVE felhom-repo. Matching on the size would have deleted a working repository. The operator was shown the corrected list and confirmed 'delete all three'. Deleted: demo-felhom orphaned-20260717 (1.4 G) + orphaned-20260718 (3.0 M); demo-hp orphaned-20260804 (43 M). Both LIVE repos untouched, confirmed by full listings before and after on each account. Proof nothing live was caught: a real off-site run on demo-hp immediately afterwards returned status ok, orphaned false, no error, 6 snapshots. Method note recorded for the next session: the storage box has a RESTRICTED shell. 'test -d X && rm -rf -- X' returns 'Command not found' and does nothing (it failed CLOSED, verified by an unchanged listing); 'rm -r <path>' as one simple command is the working form.
This commit is contained in:
1 parent
4faebe2926
commit
63e0ac01f2
3 files changed
+36
-12
No files matched your search
+25
-3
@@ -125,7 +125,7 @@ initially seeded `Status: "blocked"` through `SaveCustomerConfig`, whose INSERT
|
||||
column — so the customer was never actually blocked and the assertion would have been vacuous. It now
|
||||
goes through `SetCustomerConfigStatus` **and asserts `IsCustomerBlocked` before proceeding**.
|
||||
|
||||
## 8. Part 4 — HALTED AT ITS STOP. Nothing was deleted.
|
||||
## 8. Part 4 — STOPPED, corrected, then COMPLETED with the operator's confirmation
|
||||
|
||||
§8.7: *"If the paths do not match R-193's record exactly, STOP. A near-match on a protected endpoint
|
||||
is not a match."* **They do not match.**
|
||||
@@ -143,7 +143,29 @@ Measured read-only over SFTP, using each box's own credential, from inside its g
|
||||
The ruling says *"~1.2 GB across the two demo boxes, in set-aside stores"*. Reality: **three** set-aside
|
||||
stores totalling **~1.45 GB** — and **the figure that matches ~1.2 GB is demo-felhom's LIVE
|
||||
`felhom-repo`**. Had the size been used to identify the target, the live repository would have been
|
||||
deleted. **Filed as R-212, WAITING-ON-OPERATOR.**
|
||||
deleted. Filed as **R-212**, and the operator was asked with the corrected list.
|
||||
|
||||
**The operator confirmed *delete all three*, and all three were deleted.**
|
||||
|
||||
| Account | Deleted | Freed |
|
||||
|---|---|---|
|
||||
| `u629488-sub1` | `felhom-repo.orphaned-20260718` | 3.0 M |
|
||||
| `u629488-sub1` | `felhom-repo.orphaned-20260717` | 1.4 G |
|
||||
| `u629488-sub3` | `felhom-repo.orphaned-20260804` | 43 M |
|
||||
|
||||
**AFTER, on each account, a full listing:** `u629488-sub1` holds `.ssh` + `felhom-repo` (**1.2 G**,
|
||||
live); `u629488-sub3` holds `.ssh` + `felhom-repo` (**582 K**, live). **Nothing outside the three named
|
||||
paths was touched**, and no prune job, datastore or live repository was involved.
|
||||
|
||||
**Proof nothing live was caught:** a REAL off-site run triggered on demo-hp immediately afterwards
|
||||
(`POST /backup/offbox/run`, authenticated + CSRF) returned `status: "ok"`, `orphaned: false`,
|
||||
`last_error: ""`, `last_run: 2026-08-05T09:14:02Z`, `last_duration: 1m24s`, 6 snapshots.
|
||||
|
||||
**METHOD NOTE, worth carrying forward.** The Hetzner storage box runs a **restricted shell**: no shell
|
||||
operators, no `test`, no GNU long flags. The first attempt used `test -d X && rm -rf -- X` and got
|
||||
*"Command not found. Use 'help' to get a list of available commands."* — **it failed CLOSED, verified
|
||||
by a byte-identical before/after listing.** `rm -r <path>` issued as ONE simple command is the working
|
||||
form, and the smallest store was deleted first to confirm the syntax before the 1.4 GB one.
|
||||
|
||||
## 9. Live validation
|
||||
|
||||
@@ -153,7 +175,7 @@ deleted. **Filed as R-212, WAITING-ON-OPERATOR.**
|
||||
| 2 | **The declared state, produced live without wiping a box** | demo-felhom 9201 arranged **reversibly** into the stranded shape (settings + `offbox/` backed up first; the `offbox` key removed and `repo_password` moved aside). Report **id=16743** reached the hub carrying `{enabled:false, state:"needs_credential", quota_gb:0, repo_size_bytes:0}`. Restored the same minute; report **id=16744** is healthy again. **The single declaration was absorbed by the debounce — no self-heal event fired** — which is Scenario F demonstrated on live infrastructure rather than in a fake. |
|
||||
| 2b | **The ACK field is no longer discarded** | Both demo boxes' `settings.json` now carry `hub_escrow_identity_present = true` — the recorder working on a HEALTHY box, which is the case that used to return early. |
|
||||
| 3 | **A push from outside the workspace is refused** | A scratch clone at `/tmp/.../outside-clone`: `pre-push: PUSH REFUSED - this clone is OUTSIDE the felhom workspace`, naming `/mnt/5_hdd/felhom.eu`, **before the gates run**. **Red-proof:** with the assertion removed the same push **succeeded** (`rc=0`, new branch on a throwaway bare remote). In-workspace pushes ran normally all session. |
|
||||
| 4 | **Part 4** | **Not performed — halted at the STOP.** Listings above. |
|
||||
| 4 | **Part 4** | **Completed after the STOP and a corrected list.** Full listings before and after on both accounts; both LIVE repos intact; a real off-site run on demo-hp succeeded immediately afterwards (`status: ok`, `last_run 09:14:02Z`). |
|
||||
|
||||
**Not fired live: the hub actually re-staging a credential.** Doing so would have re-applied
|
||||
demo-felhom's off-site target mid-session and changed the very state Part 4's listing describes. It is
|
||||
|
||||
@@ -47,6 +47,14 @@ over; and the file was restored **byte for byte identical**. *(R-201)*
|
||||
|
||||
## What shipped recently
|
||||
|
||||
- **2026-08-05** — **The orphaned backups are deleted — and the list you were given was wrong, which
|
||||
is why you were asked again.** You had approved "about 1.2 GB in two set-aside stores". Measured
|
||||
before touching anything: there were **three** set-aside stores totalling **~1.45 GB** — and the
|
||||
thing that was exactly 1.2 GB was demo-felhom's **live** store. Matching on the size would have
|
||||
deleted a working backup. With the corrected list confirmed, all three were removed and both live
|
||||
stores left alone; a real off-site backup ran successfully straight afterwards to prove nothing
|
||||
working had been caught. *(R-212)*
|
||||
|
||||
- **2026-08-05** — **A rebuilt machine now asks for its storage credential, and the hub gives it back.**
|
||||
The machine says plainly what it needs — it can tell it has been rebuilt, because its data area is
|
||||
empty *and* the hub is holding a sealed recovery package for it — instead of leaving the hub to guess
|
||||
@@ -107,13 +115,7 @@ over; and the file was restored **byte for byte identical**. *(R-201)*
|
||||
(controller up to 0.135.0, hub up to 0.57.0; everything newer is safely in the registry). **Nothing
|
||||
was deleted.** Worth knowing before you spend time on it: they only account for about 27 GB against
|
||||
199 GB now free, so this is about clutter, not space. *(R-210)*
|
||||
- **The orphaned-backup deletion is STOPPED and needs your ruling — the list does not match.** You
|
||||
asked for about 1.2 GB in two set-aside stores to be deleted. Measured today, read-only: demo-felhom
|
||||
holds a **live** store of 1.2 GB plus set-aside stores of **1.4 GB** and **3 MB**; demo-hp holds a
|
||||
live store of 582 KB plus a set-aside store of **43 MB**. So there are **three** set-aside stores
|
||||
totalling ~1.45 GB, not two — **and the thing that is exactly 1.2 GB is demo-felhom's LIVE store**,
|
||||
which must not be deleted. **Nothing was deleted.** Tell me which of the three `orphaned` stores to
|
||||
remove. *(R-212)*
|
||||
- **Nothing.**
|
||||
- **The recovery screen you described has been priced, and it can be built.** A freshly installed
|
||||
machine that finds a sealed package waiting should say so, offer a box for the recovery code, and
|
||||
show what would come back before doing anything. One thing to weigh, deliberately not decided: that
|
||||
|
||||
@@ -57,7 +57,7 @@ drill needed — the fourth is R-193, and until it goes the recovery is still op
|
||||
|
||||
| ID | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|
|
||||
| **R-212** | **The orphaned-ciphertext deletion HALTED: the stores on the storage box do not match this register's record.** Ruling 2 (2026-08-04) says *"delete the orphaned ciphertext, ~1.2 GB across the two demo boxes, in set-aside stores nothing prunes"*. **Measured 2026-08-05 over SFTP with each box's own credential, read-only:** demo-felhom (`u629488-sub1`) holds `felhom-repo` **1.2 G**, `felhom-repo.orphaned-20260717` **1.4 G**, `felhom-repo.orphaned-20260718` **3.0 M**; demo-hp (`u629488-sub3`) holds `felhom-repo` **582 K**, `felhom-repo.orphaned-20260804` **43 M**. So there are **THREE** set-aside stores totalling **~1.45 GB**, not two totalling ~1.2 GB — and **the figure that DOES match ~1.2 GB is demo-felhom's LIVE `felhom-repo`**, which is the configured `repo_path` and must not be deleted (it is unopenable by the box, which is R-193's point, but it is not a set-aside store) | **WAITING-ON-OPERATOR — deletion NOT performed** | an exact path list, confirmed by the operator | Confirm which of the three `*.orphaned-*` paths to delete. **Nothing was deleted**; the listing above is the complete measured state | Operator |
|
||||
| **R-212** | **The orphaned-ciphertext deletion HALTED: the stores on the storage box do not match this register's record.** Ruling 2 (2026-08-04) says *"delete the orphaned ciphertext, ~1.2 GB across the two demo boxes, in set-aside stores nothing prunes"*. **Measured 2026-08-05 over SFTP with each box's own credential, read-only:** demo-felhom (`u629488-sub1`) holds `felhom-repo` **1.2 G**, `felhom-repo.orphaned-20260717` **1.4 G**, `felhom-repo.orphaned-20260718` **3.0 M**; demo-hp (`u629488-sub3`) holds `felhom-repo` **582 K**, `felhom-repo.orphaned-20260804` **43 M**. So there are **THREE** set-aside stores totalling **~1.45 GB**, not two totalling ~1.2 GB — and **the figure that DOES match ~1.2 GB is demo-felhom's LIVE `felhom-repo`**, which is the configured `repo_path` and must not be deleted (it is unopenable by the box, which is R-193's point, but it is not a set-aside store) | **CLOSED 2026-08-05 — all three deleted after the operator confirmed the corrected list** | an exact path list, confirmed by the operator | Confirm which of the three `*.orphaned-*` paths to delete. **Nothing was deleted**; the listing above is the complete measured state | **DONE 2026-08-05.** The operator was shown the measured listing, confirmed *delete all three*, and all three `*.orphaned-*` stores were removed: demo-felhom `felhom-repo.orphaned-20260717` (1.4 G) + `felhom-repo.orphaned-20260718` (3.0 M), demo-hp `felhom-repo.orphaned-20260804` (43 M). **~1.45 GB freed.** **Both LIVE `felhom-repo` stores are untouched** (demo-felhom 1.2 G, demo-hp 582 K) — confirmed by a full listing before AND after on each account. **Proof nothing live was caught:** a REAL off-site run on demo-hp immediately afterwards returned `status: ok`, `orphaned: false`, `last_error: ""`, `last_run 2026-08-05T09:14:02Z`, 6 snapshots. **METHOD NOTE for the next session:** the Hetzner storage box has a RESTRICTED shell — no shell operators, no `test`, no GNU long flags. `rm -r <path>` as ONE simple command works; `test -d X && rm -rf -- X` returns *"Command not found"* and silently does nothing, which is how the first attempt failed CLOSED (verified by an unchanged listing). |
|
||||
| **R-88a** | ~~Failing backup re-quiesces every 5 min, no backoff~~ | **SHIPPED** (controller v0.176.0, 2026-07-27) | — | Live on both boxes; breaker 15m→4h, per-tier, never permanent | — |
|
||||
| **R-88b** | ~~`/backup/due` cannot say *unknown*~~ | **SHIPPED + PROVEN-LIVE** (agent v0.105.0 + controller v0.178.0, 2026-07-27) | — | `age_state=unknown` captured on real hardware during a deliberate ep0 outage; controller deferred, **zero app stacks stopped** | — |
|
||||
| **E-2d** | **Prove E-2 on a fresh VM** — a real `felhom-host-install.sh` 1.22.0 run, Case B naturally, a claimable customer, then add a drive (the offer) and unplug it (`backup_target_absent` end-to-end) | **CLOSED — PARTIALLY PROVEN** (2026-07-29) | — | **C1, C2 proven** (`audits/E2D-fresh-vm-2026-07-29.md`); **C3, C4 proven live** (`audits/SESSION-C-2026-07-29.md`); **C5 FAILED → R-116** — the gate fires and an alarm reaches the hub, but it is the generic event, so the alarm and its recovery cannot be paired. **R-116 is the single named open leg**; per the Session-C runbook §9, decided in advance, a failed claim closes the item as partially proven rather than triggering a re-run. Both audits carry the full record — the `local-lvm` fence, the ISO/PAIRING derivation, the Phase 0 answers, the per-claim observables and the teardown evidence — and are the place to read it, not this cell. **The arc's actual definition of done is R-106 + R-109, R-108 and D5**, none of which this detour touched | CC |
|
||||
@@ -96,7 +96,7 @@ drill needed — the fourth is R-193, and until it goes the recovery is still op
|
||||
| **R-203** | **A customer-declared MANDATORY data directory was silently absent from the off-site snapshot while the run reported `ok`.** Measured live on demo-hp 2026-08-04: `calibre-web` declares `userdata: media/books class: mandatory`; its live bind is `/mnt/sys_drive/userdata/media/books` (the sentinel file was there), while the off-site capture set looked for `/mnt/sys_drive/felhom-data/userdata/media/books`, which does not exist. Result: `[WARN] mandatory data path missing on disk, skipped from offsite`, then `backed up calibre-web (…, 0 mandatory path(s))` and `backup OK: 3 app(s), 3 snapshot(s)` — `last_status: ok`, `last_success` stamped, nothing customer-visible, nothing hub-visible | **SHIPPED + PROVEN-LIVE 2026-08-04** (controller **v0.197.0**) | — | **THE MECHANISM, from source.** `NamespaceRoot(drivePath, inGuestDrive)` (`appbackup/paths.go:28-33`) appends the `felhom-data` segment **when the drive IS the system data path** (`m.namespaceRoot` = `NamespaceRoot(drivePath, drivePath != m.systemDataPath)`, `backup/backup.go:331`). The deploy-time bind does not: `${USERDATA_PATH}` is `<HDD_PATH>/userdata` (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive` and an app deployed at `HDD_PATH=/mnt/sys_drive`, the two produce different directories. **The same compose used BOTH roots**, from one deploy: `${IMPORT_PATH}` → `/mnt/sys_drive/felhom-data/userdata/import/calibre` (with the segment), `${USERDATA_PATH}` → `/mnt/sys_drive/userdata/media/books` (without). **WHAT IS MEASURED vs NOT, because it changes the fix.** MEASURED: the paths disagree, the mandatory directory is absent from the snapshot, the run says `ok`, and the only signal is a container-log WARN. **NOT ESTABLISHED:** whether `HDD_PATH=/mnt/sys_drive` is a SUPPORTED choice — it was used because demo-hp's only registered drive (`Felhom-Share`) is a NAS and was **correctly refused** as an app namespace (R-108 working as designed), while `/mnt/sys_drive` was **accepted** (HTTP 202). **EITHER BRANCH IS A DEFECT:** if the system drive is a supported app namespace, userdata resolution is wrong for every app on it and their mandatory directories are silently unprotected; if it is not supported, the deploy accepted a namespace it should have refused one call after refusing the NAS. **NOT a general off-site failure:** `opengist` and `privatebin` declare no mandatory userdata paths (everything of theirs is in named volumes), so they are unaffected and their snapshots are real. **Fix shape:** make the two roots one function, whichever is right — and make a skipped MANDATORY path a customer/hub-visible signal rather than a WARN, because `ok` with a missing mandatory directory is this project's own *a path the customer thinks is protected is not in the snapshot* shape. Source: `audits/DRILL-r201-offsite-recovery-2026-08-04.md` §2 **BOTH HALVES SHIPPED.** **(1) The paths.** `appbackup`'s helpers take a NAMESPACE ROOT; the census found **FIVE** bare-drive-path callers, not the four the spec named — the fifth is the **FileBrowser mount builder** (`web/handlers.go`), i.e. the customer's own file browser would have shown the wrong directory on a non-enrolled path (latent: the system drive is deliberately never a registered `StoragePath`). The rule now has **ONE expression** (`appbackup.NamespaceRootFor` / `IsEnrolledDrive`); there were already **two** copies and **they differed** — `backup.Manager.namespaceRoot` compared without `filepath.Clean`, `stacks.Manager.inGuest` with it, so a trailing slash from config would have flipped the mode in one package and not the other. `ComputeFabBuckets` now receives the namespace root, which is what `ComputeCaptureSet` has always received, so the export and the backup describe the same directories by construction. **(2) The verdict.** `last_status` gains **`incomplete`** — minted, because `ok`|`error`|`running` had nothing meaning *"it ran, and this app is not fully protected"*. **Not `error`:** the rest of the run worked, so `SnapshotCount` and the `LastSuccess` anchor still record what WAS captured. It reaches the operator through the **existing** per-run digest (`backup_run_failures`) — a new event type would be a two-repo change and the hub drops anything outside `allowedEventTypes`. **§8.4's narrowing is a NO-OP and no customer warning disappears:** `TierOffsite`'s `tierKeeps()` already admits mandatory only, demonstrated by widening the tier filter alone and watching the class check hold the line. **THE SPEC'S §8.3 RISK DOES NOT EXIST, and this is the correction owed:** `ExportDataMounts` lives in `delete.go` but is **export-only** — its single production caller is the `.fab` adapter, nothing deletes on its result, and the delete path's own guard `ProtectedHDDPaths` is layout-agnostic by construction (it protects BOTH `<hdd>/…` and `<hdd>/felhom-data/…`). It shipped as its own commit anyway. **PROVEN LIVE on demo-hp:** the bind moved `/mnt/sys_drive/userdata/Line truncated
|
||||
| **R-204** | **A rebuilt box can recover its off-site key and still cannot use it: the remedy that reconfigures the tier is the thing that blocks the recovery.** Measured end to end on demo-hp during the 2026-08-04 night drill, after a real controller-data wipe | **ALL FOUR ITEMS CLOSED 2026-08-05** (items 1–3 controller v0.198.0 + hub v0.95.0; item 4 controller v0.199.0 + hub v0.96.0) | — | **THE CHAIN, each link measured.** (1) A rebuilt controller **cannot configure its off-site target at all**: `offsite-apply: consume one-time password: no unconsumed offsite password` — the previous controller consumed it (ledger: created `07:11:51`, consumed `07:12:06`). That is R-193, reconfirmed live. (2) The documented remedy is an operator **Re-issue**, which works — and **sets `stale_at` on the escrow** (measured: `2026-08-04 20:15:49`) **while `restic_pw_sha256` is unchanged**, i.e. R-196's false staleness. (3) A stale escrow makes the hub **withhold the hash from the report ACK**, so `EscrowAutoConfirmer` can never flip `pending → escrowed`, and `OffboxRunnable` (`configured && escrowed`) **refuses every off-site run**. (4) The only documented way to clear a stale escrow is **a fresh ceremony — which supersedes the identity blob and destroys the key being recovered.** **So the recovery and its precondition are mutually exclusive as built.** The key comes back (proven — `8a9e33aa4da6…` recovered byte-identical after the wipe, and installed) and then cannot be used to open the repository. **(5) A FOURTH link, undocumented anywhere:** a rebuilt box is **unclaimed**, and the claim gate correctly intercepts every non-claim route (`claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)`), so **no controller endpoint responds at all** until the customer re-claims. That step appears in no design document and comes first. **Fix shape, not decided here:** either the Re-issue stops marking the escrow stale on evidence it does not have (R-196's own fix), or a recovered-and-verified key is allowed to confirm the escrow without a ceremony — the hash comparison that `EscrowAutoConfirmer` already performs is exactly the evidence needed, and it is being withheld precisely when it would be conclusive. **Do NOT fix by widening `OffboxRunnable`** — the atomicity guarantee it enforces (no un-recoverable ciphertext) is the reason the escrow exists. Source: `audits/DRILL-r201-night-run-2026-08-04.md` §3 **ALL FOUR WALKED AND MEASURED during the passing 2026-08-04 drill, so this row is now a known-good manual runbook AND the gap list.** (1) **R-193:** a rebuilt controller cannot configure its off-site tier — `no unconsumed offsite password` (ledger: created `07:11:51`, consumed `07:12:06` by its predecessor). Remedy: operator Re-issue. (2) **The claim gate:** a rebuilt box is unclaimed, so the gate intercepts EVERY controller endpoint (`claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)`) — a first step of every recovery that appears in no design document. **And the local escape hatch does not work unaided:** `--print-reset-code` writes the new hash to `settings.json` while the RUNNING controller keeps its old copy in memory, so `effectiveClaimCode()` never sees it and the claim fails with *"Hibás vagy lejárt kód"*. **The controller must be restarted between minting and claiming** — two attempts failed before this was diagnosed. (3) **R-196:** the Re-issue sets `stale_at` (`20:15:49`) while `restic_pw_sha256` is unchanged → the hub withholds the hash → auto-confirm can never fire → `OffboxRunnable` refuses every run. Cleared here with the **manual** confirm (`/backup/offbox/confirm-escrow`) — NOT a ceremony, which would have superseded the identity blob and destroyed the recovered key. (4) **`mode=unit` is the restore default and returns the recovery unit, NOT the userdata leg.** A customer told to "restore from off-site" gets their app definition back and not their documents, and nothing in that outcome says so. **Fix priorities, in the order they hurt:** (4) is a customer-facing trap on the last step; (3) is R-196's fix; (2) needs the escape hatch to reload settings (or the claim state to survive a rebuild); (1) is R-193's restage. Source: `audits/DRILL-r201-night-run-2026-08-04.md` §3–§4 | CC + operator **OUTCOME 2026-08-05 — controller v0.198.0 + hub v0.95.0.** **Item 1 (the reset code needs a restart) — CLOSED, proven live.** `effectiveClaimCode` reads through to the persisted claim state, so a code minted by the separate `--print-reset-code` process is seen without a restart; the precedence rule between settings and config is unchanged. Read-through, not a TTL: a TTL leaves a window in which a superseded code still works, and that is the mutation `TestClaimCode_SupersededByASecondMint_RefusedImmediately` kills. Fails closed on an unreadable state. **Live on demo-felhom 9201, nothing restarted (`restarts=0`, container older than both mints): the Line truncated
|
||||
| **R-192** | **`offsite_delivery_stuck` tells the operator the opposite of what the detector measured, and the self-heal silently refuses for exactly the reason the message denies.** demo-hp has been e-mailing this daily since 2026-08-03 06:12 UTC: *"one-time password consumed 284h ago and **500 report(s) since carry no offbox target** — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one."* **Measured against the hub's own data: all 500 of those reports DO carry an offbox target** | **CLOSED 2026-08-05 — the guard's scoping half closed BY REPLACEMENT** (hub v0.96.0) | — | **What actually happened on that box:** the credential was consumed 2026-07-23 09:53:41 and **applied successfully** — the controller reported an `offsite` object continuously until **2026-08-03 05:59:21 UTC**, then it **vanished at 06:12:19** and has been absent for **108 consecutive reports** since. So this is a **regressed apply**, not a burned credential. **Two distinct defects, and the second explains the first's invisibility.** **(a)** `maybeEmitStuck` builds its message from `status.ReportsSinceConsume` (the TOTAL) while hardcoding the phrase *"carry no offbox target"*, and never consults `status.OffsiteReportsSinceConsume` — which is the field that says the opposite. The recommended action (*Re-issue*) is aimed at a failure mode that did not occur. This is R-100's corollary again: an alarm whose text stopped matching what its verdict counts. **(b)** `maybeHeal` refuses **silently** (`OffsiteReportsSinceConsume != 0` → *"regressed-apply shape → operator's call"*, a bare `return` with no log line), so the operator gets a daily e-mail with the wrong story, no heal, and nothing anywhere saying why the heal declined. `offsite_credential_restaged` has **never** fired, on any customer. **The underlying condition is REAL and is the part that matters:** demo-hp currently reports no offsite target at all, i.e. that box's customer app-data has **no off-site copy right now** — and it has been that way since 08:12 CEST on 2026-08-03. A spot check inside the controller container found no restic environment, consistent with the report. **What removed it is not established** and is the first thing to find out. **Fix shape:** the message must state which shape was detected (burned vs regressed) and say what to do for each; the heal's refusal must log its reason; and the regressed shape probably deserves its own event type rather than borrowing the burned one. **Do NOT 'fix' it by widening the heal to restage over a regression** — the guard is right, only mute **CAUSE ESTABLISHED 2026-08-04 (operator confirms no hub-side offsite config change).** The regression is a **guest REBUILD**: at 06:09:40 `host_leaf_changed` (agent re-keyed), at 06:12:18 `controller_started (0.192.0)` — the controller went **0.187.0 → 0.192.0** with a **new config hash** (`1f725a2e843c` → `744e83d72c80`) — and the report at 06:12:19 is the first without `offsite`. The pre-rebuild object was fully healthy: `escrow_state: escrowed, last_status: ok, last_success 2026-08-03T02:16:39Z, snapshot_count 15, repo_size 40.9 MB`. → **R-193** owns the rebuild half. **AND THE HEAL'S GUARD IS WRONG FOR EXACTLY THIS CASE, which is why the automation that exists to fix it declined.** `maybeHeal` refuses when `OffsiteReportsSinceConsume != 0`, reading that as *"the apply regressed, so it is the operator's call"*. But `CountReportsOffsiteSince` counts the **OLDEST 500 reports since the consume** (`ORDER BY id LIMIT 500`) — for demo-hp all 500 predate the rebuild. **Offbox evidence from before a rebuild is not evidence that the credential still works**, so the guard reads healthy history as a reason not to heal a box that demonstrably cannot apply. The fix is to judge on RECENT evidence (e.g. the latest N reports, or evidence after the newest `controller_started`), not on everything since the consume. **SPIKE 2026-08-04 — BOTH HALVES CONFIRMED WITH NUMBERS, still OPEN, still not fixed here** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q7). The query is quoted at source (`store.go:987`): `... ORDER BY id LIMIT 500` = **the oldest 500**. Reproduced against the live hub DB with demo-hp's real consume anchor `2026-07-23 09:53:41`: the guard sees `total=500, withOffsite=500`, spanning `2026-07-23 09:53:47` → **`2026-07-28 11:17:40`** — the entire evidence set ends **six days before** the 2026-08-03 rebuild. The true window totals are `1174 / 1063` (⇒ 111 without, matching the 111 offsite-less reports). So the e-mail's *"500 report(s) since carry no offbox target"* interpolates `ReportsSinceConsume` while `OffsiteReportsSinceConsume` was **500** — the message states the precise negation of its own measurement. `offsite_credential_restaged` has **never fired for any customer** (zero rows of that type in the DB — checked, not assumed). **A NEW REASON NOT TO FIX THIS IN ISOLATION, from the samLine truncated
|
||||
| **R-193** | **A guest rebuild silently drops the off-site app-data tier, and nothing restages the credential.** demo-hp was rebuilt on 2026-08-03 (controller 0.187.0 → 0.192.0, new config hash, agent leaf re-keyed at 06:09:40). Before it, the offsite tier was healthy and working — `escrow_state: escrowed`, `last_status: ok`, last success **02:16:39Z that morning**, **15 snapshots, 40.9 MB**. After it: no `offsite` object in any of **108** reports, and **no off-site copy of that customer's app data since 08:12 CEST on 2026-08-03** | **CREDENTIAL HALF CLOSED 2026-08-05** (controller v0.199.0 + hub v0.96.0); the recovery SCREEN and the ciphertext deletion stay OPEN | — | **Mechanism, fully evidenced.** The restic credential reaches a box exactly once, as a one-time secret. demo-hp's was consumed **2026-07-23 09:53:41**; the rebuilt controller came up with a fresh data volume, no copy of it, and **no way to ask for another** — the hub is the only side that can stage one, and it will not re-stage a consumed secret on its own (the R-71c self-heal would, but it refuses — see R-192). **demo-felhom survived the SAME rebuild by luck, and the contrast is the proof:** its secret was created 2026-07-21 and still **UNCONSUMED**, so when its config hash changed at 07:17:54 and `offsite` dropped for exactly one report, it consumed the staged secret at **07:17:58** and was reporting `offsite` again by 07:19:10. One box had a spare credential staged and recovered in 76 seconds; the other did not and has been unprotected for a day. **That difference was not a design decision — it was an accident of which box happened to have an unconsumed secret lying around.** **Why this is not just "re-issue it":** the remedy (Re-issue) resets the sub-account password via the Hetzner API and, per R-39's record, **rotates the restic password and makes the escrow STALE** — so it needs the recovery-code ceremony re-run, and the continuity of the 15 existing snapshots under the new credential must be VERIFIED, not assumed (`hub v0.60.0` retains superseded escrow, and the orphan guard is move-aside-never-delete). That is an operator act with a customer-facing consequence, so it is not something to fire automatically without deciding the escrow question first. **What to design:** a rebuild is a normal, expected event on these boxes — the offsite tier must survive one, either by the hub restaging automatically when a re-enrolled box reports no offsite (the R-192 guard fix makes this safe), or by the credential being recoverable from escrow at re-bootstrap rather than delivered once and unrecoverable **RESOLVED ON THE BOX 2026-08-04 (operator-authorised).** Re-issue fired through the designed endpoint (`POST /configs/demo-hp/offsite-reissue`, HTTP 303): hub staged a fresh one-time password at **07:11:51**, the box's config hash moved `744e83d7` → `5eee0e42`, R-71a's settle-gate reported **GO** (*at/above floor 0.156.0, we are 0.194.0*), the password was **consumed 15 s later at 07:12:06**, and the controller logged *offsite configured for u629488-sub3@…:/home/felhom-repo* at 07:12:09 — the **same sub-account (275124) and the same repo path**, since Re-issue resets the sub-account password and the one-time password is only the transport credential used once to install the box's own SSH key. **THE ESCROW DID NOT RECOVER BY ITSELF — a correction to this session's own first reading.** `escrow_state` went `pending` → `escrowed` 15 s after the apply and CC inferred an automatic re-escrow; **the operator had run the ceremony**. It needed a human, on BOTH boxes: demo-hp escrowed 07:16:02, demo-felhom (whose offsite re-applied on its own the previous day but whose escrow had been `pending` ever since) escrowed 07:20:28. **A 15-second state change is not evidence of automation** — that is the same class as reading an absent log line as success. **Snapshot continuity is NOT yet established and must not be assumed from the counters:** both boxes report `snapshot_count: 0, repo_size_bytes: 0`, but the run-history keys (`last_run`, `last_status`, `last_success`) are **absent entirely** rather than zeroed — the shape of a controller that has never run an offbox backup in this lifetime, not of an empty repo. demo-hp's pre-rebuild object carried all three plus 15 snapshots / 40.9 MB. ~~**The next scheduled `offbox-backup` (04:15) decides it:** 15+ snapshots ⇒ the repo reattached; 1 ⇒ it started fresh and the old snapshots are orphaned-but-retained.~~ **SPIKE 2026-08-04 SETTLED IT WITHOUT WAITING, AND THE ITEM IS BIGGER THAN FILED** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md`). **(1) THE ONE-SHOT SECRET IS NOT WHERE THE HARM IS.** Three secrets exist; the one-time password is the *recoverable* one (the operator can reset it at the provider any time) and the box's SFTP key is regenerable by design. The **restic repository password** — the DATA key, which the agent's own source calls *"irreplaceable (unlike the SFTPLine truncated
|
||||
| **R-193** | **A guest rebuild silently drops the off-site app-data tier, and nothing restages the credential.** demo-hp was rebuilt on 2026-08-03 (controller 0.187.0 → 0.192.0, new config hash, agent leaf re-keyed at 06:09:40). Before it, the offsite tier was healthy and working — `escrow_state: escrowed`, `last_status: ok`, last success **02:16:39Z that morning**, **15 snapshots, 40.9 MB**. After it: no `offsite` object in any of **108** reports, and **no off-site copy of that customer's app data since 08:12 CEST on 2026-08-03** | **CREDENTIAL HALF CLOSED 2026-08-05** (controller v0.199.0 + hub v0.96.0); the recovery SCREEN and the ciphertext deletion stay OPEN | — | **Mechanism, fully evidenced.** The restic credential reaches a box exactly once, as a one-time secret. demo-hp's was consumed **2026-07-23 09:53:41**; the rebuilt controller came up with a fresh data volume, no copy of it, and **no way to ask for another** — the hub is the only side that can stage one, and it will not re-stage a consumed secret on its own (the R-71c self-heal would, but it refuses — see R-192). **demo-felhom survived the SAME rebuild by luck, and the contrast is the proof:** its secret was created 2026-07-21 and still **UNCONSUMED**, so when its config hash changed at 07:17:54 and `offsite` dropped for exactly one report, it consumed the staged secret at **07:17:58** and was reporting `offsite` again by 07:19:10. One box had a spare credential staged and recovered in 76 seconds; the other did not and has been unprotected for a day. **That difference was not a design decision — it was an accident of which box happened to have an unconsumed secret lying around.** **Why this is not just "re-issue it":** the remedy (Re-issue) resets the sub-account password via the Hetzner API and, per R-39's record, **rotates the restic password and makes the escrow STALE** — so it needs the recovery-code ceremony re-run, and the continuity of the 15 existing snapshots under the new credential must be VERIFIED, not assumed (`hub v0.60.0` retains superseded escrow, and the orphan guard is move-aside-never-delete). That is an operator act with a customer-facing consequence, so it is not something to fire automatically without deciding the escrow question first. **What to design:** a rebuild is a normal, expected event on these boxes — the offsite tier must survive one, either by the hub restaging automatically when a re-enrolled box reports no offsite (the R-192 guard fix makes this safe), or by the credential being recoverable from escrow at re-bootstrap rather than delivered once and unrecoverable **RESOLVED ON THE BOX 2026-08-04 (operator-authorised).** Re-issue fired through the designed endpoint (`POST /configs/demo-hp/offsite-reissue`, HTTP 303): hub staged a fresh one-time password at **07:11:51**, the box's config hash moved `744e83d7` → `5eee0e42`, R-71a's settle-gate reported **GO** (*at/above floor 0.156.0, we are 0.194.0*), the password was **consumed 15 s later at 07:12:06**, and the controller logged *offsite configured for u629488-sub3@…:/home/felhom-repo* at 07:12:09 — the **same sub-account (275124) and the same repo path**, since Re-issue resets the sub-account password and the one-time password is only the transport credential used once to install the box's own SSH key. **THE ESCROW DID NOT RECOVER BY ITSELF — a correction to this session's own first reading.** `escrow_state` went `pending` → `escrowed` 15 s after the apply and CC inferred an automatic re-escrow; **the operator had run the ceremony**. It needed a human, on BOTH boxes: demo-hp escrowed 07:16:02, demo-felhom (whose offsite re-applied on its own the previous day but whose escrow had been `pending` ever since) escrowed 07:20:28. **A 15-second state change is not evidence of automation** — that is the same class as reading an absent log line as success. **Snapshot continuity is NOT yet established and must not be assumed from the counters:** both boxes report `snapshot_count: 0, repo_size_bytes: 0`, but the run-history keys (`last_run`, `last_status`, `last_success`) are **absent entirely** rather than zeroed — the shape of a controller that has never run an offbox backup in this lifetime, not of an empty repo. demo-hp's pre-rebuild object carried all three plus 15 snapshots / 40.9 MB. ~~**The next scheduled `offbox-backup` (04:15) decides it:** 15+ snapshots ⇒ the repo reattached; 1 ⇒ it started fresh and the old snapshots are orphaned-but-retained.~~ **SPIKE 2026-08-04 SETTLED IT WITHOUT WAITING, AND THE ITEM IS BIGGER THAN FILED** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md`). **(1) THE ONE-SHOT SECRET IS NOT WHERE THE HARM IS.** Three secrets exist; the one-time password is the *recoverable* one (the operator can reset it at the provider any time) and the box's SFTP key is regenerable by design. The **restic repository password** — the DATA key, which the agent's own source calls *"irreplaceable (unlike the SFTPLine truncated
|
||||
| — | Storage Box **snapshots** on `storage-box-pool-1` — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm `size_snapshots > 0` tomorrow; until then the mitigation is armed, not proven | CC |
|
||||
| — | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | WAITING-ON-OPERATOR | operator console | Delete the box | operator |
|
||||
| **R-90** | ~~ep0 RAM headroom — 4 GiB swap survived its first reboot 2026-07-27; 3.8 GB RAM unchanged~~ | **CLOSED — the operator rescaled ep0 to a CX33 on 2026-08-03** | — | **MEASURED ON THE BOX, not read from an invoice:** `felhom-hetzner` reports `Mem: 7757` MB total (**8 GB**, was 3.8) and `nproc` **4**. **The interim lever survived and was checked rather than assumed** — a resize is a stop/start, so "the swapfile is still there" was an assumption until measured: `/swapfile`, 4 GiB, dated `Jul 27 14:40`, **active** (`swapon --show` → `/swapfile file 4G 0B -2`), 0 B in use on an idle box. **THE 40 GB LOCAL DISK DID NOT CHANGE** and must not be "corrected" alongside the RAM: `/` is 38 G, 58% used. This was a CPU/RAM resize only, so every disk figure in the runbooks still stands — the separate 98 G volume at `/mnt/pbs-datastore` (R-82 P0.3) is unaffected. **Why this was BLOCKED and no longer is:** the row recorded CX33 as *"confirmed unavailable even powered OFF"* — the Cost-Optimized line's limited availability, not a power-state problem. It became available and the operator took it. **Documentation corrected** (`RUNBOOK-ep0-datastore-volume`, `RUNBOOK-pbs-prune-serverside` ×2, `runbooks/offsite-endpoint.md` ×2, `runbooks/target-selection.md`) and **audit/evidence documents ANNOTATED, not revised** (`SPIKE-connectivity-wireguard-2026-07-03`, campaign-10 `phaseA-journal`) — they record what was true when written and that is their value. **Still open and still the operator's, deliberately untouched:** `target-selection.md`'s *"D-d did not name ep0 either way. Confirm it explicitly."* | — |
|
||||
|
||||
Reference in new issue
Block a user