All five layers gone, each verified with a positive control that must
survive and does:
VM 324 + 4 disks -> absent (VM 300 drill-r50 remains)
hub: 13 tables at 0, incl. BOTH escrow tables (demo-felhom/demo-hp/peti remain)
Storage Box u629488-sub4 -> gone (sub1/2/3 remain)
ep0 PBS ns finalwalk -> gone (demo-felhom, demo-hp remain)
WireGuard 10.77.0.5 -> gone from the LIVE wg show on ep0, not just
the hub DB (.2/.3/.4/.250 remain)
14.06 GiB reclaimed against 15 G measured before deletion.
R shredded with a planted-copy control: plant -> search finds both ->
shred -> the same search finds 0. The zero was not believed until the
instrument was proven.
R-244 (NEW): a FULL census after the cascade logged COMPLETE full teardown
found 61 rows still matching finalwalk. Four sources are deliberate
provenance; the fifth, app_log_issues (29 rows), is NOT covered by the
residue purge - and it is systematic: c11 40, rewalk 20, part4 24 still
present from the 2026-08-06 teardown, whose ledger recorded zero
occurrences. That claim used a narrower query than a census and does not
hold; the correction is recorded in both the prior ledger and the register
rather than the measurement quietly redone.
No secret material is involved. The table is a fleet-wide aggregate: 12 of
the 29 rows are finalwalk-only orphans, 17 are shared with LIVE customers
and must be de-referenced, not deleted - very likely why the leg was never
written. Not fixed; a cascade change needs its own red-proof.
Lesson, and it is the reusable part: a per-table absence query is not a
census.
This commit is contained in:
@@ -0,0 +1,194 @@
|
||||
# REPORT — R-241 spike · golden 0.205.0 · finalwalk teardown (2026-08-07)
|
||||
|
||||
*A sibling report: `REPORT.md` is overwritten per-session and a parallel session shares this clone.*
|
||||
|
||||
**Three parts, in the order the venue's perishability required.** The spike read VM 324 before
|
||||
anything else touched the fleet; the bake and the vouch followed; the teardown went last.
|
||||
|
||||
---
|
||||
|
||||
## 1. THE SESSION'S ANSWER — R-241 is a MINTING defect, not a screen-predicate defect
|
||||
|
||||
**This reverses the fix.** The recovery screen was telling the truth: there genuinely was nothing
|
||||
recoverable under the key the box held, because **the box minted that key itself, over the top of a
|
||||
sealed package it already knew the hub was holding for it.** Mending the predicate would have papered
|
||||
over a box quietly making its own backup history unopenable.
|
||||
|
||||
**Three measurements, taken from the venue this session, not copied from last night's journal:**
|
||||
|
||||
1. **`WriteOffboxSecrets` (`offbox.go:411`) mints on ONE input — does the file exist.** No settings
|
||||
read at all, while its two neighbours in the same file, `OffsiteRecoveryOffer()` (`:1412`) and
|
||||
`needsOffsiteCredential()` (`:1377`), both consult `GetHubEscrowIdentityPresent()`. **The same fact
|
||||
is available on three paths and used on two.**
|
||||
2. **That flag was not merely available — it was the precondition of the chain that reached the
|
||||
minting.** The 5-minute retry job logs only when `RetryIfDeclared` fires, which requires the
|
||||
declaration, which requires the flag. The venue logged
|
||||
`credential retry: … (the box still declares a need; retrying)` at **02:48:03Z** and five times
|
||||
after — **thirty minutes and six ticks before the mint at 03:18:06Z.**
|
||||
3. **The box computed the right answer and threw it away.** At **03:28:03Z** — thirty-five minutes
|
||||
before the customer looked — `EscrowAutoConfirmer.Reconcile` (`escrow_confirm.go:154`) logged
|
||||
`the hub's escrow blob does not cover the CURRENT repo password (hub hash 30ef574fe492… != local
|
||||
9b4a9a9dcec7…) … staying pending`. It is recomputed on every report cycle, **never persisted,
|
||||
never surfaced.**
|
||||
|
||||
**And the hub explicitly disclaims doing this** — `offsiteheal`'s package doc: *"it never runs, or asks
|
||||
for, an escrow ceremony… **credential automatic, key customer-present** — the ruling this session
|
||||
implements and must not quietly widen."* The repository key is created **in the seam between two sides
|
||||
that each honoured their contract**, by a helper doing exactly what its doc comment says. That is why
|
||||
it survived review: every individual comment is accurate.
|
||||
|
||||
**Corroboration from both sides, measured independently here:** the two key hashes on the venue
|
||||
(`30ef574f…` current, `9b4a9a9d…` in `repo_password.selfheal-aside`, mtime 03:18:06) reproduce the
|
||||
journal's figures exactly; and the hub's `host_escrow` row for `finalwalk-ed05d6` seals
|
||||
`restic_pw_sha256 = 30ef574f…` with a 572-byte `identity_blob` and **no superseded row** — so the hash
|
||||
the box logged as *"hub hash"* is confirmed from the database, not inferred.
|
||||
|
||||
**Q2 — shape (b) is structurally unreachable**, not merely unfired: `markOrphaned()` has one producer,
|
||||
`ensureOffboxRepo()`, reached only from inside a run, and every run passes `if !m.offboxEscrowed()`
|
||||
(`offbox.go:743`) **first**. The box was `pending` and could not stop being pending, because the
|
||||
auto-confirm flips only on a hash match. *Positive control, because an absent log line is not
|
||||
evidence:* the scheduler was alive throughout (241 `agent-channel-health`, 120 `stack-scan`, 48
|
||||
`offsite-credential-retry`, 16 `hub-report`), and `offbox-backup` is a `sched.Daily` leg whose slot
|
||||
fell **before** the destruction — the absence is explained, not just observed.
|
||||
|
||||
**Q7 — the „Helyreállítási kód létrehozása" button.** It does **not** destroy the data: `SaveHostEscrow`
|
||||
demotes the current row into `host_escrow_superseded` **copy-before-delete, `identity_blob` included**
|
||||
(R-198). But the read path for a retained package is **unbuilt** (R-199), so it converts a
|
||||
one-screen-away self-service recovery into one needing an operator and tooling that does not exist.
|
||||
**And it re-enables the screen while invalidating the code that screen accepts** — a worse trap than a
|
||||
plain dead end, because it looks like progress. *(Reasoned from code and the hub schema; I did not
|
||||
press it — that is a state change and would have destroyed the evidence.)*
|
||||
|
||||
**Two findings nobody asked for**, both filed: **R-243** (a box in this state silently stops backing up
|
||||
and **no alarm fires** — three individually-correct exclusions leave one state unobserved) and the trap
|
||||
in the obvious fix (`ResetOrphanedRepo` clears the orphan flag without a ceremony, so a hash-mismatch
|
||||
discriminator alone would re-offer the screen forever to a customer who declined the old data).
|
||||
|
||||
**Output:** `audits/SPIKE-r241-recovery-offer-2026-08-07.md` — question/method/measurement/ruling, the
|
||||
Q4 seven-state table, ranked options, and **four operator decisions stated and left unanswered.**
|
||||
**No product code was written, in any part of this session.**
|
||||
|
||||
---
|
||||
|
||||
## 2. GOLDEN 0.205.0 — baked, verified, VOUCHED (R-239 CLOSED)
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| version | **0.205.0** · sha256 `8f49b2e8ccbc86a49df821fee9fb00c07293758811d3d0f0512dd0cf5fd54ee8` |
|
||||
| size | 656,937,561 bytes (uncompressed 2,003,343,360) |
|
||||
| MinAgent | 0.127.0 |
|
||||
|
||||
**Round-trip verified rather than trusted:** the published bytes were fetched back (HTTP 200, byte
|
||||
count matches), re-hashed **independently of the baker** (matches), `zstd -t`'d clean, and
|
||||
`./etc/felhom-controller-image` was read **out of the downloaded archive** →
|
||||
`felhom-controller:0.205.0`. That last step is the one that matters, because `GOLDEN_VERSION` is
|
||||
derived from the tag argument and could have been right over stale content. A **third witness**: the
|
||||
hub's own dropdown lists `0.205.0` with `data-sha="8f49b2e8…"`.
|
||||
|
||||
All acceptance markers pass (`docker OK (overlay2` ×1, `including mount point` ×2 = rootfs + mp0,
|
||||
`upload OK (HTTP 201)` ×1, and 0 each for `excluding`/`FATAL`/`ERROR:`/`WARN:`); unit
|
||||
`Result=success`, `ExecMainStatus=0`. Bake VM torn down: CT 9100 purged, secrets shredded, qemu
|
||||
observed gone via `ps -eo comm`, `drill.qcow2` reverted to `virgin`.
|
||||
|
||||
**Secret hygiene:** token copied file→file; the invocation lives in an in-VM runner that reads the
|
||||
token itself, so it never reached a command line (`systemctl show … | grep -c -F <token>` → **0**);
|
||||
literal-value leak grep on the **committed** log → **0**, and **the instrument was proven first** —
|
||||
token appended to a throwaway copy → 1 hit → copy shredded → the 0 is a measurement.
|
||||
|
||||
**Vouched, with the operator's approval.** In the event it was a **ONE-field change, not three** — read
|
||||
from live `hub_settings` before and after, not assumed:
|
||||
|
||||
| field | before | after |
|
||||
|---|---|---|
|
||||
| `artifact_golden_version` | 0.203.0 | **0.205.0** |
|
||||
| `artifact_golden_sha256` | `3039c6ff…` | **`8f49b2e8…`** |
|
||||
| `artifact_agent_version` | 0.127.0 | 0.127.0 — **unchanged** |
|
||||
| `artifact_min_agent` | 0.127.0 | 0.127.0 — **unchanged** |
|
||||
|
||||
**A trap worth naming:** `wrapper_sha256` is read from the form and **cleared when omitted**. A
|
||||
headless POST that forgets it silently drops the PBS-DR wrapper hash. It was carried through
|
||||
explicitly and verified present afterwards. **The R-120 gate passed exactly** — the newest controller
|
||||
the fleet reports is 0.205.0 (demo-hp), so a **0.204.0 golden would have been REFUSED**. Vouching is
|
||||
reversible; a bake never deletes an older golden's package.
|
||||
|
||||
**§4.1's systemic half is recorded, NOT built** → **R-242**, with three proposed shapes and a stated
|
||||
earliest-catch (a `repo_gates.py` comparison of the manifest's `golden_version` against the newest
|
||||
released controller — it fires on the push that creates the gap, before any box is installed).
|
||||
|
||||
---
|
||||
|
||||
## 3. TEARDOWN — `finalwalk`, all five layers
|
||||
|
||||
Enumerated first and cross-checked against the hub's **own** delete-preview, which agreed in every
|
||||
field. Matched on **identity**, never on size.
|
||||
|
||||
The cascade refuses a live host, so VM 324 was stopped at 07:23:56Z (guarded on `qm config 324`
|
||||
reading `name: finalwalk-appliance` — `demo-hp` also carries a guest 9201) and aged past the 30 m
|
||||
`stale_threshold`, read from the deployed config. `POST /configs/finalwalk/delete` with all six gates
|
||||
→ the hub's leg-by-leg log shows host deleted, off-site sub-account 285071 deprovisioned, PBS
|
||||
namespace deprovisioned (`existed=true` — the positive observable), claim reset, residue purged
|
||||
(**124 rows, matching the preview exactly**), `COMPLETE … full teardown`. Then `qm destroy 324
|
||||
--purge`.
|
||||
|
||||
**Every layer verified absent with a positive control that must survive, and does:** VM 324 gone (VM
|
||||
300 `drill-r50` remains) · all 13 hub tables at 0 including **both** escrow tables (demo-felhom 50,823
|
||||
/ demo-hp 7,640 / peti 1,827 rows remain) · Storage Box `u629488-sub4` gone (sub1/2/3 remain) · `ep0`
|
||||
namespace `finalwalk` gone (demo-felhom, demo-hp remain) · WireGuard `10.77.0.5` gone **from the live
|
||||
`wg show` on ep0, not merely from the hub DB** (`.2/.3/.4/.250` remain). **14.06 GiB reclaimed**,
|
||||
against 15 G measured before deletion.
|
||||
|
||||
**R shredded with a planted-copy control** — plant → search finds both → shred → same search finds 0.
|
||||
The zero was not believed until the instrument was proven.
|
||||
|
||||
### ⚠ And a correction I am reporting rather than quietly fixing
|
||||
|
||||
A **full census** (every table, every column) after the "COMPLETE" cascade found **61 rows still
|
||||
matching `finalwalk`**. Four sources are deliberate (`events`, `notification_log`, `host_deletions`,
|
||||
`customer_resets` — the cascade's header says provenance outlives every tier). **The fifth is a gap:**
|
||||
`app_log_issues`, 29 rows, not covered by the residue purge — and **systematic**, with `c11` 40,
|
||||
`rewalk` 20 and `part4` 24 still present from the 2026-08-06 teardown, **whose ledger recorded "0
|
||||
occurrences"**. That claim used a narrower query than a census and does not hold. Both the prior ledger
|
||||
and the register now carry the correction.
|
||||
|
||||
**No secret material is involved.** The table is a fleet-wide aggregate; 12 of the 29 rows are
|
||||
`finalwalk`-only orphans and **17 are shared with live customers and must be de-referenced, not
|
||||
deleted** — very likely why the leg was never written. Filed as **R-244**, not fixed: a cascade change
|
||||
needs its own red-proof. **The reusable lesson: a per-table absence query is not a census.**
|
||||
|
||||
---
|
||||
|
||||
## Register
|
||||
|
||||
| ID | Movement |
|
||||
|---|---|
|
||||
| **R-241** | **RULED** — minting defect. Diagnosed, **not fixed**; four operator decisions owed |
|
||||
| **R-239** | **CLOSED** — golden 0.205.0 baked, verified and vouched |
|
||||
| **R-242** | **NEW** — a release is not delivered until a golden carries it; recorded, **not built** |
|
||||
| **R-243** | **NEW** — the R-241 state silently stops off-site backups with no alarm |
|
||||
| **R-244** | **NEW** — `app_log_issues` survives the delete cascade, across all four torn-down venues |
|
||||
|
||||
**Highest register ID moved R-241 → R-244.**
|
||||
|
||||
## Verification
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| commits | `71c43f87c240` (spike) · `08b75e602e01` (bake evidence) · `db578cd44d3e` (R-239 closed, map + STATUS) · this one |
|
||||
| CI | run **233** `71c43f87c240` success · run **234** `08b75e602e01` success — matched by `head_sha`, pulled not assumed |
|
||||
| `--no-verify` | **not used.** The pre-push hook ran `repo_gates.py --fast` on every push and reported `gates OK` |
|
||||
| gates | `python3 scripts/repo_gates.py --fast` → all six OK before each commit |
|
||||
|
||||
## What did not run, and why
|
||||
|
||||
- **No product code, deliberately** — Part 1 was a question, and the answer changes what the fix
|
||||
should be. Beginning a candidate before the operator rules on §"THE OPERATOR'S DECISION" would
|
||||
prejudge it.
|
||||
- **Q4 row 7 and Q7's post-button behaviour are reasoned from code, not measured.** Both need a state
|
||||
change on the venue; either would have destroyed the evidence for everything else. They need a fresh
|
||||
fixture and their own session — and **row 7 shapes the fix**, so it matters.
|
||||
- **R-242, R-243 and R-244 are filed and not built** — R-242 because the task scoped it record-only;
|
||||
the other two because they surfaced inside a spike and an operation, and each needs its own
|
||||
red-proof.
|
||||
- **The exact wall-clock at which `hub_escrow_identity_present` first became true is not measured** —
|
||||
`SetHubEscrowIdentityPresent` writes only on change and logs nothing. The bound that matters is
|
||||
established by control flow: true at or before **02:48:03Z**.
|
||||
@@ -172,6 +172,8 @@ the fault was real. Full observables: `tests/campaign11-evidence-2026-08-05/jour
|
||||
| **R-242** | **A controller release that changes customer-visible behaviour is not delivered until a golden carries it — and nothing enforces that.** R-239 is the symptom; this is the mechanism, recorded **2026-08-07** and deliberately **NOT built** (the task that found it scoped it as a record-only item). **Two releases went out without a golden and the gap was invisible until a walk measured it from the customer's side**: v0.204.0 (R-237) and v0.205.0 (R-234) were written, tested, pushed, and CHANGELOG'd, and every one of those steps passed while a machine installed that night received neither. The register said CLOSED; the fleet said otherwise. **Nothing in the release path knows a golden exists.** The version bump, the image push, the CHANGELOG entry and the register closure are all repo-local; the manifest's `golden_version` is edited by a separate operator act, in a different repo, with no link back. **Proposed shapes, cheapest first — the choice is the operator's and is not taken here.** (a) **A release-path checklist step** — one line in the controller's end-of-session checklist: *a release that changes customer-visible behaviour is not finished until a golden carries it or a register row says why not.* Costs nothing, catches nothing mechanically. (b) **A gate in `repo_gates.py`** comparing the manifest's `golden_version` against the newest released controller and FAILING (or warning) past a tolerance of one minor. Mechanical, runs on every push, and would have fired the morning after v0.204.0. (c) **A hub-side checker** — the hub already knows every box's running controller version from `/hosts` and the vouched golden from the manifest; a periodic comparison against the newest published image would catch drift the repo cannot see, including a vouch that was made and then rolled back. **Earliest catch: (b).** It fires on the push that creates the gap, before any box is installed, and it needs no live fleet. **(c) catches strictly more but only after boxes exist.** (a) is worth doing regardless because it is free. **Not built. No gate was written this session.** | **READY — recorded, not built** — owner Viktor |
|
||||
| **R-243** | **A box in the R-241 state silently stops backing up off-site, and NO ALARM OF ANY KIND FIRES.** Found by the R-241 spike (2026-08-07) as a by-product; **not part of the walk's finding and not previously filed.** The R-241 state is self-locking in a second, worse way than the recovery-journey dead end: `escrow_state` is stuck `pending` forever (the auto-confirm flips only on a hash match, and the hash cannot match a key the box minted itself), and `runOffboxBackup` returns at the escrow gate (`offbox.go:743`) before touching anything. **So off-site backups never run again — and the hub never notices.** All three signals that could catch it are excluded, each for its own individually-correct reason, verified in the hub this session: `offsite_stale` — `isStale` (`monitor/offsite.go:135`) returns false unless `EscrowState == "escrowed"`, and its own comment reads *"Pending/disabled = normal onboarding, never stale"*, so the box is classified as **still being set up, forever**; `offsite_delivery_stuck` — `monitor/offsite_delivery.go:91` skips the `applied` shape, and delivery genuinely IS applied (the credential was consumed and the target is in every report); `backup_failed` — never fires, because nothing fails: the run returns `nil` before it starts. **Three correct exclusions leaving one state unobserved.** This is the same class as the workspace `CLAUDE.md` "presence is not success" rule, one level up: **the absence of a failure is being read as the presence of a working tier.** **Partly subsumed by R-241's fix** — a box that recovers leaves this state — but **not for a box that does not**, and the alarm gap is what makes "does not" survivable indefinitely. **Not fixed; no code written.** | **READY** — owner Viktor |
|
||||
|
||||
| **R-244** | **The customer DELETE cascade leaves `app_log_issues` behind, and it is systematic across every venue ever torn down.** Found **2026-08-07** while verifying the `finalwalk` teardown with a **full census** (every table, every column) rather than a per-table query. After a cascade that logged `COMPLETE … full teardown`, **61 rows still matched `finalwalk`**. Four of the five sources are **deliberate and correct** — the cascade's own header states *"Provenance/events are NEVER wiped — audit outlives every tier"*: `events` 16, `notification_log` 14, `host_deletions` 1, `customer_resets` 1. **The fifth is a gap:** `app_log_issues` 29 rows, which the residue purge does not touch (its logged leg covers `reports`/`app_telemetry`/`app_log_tails`/`log_tail_requests`/`notif_prefs`/`selfbind_tokens`/`appliance_registrations` — not this table). **It is not a `finalwalk` quirk:** rows still reference **`c11` 40, `rewalk` 20, `part4` 24** — all three torn down 2026-08-06, whose ledger recorded *"0 occurrences"*. **That prior claim was measured with a narrower query and does not survive a full census; the correction is recorded rather than the measurement quietly redone.** **Why it was probably never written, established rather than assumed:** the table is a **fleet-wide aggregate** keyed on `app_name`+`fingerprint` with an `affected_customers` JSON list — of the 29 `finalwalk` rows, **12 reference only `finalwalk`** (orphans, safely deletable) and **17 are shared with LIVE customers** (`demo-felhom`, `peti-felhom`, …) and **must not be deleted, only de-referenced.** A naive `DELETE … WHERE customer LIKE` would destroy a live customer's issue history — which is very likely why the leg does not exist, and is the reason this is not a one-line fix. **Severity is LOW and stated plainly: no secret material is involved** — app name, fingerprint, message text, counts, timestamps. What survives is a deleted customer's *identifier* inside an aggregate row. **Proposed shape:** a residue leg that (a) removes the customer id from `affected_customers`/`context_customer`, and (b) deletes rows whose `affected_customers` becomes empty; plus a one-off sweep for the four already-torn-down venues. **The general lesson is the reusable part:** *a per-table absence query is not a census.* The teardown verification is now a full-schema sweep, and that is what found this. **Not fixed** — a cascade change needs its own red-proof and this session was scoped as a spike plus two operations. Evidence: `tests/teardown-finalwalk-2026-08-07.md`. | **READY** — owner Viktor |
|
||||
|
||||
**Recorded against existing rows by Phase 2:**
|
||||
|
||||
- **R-216 — §4.1 is now MEASURED, not deduced.** The previous session could only offer two absences.
|
||||
|
||||
@@ -107,3 +107,18 @@ under `~/.config/part4/` on DooPlex were shredded.
|
||||
**One operational note carried over from the walk:** the `part4` retrieval passphrase was leaked into
|
||||
a session transcript on 2026-08-06 and rotated at the time; the customer it belonged to no longer
|
||||
exists.
|
||||
|
||||
---
|
||||
|
||||
## ⚠ CORRECTION, 2026-08-07 — the "0 occurrences" claims above are narrower than they read
|
||||
|
||||
Verifying the `finalwalk` teardown with a **full census** (every table, every column) rather than a
|
||||
per-table query found rows this ledger's method could not see. `app_log_issues` is **not** covered by
|
||||
the cascade's residue purge, and it still holds **`c11` 40 rows, `rewalk` 20, `part4` 24** — plus 29
|
||||
for `finalwalk`.
|
||||
|
||||
The tables this ledger *did* check are genuinely at zero; the claim that is wrong is the implied
|
||||
completeness of the sweep. **A per-table absence query is not a census.** Filed as **R-244**, with the
|
||||
reason the leg is not trivial (the table is a fleet-wide aggregate whose rows are shared with live
|
||||
customers, so the fix is a de-reference plus an orphan sweep, not a delete). No secret material is
|
||||
involved. Full working: `teardown-finalwalk-2026-08-07.md`.
|
||||
|
||||
@@ -0,0 +1,171 @@
|
||||
# Teardown ledger — `finalwalk`, the final-walk venue (2026-08-07)
|
||||
|
||||
The last campaign venue. **Kept standing until the R-241 spike had read it and its findings were
|
||||
pushed** (`71c43f87c240`) — it was the only machine standing in the failed state, and tearing it down
|
||||
first would have destroyed the evidence. Destroyed only after that, with the operator's confirmation.
|
||||
|
||||
Predecessor ledger: `teardown-2026-08-06.md` (venues `c11`, `rewalk`, `part4`).
|
||||
|
||||
## What was matched, and how
|
||||
|
||||
Matched on **identity** — customer id → host id → off-site sub-account → PBS namespace → wg peer —
|
||||
**never on size**. On 2026-08-06 an item that measured exactly the expected size turned out to be a
|
||||
working store, which is why size is not a key here.
|
||||
|
||||
The hub's own delete-preview (`GET /configs/finalwalk/delete`) was used as an **independent
|
||||
enumeration** and agreed with the hand-built one in every field:
|
||||
|
||||
```json
|
||||
{"customer_id":"finalwalk","customer_name":"Final walk R-201","host_count":1,
|
||||
"hosts":[{"host_id":"finalwalk-ed05d6","online":true,"status":"ok"}],
|
||||
"offsite_enabled":true,"offsite_identifier":"u629488-sub4","offsite_type":"shared",
|
||||
"pbs_tenancy_configured":true,"claim_present":true,"dr_recipe_present":true,
|
||||
"one_time_secret":true,"superseded_blobs":0,"residue_total":124}
|
||||
```
|
||||
|
||||
`superseded_blobs: 0` matches the hand query on `host_escrow_superseded` (no rows) — consistent with
|
||||
the spike's finding that the box never re-escrowed, which is exactly why its key never matched.
|
||||
|
||||
## The enumeration, before anything was deleted
|
||||
|
||||
| Layer | Item | Identified by |
|
||||
|---|---|---|
|
||||
| **machine** | `demo-hp` VM **324** `finalwalk-appliance` — 4 disks (efidisk + 200 G + 50 G + 50 G), **15 G actual** on `/mnt/nvme-1tb/images/324` | `qm config 324` → `name: finalwalk-appliance` |
|
||||
| **hub** | host `finalwalk-ed05d6`, customer `finalwalk` — **221 rows across 19 tables** (full census, no truncation) | census below |
|
||||
| **off-site** | Storage Box sub-account **`u629488-sub4`**, home `felhom-finalwalk`, id `285071` | Hetzner API, matched on the **home directory naming the customer** |
|
||||
| **off-site** | `ep0` PBS namespace **`finalwalk`** in datastore `felhom-offsite` — 1 CT snapshot (`ct/9201/2026-08-06T22:17:08Z`), 88 K of metadata | `ls /mnt/pbs-datastore/ns` |
|
||||
| **network** | WireGuard peer **10.77.0.5** | `wg_peers.host_id = finalwalk-ed05d6` |
|
||||
|
||||
**Full hub-DB census — every table, every column, matching `finalwalk`** (221 rows / 19 tables):
|
||||
`app_log_issues` 29 · `app_telemetry` 70 · `appliance_registrations` 1 · `customer_claims` 1 ·
|
||||
`customer_configs` 1 · `customer_notifications` 1 · `dr_recipe` 1 · `events` 14 · `guests` 1 ·
|
||||
`host_escrow` 1 · `host_pbs_secrets` 1 · `host_recovery` 1 · `host_reports` 37 · `hosts` 1 ·
|
||||
`notification_log` 13 · `one_time_secrets` 1 · `reports` 45 · `selfbind_tokens` 1 · `wg_peers` 1.
|
||||
|
||||
**`pvesm status` on `demo-hp` BEFORE:**
|
||||
|
||||
```
|
||||
c11-scratch dir active 983379700 KiB total 21010360 KiB used 912342728 KiB avail 2.14%
|
||||
```
|
||||
|
||||
**A note on the `ep0` datastore path.** The live datastore is `/mnt/pbs-datastore` (the 100 GB volume).
|
||||
`/srv/pbs-felhom` still exists on the root filesystem and holds **stale** namespaces
|
||||
(`demo-felhom`, `demo-felhom-01`, `demo-hp`) from before the volume move. It is **not** the live store
|
||||
and was not touched. Reading namespaces from it would have produced a wrong answer in both directions —
|
||||
recorded here because the first listing this session did exactly that.
|
||||
|
||||
## The gate that shaped the operation
|
||||
|
||||
The cascade **refuses to delete a live host** — there is no hub decommission endpoint; the word appears
|
||||
only in the refusal. So VM 324 was **stopped** at **07:23:56Z** (guarded on `qm config 324` reading
|
||||
`name: finalwalk-appliance` first — `demo-hp` also carries a guest 9201) and the hub was allowed to age
|
||||
it past its `stale_threshold` (**30m**, read from the deployed `hub-config`, not assumed) before the
|
||||
cascade would accept it.
|
||||
|
||||
---
|
||||
|
||||
## The cascade
|
||||
|
||||
`POST /configs/finalwalk/delete` with the six gates satisfied deliberately — `ack_hosts`,
|
||||
`ack_reset`, `ack_purge`, `confirm_id=finalwalk` (typed), `expect_hosts=1` (the stale-preview race
|
||||
guard), and the online-host gate lifted by ageing. → **`303 /configs?flash=deleted`**.
|
||||
|
||||
**The hub's own leg-by-leg log, which is the evidence — not the flash:**
|
||||
|
||||
```
|
||||
09:54:02 customer DELETE cascade started for finalwalk (journal #14, 1 host(s))
|
||||
09:54:02 delete finalwalk: host finalwalk-ed05d6 deleted (escrow DEMOTED to retained custody)
|
||||
09:54:10 [offsite] deprovisioned shared sub-account 285071 for finalwalk (repo data destroyed)
|
||||
09:54:10 reset finalwalk: offsite deprovisioned (repo data destroyed)
|
||||
09:54:10 tenantsync: deprovision ok for finalwalk (ns=finalwalk, existed=true)
|
||||
09:54:10 reset finalwalk: PBS tenancy deprovisioned
|
||||
09:54:10 [claim] reset to unclaimed for finalwalk (customer RESET)
|
||||
09:54:11 delete finalwalk: residue purged (reports=48 app_telemetry=73 app_log_tails=0
|
||||
log_tail_requests=0 notif_prefs=1 selfbind_tokens=1 appliance_registrations=1)
|
||||
09:54:11 customer DELETE cascade COMPLETE for finalwalk (journal #14) — full teardown
|
||||
```
|
||||
|
||||
`existed=true` on the PBS leg is the positive observable that it deleted something rather than
|
||||
no-op'ing on an absent namespace. The residue counts sum to **124**, matching the preview exactly.
|
||||
|
||||
Then, guarded a second time on `qm config 324` reading `name: finalwalk-appliance`:
|
||||
`qm destroy 324 --purge` → absent from `qm list`, and `/mnt/nvme-1tb/images/` is now empty.
|
||||
|
||||
## After — every layer, each with a positive control
|
||||
|
||||
| Layer | Item | After | Positive control (must survive, and does) |
|
||||
|---|---|---|---|
|
||||
| machine | VM 324 + 4 disks | **absent** from `qm list`; image dir empty | VM **300** `drill-r50` still listed |
|
||||
| hub | `hosts`, `customer_configs`, `host_escrow`, `host_escrow_superseded`, `host_recovery`, `host_pbs_secrets`, `one_time_secrets`, `dr_recipe`, `customer_claims`, `guests`, `reports`, `host_reports`, `wg_peers` | **0 rows each** | `demo-felhom` 50,823 · `demo-hp` 7,640 · `peti` 1,827 rows still present |
|
||||
| off-site | Storage Box sub-account `u629488-sub4` (id 285071) | **gone** — 3 sub-accounts remain | `u629488-sub1/2/3` (demo-felhom, peti-felhom, demo-hp) all present |
|
||||
| off-site | `ep0` PBS namespace `finalwalk` | **gone** | namespaces `demo-felhom`, `demo-hp` both present |
|
||||
| network | WireGuard peer `10.77.0.5` | **gone from the LIVE `wg show` on ep0**, not merely from the hub DB | peers `10.77.0.2`, `.3`, `.4`, `.250` all still on `wg0` |
|
||||
|
||||
**Space returned, measured.** `c11-scratch` used **21,010,360 → 6,262,060 KiB = 14,748,300 KiB
|
||||
(14.06 GiB)** reclaimed, against the 15 G measured on disk before deletion. Available went
|
||||
912,342,728 → 927,091,028 KiB.
|
||||
|
||||
**`demo-hp`'s own guest 9201 is still running** — the host itself was untouched.
|
||||
|
||||
## ⚠ WHAT IS **NOT** GONE, and it is not what the previous ledger implied
|
||||
|
||||
A full census (every table, every column) after the cascade returns **61 rows still matching
|
||||
`finalwalk`**, not 0. Four of the five sources are **deliberate and correct** — the cascade's own
|
||||
header says *"Provenance/events are NEVER wiped — audit outlives every tier"*:
|
||||
|
||||
| table | rows | verdict |
|
||||
|---|---|---|
|
||||
| `events` | 16 | **by design** — audit |
|
||||
| `notification_log` | 14 | **by design** — audit |
|
||||
| `host_deletions` | 1 | **by design** — the deletion tombstone |
|
||||
| `customer_resets` | 1 | **by design** — the journal row (`#14`, completed) |
|
||||
| **`app_log_issues`** | **29** | **A GAP — see below** |
|
||||
|
||||
**`app_log_issues` is not covered by the residue purge, and it is systematic, not a finalwalk
|
||||
quirk.** Rows still reference every venue ever torn down: **finalwalk 29, `c11` 40, `rewalk` 20,
|
||||
`part4` 24** — the last three from the 2026-08-06 teardown, whose ledger recorded "0 occurrences"
|
||||
for those customers. **That claim was measured with a narrower query than the one used here and does
|
||||
not hold under a full census.** Recording the correction rather than quietly re-measuring.
|
||||
|
||||
**Why it was probably never written, established rather than assumed:** the table is a **fleet-wide
|
||||
aggregate**, keyed on `app_name`+`fingerprint` with an `affected_customers` JSON list. Of the 29 rows,
|
||||
**12 reference only `finalwalk`** (orphans, safely deletable) and **17 are shared with live
|
||||
customers** (`demo-felhom`, `peti-felhom`, …) and **must not be deleted — only de-referenced**. A
|
||||
naive `DELETE … WHERE customer LIKE` would destroy live customers' issue history, so the fix is a
|
||||
de-reference plus an orphan sweep, not a delete.
|
||||
|
||||
**It carries no secret material** — app name, fingerprint, message text, counts and timestamps. What
|
||||
survives is the deleted customer's *identifier* inside an aggregate. Filed as **R-244**; **not fixed
|
||||
here**, because this session's scope is a spike and two operations, and a cascade change needs its own
|
||||
red-proof.
|
||||
|
||||
## Side effect, recorded
|
||||
|
||||
Stopping the box to age it past the staleness gate fired a genuine operator alert at 07:43:11Z —
|
||||
`Staleness: finalwalk ok → stale (node_stale)` and `Operator email sent for finalwalk/node_stale`.
|
||||
Expected, and correct behaviour: the hub cannot know a stop was deliberate. Noted so the mail is not
|
||||
later read as an incident.
|
||||
|
||||
## R, and the other session credentials
|
||||
|
||||
**Shredded, with a planted-copy control** — because a sweep that finds nothing proves nothing until
|
||||
the instrument is shown to work:
|
||||
|
||||
1. A copy of `R` was **planted** at a second path.
|
||||
2. The search found **both** the real `R` (`~/.config/finalwalk/R_finalwalk.txt`) and the plant —
|
||||
the instrument works.
|
||||
3. Both were `shred -u`'d, along with the venue's other session credentials (appliance root, dashboard
|
||||
passwords, claim code, retrieval password) and the working copy of the appliance root password.
|
||||
`~/.config/finalwalk/` was removed.
|
||||
4. The **same** search then returned **0 hits**.
|
||||
|
||||
On 2026-08-05 a sweep pointed at a path that did not exist and its zero meant nothing; on 2026-08-06 a
|
||||
control found a copy nobody had created. This one was proven in both directions before the zero was
|
||||
believed.
|
||||
|
||||
## Untouched, as scoped
|
||||
|
||||
`demo-hp` itself and its guest 9201, `demo-felhom`, `drill-r50` (VM 300), DooPlex, **the parked
|
||||
tester's box (`peti-felhom` / `u629488-sub2`)**, and every `ep0` namespace that is not the venue's own.
|
||||
The stale pre-volume-move tree at `/srv/pbs-felhom` on ep0 was read and left alone — it is not the live
|
||||
datastore and contains no `finalwalk` namespace.
|
||||
Reference in New Issue
Block a user