DRILL R-95: the recovery route does not exist — stopped before the destructive phase
gates / gates (push) Successful in 19s
gates / gates (push) Successful in 19s
The drill was to delete demo-hp's off-site history and get it back out of a Storage
Box snapshot, filling row 10's blank RTO. Phase 1 found there is nothing to get it
back from: no snapshot is reachable from a sub-account BY ANY NAME.
Measured, read-only, no delete verb issued against any live store:
- 777,600 exact names in the vendor form YYYY-MM-DDTHH-MM-SS, nine full days at
second granularity, plus 126 alternative shapes -> ZERO hits.
- The control is what makes that mean anything: the identical 600-name batch shape
with one real path appended returned it, 6 of 6.
- Structural cause: /home (u629488-sub3) is st_dev 0,82; /.zfs/snapshot is st_dev
0,276; /home/.zfs does not exist. A snapshot under /.zfs/snapshot belongs to a
different dataset than the one holding felhom-repo.
- Three tools agree with controls in the same run: SFTP, the port-23 shell,
rsync --list-only.
So yesterday's re-scope splits: clause (a) "the box cannot write into the snapshot
area" STANDS and is re-confirmed; clause (b) "the rest is recoverable file by file"
is NOT SUPPORTED. STOPPED before Phase 2 on the operator's ruling — with no recovery
leg the deletion would have destroyed real history to buy only an alarm test that
could not fire at the specified size. Store verified untouched at 69 snapshots.
R-432 ANSWERED (negatively; its panel-read next step withdrawn as unnecessary).
R-433 no snapshot reachable by any name — decides R-95's remedy and its rank.
R-434 the drop alarm's text promises a file-by-file recovery that cannot be performed.
R-435 the drop detector is blind to a single-app deletion (>50% of 69 needed, ~9 given).
R-436 LEAD: the provider offers `rclone serve restic --stdio` and restic 0.14.0 speaks
`rclone:` (measured, controlled) — real prevention may need no new machine, IF
the vendor pins --append-only. Ask before building.
07 §8 row 10: text corrected, status NOT moved, RTO still blank.
No code, no version bump, no image, no golden. REPORT.md's only copy of the R-331
report preserved as REPORT-r331-backup-card.md before overwrite.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LB8FmJaGd2cyjvy6dbEjpM
This commit is contained in:
@@ -1,187 +1,166 @@
|
||||
# REPORT — R-331: the operator Backup card said every customer had no backups
|
||||
# REPORT — the deletion we said is survivable: the recovery route does not exist (R-95 drill, 2026-09-01)
|
||||
|
||||
**Hub v0.109.0 (with controller v0.225.0) · 2026-08-30**
|
||||
**RUNBOOK, destructive class, `demo-hp` only. STOPPED at the end of Phase 1 on the operator's ruling,
|
||||
before any destructive step. No delete verb was issued against any live store; no byte on either
|
||||
Storage Box sub-account was written, moved or removed.** No production code, no version bump, no
|
||||
image, no golden. Evidence: `documentation/audits/evidence-drill-r95-recovery-2026-09-01/`.
|
||||
|
||||
| # | phase | verdict | one sentence |
|
||||
|---|---|---|---|
|
||||
| 1 | snapshot reachable, and its name | **NO — and it has no reachable name** | 777,600 exact names across nine days in the vendor's own format, plus 126 alternative shapes; zero resolve, with a control proving the sweep detects a path that exists. |
|
||||
| 2 | the deletion | **NOT RUN — operator ruling** | With no recovery route, the deletion would have destroyed real history to buy nothing; put as a two-option decision, the ruling was stop. |
|
||||
| 3 | the alarm fired | **NOT RUN — and it could not have fired at the specified size** | The shipped threshold needs a fall of more than half; one app's tag is ~9 of 69. → **R-435** |
|
||||
| 4 | **the recovery** | **NOT RUN — no route exists that is not fenced** | Box-side: proven impossible. Panel: fenced and browserless. Hetzner API: fenced (§11-D), and the hub's client has no snapshot method at all. |
|
||||
| — | **RTO from T₀** | **STILL BLANK** | Row 10's RTO cell is unchanged and remains a finding. |
|
||||
| — | **data lost, quantified** | **NOT MEASURABLE THIS WAY** | The quantity only has meaning if the rest is recoverable, and the route that would recover it is not reachable. |
|
||||
| 5 | re-arm | **NOT RUN** | Depended on Phase 4. |
|
||||
| 6 | teardown | **PASS** | Store untouched at 69 snapshots; all three scratch layers removed; hub DB copies shredded; both boxes healthy. |
|
||||
|
||||
---
|
||||
|
||||
## 1. What was wrong
|
||||
## 1. Did the recovery work — and does yesterday's re-scope survive?
|
||||
|
||||
The hub customer page's **Backup** card read, for **every customer, indefinitely**:
|
||||
**The recovery was never reachable, and the re-scope does not survive intact. Its first half stands;
|
||||
its second half does not.**
|
||||
|
||||
```
|
||||
Enabled Yes Snapshots 0
|
||||
Repo Size 0 MB Integrity Unknown
|
||||
```
|
||||
Yesterday's re-scope has two clauses. They must now be separated:
|
||||
|
||||
Measured on `demo-hp` 2026-08-30, at which moment the truth was:
|
||||
* **(a) "The box can delete its live repository, but cannot write to the daily snapshots of it."**
|
||||
**STANDS.** Re-confirmed here: `/.zfs/snapshot` is reachable and the write-refusal measurement is
|
||||
unchanged. Nothing in this drill weakens it.
|
||||
* **(b) "…so the rest is recoverable — file by file, one customer at a time."** **NOT SUPPORTED.**
|
||||
A snapshot that cannot be opened cannot be copied out of. R-432 recorded the directory listing
|
||||
empty and named the cheapest next step: *"a single `ls /.zfs/snapshot/<name>` from a box then
|
||||
settles whether a named snapshot can be entered even though the directory does not list (ZFS
|
||||
allows exactly that)."* **That step is now done, exhaustively, and the answer is no.**
|
||||
|
||||
| source | value |
|
||||
|---|---|
|
||||
| the box's own `settings.json` | `snapshot_count: 67, repo_size_bytes: 140829678, stats_known: true` |
|
||||
| that night's controller log | `[offbox] backup OK: 8 app(s) backed up, 67 snapshot(s), 2m14s` |
|
||||
| **this hub's own Offsite page** | `0.1 GB` used of a `50 GB` quota — read from the same stored report |
|
||||
**What was measured.** The port-23 restricted shell accepts a batched `stat`, which makes a cheap
|
||||
existence oracle: 500–600 paths per round trip, stdout carrying only paths that exist.
|
||||
|
||||
**A card that reads "no backups" over a working backup is worse than no card.** It is the R-88
|
||||
direction of failure — degrading to *no backup* rather than to *unknown* — on the one screen an
|
||||
operator consults to answer "is this customer protected?".
|
||||
|
||||
## 2. Root cause
|
||||
|
||||
The card rendered the report's **`backup`** object. Its `snapshot_count`, `repo_size_mb` and
|
||||
`integrity_ok` fields have had **no producer** since disk-tier restic moved to the host agent (slice
|
||||
8C) — the controller's `buildBackupReport` leaves them zero *deliberately* and says so in a comment.
|
||||
The zeros were correct values for dead fields, rendered as if live.
|
||||
|
||||
**The data was never missing.** The live numbers ride in the report's **`offsite`** object, which this
|
||||
package **already** reads for the Offsite page (`offsiteUsageBytes`) and which `monitor.OffsiteChecker`
|
||||
**already** drives fill and staleness alarms from. That the Offsite page rendered demo-hp's real usage
|
||||
from the same stored report, at the same moment the Backup card said `0 MB`, is the proof the bytes
|
||||
were arriving. This is a **render fix over an existing feed**, not a new pipeline.
|
||||
|
||||
## 3. Why it was not a one-line template swap
|
||||
|
||||
`snapshot_count: 0` means two opposite things — *this repository holds nothing* and *nobody has ever
|
||||
measured this repository*. **R-225 measured that confusion one layer down**: a rebuilt box rendered
|
||||
„Tarolo meret · 0 pillanatkep" over a store that really held snapshot `f3d9cd67`, and the controller's
|
||||
`StatsKnown` fixed it there. It was never on the wire, so rendering the count without it would have
|
||||
**moved R-225 up to the hub instead of fixing anything**. Controller v0.225.0 now forwards
|
||||
`stats_known`.
|
||||
|
||||
## 4. What changed
|
||||
|
||||
`hub/internal/web/backup_card.go` builds a typed `backupCardView` — resolved in Go, because the card's
|
||||
whole subject is a distinction a template `{{if}}` chain over `map[string]interface{}` float64s cannot
|
||||
keep:
|
||||
|
||||
| report state | card shows |
|
||||
|---|---|
|
||||
| no `offsite` object at all | "No off-site data reported" — **and says explicitly this is not the same as "no backups"** |
|
||||
| `enabled:false` + declared `state` | the blocker by name (`needs_credential`) — a different operator action from "not enabled" |
|
||||
| enabled, `stats_known:false` | **—**, plus "never been measured". Never `0` |
|
||||
| enabled, `stats_known:true` | the real count and size, **including a real `0`** — measured empty is knowledge |
|
||||
|
||||
**A pre-v0.225.0 controller sends no `stats_known`, which unmarshals to false → "unknown".** That is
|
||||
the fail-safe direction: upgrading the hub ahead of the fleet must not tell the operator that every
|
||||
un-upgraded customer has zero backups. Pinned by a test.
|
||||
|
||||
**The Integrity row is deleted, not re-sourced.** Nothing produces it: the controller runs no integrity
|
||||
check, and `NotifyIntegrityOK` / `NotifyIntegrityFailed` exist and are **called from nowhere**. A row
|
||||
that can only ever read "Unknown" is not information, and one that could read "OK" from an unwritten
|
||||
field would be a lie.
|
||||
|
||||
`fmtBytesAuto` is new rather than reusing `fmtBytesGB`: that one is fixed at GB because it renders
|
||||
against GB quotas, and it turns demo-hp's real 140 829 678 bytes into `0.1 GB` — which on a card whose
|
||||
entire defect was under-reporting a real backup reads as "nearly nothing".
|
||||
|
||||
## 5. Tests and the red-proof
|
||||
|
||||
`r331_backup_card_test.go` asserts the **rendered page**, using demo-hp's real reported values, so a
|
||||
regression fails against the same numbers the defect was measured against. **The defect lived in the
|
||||
template's choice of source object, so a test one layer below it would have been green against the
|
||||
shipped bug** — which is why these drive `handleCustomerUnified` and grep the HTML.
|
||||
|
||||
**RED-PROOF (run 2026-08-30):** restoring the pre-fix card markup fails all four tests —
|
||||
`the rendered Backup card does not contain demo-hp's real snapshot count (67)`,
|
||||
`the card does not carry the real repository size (134.3 MB ...)`,
|
||||
`the card still shows an Integrity row`, plus every branch of the three-way ruling. Restored
|
||||
immediately; `git diff` clean.
|
||||
|
||||
**Green gate:** `go build ./... && go vet ./... && go test ./...` in `hub/` — 18 packages, rc 0.
|
||||
|
||||
## 6. Deployment and live verification
|
||||
|
||||
Hub **0.109.0** built, pushed, manifest bumped, ArgoCD hard-refreshed and synced. Controller
|
||||
**0.225.0** deployed to both demo boxes. Verified from the live objects, not from a rollout message
|
||||
(an ArgoCD "rolled out" can name the old image):
|
||||
|
||||
```
|
||||
argocd: sync=Synced rev=36f86300209e9ce914f2a97dac16ee9eeea249e2 (== HEAD)
|
||||
deploy: gitea.dooplex.hu/admin/felhom-hub:0.109.0
|
||||
pod: hub-6795879c4b-pf4lz ...felhom-hub:0.109.0 Running
|
||||
boxes: ...felhom-controller:0.225.0 Up (healthy) on demo-felhom AND demo-hp
|
||||
```
|
||||
|
||||
**The card, fetched from the live hub** (endpoint-level: the exact URL the operator's browser
|
||||
requests; the residual is client-side rendering only — there is no browser on DooPlex):
|
||||
|
||||
| | demo-hp | demo-felhom |
|
||||
| sweep | candidates | hits |
|
||||
|---|---|---|
|
||||
| Off-site snapshots | **67** | **10** |
|
||||
| Repo size | **134.3 MB** | **132.5 KB** |
|
||||
| Last successful run | 15h ago | 15h ago |
|
||||
| Soft quota | 50 GB | 50 GB |
|
||||
| Integrity row | **absent** (grep count 0) | **absent** (grep count 0) |
|
||||
| `/.zfs/snapshot/YYYY-MM-DDTHH-MM-SS`, nine full days, second granularity | **777,600** | **0** |
|
||||
| 126 alternative name shapes and snapshot paths (`daily`, `snapshot-1`, colon and compact time forms, `/home/.snapshot`, …) | 126 | 0 |
|
||||
| **control — the identical 600-name batch shape with one real path appended** | 6 batches | **6/6 returned it** |
|
||||
|
||||
Both read `Snapshots 0 · Repo Size 0 MB · Integrity Unknown` before this change.
|
||||
|
||||
**Cross-checked against the source, not just against itself** — the numbers on the card are the
|
||||
numbers on the boxes:
|
||||
**And there is a structural reason, which is why I stopped sweeping.** The customer's data and the
|
||||
snapshot door are on **different filesystems**:
|
||||
|
||||
```
|
||||
demo-hp settings.json offbox: snapshot_count 67, repo_size_bytes 140829678, stats_known true
|
||||
demo-felhom settings.json offbox: snapshot_count 10, repo_size_bytes 135635, stats_known true
|
||||
135635 / 1024 = 132.5 KB → matches the rendered value
|
||||
df → u629488-sub3 mounted on /home
|
||||
stat /home → Device 0,82
|
||||
stat /.zfs/snapshot → Device 0,276 ← a different device
|
||||
stat /home/.zfs → cannot statx: No such file or directory
|
||||
```
|
||||
|
||||
**One honest detail worth keeping:** demo-felhom's "Last DB dump" reads `—`. That is correct, not a
|
||||
regression — its only app (`opengist`) has no database, so the box has never taken a DB dump.
|
||||
A ZFS snapshot under `/.zfs/snapshot` belongs to the dataset that owns that `.zfs` — not to the
|
||||
child mounted at `/home`. **So even a correctly named snapshot there could not contain
|
||||
`felhom-repo`,** and the dataset that does hold it exposes no `.zfs` at all to this account. The
|
||||
empty listing is not a display toggle hiding a reachable tree; from a sub-account there is no tree.
|
||||
|
||||
**Not verified live: the "never measured" branch.** Both boxes report `stats_known: true`, so the
|
||||
degradation path could not be exercised on real hardware without falsifying a box's state. It is
|
||||
covered by `TestBackupCard_ThreeWayRuling` and `TestBackupCard_OldControllerDegradesToUnknownNotEmpty`
|
||||
at render level, and this is stated rather than implied.
|
||||
**Three tools agree, each with controls in the same run:** SFTP, the port-23 shell, and
|
||||
`rsync --list-only`.
|
||||
|
||||
## 7. The push bypassed a gate, deliberately, and here is the declaration
|
||||
**What that does to R-95.** Its *exposure* is unchanged and its *remedy* is not. Yesterday the row
|
||||
could say a deletion costs about a day because the rest comes back per-file. Today the only routes
|
||||
to "the rest" are a whole-box panel rollback (which deletes newer snapshots and hits every customer
|
||||
on the box) and the provider API (fenced, and unimplemented in the hub's client). **The re-scope's
|
||||
comfort was resting on a route nobody had walked — which is precisely the standard this project
|
||||
applies, and it is the reason this drill was called.**
|
||||
|
||||
**`git push --no-verify` was used for this change.** `repo_gates.py`'s `golden-currency` gate was
|
||||
CONVICTED and it was RIGHT: controller **v0.224.0** and **v0.225.0** are released and the newest golden
|
||||
bake carries **0.223.0**, so a machine installed right now receives neither fix.
|
||||
**The ranking is Viktor's and I am not re-ranking it.** What I will say plainly: the argument that
|
||||
moved R-95 down yesterday is the argument this drill removed. On these facts I would put it back
|
||||
where it was.
|
||||
|
||||
**This is a BYPASS, not a waiver.** The gate offers a waiver only for a release that *deliberately needs
|
||||
no golden*; these need one. **The operator was asked and ruled bypass-now-bake-later**, on the stated
|
||||
ground that neither fix bites a day-0 box — R-330 is a nightly false alarm about apps a new box has not
|
||||
installed yet, R-331 is a hub-side display over backups a new box has not taken yet — and both arrive by
|
||||
self-update afterwards. That ground is recorded on R-242 precisely because it is the thing to re-check:
|
||||
**it does not extend to a release that changes first-boot behaviour.**
|
||||
## 2. The RTO
|
||||
|
||||
**A golden carrying 0.225.0 is OWED** (`RUNBOOK-manual-build.md` §4.1; the vouch is a three-field change,
|
||||
`MinAgent 0.129.0`). This is the **fourth** bypass of this gate, and the gap it names is now two releases
|
||||
wide rather than one.
|
||||
**Still blank, and it stays a finding.** `07` §8 row 10's RTO cell has been empty since July and this
|
||||
drill did not fill it. Nothing was recovered, so nothing was timed. The summary line at
|
||||
`07-backup-architecture.md:948` — *"no ransomware-shaped recovery has ever been run"* — is still
|
||||
true, and is now true for a sharper reason: **not "nobody has run it" but "from the box, it cannot
|
||||
be run."**
|
||||
|
||||
**The other failing gate was fixed, not bypassed.** `due-checks` was red on R-341's `+7 d` measurement,
|
||||
five days overdue. It was **taken** during this session — see §8.
|
||||
## 3. R-432's answer, and the naming scheme
|
||||
|
||||
## 8. R-341's overdue check was taken, and its premise did not survive
|
||||
**R-432 is ANSWERED, and negatively. It did not need the panel read it was waiting on.**
|
||||
|
||||
Unrelated to R-331; it blocked the same push, so it was done rather than deferred. Evidence:
|
||||
`documentation/audits/evidence-r341-plus7d-2026-08-30/step1-fd-and-sockets.txt`.
|
||||
* **The naming scheme is `YYYY-MM-DDTHH-MM-SS`** — vendor-documented examples `2025-12-03T13-47-47`,
|
||||
`2025-02-12T11-35-19`. Recorded so nobody hunts a console again.
|
||||
* **Knowing it does not help.** Every name in that format for nine days is refused, and the st_dev
|
||||
split above says why. **Per-file recovery is not operator-only — from the box it is nobody's,** and
|
||||
for the operator it is a browser act against the main account that no credential in this project
|
||||
can perform.
|
||||
* **The panel cannot supply the missing piece either.** It offers Restore and Delete on a row and
|
||||
does not show names; and the one name-shaped thing it could give would be tried against a door
|
||||
that leads to the wrong dataset.
|
||||
|
||||
**Precondition passed**, which is what makes the reading interpretable: `proxmox-backup-proxy` still
|
||||
`MainPID 551655`, `ps -o lstart=` still `2026-08-18 09:51:04`, `NRestarts=0` — the same proxy generation
|
||||
as t0, so nothing restarted and re-based the count. (The anchor is `ps`, not `ActiveEnterTimestamp`,
|
||||
which reads 03:54:54Z here — R-346's trap, avoided.)
|
||||
## 4. The alarm's first real firing
|
||||
|
||||
**Result: fd = 17.** Not 17 more — seventeen total, exactly the documented baseline, against **405** at
|
||||
the first check on 2026-08-20. The socket histogram holds **one LISTEN and nothing else**: ESTAB 0,
|
||||
CLOSE-WAIT 0.
|
||||
**It did not happen, and the drill as written could not have produced it.** The detector fires on a
|
||||
fall of **more than half** the previous count **and at least 5** (`hub/internal/monitor/offsite.go`,
|
||||
`snapshotDropFraction = 0.5`, `snapshotDropFloor = 5`). demo-hp's baseline is **69**. Phase 2 deletes
|
||||
**one app's** history — about **9** snapshots. 9 is over the floor and nowhere near half, so the
|
||||
alarm stays silent, **correctly and by design**. Firing it for real needs ~35+ snapshots destroyed,
|
||||
i.e. most of demo-hp's off-site history. That trade is what the operator was asked to rule on. → **R-435**
|
||||
|
||||
**The verdict is "unanswerable", not "the upgrade fixed it".** R-341 asks whether the PBS 4.2.5-1
|
||||
upgrade changed the fd slope. Inside this interval **we removed the leak ourselves** (R-344, agent
|
||||
0.130.0, now live on both boxes). A slope of ~0 measures our fix, not the upgrade — and reading it the
|
||||
other way would credit a changelog that was read in advance and found to contain no such mechanism. The
|
||||
perturbation pre-registered for this window was Phase C at ~3%; the actual perturbation was the removal
|
||||
of the entire phenomenon. The question is now **moot**, and the row is closed as such.
|
||||
**One thing the alarm says is now wrong.** Its message, live in hub 0.111.0, reads:
|
||||
|
||||
**What it does establish, which is worth more than the original question:** twelve days after the R-344
|
||||
fix, on the same proxy generation with no restart to hide behind, ep0 sits at baseline with zero
|
||||
established connections. The 388-descriptor accumulation has not returned, and R-336's ~323-day runway
|
||||
concern retires with it.
|
||||
> "The daily Storage Box snapshots are read-only and still hold the older copy, **so this is
|
||||
> recoverable file-by-file**; it is NOT confirmed data loss."
|
||||
|
||||
## 9. Not done, and why
|
||||
The first clause is true; **the second promises a recovery the product cannot perform and the
|
||||
operator cannot perform without a browser and the main account.** This is this project's own
|
||||
corollary — *when a verdict changes which field it counts from, the alarm text has to change with
|
||||
it* — landing on the alarm shipped the same day. → **R-434**
|
||||
|
||||
- **No staleness verdict on the card.** `monitor.OffsiteChecker` already owns that and alarms on it. A
|
||||
second verdict over the same data is two things that can disagree — a shape this codebase has already
|
||||
paid for (`LastRun` vs `LastSuccess`, R-100).
|
||||
- **The dead `backup` fields were not removed from the controller's wire format.** Removing them would
|
||||
stop historical reports already in this hub's store from parsing, for no gain — nothing renders them
|
||||
now, and a controller-side test fails if anything starts producing them.
|
||||
## 5. Findings, as register rows
|
||||
|
||||
All four filed in `documentation/backlog/OPEN-ITEMS.md`.
|
||||
|
||||
| row | finding |
|
||||
|---|---|
|
||||
| **R-433** | A sub-account cannot reach any Storage Box snapshot **by any name**; `/home` and `/.zfs` are different filesystems and `/home/.zfs` does not exist. Answers R-432 negatively and removes clause (b) of the R-95 re-scope. |
|
||||
| **R-434** | `emitSnapshotDrop`'s message promises file-by-file recovery that is not reachable. Live in hub 0.111.0. |
|
||||
| **R-435** | The drop detector cannot see a single-app deletion (>50% of 69 ⇒ ~35 needed). `offbox.go:1388` forgets **by tag**, so a single-tag wipe is exactly the shape the detector is blind to. Deliberate insensitivity, but the blind spot should be stated where the operator reads it. |
|
||||
| **R-436** | **LEAD, not a defect.** Hetzner's port-23 shell offers `rclone serve restic --stdio` as a server-side backend, and restic 0.14.0 recognises the `rclone:` backend (measured; control `banana:` → invalid backend; rclone is absent from the controller image). `rclone serve restic` carries `--append-only`. **This could make R-95's real prevention far cheaper than the spike concluded — no new always-on machine, no data migration.** Caveat stated up front: the **client** supplies the server command line, so a compromised box could omit the flag unless the provider pins it. Settling that is a vendor question, not a code change. |
|
||||
|
||||
**R-432 is marked ANSWERED**; its "one panel read settles it" next step is withdrawn as unnecessary.
|
||||
|
||||
## 6. Does `07` §8 row 10 move?
|
||||
|
||||
**No. It stays `PARTIAL`, and its RTO stays blank.** The status was already correct for the right
|
||||
reason — *"the recovery ROUTE has never been walked, which is what PARTIAL means"* — and this drill
|
||||
found the route is not walkable from the box at all. **What the row needs is a text correction, not a
|
||||
status change:** its clause *"recoverable per-file (vendor)"* and its limit *"per-file recovery is
|
||||
operator-only today (R-432)"* both overstate what exists. Updated in place with the citation. Moving
|
||||
it only as far as the evidence goes means not moving it.
|
||||
|
||||
## 7. What could not be tested, and why
|
||||
|
||||
* **Whether the main account can see the snapshots.** No main-account credential exists in this
|
||||
project — the hub holds only per-customer sub-accounts. This is the one question that would decide
|
||||
whether per-file recovery exists *at all*, for anyone.
|
||||
* **Whether the Hetzner API can list or read a snapshot.** Fenced by the runbook (§11-D). Separately,
|
||||
`hub/internal/hetznerapi/hetznerapi.go` has **no snapshot method** — so this route needs new code
|
||||
regardless of the fence.
|
||||
* **The deletion, the alarm's first real firing, the recovery, the RTO, the re-arm.** Phases 2–5, not
|
||||
run, on the operator's ruling.
|
||||
* **Whether `rclone serve restic --stdio` is pinned server-side with `--append-only`** (R-436).
|
||||
|
||||
## 8. My own mistakes
|
||||
|
||||
* **I ran Phase 1 before finishing Phase 0's subject-app choice, and did not say so up front.** The
|
||||
choice depends on the snapshot's timestamp, so the order was right, but the runbook's order is the
|
||||
runbook's and a silent reordering is the thing this project keeps getting caught by. Stated at the
|
||||
time in the session, recorded here.
|
||||
* **My first sweep guessed the schedule instead of establishing it.** I probed 00:00 UTC and 22:00
|
||||
UTC — 600 names — on the strength of a register line reading *"daily 00:00"*, got nothing, and only
|
||||
then widened to whole days. The narrow sweep was worth nothing on its own: a zero over a guessed
|
||||
window is not evidence, and I should have gone to full days first or not run it at all.
|
||||
* **I nearly reported the empty listing as "the display toggle is hiding it".** The vendor documents
|
||||
exactly such a toggle and it fitted. The st_dev comparison — which I only ran because `df` printed
|
||||
a filesystem name I did not expect — says the tree is on another dataset entirely. **A plausible
|
||||
cause that fits the symptom is not a measured one**, and I had the wrong one for about ten minutes.
|
||||
* **`REPORT.md` held the only copy of the R-331 report** (hub v0.109.0, 2026-08-30) — durable content
|
||||
living only in the overwritten file, which `CLAUDE.md:82-87` forbids. Preserved as
|
||||
`REPORT-r331-backup-card.md` before this report replaced it.
|
||||
|
||||
Reference in New Issue
Block a user