RECON: what DooPlex's own backup actually protects (R-232) — read-only, nothing changed
gates / gates (push) Successful in 9s
gates / gates (push) Successful in 9s
Five sets, 14/14 successful runs in 14 days, and a file restored byte-identical from the data repo -- the first time a restore has ever been demonstrated here. What the failure matrix exposes: notify_failure is a no-op (NOTIFY_ON_FAILURE=true, webhook commented out) so a failed backup tells nobody; NOTHING leaves the box (Longhorn's target is DooPlex itself over NFS, the only outbound-looking cron pulls inbound for another project); the backup tree is a single writable path with no append-only; two sets are same-disk (.claude-memory and the postgres dumps, whose source sits inside the backup tree); Longhorn retains 1 generation; secrets/restic-repo has never held a snapshot because backup-secrets.sh never calls restic; and /opt/backup/docs/BACKUP-RESTORE.md does not exist though the unit advertises it. NOT a finding: the restic passphrase. On-box copy is on a different disk from the backups and the operator holds an offline copy out of band, so disk loss is recoverable. The residual is only that it is operator-held rather than system-held escrow. A stale lock in the data repo is MINE, from the previous task's verification. Fenced from clearing it; recorded with its impact and the one command that fixes it.
This commit is contained in:
@@ -17,6 +17,14 @@
|
||||
|
||||
## Standing rulings
|
||||
|
||||
**DooPlex's own backup, surveyed 2026-08-06 (R-232, read-only).** Five sets nightly, 14/14 in 14
|
||||
days, single-file restore proven byte-identical. Destination for **every** set is `/dev/sda1`;
|
||||
`.claude-memory` and the PostgreSQL dumps are the two same-disk cases. **Nothing leaves the box** —
|
||||
Longhorn's target is DooPlex itself over NFS, and the only outbound-looking cron pulls inbound.
|
||||
`notify_failure` is a no-op (no webhook), so a failure is silent. The restic passphrase has an
|
||||
**offline operator-held copy**, so disk loss is recoverable — it is not system-held escrow, which is
|
||||
the only residual. Full survey: `audits/RECON-dooplex-backup-2026-08-06.md`.
|
||||
|
||||
**S-37 — A CLAIM IN AN INSTRUCTION FILE IS CHECKED, NOT TRUSTED (2026-08-06, R-229/R-230 close-out).**
|
||||
|
||||
1. **The workspace-root `CLAUDE.md` is a SYMLINK** to `documentation/runbooks/workspace-CLAUDE.md`.
|
||||
|
||||
@@ -91,7 +91,13 @@ failed until someone walks it end to end with no help from us.
|
||||
held. It is now included in that machine's nightly backup. **Two things you should know before
|
||||
treating that as solved:** the copy lands on the **same physical disk** as the original, so it
|
||||
survives a mistake but not a dead drive, and the build machine's backups have **no off-site copy at
|
||||
all**. Also tidied the same day: a quarter of those notes had become unreachable — filed but listed
|
||||
all**. **A full survey of that machine's backup, done the same day, confirmed both and found two
|
||||
more things worth knowing.** The good news first: it has run every night without missing a set, and
|
||||
we pulled a file back out of it and checked it matched the original exactly — the first time that
|
||||
has ever been demonstrated. The rest: **if a backup ever fails, nobody is told** — the alert was
|
||||
configured but never given anywhere to send to — and **every copy it makes stays inside that one
|
||||
box**, so it survives any single disk dying but not the room. Nothing was changed; the survey was
|
||||
read-only and the decisions are yours. Also tidied the same day: a quarter of those notes had become unreachable — filed but listed
|
||||
nowhere, so nothing would ever read them — and the instructions the assistant reads at the start of
|
||||
every session were cut roughly in half, with a check added so they cannot quietly grow back. Nothing
|
||||
was deleted. *(R-229, R-230, R-231)*
|
||||
|
||||
@@ -0,0 +1,214 @@
|
||||
# RECON — what DooPlex's own backup actually protects
|
||||
|
||||
**2026-08-06, read-only.** Nothing was modified, stopped, pruned or deleted. One scratch restore into
|
||||
`/tmp/recon-restore-20260806-121056`, deleted afterwards. No customer machine touched.
|
||||
|
||||
**State at start:** no backup running, no restic process. `dooplex-backup.timer` **enabled + active**,
|
||||
next fire 2026-08-07 03:18:49. The service showing `disabled` is **expected** for a timer-driven unit
|
||||
and is **not a finding** — recorded here so nobody chases it later. Last run 03:06:07 → 03:12:51,
|
||||
`Result=success`, `ExecMainStatus=0`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Set inventory
|
||||
|
||||
Five sets run from `backup-all.sh`. Two more things write into the same tree and are **not** part of
|
||||
that service. Every "last success" below is from the journal or the repo, not from a config line.
|
||||
|
||||
| Set | Source paths | Source dev | Mechanism | Destination | Dest dev | Schedule | Last success | Snaps | Oldest → newest | Verified? | Off-site |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| **K3s State** | `/var/lib/rancher/k3s/server/{db,token,node-token,cred,tls}` · `/etc/rancher/k3s` | `sdb1` | restic | `backup/k3s/restic-repo` | **`sda1`** | nightly 03:00 | 2026-08-06 03:06:07 | 26 (13+13) | 03-31 → 08-06 | **yes** — `check --read-data-subset=5%` every run | no |
|
||||
| **Kubernetes Secrets** | k8s API (all secrets + configmaps, per-namespace) | n/a | **GPG files — NO restic** | `backup/secrets/exports/` | **`sda1`** | nightly | 2026-08-06 03:06 | **0** | 07-06 → 08-06 (96 files, 134 MB, 30-day `find -mtime +30 -delete`) | no | no |
|
||||
| **Git Manifests** | Gitea repos via API token | n/a | clone/mirror | `backup/homelab-manifests/` | **`sda1`** | nightly | 2026-08-06 03:06 | — | — | no | no |
|
||||
| **User Data** | `/mnt/4_hdd/data` · `/mnt/5_hdd/felhom.eu/git/.claude-memory` | `sdg2` · **`sda1`** | restic | `backup/data/restic-repo` | **`sda1`** | nightly | 2026-08-06 10:51:17 (manual) / 03:0x nightly | 14 | 03-31 → 08-06 | **no** | no |
|
||||
| **PostgreSQL Dumps** | `/mnt/5_hdd/backup/postgresql/dumps` | **`sda1`** | restic | `backup/postgresql/restic-repo` | **`sda1`** | nightly | 2026-08-06 03:09:52 | 11 | 06-25 → 08-06 | **no** | no |
|
||||
| *(separate)* **Longhorn PVCs** | 77 volumes, replicas under `/var/lib/longhorn` | `sdb1` | Longhorn backupstore | `nfs://192.168.0.180:/mnt/5_hdd/backup/longhorn-pvc` | **`sda1`** | daily 04:00 + weekly, **retain=1** | 2026-08-06 04:10 (sync 10:08) | 77 vols / 88 GB | — | no | no |
|
||||
| *(separate)* **jarrs.eu** | Hetzner → DooPlex | remote | rsync **inbound** | `backup/jarrs.eu/` | `sda1` | cron 04:00 | 2026-08-06 04:00 | 33 files / 19 MB | — | n/a | **inbound only** |
|
||||
|
||||
**Dormant destination directories** (nothing writes them any more): `demo-felhom` 25 GB, newest
|
||||
**2026-07-10** (27 days); `sparkyfitness` 652 KB, newest **2026-06-24** (43 days);
|
||||
`secrets/restic-repo` — initialised 2026-01-05, **zero snapshots ever**.
|
||||
|
||||
Tree total ≈ 167 GB. `/mnt/5_hdd` is 24% used, 6.6 TB free — **no space pressure**.
|
||||
|
||||
**Per-set reliability: 14 runs in 14 days, all five sets `✓ completed successfully` every run.** The
|
||||
routine works; the questions below are about what it would survive, not whether it runs.
|
||||
|
||||
---
|
||||
|
||||
## 2. The same-disk answer, stated directly
|
||||
|
||||
**The destination for every set is `/dev/sda1`** (WDC WD103KFBX-68CCLN0, serial `WD-B1036H8J`).
|
||||
|
||||
| Set | Source device | Same disk as destination? |
|
||||
|---|---|---|
|
||||
| K3s State | `sdb1` (Apacer SSD, root) | **No** — cross-disk |
|
||||
| Longhorn PVCs | `sdb1` (replicas under `/var/lib/longhorn`) | **No** — cross-disk |
|
||||
| User Data → `/mnt/4_hdd/data` | `sdg2` (ST3000VX010 `Z6A07P2G`) | **No** — cross-disk |
|
||||
| **User Data → `.claude-memory`** | **`sda1`** | **YES — same physical disk** |
|
||||
| **PostgreSQL Dumps** | **`sda1`** (`/mnt/5_hdd/backup/postgresql/dumps`) | **YES — and the source is *inside the backup tree*** |
|
||||
| Kubernetes Secrets | k8s API (no disk source) | destination `sda1`, single copy |
|
||||
| Git Manifests | Gitea (which is a Longhorn PVC on `sdb1`) | No — cross-disk |
|
||||
|
||||
**The operator's suspicion was right, and it is narrower than feared.** Two sets are same-disk:
|
||||
the memory store (added 2026-08-06) and the PostgreSQL dumps. The postgres case is the stranger of
|
||||
the two — the dump directory lives *inside* `/mnt/5_hdd/backup/`, so restic is deduplicating a copy
|
||||
of files that are already in the backup tree, onto the same disk. It protects against a bad dump, not
|
||||
against losing `sda1`.
|
||||
|
||||
Everything else is genuinely cross-disk. **But cross-disk is not off-site — see §4.**
|
||||
|
||||
---
|
||||
|
||||
## 3. Failure matrix
|
||||
|
||||
**"Demonstrated" means someone has actually done it and checked the result.** Only one cell in this
|
||||
column is a yes, and it was earned today.
|
||||
|
||||
| Failure | What is lost | What is recoverable | From where | How long | Ever demonstrated? |
|
||||
|---|---|---|---|---|---|
|
||||
| **A file deleted by mistake** | the file | **all of it**, to the previous night (or to the 6-hourly dump for a DB) | the relevant restic repo / Longhorn backup on `sda1` | minutes | **YES — today.** A file was restored from the `data` repo and matched the live original byte for byte (§5) |
|
||||
| **`sda1` (5_hdd) fails** | **every backup, all of it** — all four restic repos, the Longhorn backupstore, the secrets exports, the manifest mirror | the *live system* keeps running: k3s, Longhorn replicas, Gitea, the registry and the hub are all on `sdb1`. `/mnt/4_hdd/data` survives on `sdg2`. **`.claude-memory` is lost** (source and only copy were both on `sda1`) | nothing — there is no second copy of the backups | n/a for the backups; the live system is unaffected | no |
|
||||
| **`sdg2` (4_hdd) fails** | the live `/mnt/4_hdd/data` | **all of it** | `data` restic repo on `sda1` | hours (405 GiB) | **partially** — the single-file probe worked; a full-set restore has never been run |
|
||||
| **DooPlex's board or PSU dies** | nothing permanently, if the disks survive | **everything**, by moving the disks or rebuilding and reattaching | `sda1` + `sdb1` in hand | days — and see the note below | no |
|
||||
| **The flat burns** | **everything** — every disk is in the same box, and **nothing leaves it** | **nothing from this backup system.** Separately: customer data has a genuine off-site tier (Hetzner + ep0); the felhom *repos* exist on Gitea, which is on DooPlex; some repos have working clones on this same machine | — | — | no |
|
||||
| **A bad script or ransomware writes over `/mnt/5_hdd/backup/`** | every backup at once | the live system only | nothing — the tree is a single writable path, and the restic repos are **not** `--append-only` | n/a | no |
|
||||
|
||||
**The note on "board or PSU dies":** recovery needs both the disks *and* the restic passphrase. The
|
||||
on-box copy is `/etc/backup/restic-password` on **`sdb1`** — a different disk from the backups, which
|
||||
is good. **The operator has confirmed an offline copy exists out of band**, so this row is
|
||||
recoverable rather than fatal. See §6 for the one residual that remains.
|
||||
|
||||
---
|
||||
|
||||
## 4. Does anything leave the box?
|
||||
|
||||
**No. Nothing.** Stated without softening, because it is the row the matrix turns on.
|
||||
|
||||
- No `rclone`, no remote restic repository, no rsync-to-remote, no S3/B2/SFTP target anywhere in
|
||||
`/opt/backup/scripts/`.
|
||||
- The only outbound-looking job, `sync-hetzner-backups.sh` (cron 04:00), is **inbound**: it pulls
|
||||
*jarrs.eu* database backups **from** Hetzner **to** DooPlex. It is a different project and it moves
|
||||
data the wrong way to help here.
|
||||
- Longhorn's backup target is `nfs://192.168.0.180:/mnt/5_hdd/backup/longhorn-pvc` — **DooPlex
|
||||
writing to itself over NFS**, landing on `sda1`.
|
||||
|
||||
The felhom *product* has a real off-site chain (Hetzner Storage Box + ep0/PBS-DR) for customer boxes.
|
||||
**DooPlex, which runs the hub that manages that chain, has none of it.**
|
||||
|
||||
---
|
||||
|
||||
## 5. The restore probe
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Repo | `/mnt/5_hdd/backup/data/restic-repo` |
|
||||
| Snapshot | `b587f775` (2026-08-06 10:51:17) |
|
||||
| File | `.claude-memory/hetzner-storagebox-api.md` (3.812 KiB) |
|
||||
| Restored sha256 | `5b4d66b0341fe38c21b3a77f2f0af59ad4389fc21a8ce941c729fb807a3dd953` |
|
||||
| Live sha256 | `5b4d66b0341fe38c21b3a77f2f0af59ad4389fc21a8ce941c729fb807a3dd953` |
|
||||
| **Result** | **MATCH — a file came back, byte for byte** |
|
||||
|
||||
Scratch directory `/tmp/recon-restore-20260806-121056` **deleted**.
|
||||
|
||||
A first attempt used `MEMORY.md` and the hashes **differed** — correctly: the restored copy was
|
||||
17,977 bytes, the file's exact size *at snapshot time*, and the live file had since been edited to
|
||||
18,078. That is the snapshot behaving as a point-in-time copy, not a fault, and the probe was redone
|
||||
against a file this session never touched.
|
||||
|
||||
---
|
||||
|
||||
## 6. The password question
|
||||
|
||||
**Measured on the box:** `/etc/backup/restic-password` — 45 bytes, `0600 root:root`, on **`/dev/sdb1`**
|
||||
(the root SSD, *not* the disk holding the backups). It is **not a source path in any set**. The same
|
||||
file is the passphrase for the GPG-encrypted secrets exports (`--passphrase-file "${RESTIC_PASSWORD_FILE}"`),
|
||||
so one secret opens both the restic repos and the secrets archive.
|
||||
|
||||
**Off the box:** the operator confirms **an offline copy exists out of band.** A recon on the machine
|
||||
cannot observe that, and without it this section would have concluded that an `sdb1` loss makes every
|
||||
repo permanently unopenable — which would have been wrong. The failure matrix above is written on the
|
||||
corrected basis.
|
||||
|
||||
**The residual, stated narrowly and without alarm:** the offline copy is *operator-held*, not
|
||||
*system-held*. The felhom product solves the equivalent customer problem with hub-vaulted escrow and a
|
||||
zero-knowledge recovery code; the operator case has no equivalent mechanism, so recovery time in the
|
||||
"board dies" and "flat burns" rows depends on that copy being **current** and **findable by someone
|
||||
other than the operator**. Worth confirming both; not worth engineering around.
|
||||
|
||||
---
|
||||
|
||||
## 7. Coverage gaps
|
||||
|
||||
| Item | Status | Evidence |
|
||||
|---|---|---|
|
||||
| **Gitea repositories** | **covered** (twice) | `gitea-data` is a 50 Gi Longhorn PVC → backupstore; plus `backup-manifests.sh` mirrors repos via the API to `backup/homelab-manifests/` |
|
||||
| **Container registry** | **covered** | part of `gitea-data` (Gitea's built-in registry) |
|
||||
| **Gitea database** | **covered** | CNPG → 6-hourly dumps → `postgresql` restic repo |
|
||||
| **k3s state** | **covered, and the only verified set** | `db`, `token`, `node-token`, `cred`, `tls` + `/etc/rancher/k3s`, `check --read-data-subset=5%` each run |
|
||||
| **Longhorn volumes** | **covered, thinly** | 77 volumes, but **`retain=1`** — one generation per volume. A corruption noticed after the next run has no earlier copy |
|
||||
| **Hub database** | **covered** | `hub-data` (1 Gi) is a Longhorn PVC |
|
||||
| **TLS material / secrets** | **partially** | exported and GPG-encrypted nightly, 30-day retention, on `sda1` only, **never entered into any restic repo** — `backup-secrets.sh` contains no `restic` call at all |
|
||||
| **`/opt/backup` itself** | **NOT covered** | on `sdb1`, in no set. The scripts that perform every backup are backed up by nothing (R-231) |
|
||||
| **`~/.claude/settings.json`** | **NOT covered** | on `sdb1`, in no set. Now reconstructible via `install_workspace.py` |
|
||||
| **ISO build inputs** | **NOT covered** | `/mnt/5_hdd/felhom.eu/build`, on `sda1`, in no set |
|
||||
| **Local git clones** | **NOT covered — and circular** | `/mnt/5_hdd/felhom.eu/git` on `sda1`, in no set. Recoverable from Gitea — **which runs on this machine.** If DooPlex is gone, both copies are gone together |
|
||||
| **`.claude-memory`** | covered, **same-disk only** | §2 |
|
||||
|
||||
---
|
||||
|
||||
## 8. Observations — noticed, not acted on
|
||||
|
||||
1. **`/opt/backup/docs/` is EMPTY.** The systemd unit advertises
|
||||
`Documentation=file:///opt/backup/docs/BACKUP-RESTORE.md` and that file does not exist. The
|
||||
restore procedure you would reach for while the machine is on fire is absent, and the unit points
|
||||
at it anyway.
|
||||
2. **`notify_failure` is a no-op.** `NOTIFY_ON_FAILURE="true"` but `NOTIFY_WEBHOOK_URL` is commented
|
||||
out, and the function requires it to be non-empty. **A failed backup notifies nobody.** The five
|
||||
sets have succeeded 14/14, so this has never mattered yet — which is exactly when it is cheapest
|
||||
to fix.
|
||||
3. **`secrets/restic-repo` exists and has never held a snapshot.** A directory that looks like a
|
||||
backup and is not one. `backup-secrets.sh` never calls restic; its only match for the word is a
|
||||
comment about the password file.
|
||||
4. **A stale lock is sitting in the `data` repo, and it is MINE.** `pid 3524524`, non-exclusive,
|
||||
created 11:20:26, process gone — left by a `timeout`-wrapped `restic ls` during the previous
|
||||
task's verification. **I am fenced from clearing it (§4 of the task) and have not.** Its impact is
|
||||
concrete: `restic backup` needs only a non-exclusive lock and will succeed, but `restic check`
|
||||
already refused (§ the check run below), and `apply_retention` → `forget --prune` needs an
|
||||
**exclusive** lock. With `set -euo pipefail` and the `ERR` trap, tonight's User Data set would
|
||||
therefore fail — and per observation 2, silently. One command clears it:
|
||||
`sudo RESTIC_PASSWORD_FILE=/etc/backup/restic-password restic -r /mnt/5_hdd/backup/data/restic-repo unlock`
|
||||
5. **Integrity checks:** `k3s` and `postgresql` pass `restic check` cleanly; `secrets` is empty so
|
||||
passes trivially; **`data` could not be checked** because of observation 4. Only `k3s` is verified
|
||||
on a schedule — the others have never been verified except by hand.
|
||||
6. **Two dormant destination directories** hold 25 GB (`demo-felhom`, last written 2026-07-10) and
|
||||
652 KB (`sparkyfitness`, 2026-06-24). Nothing writes them; nothing prunes them.
|
||||
7. **The postgres dumps are produced by a k8s CronJob** (`database-system/postgresql-backup`, every
|
||||
6 h) that the backup service does not control. `backup-postgresql.sh` refuses only if the
|
||||
directory is *missing* — if the CronJob stopped, the set would keep succeeding over stale dumps.
|
||||
Currently fresh: newest dump `20260806-100000` at 12:00 today.
|
||||
8. **`restic check --read-data-subset` was not run** beyond what `backup-k3s.sh` does itself, per the
|
||||
task's fence. No repository has ever had its data blocks read back in full.
|
||||
|
||||
---
|
||||
|
||||
## 9. Recommendations, ranked — recommend only, nothing implemented
|
||||
|
||||
| # | Recommendation | Cost | Failure it removes |
|
||||
|---|---|---|---|
|
||||
| 1 | **Clear the stale lock in the `data` repo** (command in §8.4) | one command | tonight's User Data set failing, silently |
|
||||
| 2 | **Give `notify_failure` somewhere to go** — the project already has a working transactional mail path (Resend, used by CI and the hub) | ~10 lines | a backup failing and nobody learning it. This is the cheapest item here and the one that makes every other failure visible |
|
||||
| 3 | **Get one copy off the box.** Even one set — k3s + secrets + postgres dumps is ~1.2 GB — to the Storage Box or ep0 | small; the credentials and the tooling already exist | the "flat burns" row, which is currently total loss |
|
||||
| 4 | **Make `/mnt/5_hdd/backup` non-trivially destroyable** — restic `--append-only` via a REST/rclone-served repo, or a second target the writer cannot reach | moderate | the ransomware / bad-script row |
|
||||
| 5 | **Move the postgres dump directory out of the backup tree**, and add `.claude-memory` to a cross-disk destination | small | the two same-disk cases in §2 |
|
||||
| 6 | **Raise Longhorn `retain` above 1** | disk only, and there is 6.6 TB free | a corruption noticed one day late having no earlier copy |
|
||||
| 7 | **Write `/opt/backup/docs/BACKUP-RESTORE.md`**, or drop the `Documentation=` line | an hour | reaching for a restore procedure that is not there |
|
||||
| 8 | **Do one full-set restore drill** and record it | half a day | the entire "Ever demonstrated?" column being empty |
|
||||
| 9 | **Put the secrets exports into their empty restic repo**, or delete the repo so it stops looking like a backup | small | a directory that reads as protection and is not |
|
||||
|
||||
---
|
||||
|
||||
## 10. The one-sentence answer
|
||||
|
||||
**The backup runs reliably, restores a file correctly, and would survive any single disk failing —
|
||||
but every copy it makes lives in one box, and if that box is gone, so is all of it.**
|
||||
@@ -55,6 +55,44 @@ v0.93.0 made the key survive. v0.94.0/v0.125.0/v0.195.0 make it come back. v0.19
|
||||
the snapshot and the drill got it out again. **v0.198.0/v0.95.0 remove three of the four crutches the
|
||||
drill needed — the fourth is R-193, and until it goes the recovery is still operator-assisted.**
|
||||
|
||||
## R-201 — THE RE-WALK, 2026-08-06 (attended)
|
||||
|
||||
**The question was asked a second time, on the fixed build, on a brand-new appliance built from the
|
||||
published ISO. The answer is still no — but it is a nearer no.**
|
||||
|
||||
| half | verdict |
|
||||
|---|---|
|
||||
| **the data** | **PASS** — all three sentinels byte-identical, including a 12 MB binary and an accented Hungarian filename **whose NAME BYTES are also byte-identical** (verified as hex, not as rendered text). Restored in **23 s** out of the pre-destruction snapshot `a7bc23bd`, through the customer's own two-step full-restore flow |
|
||||
| **the journey** | **FAIL — two dead ends**, against Phase 1's four. One needed a command line **inside the guest**, one a **Proxmox-host** action |
|
||||
|
||||
**RTO: the unaided figure is STILL UNDEFINED**, because the unaided journey still does not complete.
|
||||
Attended: login 11:42:22 → key placed **+45 s** → tier up **+24 m 12 s** (after intervention 1) →
|
||||
all three sentinels restored and verified **+30 m 13 s** (after intervention 2). **The 30 m figure must
|
||||
not be quoted as the customer number.** The only segment that reflects the product working alone is
|
||||
**23 seconds** to pull 12.8 MB back once everything was in place.
|
||||
|
||||
**The two dead ends:** **R-218's consume half** (row corrected above) and **R-220** (drives
|
||||
unenrollable after a rebuild — reproduced and red-proved again; without it no app can be redeployed,
|
||||
and without a redeployed app the restore page is empty, which is R-213's territory and follows from
|
||||
R-220 rather than being separate).
|
||||
|
||||
**What PASSED and is worth keeping:** the recovery screen **appeared without being sought**
|
||||
(`/` → `/launcher` → `/recovery`); it answered all three of its questions and its seal date matched the
|
||||
hub's `created_at` exactly; the emailed reset code worked **first try**; the unlock took **1.528 s** —
|
||||
a real unseal — and placed the key; and **R-225's fix was seen working in the wild** (the store read
|
||||
„a pillanatképek száma még ismeretlen" rather than a false zero).
|
||||
|
||||
**R-216 part 4 reproduced live:** the reinstall **downgraded the agent 0.126.0 → 0.125.0**, back to the
|
||||
vouched version — an operator's hand-fix undone by the very event that makes recovery necessary.
|
||||
|
||||
⚠ **THE DELIVERY GAP, and it is owed.** A fresh install landed on controller **0.201.0** / agent
|
||||
**0.125.0** — the vouched versions, **neither carrying the fixes**. They were installed **by hand**.
|
||||
Fleet delivery needs a golden carrying 0.202.0 **and** a vouched agent 0.126.0. **Nothing was vouched;
|
||||
that is the operator's act.** **This re-walk proves the JOURNEY on the fixed build; it does NOT prove a
|
||||
customer would receive that build.**
|
||||
|
||||
Evidence: `tests/rewalk-r201-2026-08-06/journal.md`.
|
||||
|
||||
## CAMPAIGN 11 — the recovery journey, 2026-08-05
|
||||
|
||||
**The whole journey was walked end to end for the first time, on a throwaway appliance built from the
|
||||
@@ -73,7 +111,7 @@ and `journal-phase24.md` (Phases 2/4). Campaign document:
|
||||
| ID | What | State |
|
||||
|---|---|---|
|
||||
| **R-216** | **A correct recovery code was reported to the customer as wrong.** The unseal needs agent v0.125.0; on an older agent the route 404s and the unlock was attempted anyway, producing *„…nem fogadtuk el. Ellenőrizd, hogy mind a tíz szót pontosan…"* in 0.134 s. **The default state** — the Day-0 manifest vouches 0.120.0, and a reinstall actively DOWNGRADES a hand-fixed box back to it. The hub's own guard could not catch it: `ResolveManagedFloor` compared against the GOLDEN's MinAgent while serving a FLOOR that pointed elsewhere (the 8th entry in `CLAUDE.md`'s comment-vs-code table, and the first where the false invariant was a guard) | **SHIPPED** (controller v0.201.0 + hub v0.97.0/0.97.1) — **but see R-223**: the feature does not work on a NEW box until the manifest vouches agent 0.125.0. Until then such a box is correctly HELD, not lied to |
|
||||
| **R-218** | **Succeeding at recovery stopped the box asking for what it still needed.** `needsOffsiteCredential` short-circuited on a repository password existing — and installing one is the recovery screen's whole job. 32 s after the hub re-staged the credential, the customer's success switched the mechanism off; the hub held an unconsumed credential the box had no reason to collect, and nothing ever asked again | **SHIPPED** (controller v0.201.0) — the declaration now stops when the TIER WORKS, not when a key exists |
|
||||
| **R-218** | **Succeeding at recovery stopped the box asking for what it still needed.** `needsOffsiteCredential` short-circuited on a repository password existing — and installing one is the recovery screen's whole job. 32 s after the hub re-staged the credential, the customer's success switched the mechanism off; the hub held an unconsumed credential the box had no reason to collect, and nothing ever asked again | **REOPENED 2026-08-06 — the fix covers the DECLARATION half only, and this row over-claimed it.** Measured on the R-201 re-walk: the box declared, and **`offsiteheal` re-staged the secret at 11:44:57** saying *"the box re-consumes on its next cycle"* — **the next cycle came and went** (`host-report` 11:55:46, `Received report` 11:55:54, a full cycle **with a positive control that it ran**) **and the credential was still not consumed.** 23 minutes after the re-stage the box's last off-site-apply attempt was still the pre-re-stage one. A census of the customer-reachable actions on `/backups/remote` (`config`, `reset`, `run`, `toggle`) found **none that fetches a staged credential**, and the only lever is `systemctl restart felhom-controller-bootstrap.service` **inside the guest** — which worked in **18 s** (Campaign 11 measured 17), confirming nothing was wrong with the credential, the target or the key: **the only thing missing is anything at all to trigger a retry.** **This is the FIRST of the two dead ends that keep the recovery journey failing** |
|
||||
| **R-219** | **The listing the screen promises could never render on the shape it exists for.** Listing needs a target; a target cannot exist without a repository password; shape (a) is defined by having none. And placing the key flipped the offer false, so the unlock response was the only chance and was guaranteed not to contain it | **SHIPPED** (controller v0.201.0) — the unlock now places the key, brings the tier up, then lists |
|
||||
| **R-217** | **An unreadable store reported as "opened, with unattributable content".** The failure path passed `backup.OffsiteInventory{}`, whose `Empty=false` the template read as `InvUntagged`. The field built to prevent exactly this names the hazard in its own doc comment | **SHIPPED** (controller v0.201.0) — opened / empty / unreadable are three distinguishable states |
|
||||
| **R-222** | **Reaching for a RETAINED earlier package read as a wrong code.** The engine is right (it fails closed against the current package); the message was not. Proven live: the correct code for the orphaned history got *„check your ten words"* | **SHIPPED** (controller v0.201.0 + hub v0.97.0) — the ACK carries `superseded_present`/`superseded_at` and the screen names the situation. **It states what the hub knows and promises nothing** — the read path is still unbuilt (R-199's inventory) |
|
||||
@@ -115,6 +153,7 @@ the fault was real. Full observables: `tests/campaign11-evidence-2026-08-05/jour
|
||||
|---|---|---|
|
||||
| **R-229** | **The instruction-file rightsizing landed for `felhom-controller` and the workspace root; three pieces were deliberately deferred.** Done 2026-08-06: controller split into a 92-effective-line core plus four `paths:`-scoped `.claude/rules/*.md`; workspace root 208→142 effective lines with its versioned copy kept byte-identical; surgical corrections to `felhom-agent` and `felhom.eu` (expired TEMPORARY block, every version literal, the Legacy-Windows copies, the duplicated health-check rule); five contradictions resolved — including a drill-VM claim **measured live** (`qm list` on demo-hp shows VM 300 `drill-r50`; `felhom-agent` was right, `felhom-controller` was wrong); new shared `felhom.eu/scripts/instructions_gate.py` registered in `controller_gates.py` and `agent_gates.py`, 20 fixture tests + red-proof. **Leg (a) CLOSED 2026-08-06 (part 2):** `felhom.eu/CLAUDE.md` **227 → 115 effective lines**, split into a core plus `.claude/rules/{hub,website,manifests,docs}.md`; `instructions_gate` **registered in `scripts/repo_gates.py`** (six gates, all OK) in the required order — trim first, register second, because a registered-but-failing gate refuses every push. Scoping proven from the `InstructionsLoaded` hook log in two fresh sessions, not from frontmatter. **Still deferred:** (b) **CLOSED 2026-08-06 (close-out)** — `felhom-agent/CLAUDE.md` **175 → 99 effective lines** (measured 175, not 173: the CI correction added two), split into a core plus `.claude/rules/{proxmox,localapi,backup,storage}.md` beside the existing `health-checks.md`. The release section now points at the `felhom-build-deploy` skill instead of restating a table that drifts from the script. **Every `CLAUDE.md` in the workspace is now ≤120 effective lines except the workspace root at 142, which is deliberate — it is the only file re-injected after `/compact`.** (c) **CLOSED 2026-08-06 (part 2)** — all 44 orphans resolved with **zero deletions** (file count 158 before and after): 4 durable `reference`-type files indexed, 40 dated episode records moved to `.claude-memory/archive/`. `MEMORY.md` 145 → **150 lines / 17,977 bytes**, and `instructions_gate` check 6 now watches it (over-limit FAILS, orphan WARNS, absent store PASSES *printing its reason*). (d) **The spec-as-failing-test pilot** — moved to R-230. Full accounting: `audits/LEDGER-instruction-trim-2026-08-06.md` + `audits/LEDGER-instruction-trim-part2-2026-08-06.md` | **READY** — owner Viktor |
|
||||
| **R-230** | **Three instruction/memory follow-ups deliberately left by the part-2 session (2026-08-06), each needing a decision rather than an implementation.** (a) **A ruling is owed on auto-written staleness.** The hand-written `CLAUDE.md` files are now clean of version literals and expired blocks — the gate enforces it — but `MEMORY.md`, which Claude writes and which is the LARGER half of what loads (8.4k tokens vs the root file's 6.6k), carries **21 lines with component version literals**, **5 with bare host addresses**, and an entry still reading *"demo boxes REMOTE till ~08-02"* — the same expired-TEMPORARY class the gate was built to kill, now surviving in the one file the gate's content rules do not cover. **Partly actioned 2026-08-06 (close-out), and the ruling is STILL OWED:** the **three statements that were actively false** were corrected — `R-193 decision open` (closed 2026-08-05), `demo boxes REMOTE till ~08-02` (the box answers on the home LAN), `OPEN R-25b` (shipped 2026-07-21) — and gate check 6 now **WARNs** on version literals, host addresses, expired statements and stale-open citations in the index. WARN, never FAIL: Claude writes that file between sessions, so a hard failure would refuse a human's push over a line no human typed, and the warning is read by the model that will next edit it. **The remaining 32 version literals and 4 host addresses were deliberately left** for that loop. What is still owed is the bulk-correction ruling. **Correcting the premise:** the earlier report's "three expired statements" were all FALSE POSITIVES — each matched an ISO date inside a markdown link target, i.e. a filename — while the one real expired claim carried no ISO date at all. (b) **CLOSED 2026-08-06 (close-out)** — the workspace-root `CLAUDE.md` **is now a relative symlink** to the versioned copy, so the divergence class is gone rather than policed. Check 5 learned two shapes: for a link it asserts the target resolves to a real file (**a dangling link is worse than a diverged copy — the instructions load NOTHING and there is no content left to notice is wrong**), for two files byte-identity as before, so a clone elsewhere is unaffected. **Proven, not assumed:** three fresh sessions logged `session_start` for the link path, and a fourth **with no tools at all** quoted standing rule 1 verbatim — the content reaches the model, not just the path. (c) **The spec-as-failing-test pilot**, approved in principle and not started (was R-229(d)). | **READY** — owner Viktor |
|
||||
| **R-232** | **DooPlex's backup makes every copy inside the same box — and nothing tells anyone when it fails.** Surveyed read-only 2026-08-06 (`audits/RECON-dooplex-backup-2026-08-06.md`). **What works:** five sets, 14/14 successful runs in 14 days; a file was restored from the `data` repo and matched the live original **byte for byte**; every set except two is cross-disk; k3s is integrity-checked on every run. **What the matrix exposes, ranked:** (a) **`notify_failure` is a no-op** — `NOTIFY_ON_FAILURE=true` but `NOTIFY_WEBHOOK_URL` is commented out, so a failed backup notifies **nobody**; the project already has a working Resend path that CI uses. Cheapest item, and it makes every other failure visible. (b) **Nothing leaves the box** — no rclone, no remote repo, no off-site target anywhere; Longhorn's target is `nfs://192.168.0.180:` pointing at DooPlex itself, and the only outbound-looking cron pulls *inbound* from Hetzner for a different project. The machine that runs the hub managing the customers' off-site chain has no off-site copy of its own. (c) **The backup tree is a single writable path** and the restic repos are not append-only — one bad script or ransomware destroys every copy at once. (d) **Two same-disk sets**: `.claude-memory` and the PostgreSQL dumps, whose source directory sits *inside* the backup tree. (e) **Longhorn `retain=1`** — one generation per volume, so a corruption noticed a day late has no earlier copy. (f) **`/opt/backup/docs/BACKUP-RESTORE.md` does not exist** though the systemd unit advertises it. (g) **`secrets/restic-repo` has never held a snapshot** — `backup-secrets.sh` contains no `restic` call; the secrets are GPG files on `sda1` only. (h) **No restore has ever been run** beyond today's single-file probe — the matrix's "ever demonstrated?" column is otherwise entirely empty. **Not a finding:** the restic passphrase. The on-box copy is on `sdb1`, a different disk from the backups, and the **operator holds an offline copy out of band** — so a disk loss is recoverable. The narrow residual is that it is operator-held rather than system-held, unlike the customer case's hub-vaulted escrow, so it should be confirmed current and findable by someone else. **Nothing was changed by the recon.** | **READY** — owner Viktor |
|
||||
| **R-231** | **`/opt/backup/scripts/` on DooPlex is unversioned host state** — found 2026-08-06 while adding the auto-memory store to the backup set. No repository tracks the scripts that protect the recovery chain, so the edit made that day (`CLAUDE_MEMORY_DIR` in `backup-config.sh`, multi-path restic call in `backup-data.sh`) exists only on the box. This is the same class the part-2 session was closing, found inside the fix for it; the change is transcribed in `felhom.eu/workspace/README.md` so it is at least *recorded*. **Two related facts, both understating current safety:** the backup destination (`/mnt/5_hdd/backup`) is on the **same physical disk** as the workspace it protects, and the DooPlex backup set has **no off-site leg** (`sync-hetzner-backups.sh` is jarrs.eu and pulls *from* Hetzner *to* DooPlex). Bringing a root-owned production backup script under version control, and deciding what installs it, is its own scoped change. | **READY** — owner Viktor |
|
||||
|
||||
**Recorded against existing rows by Phase 2:**
|
||||
|
||||
Reference in New Issue
Block a user