R-178 CLOSED: both demo boxes reinstalled from the merged golden and proven
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
Two boxes, two DIFFERENT supply paths, so the session proved the disk shape and the delivery route rather than one of them twice. demo-hp (layout proof, --golden <local volid>): mp0 at /var/lib/felhom, backup=1, 70G, no mp1; /var/lib/docker and /mnt/sys_drive both real mounts of its subdirectories via fstab; one df figure and one device id (64519) on all three paths; reboots 3/3 with the binds surviving each. demo-felhom (pipeline proof, --force-gitea-golden): fetch_verify succeeding against the vouched manifest for BOTH artifacts -- 'verified sha256 54e2a4c431daf580... matches the hub manifest' for the golden, a7763d31... for the agent. 250G single volume, grep -c '^mp1:' = 0, reboots 3/3. Journey proven on both, endpoint-level: claim -> deploy -> back up -> restore, with a planted marker returning byte-identical on each box. Ceiling measured gone: 65 GiB and 233 GiB available to a recovery unit, against 19 and 45. R-165 -> IMPLEMENTED, not PROVEN-LIVE, on the operator's ruling. B2, which that row records as the bulkhead's replacement, fired live for the first time and does refuse per app, delete nothing and alert -- but it is checked only in captureAllRecoveryUnits while runVolumeDumps writes the bulk unguarded, and its 'the previous unit is untouched' claim was measured false (182,272 B dump replaced by 2,147,666,432 B under a manifest still dated 06:34:26). -> R-181. New: R-179 (uninstall leaves NAS network-storage units), R-180 (--archive-storage not cross-checked against the ACL grant; 403 at step 8/8 after root@pam is rotated), R-181. Third instance of R-115 recorded (agent 0.120.0 unpublished). No code written, no version bumps -- this was a runbook.
This commit is contained in:
@@ -1,83 +1,342 @@
|
||||
# REPORT — R-165: the `mp1` merge, built and proven at the bake (2026-08-03)
|
||||
# REPORT — R-178: both demo boxes reinstalled from the merged golden and proven (2026-08-03)
|
||||
|
||||
**Overwritten** per the standing rule. The prior contents (hub v0.89.0 / R-167, 2026-08-02) have their
|
||||
durable record in `hub/CHANGELOG.md` and `CONTEXT.md` S-11/S-12.
|
||||
**Overwritten** per the standing rule. The prior contents (R-165, the bake, 2026-08-03) have their
|
||||
durable record in `documentation/backlog/OPEN-ITEMS.md` R-165 and the per-repo CHANGELOGs.
|
||||
|
||||
**Companion:** `felhom-agent/REPORT.md` holds the full session detail — probes, variant evidence, the
|
||||
bake transcript, red-proofs and teardown. **This file covers what changed in THIS repo, and the CI
|
||||
verification for all three.**
|
||||
|
||||
> **Scope, stated first.** The merge is **built and green through Phase 5**. **Phases 6–7 — reinstalling
|
||||
> the two demo boxes from the merged golden and proving one end to end — were NOT done, and nothing
|
||||
> was wiped.** The golden is therefore deliberately **unvouched**. Remaining work: **R-178**.
|
||||
**Runbook, not a task.** No repo got a version bump and nothing was built. One artifact was
|
||||
**published** (agent 0.120.0) on an explicit operator ruling — see §2. All code findings are filed as
|
||||
register rows, not commits, per the runbook's §7.
|
||||
|
||||
---
|
||||
|
||||
## 1. What changed in `felhom.eu`
|
||||
## 1. Preconditions P1–P6, each as measured
|
||||
|
||||
**No hub change; no hub version bump.** This repo changed in two places, one of them forced.
|
||||
| # | Precondition | Measurement |
|
||||
|---|---|---|
|
||||
| **P1** | Golden sha256 matches R-178's record | **PASS** — computed from the artifact itself on demo-hp: `sha256sum` → `54e2a4c431daf580d2807b82d810be36a7c6a9f697b27094dc2912d26a43b3e0`, 649,547,835 B. Matches R-178's recorded prefix and the hub manifest's stored value exactly. Re-verified after copying to `local`: identical |
|
||||
| **P2** | A rollback golden still exists | **PASS, three depths.** Split-layout golden **0.188.0** fetchable from Gitea (`HTTP 200`, 649,310,288 B). A **local** split-layout golden sat on each box — `local:backup/vzdump-lxc-9100-2026_07_21-18_24_49.tar.zst` (demo-hp), `…2026_07_20-17_50_57…` (demo-felhom). Best: **full guest vzdumps**, three per box, newest `vzdump-lxc-9201-2026_08_03-04_54_28.tar.zst` (2,351,557,870 B, demo-hp) and `…2026_08_03-04_44_50…` (6,323,506,165 B, demo-felhom), both ~3 h old — so each box could be put back exactly as it was |
|
||||
| **P3** | Neither demo box holds anything wanted | **PASS, stated explicitly.** demo-hp: paperless-ngx (webserver/postgres/redis) + filebrowser + samba shares. demo-felhom: immich (×4), docmost (×3), calibre-web, bookstack (×2) + filebrowser + samba. Both are demo customers (`Demo HP`, `Demo Ügyfél`); no real customer data. **All of it was destroyed by the wipes and none of it was restored** — that was the point, and the operator confirmed each wipe |
|
||||
| **P4** | The colleague's box untouched | **PASS** — `peti-felhom` is Tier 2 and out of scope. No command was sent to it; it appears in the hub customer list as `DOWN`, exactly as before |
|
||||
| **P5** | Both hosts' agent is v0.120.0 | **PASS** — `felhom-agent 0.120.0` on both, before and after |
|
||||
| **P6** | The hub's host register, BEFORE | **Captured**: `demo-felhom-8363b5 / Demo Ügyfél / 0.120.0 / ONLINE / 1-of-2 guests`; `demo-hp-bb76ea / Demo HP / 0.120.0 / ONLINE / 1-of-2`; `drill-r50-0a4f9a / drill-r50 / 0.113.0 / DOWN` (the fixture, untouched throughout) |
|
||||
|
||||
**`scripts/felhom-host-install.sh` — forced by a census, not planned.** `step_grows` computed **two**
|
||||
volume sizes and the install call passed both, so it had to change with the agent or every install
|
||||
would have provisioned a half-sized box. It now computes ONE total, summing the old 80/20 split
|
||||
(`226` = the previous `184 + 42`), so **a standard appliance keeps exactly the 250 G it had** — it is
|
||||
simply no longer split by a wall. **The size still comes from the physical disk**: `step_grows`
|
||||
already read the thin pool's real free space, and the merge only collapsed its two outputs into one.
|
||||
That is the answer to the task's §8.2 — no row needed filing. `--sysdata-grow` is deprecated but still
|
||||
honoured, because the agent **folds** a hand-passed value in rather than dropping it.
|
||||
---
|
||||
|
||||
**Documentation**, per the coupling rule — see §3.
|
||||
## 2. Two findings that contradicted the runbook's premise, both surfaced before any wipe
|
||||
|
||||
## 2. The bake, as this repo's audit trail records it
|
||||
**(a) The golden was ALREADY vouched.** Hub log, a positive observable:
|
||||
`2026/08/03 07:23:26 [INFO] Artifact manifest set: agent=0.119.0 golden=0.192.0 min_agent="0.113.0"
|
||||
wrapper_sha=true` — roughly ten minutes before this session's first hub read, and this session had
|
||||
POSTed nothing. The configuration page confirmed `Artifacts.GoldenVersion=0.192.0`,
|
||||
`GoldenSHA256=54e2a4c4…3b3e0`. So §7's prove-then-vouch order was already spent, and Phase A's stated
|
||||
safety ("nothing is official yet, so a failure reaches nobody") was void. **Operator ruling: accept it
|
||||
and proceed**, keeping the two different supply paths. Recorded on `CONTEXT.md` S-14, now marked SPENT
|
||||
with the reason — the rule lived only in prose and nothing enforced it.
|
||||
|
||||
`documentation/audits/SPIKE-r165-phase0-2026-08-03.md` (new) holds P1, P2 and P3 with method,
|
||||
measurement and ruling, including the teardown of every probe artefact at all three layers. The
|
||||
headline measurements:
|
||||
**(b) Agent v0.120.0 had never been published — the vouched agent was 0.119.0.**
|
||||
`GET …/generic/felhom-agent/0.120.0/felhom-agent` → **HTTP 404** (0.119.0 → 200). Installer step 5's
|
||||
idempotent skip requires `installed == vouched` exactly, so a documented-path reinstall would have
|
||||
**downgraded both boxes** to the pre-merge 0.119.0 — and would have *succeeded* while doing it, since
|
||||
the current `step_grows` sets `SYSDATA_GROW=0` and 0.119.0's `mp1` resize (`bringup.go` 4c, fatal on
|
||||
error) therefore never fires. The session would have proven a stack nobody ships. **Operator ruling:
|
||||
publish and vouch first.** `scripts/publish-agent.sh 0.120.0` from a clean tree at `4bb84fc3`
|
||||
(`git status --porcelain` empty, `HEAD == origin/main`): upload HTTP 201, **round-trip GET verified**,
|
||||
`AGENT_SHA256=a7763d31b55b5ce75457b4dba7b06aa300325811834b0be78af4587b47110b9d`. Vouched via
|
||||
`POST /configuration/artifacts` → `303 …flash=artifacts_set`, hub log
|
||||
`Artifact manifest set: agent=0.120.0 golden=0.192.0`; the hub resolved the sha authoritatively from
|
||||
Gitea rather than trusting the submitted value. Filed as the **third instance of R-115**, not a new
|
||||
ID — R-115 is that row, and it has been WAITING-ON-OPERATOR since 2026-07-29.
|
||||
|
||||
- **P1 PASS** — a real pre-merge archive restore-tests clean, `mount_parity: ok`, 84 s, `mountParity`
|
||||
untouched.
|
||||
- **P2** — all three variants boot and reboot 3/3; they are separated by **scoping**, not mechanics.
|
||||
The container's view of `/mnt` is 8.0K under V-a and V-c, and **17.9M — Docker's entire data-root —
|
||||
under V-b**. The operator chose **V-c**.
|
||||
- **P3** — the four retargeted golden assertions, run against a deliberately wrong shape: **8/8**.
|
||||
---
|
||||
|
||||
## 3. Documentation coupling
|
||||
## 3. demo-hp — the layout proof
|
||||
|
||||
| File | Change |
|
||||
**Install path.** Uninstall: `./felhom-host-install.sh --uninstall --vmid 9201` (typed-vmid
|
||||
confirmation supplied over a pty). Install:
|
||||
|
||||
```
|
||||
./felhom-host-install.sh --customer-id demo-hp --mode appliance --vmid 9201 \
|
||||
--cores 7 --memory 26906 \
|
||||
--golden local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst \
|
||||
--passphrase-file /root/.pp-demo-hp
|
||||
```
|
||||
|
||||
Script fetched from `https://felhom.eu/scripts/felhom-host-install.sh`, **v1.22.0**, sha256
|
||||
`ed02acb2da46c8d2b5c486ce99d5b9a2747e8786c6eb03652cf755ed1abdd9f4` — byte-identical to the repo copy
|
||||
that was read. The passphrase went file→file into a 0600 file and never onto a command line. A
|
||||
`--dry-run` preceded the real run and resolved `manifest: agent v0.120.0 (sha a7763d31…), golden
|
||||
v0.192.0` with `grows: rootfs +0G (->32G), data +46G (->70G, ONE volume)`.
|
||||
|
||||
**One deviation, mine, and it cost a restart.** The first attempt staged the golden on
|
||||
`felhom-backup` with `--archive-storage felhom-backup`. Pre-flight passed; **step 8/8** failed:
|
||||
`HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.AllocateSpace)` —
|
||||
the default pool-scoped ACL grants `local local-lvm felhom-pbs` only. Fixed by copying the golden to
|
||||
`local` (sha re-verified after the copy) and `--resume`. Filed as **R-180**: the condition is
|
||||
statically checkable in pre-flight, and the failure lands *after* step 4b has rotated and vaulted
|
||||
root@pam.
|
||||
|
||||
**Layout evidence.**
|
||||
|
||||
```
|
||||
mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G # and NO mp1 line
|
||||
rootfs: local-lvm:vm-9201-disk-0,size=32G
|
||||
|
||||
/var/lib/felhom /dev/mapper/pve-vm--9201--disk--1 ext4 rw,relatime,stripe=16
|
||||
/var/lib/docker /dev/mapper/pve-vm--9201--disk--1[/docker] ext4
|
||||
/mnt/sys_drive /dev/mapper/pve-vm--9201--disk--1[/sys_drive] ext4
|
||||
|
||||
Filesystem Size Used Avail Use% Mounted on
|
||||
/dev/mapper/pve-vm--9201--disk--1 69G 977M 65G 2% /var/lib/felhom
|
||||
/dev/mapper/pve-vm--9201--disk--1 69G 977M 65G 2% /var/lib/docker
|
||||
/dev/mapper/pve-vm--9201--disk--1 69G 977M 65G 2% /mnt/sys_drive
|
||||
|
||||
stat -c %d → 64519 for all three paths
|
||||
/etc/fstab: /var/lib/felhom/docker → /var/lib/docker ; /var/lib/felhom/sys_drive → /mnt/sys_drive
|
||||
both binds writable (touch succeeded on each)
|
||||
```
|
||||
|
||||
`/mnt/sys_drive` shows a second `findmnt` row — it is the controller container's
|
||||
`-v /mnt:/mnt:rslave` propagation (peer group 383 vs the fstab bind's 333), the same shape the split
|
||||
layout had, not a stacked bind.
|
||||
|
||||
**Reboots — each individually, minimum three:**
|
||||
|
||||
| # | started | controller healthy | outcome |
|
||||
|---|---|---|---|
|
||||
| 1 | 08:13:33 | 08:13:55, `Up 12 seconds (healthy)` | one filesystem, 69G/65G on both paths |
|
||||
| 2 | 08:13:59 | 08:14:15, `Up 7 seconds (healthy)` | same |
|
||||
| 3 | 08:14:19 | 08:14:35, `Up 7 seconds (healthy)` | same |
|
||||
|
||||
After all three, `mountpoint -q` returns true for **all three paths** and both binds still resolve to
|
||||
the single volume's subdirectories — which is what the reboots exist to test. `uptime -s` =
|
||||
`2026-08-03 06:14:24 UTC`, matching reboot 3, so these were real reboots.
|
||||
|
||||
**The journey — method stated: endpoint-level, not a browser.** `claude-in-chrome` does not exist on
|
||||
DooPlex; every step below invoked the exact endpoint the dashboard's own JavaScript calls.
|
||||
|
||||
| Leg | Endpoint | Observable |
|
||||
|---|---|---|
|
||||
| Claim | `POST /claim` with the pre-auth HMAC CSRF token (64 chars) **and** its `felhom_claim_csrf` cookie | `302 → /`; gate discriminator flipped `{"error":"dashboard not yet claimed"}` → `{"error":"authentication required"}`. **The code is emailed-only (R-119) — the operator supplied it**, after an operator resend rotated the generation (the previous code had been consumed 12 d earlier) |
|
||||
| Deploy | `POST /api/stacks/<n>/deploy` with `{"values":{…}}`, field values scraped from the deploy form exactly as the form's own `fetch` does | `{"ok":true}`; `privatebin Up (healthy)`, then `opengist Up (healthy)` |
|
||||
| Back up | `POST /api/debug/backup/dbdump` — this runs the **production** `RunDBDumps` path (DB leg → `runVolumeDumps` → `captureAllRecoveryUnits`); the debug route only starts it instead of waiting for 03:30 | `Volume dump: opengist/… → 178.0 KB`, `privatebin/… → 2.5 KB`, `App-data backup completed … 2 volume dump(s) (3.962s)`, `Recovery unit captured for …` ×2 |
|
||||
| **Restore** | `POST /backup/restore` `stack_name=privatebin snapshot_id=primary` | A marker planted in the live volume was **deleted**, then restored: `{"ok":true,"message":"privatebin visszaállítva (primary)."}` in **9.2 s**, and the marker returned with an **identical sha256 `ac1faae6134871d640d5cb1bbc6b5092d1e96b7e46c6c3bcb6e8236cc33ae861`**. App `Up (healthy)` afterwards |
|
||||
|
||||
**Recovery unit path, and it lands on the single volume:**
|
||||
`/mnt/sys_drive/felhom-data/backups/primary/<app>/{manifest.json,compose/,volume-dumps/}`, whose `df`
|
||||
is `/dev/mapper/pve-vm--9201--disk--1` — the merged volume.
|
||||
|
||||
**Note on the first app chosen.** `privatebin`'s catalog entry declares no `backup:` section, so its
|
||||
first per-stack capture produced `"volume_dumps": null` — correct for that declaration, not a defect,
|
||||
but it means a per-stack `POST /stacks/<n>/backup` writes compose+config only; the volume leg lives in
|
||||
the full pass. A second app (`opengist`) was deployed so the run had both a capture and, later, a
|
||||
refusal.
|
||||
|
||||
**The ceiling is gone, measured:** a recovery unit can use **65 GiB** — the whole volume — against the
|
||||
**19 GiB** the pre-wipe `mp1` slice offered (`/dev/mapper/pve-vm--9201--disk--2 20G 95M 19G 1%`).
|
||||
|
||||
---
|
||||
|
||||
## 4. The floor's first live firing — and it does not do what it says
|
||||
|
||||
**Instrument, proven before use.** demo-hp's thin pool is **53.93 GiB** and the auto-sized volume is
|
||||
70 G, so a real fill to 97 % would have exhausted the pool and corrupted every guest on the box,
|
||||
including the protected `drill-r50` fixture. A 5 GiB `fallocate` probe moved guest `df` from
|
||||
`977M used / 65G avail` to `6.0G / 60G` while thin-pool `data_percent` stayed **29.03 → 29.03** —
|
||||
zero blocks allocated — and cleanup returned both to baseline. The floor reads `statfs`, which is
|
||||
exactly what `fallocate` moves, so the condition it guards is genuinely present.
|
||||
|
||||
**Setup.** A **real** 2 GiB file in opengist's data volume (so its capture writes 2 GiB), then
|
||||
`fallocate` to leave `69G / 63G used / 3.0G avail / 96%` — **both floor terms deliberately still
|
||||
clear**, so the run would start. Capture order was established empirically from the previous run's log
|
||||
(opengist first, privatebin second), not assumed.
|
||||
|
||||
**What happened, 06:40:03:**
|
||||
|
||||
```
|
||||
Volume dump: opengist/opengist_opengist_data → 2.0 GB # unguarded
|
||||
Volume dump: privatebin/privatebin_privatebin_data → 2.5 KB
|
||||
App-data backup completed … 2 volume dump(s) (20.737s)
|
||||
[WARN] Recovery unit capture REFUSED for opengist — … 1.0 GB free; the previous unit is untouched and NOTHING was deleted
|
||||
[WARN] Recovery unit capture REFUSED for privatebin — … 1.0 GB free; the previous unit is untouched and NOTHING was deleted
|
||||
[INFO] Event pushed: recovery_unit_capture_failed (error) — … ×2 (HTTP 200)
|
||||
```
|
||||
|
||||
**What holds:** it refuses **per app** rather than aborting the run; **nothing was deleted** (both
|
||||
units present afterwards); and the operator alert reached the hub — `recovery_unit_capture_failed`,
|
||||
severity `error`, accepted `HTTP 200`, twice.
|
||||
|
||||
**What does not hold — measured, not inferred.** The refusal message claims *"the previous unit is
|
||||
untouched"*:
|
||||
|
||||
| unit file | before | after |
|
||||
|---|---|---|
|
||||
| `privatebin/volume-dumps/privatebin_privatebin_data.tar` | `26c546c2…` | **`b538ab89…`** |
|
||||
| `opengist/volume-dumps/opengist_opengist_data.tar` | 182,272 B | **2,147,666,432 B** |
|
||||
|
||||
Both were rewritten by the earlier leg, while each `manifest.json` kept
|
||||
`created_at: 2026-08-03T06:34:26Z` and its `checksums` block covers only the three compose files — so
|
||||
a unit's payload can be swapped under a stale descriptor and nothing inside the unit can detect it.
|
||||
|
||||
**Cause, in the code and not the log.** The floor is consulted in exactly one place —
|
||||
`m.unitFloorBlocked(stack.Name)` at `recovery_unit.go:328`, inside `captureAllRecoveryUnits`, which
|
||||
writes a manifest and a compose copy: a few KB. `runVolumeDumps` (`backup.go:535`) — the leg that
|
||||
writes the bulk, and the leg that consumed the reserve — has **no floor check at all**; its gates are
|
||||
protected-stack, volume-less, disconnected, decommissioned. And it runs first *by design*
|
||||
(`backup.go:483`). **The floor guards the cheap leg and not the leg that fills the volume.**
|
||||
|
||||
This is why R-165 is IMPLEMENTED and not PROVEN-LIVE: that row records B2 as the deliberate
|
||||
replacement for the bulkhead the `mp1` partition provided, and pre-merge the unguarded leg could only
|
||||
fill a dedicated 20 G partition — post-merge it can reach Docker's data-root. Filed as **R-181**.
|
||||
**No code was written**, per the runbook's §7.
|
||||
|
||||
**Cleanup:** fill file and payload removed; `df` back to `1.2G used / 65G avail`; a clean re-run left
|
||||
both units valid (`Volume dump … 178.0 KB` / `2.5 KB`, `App-data backup completed … (2.399s)`).
|
||||
|
||||
---
|
||||
|
||||
## 5. demo-felhom — the pipeline proof
|
||||
|
||||
**Install path — deliberately different, and this is the reason the second box exists.**
|
||||
|
||||
```
|
||||
./felhom-host-install.sh --customer-id demo-felhom --mode appliance --vmid 9201 \
|
||||
--cores 3 --memory 12288 \
|
||||
--force-gitea-golden \
|
||||
--passphrase-file /root/.pp-demo-felhom
|
||||
```
|
||||
|
||||
A copy of golden 0.192.0 already sat on this box's `local` storage — it is where the golden was
|
||||
**baked** at 06:58 (a `.log` beside it), sha `54e2a4c4…`. `--force-gitea-golden` is the documented
|
||||
C.3 customer path and overrides local discovery in **both** pre-flight and step 7, which pre-flight
|
||||
confirmed: `golden: none local — will fetch + verify from Gitea in step 7/8`. The bake artifact was
|
||||
left untouched.
|
||||
|
||||
**`fetch_verify` succeeding against the vouched sha — the observable this box exists to produce:**
|
||||
|
||||
```
|
||||
5/8 fetching agent binary v0.120.0 from Gitea …
|
||||
verified sha256 a7763d31b55b5ce7… matches the hub manifest
|
||||
7/8 fetching golden v0.192.0 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_08_03-09_15_34.tar.zst
|
||||
verified sha256 54e2a4c431daf580… matches the hub manifest
|
||||
golden imported + verified: local:backup/vzdump-lxc-9100-2026_08_03-09_15_34.tar.zst
|
||||
Day-0 provision SUCCESS — vmid=9201 host_id=demo-felhom-8363b5 customer=demo-felhom
|
||||
```
|
||||
|
||||
Both artifacts — agent and golden — were fetched anonymously (the normal customer shape) and
|
||||
sha-verified against the manifest. Controller `0.192.0` healthy.
|
||||
|
||||
**Layout:** `mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=250G`;
|
||||
`pct config 9201 | grep -c '^mp1:'` → **0**. Both binds real mounts
|
||||
(`…disk--1[/docker]`, `…disk--1[/sys_drive]`), `df` one figure — `246G 977M 233G 1%` on all three
|
||||
paths — `stat -c %d` = `64519` on all three, fstab carries both binds.
|
||||
|
||||
**Reboots:** 1 — 09:18:55 → healthy 09:19:09; 2 — 09:19:12 → 09:19:26; 3 — 09:19:30 → 09:19:45. All
|
||||
three paths still mountpoints after each; `uptime -s` = `2026-08-03 07:19:34 UTC`, matching reboot 3.
|
||||
|
||||
**Journey:** claim (`302 → /`, discriminator flipped to `authentication required`; code supplied by
|
||||
the operator after a resend) → deploy `opengist` (`{"ok":true}`, `Up (healthy)`) → capture
|
||||
(`Volume dump: opengist/… → 178.0 KB`, `Recovery unit captured`, unit on
|
||||
`/dev/mapper/pve-vm--9201--disk--1`, marker present inside the tar) → **restore**
|
||||
(`{"ok":true,"message":"opengist visszaállítva (primary)."}` in **9.4 s**, marker back with identical
|
||||
sha256 `bc5507987f3f56dc19a9c24785826c304f1a986998d4aeb0b06d1a810b59e939`, app healthy).
|
||||
|
||||
**Ceiling:** 233 GiB available to a recovery unit, against the 45 GiB the pre-wipe `mp1` offered.
|
||||
|
||||
**Step 8 was NOT repeated here, deliberately** — stating it rather than leaving it ambiguous. The
|
||||
floor fired on demo-hp and R-181 characterises it fully; re-firing would add no information and would
|
||||
mean filling a 246 G volume.
|
||||
|
||||
---
|
||||
|
||||
## 6. Vouching
|
||||
|
||||
Already done before the session (§2a), at **07:23:26 CEST on 2026-08-03**, by an operator action in
|
||||
the hub UI. The session's own manifest write was the **agent** half, at **07:44:31**:
|
||||
`Artifact manifest set: agent=0.120.0 golden=0.192.0 min_agent="0.113.0" wrapper_sha=true`. The
|
||||
manifest afterwards, read back: `agent_sha256=a7763d31b55b5ce7…10b9d`,
|
||||
`golden_sha256=54e2a4c431daf580…43b3e0`, `min_agent=0.113.0`, `wrapper_sha256=104db0a4…` (preserved
|
||||
verbatim). The R-120 gate did not block: golden 0.192.0 equals the newest controller the fleet
|
||||
reports.
|
||||
|
||||
---
|
||||
|
||||
## 7. Teardown — all three layers
|
||||
|
||||
1. **The machine.** No throwaway guest was created this session, so there is none to delete. VM 300
|
||||
`drill-r50` on demo-hp — the protected drift fixture — was never touched and is still `stopped`;
|
||||
it is not in the `felhom` pool (`pvesh get /pools/felhom` listed only `lxc/9201`), so the
|
||||
uninstall's shared-box logic never reached it.
|
||||
2. **The host.** `local-lvm`, before → after: **demo-felhom 29.31 % → 1.37 %** (the old 200 G + 50 G
|
||||
volumes returned; the new 250 G volume is thin and barely allocated). **demo-hp 39.13 % →
|
||||
36.75 %** — *higher than a clean reinstall would leave it*, because the floor test's 2 GB tar
|
||||
blocks cannot be reclaimed: `fstrim` inside an unprivileged LXC returns
|
||||
`FITRIM ioctl failed: Operation not permitted`. No operational impact (the guest shows 65 G free of
|
||||
69 G, the host 35.7 GiB free of 53.93), but it is real residue and is stated rather than rounded
|
||||
away. Each box's data volume is now the only data volume; no old guest volumes remain.
|
||||
**Residue found and cleared by hand on demo-hp:** `--uninstall` left the NAS network-storage units
|
||||
`mnt-felhom\x2ddrives-Felhom\x2dShare.{mount,automount}` behind (automount `failed`, parent bind
|
||||
still mounted) → **R-179**. demo-felhom left none, because it had no network share configured.
|
||||
3. **The hub.** **No old records exist to dispose of, and this is the honest finding, not an
|
||||
omission:** both enrollments were **idempotent** — `host REUSED (idempotent — existing
|
||||
credential)`, `host_id: demo-hp-bb76ea` and `demo-felhom-8363b5`, the same ids as before. The
|
||||
reinstalls therefore produced **no new host records**, so nothing was orphaned and nothing needed
|
||||
deleting. Final register: `demo-felhom-8363b5 ONLINE 0.120.0`, `demo-hp-bb76ea ONLINE 0.120.0`,
|
||||
`drill-r50-0a4f9a DOWN 0.113.0` — the same three rows as at P6. No scratch customers were created.
|
||||
`/appliances` returns 404 on hub 0.89.0 — there is no appliance-record surface to clean.
|
||||
|
||||
**Secrets:** both retrieval passphrases were moved file→file into 0600 files, used via
|
||||
`--passphrase-file`, and `shred -u`'d afterwards on both hosts along with the session cookie files and
|
||||
helper scripts; the local scratch copies were deleted. Nothing was written to a committed file and
|
||||
`curl -w '%{redirect_url}'` was never used (R-132).
|
||||
|
||||
---
|
||||
|
||||
## 8. Registers changed
|
||||
|
||||
| Row | Change |
|
||||
|---|---|
|
||||
| `documentation/architecture/07-backup-architecture.md` | **S-1: the contract changed in the same session.** New **§7.5.1** — the ceiling §7.5 describes no longer exists for a box built from golden ≥ 0.192.0, the bulkhead's replacement (B2) is recorded, and **R-175 is FIXED here**: the bound is restated as a function of `mp1` and scoped to split-layout boxes, naming all three real shapes |
|
||||
| `documentation/architecture/00-capability-map.md` | new row — **IMPLEMENTED, not PROVEN-LIVE**, with the missing leg named (no box reinstalled, R-178) and the bake cited as the evidence it is |
|
||||
| `documentation/backlog/OPEN-ITEMS.md` | **R-165** → shipped-not-yet-proven-live; **R-163 CLOSED** (its ceiling no longer exists); **R-175 CLOSED**; **R-176** (a) answered by P1, (b) **withdrawn** — every node is reinstalled, not migrated; **R-178 NEW** |
|
||||
| `CONTEXT.md` | **S-13** (the variant, chosen on measurement; B2's floor; pruning rejected with its reason; no migration exists) and **S-14** (prove first, then vouch) |
|
||||
| `STATUS.md` | rewritten section in plain language; the *Waiting on you* item cleared — both questions are answered; **two older entries trimmed** so the page did not grow |
|
||||
| `scripts/CHANGELOG.md` | the host-install change, with why it was forced |
|
||||
| **R-178** | **CLOSED** — both boxes reinstalled and proven, by two different supply paths |
|
||||
| **R-165** | **IMPLEMENTED**, not PROVEN-LIVE — operator ruling; the layout half is proven, the B2 half is not (→ R-181) |
|
||||
| **R-115** | **Third instance recorded** — agent 0.120.0 built, deployed to both hosts, never published |
|
||||
| **R-181** *(new)* | The capture floor guards the recovery-unit leg and not `runVolumeDumps`; its "previous unit is untouched" claim measured false |
|
||||
| **R-180** *(new)* | `--archive-storage` is not cross-checked against the ACL grant; the 403 lands at step 8/8, after root@pam has been rotated |
|
||||
| **R-179** *(new)* | `--uninstall` leaves NAS network-storage systemd units behind when a share was configured |
|
||||
|
||||
## 4. CI — run ids and conclusions
|
||||
**IDs established free before minting:** `grep -ro "R-179\b\|R-180\b\|R-181\b\|R-182\b"` over
|
||||
`documentation/` and `*.md` → **0 hits**, and over all four repo roots (`felhom-agent`,
|
||||
`felhom-controller`, `felhom.eu`, `app-catalog-felhom.eu`) → **0 hits**. `R-182` was checked and left
|
||||
unused.
|
||||
|
||||
Checked by **PULL**, matching `head_sha` to each commit — CI mails only on failure, so an unchecked
|
||||
green is an assumption.
|
||||
---
|
||||
|
||||
**Every commit in this session, across all three repos, is green.**
|
||||
## 9. CI
|
||||
|
||||
| Repo | Commit | Task id | Run # | Conclusion |
|
||||
|---|---|---|---|---|
|
||||
| `felhom-controller` | `4be6467` (v0.192.0, the B2 floor) | 39 | 15 | **success** |
|
||||
| `felhom-agent` | `cd6e267` (v0.120.0 + golden 3.0.0) | 41 | 3 | **success** |
|
||||
| `felhom-agent` | `4bb84fc` (REPORT) | 43 | 4 | **success** |
|
||||
| `felhom.eu` | `7406ac7` (phase-0 audit) | 40 | 21 | **success** |
|
||||
| `felhom.eu` | `e3525e6` (host-install one grow) | 42 | 22 | **success** |
|
||||
| `felhom.eu` | `14d8c00` (docs + registers) | 44 | 23 | **success** |
|
||||
Appended after the push — run id and conclusion, per `CLAUDE.md`'s pull-check rule.
|
||||
|
||||
## 5. `--no-verify`
|
||||
---
|
||||
|
||||
**Not used anywhere.** Every push in this session ran its repo's `.githooks/pre-push` and passed.
|
||||
## 10. Observations — noticed and NOT acted on
|
||||
|
||||
## 6. What remains — R-178
|
||||
|
||||
1. Reinstall **demo-hp** from golden 0.192.0 through the real installer path; prove claim → deploy an
|
||||
app → back up → restore; show `df` proving one filesystem and a recovery unit landing on it.
|
||||
2. Only then reinstall **demo-felhom** (it carries the PBS-DR/offsite tier, so it is the box whose
|
||||
backup chain a reinstall actually disturbs).
|
||||
3. **Then** vouch golden 0.192.0 in the hub, and flip the capability-map row to PROVEN-LIVE.
|
||||
4. Re-run P1's restore-test against agent v0.120.0 — one command, turns a sound inference into an
|
||||
observation.
|
||||
- **The runbook's central claim was wrong in a way that mattered.** R-178 said *"a reinstall is now a
|
||||
self-contained piece of work with no code left to write."* True about code; false about the artifact
|
||||
channel — the agent half of the merge was unpublished, and following the documented path without
|
||||
checking would have downgraded both boxes and produced a green, meaningless result. **No code change
|
||||
was needed to make any step pass** — §10 asks this loudly, and the answer is no. What was needed was
|
||||
a publish.
|
||||
- **Prove-then-vouch was already spent when the session opened.** Not a defect in anything, but the
|
||||
rule protected nothing because nothing enforced it. R-120's gate is the shape that would.
|
||||
- **A per-stack `POST /stacks/<n>/backup` does not produce volume dumps** — the volume leg lives in the
|
||||
full app-data pass. Not wrong, but the endpoint's name suggests otherwise and it cost time here.
|
||||
- **privatebin's recovery unit carries no user data** (its catalog entry declares no `backup:`
|
||||
section), so restoring it loses every paste. That may be intentional for an expiring, E2E-encrypted
|
||||
paste bin — but nothing in the app's description tells the customer so. Catalog question, not filed.
|
||||
- **V-c doubles systemd's mount-unit count** — every docker overlay appears twice,
|
||||
`var-lib-docker-…-merged.mount` and `var-lib-felhom-docker-…-merged.mount`, because `/var/lib/docker`
|
||||
is a bind of `/var/lib/felhom/docker`. Cosmetic, inherent to the chosen variant, no action.
|
||||
- **`fstrim` cannot run inside the guest** (`EPERM`, unprivileged LXC), so space freed inside the guest
|
||||
is not returned to the thin pool. It did not matter here; on a box that fills and empties repeatedly
|
||||
it would.
|
||||
- **The floor's own message mixes units** — it reports `64.2/68.7 GB used (93%)` against a threshold
|
||||
stated as `97% used or 1.0 GiB free`, while `df` showed 96 %. GB-vs-GiB, so the percentage term
|
||||
fires later than an operator reading `df` would expect. Minor; noted on R-181's fix shape rather
|
||||
than filed separately.
|
||||
|
||||
Reference in New Issue
Block a user