DRILL: the retained key works, and the customer cannot reach it
gates / gates (push) Successful in 23s

Three verdicts, kept separate because collapsing them is how this assumption
survived a week.

(a) The material IS retained. host_escrow_superseded id 11 is the first retained
row in fleet history to carry identity_blob (572 B), byte-identical to the
pre-supersession row (sha256 a10032341c8584ed...).

(b) The retained material DOES open the old store. Unsealed with the old recovery
code it yielded a password byte-identical to the pre-change one, and restored
three planted files byte-identical from a store the box itself could no longer
open - including a Hungarian accented filename verified as raw bytes. Negative
control ran first and failed closed.

(c) The customer has NO route, and is misinformed. ListSupersededEscrow has zero
production callers; the recovery path selects FROM host_escrow. Asked with the
code that had just worked by hand, the product answered "the recovery code did
not open the sealed bundle". A valid code for retained history is reported as a
bad code - the R-224 class again. R-304, rank 1.

Both installer faults were watched happening first, so installer-v1.27.0 is now
published (tag + both webpage.yaml refs). Pre-fix: the box came up on controller
0.98.3 against a vouched 0.213.0, below the floor and below the version carrying
the recovery screen; and our own uninstall left dnsmasq on 0.0.0.0:53 so our own
next install refused. R-297 and R-300 CLOSED.

Also filed R-305 (the dnsmasq fix fires once per machine - the leftover returns
on the second reinstall, proven), R-306 (--preflight-only writes state it says it
does not), R-307 (a live abandon countdown on demo-felhom, firing 2026-08-24 -
operator decision), R-308 (stored controller password stale), R-309 (the day-0
runbook's publication claim has been false since R-110), R-310 (two edges).

Ceiling R-303 -> R-310. Capability map moved: the retention claim is now marked
operator-only. Phase A logs did not survive the intermediate revert; recorded.
This commit is contained in:
2026-08-12 17:41:56 +02:00
parent fbe1155fbb
commit 1d5f2b8bb6
12 changed files with 2445 additions and 90 deletions
File diff suppressed because one or more lines are too long
@@ -0,0 +1,514 @@
# DRILL — the retained key, and the two fixes nobody had watched work
**Date:** 2026-08-12 · **Class:** drill (unattended, destructive on Tier 0; spike for Phase C's first step)
**Venues:** `drill-r50` (nested PVE on DooPlex) · `demo-felhom` (guest 9201 on felhom-pve) — both Tier 0
**Baselines re-confirmed live on arrival, not assumed:** controller **0.213.0** on both demo boxes
(`pct exec 9201 -- docker ps`), agent **0.128.0** (hub `hosts`), hub **0.102.0** (deployment image),
register ceiling **R-303** (grep), `felhom.eu` clean at `fbe1155f` and level with `origin/main`.
---
## 1. The three verdicts, first, in plain language
> **(a) Is the material retained? — YES.** Proven for the first time in the fleet's history.
>
> **(b) Does the retained material, once unsealed, yield a password that actually opens the old
> store? — YES.** Three planted files came back byte-identical from a store the box itself could no
> longer open, including the Hungarian accented filename.
>
> **(c) Does the customer have a route to (b) through the product? — NO. And the failure is worse
> than absence:** the customer's old recovery code — the one that demonstrably opens their old
> backups — is rejected by the product with *"the recovery code did not open the sealed bundle"*.
> They are told their correct code is wrong, and given no next step.
**The headline the operator asked for.** The brief said to be ready for the answer to be no, and our
own records predicted the retention would be *"a box we fill and cannot open"*. That prediction was
**half right, and the wrong half was the one nobody checked.** The box can be opened — the crypto,
the retention and the copy are all sound. What does not exist is the door: no code path reads a
retained row, so the only way in is an operator with SQLite, `age` and a shell. Everything I did in
§4 a customer cannot do.
That distinction matters for the three pieces of work resting on this:
| Rests on | Verdict |
|---|---|
| The census answer ("nobody else is exposed") | **Stands.** It was a statement about retention, and retention works. |
| The countdown banner's promise (old backups stay recoverable) | **True in substance, false in practice.** The bytes are recoverable; the customer cannot recover them. The promise is made to someone who has no way to act on it. |
| The capability map's recovery claims | **Must move.** Any claim of the form "the customer can recover the old history with their recovery code" is false today. |
---
## 2. §7.0 — the probe, run first and read-only
Run before anything was installed or planted, exactly as the order required.
**Q1 — does any host today hold a superseded record with key material present? NO. Zero, fleet-wide.**
```
host_id kfp16 blob_len identity_blob superseded_at
demo-hp-bb76ea 3f:4f:65:c0:d8:f 383 NULL 2026-08-04 07:15:36
demo-felhom-8363b5 7e:a6:af:f7:ea:6 383 NULL 2026-08-04 07:20:08
```
Both are the pre-R-198 rows written hours before the fix. `SELECT COUNT(*) … WHERE identity_blob IS
NOT NULL` returned **0**. The retention as fixed had never once been exercised.
*Instrument:* hub SQLite copied WITH `hub.db-wal` + `-shm` (the standing gotcha). Freshness proved by
a **positive observable** — newest `host_reports` row 61 s old — not by "the query returned no error".
`PRAGMA integrity_check` = ok.
**Q2 — what act produces a retaining supersession? Two, both read from source:**
- `hub/internal/store/store.go:2807``SaveHostEscrow` demotes when the newly sealed
`restic_pw_sha256` **differs** from the stored one.
- `hub/internal/store/store.go:2636``DeleteHost` demotes the current row unconditionally.
**Q3 — is there any code path that USES a retained record to recover a password? NO.**
The product's recovery route is real and wired (R-199) but reaches the **current** row only:
```
controller → agent POST /escrow/recover-offsite-password
→ escrow.OffsiteKeyRecoverer.Fetch felhom-agent/cmd/felhom-agent/main.go:1757
→ hubClient.FetchIdentityEscrow
→ hub GET identity-escrow handler hub/internal/api/handler.go:1325
→ store.GetHostDRBundle hub/internal/store/store.go:3152
SELECT blob, identity_blob, directive_json FROM host_escrow WHERE host_id = ?
```
`ListSupersededEscrow` (`store.go:2841`) is the **sole** reader of a retained `identity_blob`. It has
**zero production callers** — five call sites, all in `_test.go`. `CountSupersededEscrow` is used
twice and only to render a *count* (`api/handler.go:1209`, `web/hosts.go:572`).
**Feasibility verdict: producible.** Phase C proceeded.
---
## 3. Phase C — producing the state
### 3.1 The planted dataset
Planted in `/mnt/sys_drive/felhom-data/userdata/drill-20260812` on `demo-felhom`.
| file | sha256 | bytes |
|---|---|---|
| `payload.bin` | `6863824a67bf6cb636defca3cbddc1d58092bcdda3af939224dfef4663c15359` | 65536 |
| `plain-ascii.txt` | `e294e0ee5b8fd131dc0fa71f5cf1253dc39473241aa090be36404a568285c83a` | 56 |
| `árvíztűrő-tükörfúrógép.txt` | `a1957a87104c63be22b34e4f937e700edcdc925da5984bfcca407df4989f7190` | 37 |
**The accented filename never crossed a shell as text** (rule 6). It was built from explicit bytes and
verified as hex, both as intended and as listed on disk — identical:
```
c3 a1 72 76 c3 ad 7a 74 c5 b1 72 c5 91 2d 74 c3 bc 6b c3 b6 72 66 c3 ba 72 c3 b3 67 c3 a9 70 2e 74 78 74
```
### 3.2 Positive control on the comparator — done BEFORE it was relied on
One byte of a copy of `payload.bin` flipped at offset 40000:
```
unaltered f5de179b5ca941645819a879187db953db5c973d4c1a07b2a3ba96d3d79bfb6c
altered 515a65d96326bd847a87ea35c9fb5e6afebc9499ec0c1cd80ca3531b22cb2324
payload.bin: FAILED · sha256sum: WARNING: 1 computed checksum did NOT match
```
The comparator convicts. The copy was discarded and the original re-hashed unchanged.
### 3.3 The store, written under P1
`restic backup` into demo-felhom's **own** repository path
(`sftp:u629488-sub1@…your-storagebox.de:/home/felhom-repo`) → snapshot **`6ea85413`**, confirmed by
identity and by **listing its contents**, not by the success line:
```
6ea85413 2026-08-12 15:15:33 demo-felhom felhom-offbox,drill-retained-key-20260812
…/drill-20260812/payload.bin
…/drill-20260812/plain-ascii.txt
…/drill-20260812/M-CM-!rvM-CM--ztM-EM-1rM-EM-^Q-tM-CM-<kM-CM-6rfM-CM-:rM-CM-3gM-CM-)p.txt
```
**Key fingerprint of the store's password (P1): sha256 `c60c8bc737a6b7c6…`** — value never recorded.
### 3.4 The supersession, by the product's own ceremony
The repository password was replaced (P1 → P2, `d4aac4f1c0ef9ac0…`), making `/home/felhom-repo` the
**old** store, then the ceremony was driven through the same local-API calls the dashboard wizard
makes: `POST /escrow/stage-secret``POST /escrow/ceremony``…/status``…/claim`.
```
phase=done restic_pw_sealed=true uploaded=true
```
### 3.5 (a) — the material IS retained
The hub grew its first ever retained row carrying key material:
```
id host_id identity_blob restic_pw_sha256 superseded_at
11 demo-felhom-8363b5 572 c60c8bc737a6b7c6… 2026-08-12 15:18:55
```
Confirmed **by identity and byte length**, not by presence:
```
a10032341c8584edfd87ad941a50d9fd6f34c23a4802f55885cb1d4cab086bd3 current-identity.age (pre-supersession)
a10032341c8584edfd87ad941a50d9fd6f34c23a4802f55885cb1d4cab086bd3 retained-identity.age (retained row 11)
572 bytes both — IDENTICAL
```
### 3.6 (b) — the retained key DOES open the old store
**Negative control first** — the box's current password against the old store:
```
restic exit code: 1
Fatal: wrong password or no key found
```
**Then the retained key.** Retained blob → `age -d` with the old recovery code → bundle →
`restic_repo_password`, sha256 `c60c8bc737a6b7c6…`, **byte-identical to the P1 recorded before the
supersession**:
```
restic snapshots → exit 0, snapshot 6ea85413 listed
restic restore → exit 0
restored accented filename bytes: c3 a1 72 76 c3 ad 7a 74 c5 b1 72 c5 91 2d … (identical)
payload.bin: OK · plain-ascii.txt: OK · árvíztűrő-tükörfúrógép.txt: OK (compare exit code 0)
```
**All three files byte-identical, from a store the box itself could not open.**
### 3.7 (c) — the customer has no route, and is actively misinformed
The product's own recovery endpoint was asked for the old password, using the same old recovery code
that had just worked by hand:
```
OLD code (opens the retained row by hand → P1) REFUSED: "the recovery code did not open the
sealed bundle — nothing was written"
NEW code (today's ceremony) RETURNED a password, sha d4aac4f1c0ef9ac0 = P2
```
P2 does not open the old store. **The one code that works is reported as the code that does not.**
This is the R-224 defect class in a new guise: there, an unreachable hub was reported as a bad
recovery code; here, a *valid code for retained history* is reported as a bad code. The message is
not merely unhelpful — it is wrong, and it terminates the customer's attempt.
---
## 4. Every step taken off the customer's path
Stated plainly, because (b)'s "yes" is worth exactly as much as this list is short.
| Step | Off-path how | Cost to the walk's fidelity |
|---|---|---|
| Read `identity_blob` out of the hub's SQLite | No API serves a retained row — **this is (c)** | None to the finding; it *is* the finding |
| `age -d` via a hand-written pty wrapper on DooPlex | The product's unwrap is reachable only for the CURRENT blob | None — same `age -p` primitive the agent uses |
| `restic` invoked directly with the recovered password | The controller's recovery path **compares**, it never installs | None — same restic, same repo, same credentials |
| Recovery code taken from the operator's credentials file | A customer would read it from their card | None — a customer plausibly has it |
| Planted dataset pushed with `restic backup` directly | The dashboard button was unreachable (see F6) | The unit-snapshot chain was not exercised; the *store* is identical |
| `--skip-provision` + `--force` on Phase A installs | Saved a guest restore per cycle | None — dnsmasq handling is independent of provisioning |
---
## 5. Phase A — our removal undoes what our installation did
**Venue `drill-r50`, reverted to `virgin` before each clean run.**
### 5.1 The chain re-established, not assumed
Virgin baseline: `dnsmasq` **not-installed**, unit **not-found**, **`:53` free**. After an appliance
install the **agent** installed dnsmasq (`internal/lanresolver`, not the script) and the Felhom
snippet **constrained** it to `10.0.2.15:53` + `127.0.0.1:53`.
### 5.2 The wrong outcome, demonstrated first
Pre-fix uninstall (v1.26.0) — snippet removed, `README` only left, and:
```
enabled: enabled · active: active
udp UNCONN 0.0.0.0:53 · udp UNCONN [::]:53 · tcp LISTEN 0.0.0.0:53 · tcp LISTEN [::]:53
```
Pre-fix byo install, **exit 1** — the refusal the publication had been waiting for since Sunday:
```
[ERROR] a resolver is already bound to :53 on this host:
udp UNCONN 0 0 0.0.0.0:53 … users:(("dnsmasq",pid=4934,fd=4)) …
[ERROR] a resolver is already bound to :53 on this host — Felhom needs the guest reachable by name on your LAN.
Stop or reconfigure that resolver, OR point your LAN DNS at the guest's address, then re-run.
(Felhom does NOT touch DNS services on a host it does not own — this is a refusal, not a change.)
[ERROR] PRE-FLIGHT FAIL (exit 1) — fix the finding above and re-run
```
Nothing in it suggests the resolver is Felhom's own leftover.
### 5.3 The pass
Fixed path (v1.27.0) from virgin: preflight recorded `dnsmasq: not present before Felhom`; uninstall
logged `dnsmasq was installed by Felhom (recorded at install) — stopping + disabling it`; result
`enabled: disabled · active: inactive · :53 FREE`; and the second install:
```
[INFO] host DNS (:53): free
[OK] pre-flight passed
[OK] PRE-FLIGHT PASS (mode=byo) — no state written, no install step executed
```
### 5.4 Both sides of the ownership record
- **`no` (Felhom's):** stop + disable. Quoted above.
- **`yes` (the owner's):** `dnsmasq pre-dates Felhom (recorded at install) — leaving it running,
restarting only` → left `enabled`/`active`. **Felhom does not disable a resolver it did not
install.** Correct.
- **No record (the field case):** every box installed before this change. Exercised by running the
fixed byo preflight on a box whose state file the uninstall had already deleted — the improved
refusal fired (§5.5).
### 5.5 The improved refusal, judged
```
THIS LOOKS LIKE OURS. A previous Felhom install leaves the dnsmasq PACKAGE installed and its unit
enabled (only our config snippet is removed), and unconstrained it binds 0.0.0.0:53 — which is what
this gate is seeing. If this host had no dnsmasq before Felhom, clear it with:
systemctl disable --now dnsmasq
Then re-run this installer. If dnsmasq is YOURS, leave it and use one of the two routes above.
```
It names the leftover, gives the exact command, and does not overclaim — it says *looks like*, and
leaves the owner's case intact. **Good.** One weakness: it asks the reader to answer *"did this host
have dnsmasq before Felhom?"* — a question the installer itself now records the answer to, but in a
state file the uninstall has already deleted. English throughout, consistent with the rest of the
installer (this surface is operator/tester-facing, not the customer dashboard).
### 5.6 The finding this phase was not looking for — the fix works exactly once per machine
Proven, not deduced. Cycle 2 on the same box:
```
install : [INFO] dnsmasq: already installed BEFORE Felhom — recorded; uninstall will not touch it
uninstall: [INFO] dnsmasq pre-dates Felhom (recorded at install) — leaving it running, restarting only
after : enabled: enabled · active: active · 0.0.0.0:53
cycle-3 byo preflight: PRE-FLIGHT FAIL (exit 1) — "a resolver is already bound to :53"
```
**Why:** the uninstall disables the unit but never **purges** the package (deliberately — purging on
a host we may not own is the wrong blast radius). So on every later install `dpkg-query` says
installed, preflight records `yes`, the agent re-enables it, and the uninstall then only restarts it.
The wall returns on the second reinstall. → **F2**
---
## 6. Phase B — an install takes the image that was approved
**Venue `drill-r50`, reverted to `virgin`.** A genuinely stale golden (`golden-0.98.3.tar.zst`,
controller **0.98.3**, marker verified before use) was placed as
`local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst` — newest by filename, which is exactly what
discovery sorts on.
### 6.1 The fault, observed
Pre-fix (v1.25.0, the published tag) reported both facts and used the wrong one:
```
[INFO] manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.213.0
[STEP] 7/8 golden archive
[SKIP] using local golden: local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
[OK] Day-0 provision SUCCESS — vmid=120 … golden=local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
```
No digest, no version compare, no warning. **The box came up on controller 0.98.3** — confirmed twice,
from the running container and from the in-guest marker:
```
gitea.dooplex.hu/admin/felhom-controller:0.98.3 (docker ps)
gitea.dooplex.hu/admin/felhom-controller:0.98.3 (/etc/felhom-controller-image)
```
**The sharp end (§6 item 4).** Vouched golden **0.213.0**; floor **0.213.0**; the box landed on
**0.98.3** — below the floor, and below the version carrying the off-site recovery screen. A box born
this way cannot run the ceremony its own data depends on, and is born below the update floor. **The
row was right.**
### 6.2 Both fixed shapes
**Explicitly named archive → refuses** (exit 1):
```
[STEP] 7/8 golden archive
[ERROR] refusing the golden you named (local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst):
it is controller 0.98.3, but the vouched golden is 0.213.0.
The vouched golden is 0.213.0. Either pass the archive that matches it,
or re-run with --force-gitea-golden to fetch the vouched one from Gitea.
```
**Auto-discovered archive → warns and re-fetches:** see §6.3.
No local goldens were deleted and teardown's keep-behaviour was not changed.
### 6.3 Auto-discovery result
```
[STEP] 7/8 golden archive
[WARN] ignoring the local golden local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
— it is controller 0.98.3, but the vouched golden is 0.213.0
[WARN] fetching the vouched golden instead (this is what the manifest is for)
[INFO] fetching golden v0.213.0 from Gitea → …/vzdump-lxc-9100-2026_08_12-17_31_32.tar.zst
[OK] verified sha256 77429bfc7e39ffc8… matches the hub manifest
```
**Landed on 0.213.0**, confirmed from both the running container and the in-guest marker.
**Observation, not a fault:** the re-fetched archive is named by its real timestamp
(`17_31_32`), which sorts *below* the planted stale one (`23_59_59`). So the stale archive stays the
newest-by-filename and is re-discovered — and correctly re-rejected — on every subsequent install.
The fix is not defeated; it just never displaces the bad archive, so the warning recurs forever.
---
## 7. Findings, ranked by what they cost the person in front of you
| # | Rank | Finding |
|---|---|---|
| **F1** | **1 — stops the visit** | The retained key has **no product route**, and the product tells a customer holding the **correct** old recovery code that it *"did not open the sealed bundle"*. Recoverable data, valid code, and a dead end with no next step. (§3.7) |
| **F2** | 2 — costs time | The R-300 cleanup fires **only on a machine that never had Felhom**. The package is never purged, so every later install records `dnsmasq_preexisting=yes` and the leftover returns on the second reinstall. (§5.6) |
| **F3** | 3 — misleads | `--preflight-only` claims *"no state written"* (banner and the flag's own comment) but **writes `/var/lib/felhom-install/state.json`**`_state_put` short-circuits on `DRY_RUN` only. Worse, on a box carrying a Felhom leftover it records `dnsmasq_preexisting=yes` **before** the real install, baking in the wrong ownership answer. |
| **F4** | 3 — misleads | The pre-fix uninstall's closing note called dnsmasq a *"system package"*. On these boxes Felhom installed it. (Already corrected in v1.27.0's wording; confirmed live.) |
| **F5** | 4 — cosmetic | The shape-2 refusal states the vouched version twice in consecutive sentences. |
| **F6** | 2 — costs time | The stored `PASSWORD` credential no longer opens **demo-felhom**'s dashboard (`Hibás jelszó`). Not the known quoting trap — quotes were stripped and the value is unquoted. The dashboard was unreachable headlessly for this drill. |
| **F7** | — operator decision | **`demo-felhom` has a live abandon countdown**: `abandon_started_at 2026-08-10T08:06:31Z`, `abandon_at 2026-08-24T08:06:31Z`, `abandon_repo_path /home/felhom-repo.orphaned-20260810`. Not started by this drill. See §9. |
| **F8** | 4 — cosmetic | `--uninstall` requires `/dev/tty` for the typed vmid confirmation when a guest exists, and `--force` does **not** bypass it (deliberate, and correct for an irreversible destroy) — but it means teardown cannot be scripted without a pty. Worth one line in the runbook. |
**A drill that finds nothing is a suspicious result.** This one found eight, and the instrument was
demonstrably working: the comparator was shown convicting a one-byte change before it was trusted
(§3.2), the negative control on the old store failed closed before the positive one succeeded (§3.6),
and both installer faults were reproduced from a virgin machine before either fix was watched.
---
## 8. Publication
Both Phase A's and Phase B's faults were observed live. **The reward is earned.**
- Phase A fault: observed (§5.2) and fixed path quoted (§5.3).
- Phase B fault: observed (§6.1) — the box came up on 0.98.3 against a vouched 0.213.0.
**`installer-v1.27.0` cut, covering both fixes** (R-297 stale-golden comparison, R-300 removal
leftover — the only two commits touching the installer since `installer-v1.25.0`).
**And publishing is not what the day-0 runbook says it is.** §C.1 states the URL *"is always the
current `main` script … pushing `scripts/felhom-host-install.sh` publishes it."* That has been false
since R-110: `manifests/webpage.yaml` runs two git-syncs, and `/scripts/` follows the **installer
tag**, not `main`. Measured before publishing — `https://felhom.eu/scripts/felhom-host-install.sh`
served `SCRIPT_VERSION="1.25.0"` while `main` carried `1.27.0`, three and a half hours after the push.
So the fixes were genuinely unpublished, as the brief assumed — but for a reason the runbook denies.
Publication here therefore meant: cut the tag **and** bump **both** `--ref`s (sidecar at line 327 and
init container at line 372). → **R-309**
---
## 9. The countdown — an operator decision, deliberately not taken
`demo-felhom` carries a **running abandon countdown**, started **2026-08-10**, firing **2026-08-24**,
for `/home/felhom-repo.orphaned-20260810`. **This drill did not start, shorten or trigger it**, per
the fence.
But §9 of the brief requires an end state of *"no abandon countdown anywhere"*, and one exists. The
two ways to satisfy that are not equivalent and the choice is not mine:
1. **Cancel it** — the orphaned repository is kept indefinitely (storage cost, no data risk).
2. **Let it run** — on 2026-08-24 the orphaned repository is **deleted, irreversibly**.
Doing nothing selects option 2 by default. **Flagged, not resolved.**
---
## 10. Teardown — four layers
| Layer | State |
|---|---|
| **The machine** | `demo-felhom`: planted dataset and working directory removed; eight secret-bearing files **shredded**, not unlinked; `repo_password` restored to P1 (`c60c8bc737a6b7c6…`); escrow re-sealed and uploaded; offsite repo reachable (`restic snapshots` exit 0, 4 snapshots). The agent's `escrow-stage/` directory is **empty** — the ceremony wiped the staged secret (positive observable). `demo-hp`: **not touched at any point**. `drill-r50`: see §11. |
| **The host** | `felhom-pve`: no drill artefacts left outside guest 9201. `drill-r50` host: see §11. |
| **The hub** | Two new retained rows (id 11 = the P1 blob, id 12 = the P2 blob), both `identity_blob` 572 bytes — **deliberately kept as the fixture that proves the retention works**; they are append-only, tiny, and exactly what the design intends. `drill-r50-0a4f9a` host record: pre-existing since 2026-07-25, **re-used rather than duplicated** — no new hub-side sprawl. `customer_configs` gained nothing. |
| **The off-site side** | Only `demo-felhom`'s own repository path was touched. **No `prune`, no `forget`, no delete, no rename** — anywhere, at any point. One snapshot was **added**: `6ea85413`, 66 KiB, tagged `drill-retained-key-20260812`. **Deliberate residue** — removing it would have meant a `forget` on the endpoint, which the fence forbids in spirit; the operator can remove it by ID if desired. Nothing outside `/home/felhom-repo` was read or written; `peti-felhom` was never contacted. |
**How I know the off-site side is untouched outside this machine's path:** every restic invocation in
this drill went through one wrapper pinned to
`sftp:u629488-sub1@u629488-sub1.your-storagebox.de:/home/felhom-repo` — a per-customer sub-account
whose credentials reach nothing else — and the only mutating verb used was `backup`.
---
## 11. Close of run
### 11.1 Wall clocks (CEST)
| Phase | Span | Elapsed |
|---|---|---|
| §7.0 probe (read-only, source + hub) | 16:42 → 16:52 | **~9 min** |
| Phase A — removal leftover | 16:50 → 17:12 | **~22 min** (4 installs, 4 uninstalls, 3 preflights) |
| Phase B — stale golden | 17:13 → 17:34 | **~21 min** (2 full provisions) |
| Phase C — retained key | 17:14 → 17:23 | **~25 min** (overlapped Phase B; different machine) |
| Record, register, publication, teardown | 17:35 → 17:55 | **~20 min** |
| **Total** | **16:42 → 17:55** | **≈ 1 h 13 min** |
Phase C ran concurrently with Phase B deliberately — different machines, no shared state. The
45 hour envelope was not needed; nothing was hurried and nothing was dropped for time.
### 11.2 Register
**Ceiling moved R-303 → R-310.** Opened: **R-304** (retained key has no product route — rank 1),
**R-305** (R-300 fix is single-shot per machine — rank 2), **R-306** (`--preflight-only` writes state
it says it does not — rank 3), **R-307** (live abandon countdown, operator decision), **R-308**
(stored controller password stale for demo-felhom — rank 2), **R-309** (day-0 runbook's publication
claim false since R-110 — rank 3), **R-310** (two installer edges — rank 4).
**Closed: R-297 and R-300**, both observed live and published.
### 11.3 Teardown
`drill-r50`: **reverted to snapshot `virgin` and left powered off**`qemu-img snapshot -a virgin`,
qemu exited first, `qemu.pid` removed. Its three in-VM goldens went with the revert (the VM's normal
lifecycle); the **source** archive on DooPlex, `/mnt/5_hdd/felhom.eu/drill/golden-0.98.3.tar.zst`, is
untouched (mtime still 2026-07-03). Full four-layer detail in §10.
### 11.4 What was dropped, named plainly
**Nothing was dropped.** Phase A, Phase B and Phase C all ran in full, in the required order, and the
publication was earned rather than assumed.
**One thing was not done and it was not in scope:** the planted dataset reached the off-site store via
`restic` directly rather than via the dashboard's own button, because the stored controller password
no longer opens demo-felhom (**R-308**). The store, the credentials and the restic binary were the
product's own; what went unexercised is the *app-backup → recovery-unit → offsite* chain, which is not
what this drill was measuring.
### 11.5 Evidence, including a gap
Logs at `audits/evidence-drill-retained-key-2026-08-12/`: `B1-install-preB.log`, `B2-uninstall.log`,
`B3-explicit-stale.log`, `B4-autodiscover-stale.log`, `Z-final-state.txt`.
**The Phase A logs did not survive.** They lived on the drill VM's disk and were destroyed by the
revert to `virgin` between Phase A and Phase B — I copied evidence out before the *final* teardown but
not before the *intermediate* one. Every Phase A quotation in §5 is verbatim from the live run as read
at the time, but the raw files are gone and cannot be re-read. **This is the GL-1 discipline the golden-bake
runbook already states ("scp the log OUT first"), applied to the last revert and not the middle one.**
Recorded rather than glossed; the fix is procedural, not a code change.
### 11.6 Observations — noticed, not acted on
- **The stale archive is never displaced.** A re-fetched golden carries its real timestamp, which sorts
below a planted `23_59_59`, so the bad archive stays newest-by-filename and is re-rejected on every
future install. Correct, but the warning recurs forever and nothing removes the cause.
- **The ceremony wipes its staged secret.** After every run `/var/lib/felhom-agent/escrow-stage/` was
**empty** — checked as a positive observable, not inferred from an absent log line.
- **`restic_pw_sealed: true` in the ceremony status is a genuinely useful positive observable** — it
distinguishes "a ceremony ran" from "a ceremony sealed the thing that matters".
- **`demo-felhom` already had an orphaned store** (`/home/felhom-repo.orphaned-20260810`) from the
10 August rebuild, and its password never changed — the current escrow's `restic_pw_sha256` matched
the box's live `repo_password` exactly on arrival. So that orphaning was a rename, not a key change.
- **R-241's mint guard makes password-changing supersessions rare by design.** A rebuilt guest no longer
mints a fresh repository password while the hub holds a sealed package, which is why the state Phase C
needed had to be produced deliberately rather than found.
- The Hungarian locale warning (`perl: LC_CTYPE = "UTF-8"`) prefixes almost every `pct exec` on
felhom-pve. Harmless, but it is noise on every operator transcript.
@@ -0,0 +1,115 @@
[INFO] felhom-host-install v1.25.0 — mode=appliance customer=drill-r50 vmid=120
[STEP] 1/8 pre-flight
[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
[INFO] node: drill-pve (auto)
[INFO] agent config: /etc/felhom-agent/agent.json
[INFO] agent: not installed yet — will be fetched + installed in step 5/8
[INFO] local-lvm free: ~75 GiB
[WARN] local-lvm free ~75 GiB < hard min 120 GiB
[INFO] free RAM: ~6305 MiB
[INFO] existing guests on this host: 0 (pct+qm)
[INFO] archive storage 'local' present
[INFO] acl storage 'felhom-pbs' not present yet — expected: the PBS-DR tier creates it; the grant is pre-positioned deliberately
[INFO] hub reachable (https://hub.felhom.eu)
[OK] customer 'drill-r50' exists + passphrase valid
[INFO] golden (local): local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
[OK] pre-flight passed
[STEP] 2/8 Proxmox API token
[OK] token minted (secret captured, not logged)
[OK] scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm felhom-pbs])
[SKIP] old broad role FelhomAgent already absent
[STEP] 3/8 compute volume grows
[INFO] auto-computed from ~75 GiB free (ONE volume since R-165)
[INFO] grows: rootfs +0G (->32G), data +46G (->70G, ONE volume)
[STEP] 4/8 host enrollment (POST /host-enroll)
[OK] host REUSED (idempotent — existing credential)
[INFO] host_id: drill-r50-0a4f9a (api_key captured, not logged)
[STEP] 4b/8 break-glass credential (root@pam console password → hub vault)
[OK] root@pam password set + vaulted to the hub (retrieve via the operator /admin path; never logged here)
[WARN] NOTE: the root@pam password just CHANGED — the old one now fails at the PVE web GUI (:8006).
[WARN] Retrieve the new one at hub → host page (vaulted recovery credential).
[STEP] 5/8 agent install (fetch + verify + install)
[INFO] disabled pve-enterprise.sources (Enabled: no)
[INFO] disabled ceph.sources (Enabled: no)
[INFO] added pve-no-subscription.sources (suite=trixie)
[OK] apt repos aligned to no-subscription (changed; apt-get update OK)
[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
[INFO] manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.213.0
[INFO] fetching agent binary v0.128.0 from Gitea …
[OK] verified sha256 c6eba73bf9b9ad69… matches the hub manifest
[OK] installed /usr/local/bin/felhom-agent (felhom-agent 0.128.0)
[INFO] installing the 'sudo' package (required for the non-root agent model) …
[OK] sudo installed (Sudo version 1.9.16p2)
[INFO] installing the 'age' package (escrow ceremony identity-wrap dependency) …
[OK] age installed (1.2.1)
[OK] created service user felhom-agent
[OK] added felhom-agent to systemd-journal (unprivileged journal read for NAS verify)
[OK] installed /usr/local/sbin/felhom-mkfs-guarded (0755, the guarded mkfs path)
[OK] installed /usr/local/sbin/felhom-selfupdate-guarded (0755, the guarded A/B binary-swap path)
[OK] installed /usr/local/sbin/felhom-pbs-apply (0755, the guarded PBS-DR apply path)
[OK] installed /usr/local/sbin/felhom-backup-target-apply (0755, the guarded backup-target path)
[OK] installed /etc/sudoers.d/felhom-agent (0440, visudo-validated)
[OK] installed /etc/systemd/system/felhom-agent.service + enabled (started in step 6 after config)
[OK] installed self-update rollback unit + start-limit drop-in (auto-rollback armed)
[OK] installed break-glass layers 1+2 (tmpfiles /run/sshd + agent-independent watchdog timer)
[OK] installed OOB felhom-sshd instance + static belt (agent renders config + fills sets once oob.enabled)
[STEP] 6/8 agent config + service
[WARN] backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive.
[WARN] It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard.
[INFO] creating island bridge vmbr9 (portless, 169.254.253.1/30)
[OK] vmbr9 up: 169.254.253.1/30
[INFO] R-50 island ON: local_api=169.254.253.1:8443 (vmbr9); guest net1=169.254.253.2/30; lan_resolver.host_ip=10.0.2.15
[INFO] node=drill-pve local_api=169.254.253.1:8443 tls_fp=F7:CC:33:E6:CA:F5…
[OK] wrote /etc/felhom-agent/agent.json (0600 felhom-agent)
[OK] agent --selftest (read-only) passed
[OK] felhom-agent service active (non-root felhom-agent reads the config OK)
[STEP] 7/8 golden archive
[SKIP] using local golden: local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
[STEP] 8/8 provision guest 120
[SKIP] pool felhom already exists
=== felhom-agent 0.128.0 selftest=provision (vmid=120 customer=drill-r50 hostname=drill-r50) ===
--- front half: bring-up (provision) local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst → vmid 120 ---
time=2026-08-12T17:18:55.237+02:00 level=INFO msg="bring-up: pool membership re-asserted" vmid=120 pool=felhom
[OK] front half: vmid 120 up (boot+running) in 1m20s; MAC=BC:24:11:6A:90:26
--- back half: mint per-guest token + populate bootstrap config mount ---
time=2026-08-12T17:19:01.173+02:00 level=INFO msg="provision: back-half complete" vmid=120 mount=mp9 guest_path=/etc/felhom-bootstrap endpoint=169.254.253.1:8443
[OK] back half: bootstrap mount mp9 → /etc/felhom-bootstrap on vmid 120 (host dir /var/lib/felhom-agent/guests/120/bootstrap)
local-api endpoint 169.254.253.1:8443 · leaf fp e4cba31879281d094ea2ab492e412a9aa380854b99fd5792134658f7f0971dc5 · token: minted (not printed)
=== selftest=provision OK — guest 120 provisioned + bootstrap-mounted (KEPT) ===
next: reboot the guest → the golden's baked controller-bootstrap unit deploys the controller,
which PULLS its controller.yaml from the hub (retrieval passphrase) and merges in this local_api.
[OK] provision completed
[INFO] rebooting guest 120 so the baked controller-bootstrap unit picks up the mount
[STEP] verify
[OK] pct status: running
[OK] onboot: 1
mp0: local-lvm:vm-120-disk-1,mp=/var/lib/docker,backup=1,size=62G
mp1: local-lvm:vm-120-disk-2,mp=/mnt/sys_drive,backup=1,size=8G
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
rootfs: local-lvm:vm-120-disk-0,size=32G
[OK] pool: guest 120 is a member of felhom
[OK] acl: FelhomAgentBase@/ present (user+token)
[OK] acl: FelhomAgentGuest@/pool/felhom present (user+token)
[OK] acl: FelhomAgentStore@/storage/local present (user+token)
[OK] acl: FelhomAgentStore@/storage/local-lvm present (user+token)
[OK] acl: FelhomAgentStore@/storage/felhom-pbs present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990000 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990001 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990002 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990003 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990004 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990005 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990006 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990007 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990008 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990009 present (user+token)
[OK] authz signers: 2 (operator-signed self-update armed)
[OK] controller: Up 22 seconds (healthy) (after ~0s)
[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.98.3
[WARN] cloudflared not visible yet
[INFO] (confirm in the hub UI that host drill-r50-0a4f9a reports guest 120)
[OK] Day-0 provision SUCCESS — vmid=120 host_id=drill-r50-0a4f9a customer=drill-r50 golden=local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
[INFO] root@pam was rotated + vaulted at step 4b — retrieve at hub → host page (the old GUI password no longer works).
@@ -0,0 +1,790 @@
[INFO] felhom-host-install v1.25.0 — mode=uninstall
[STEP] UNINSTALL — local host teardown
[INFO] vmid 120 looks like a Felhom guest (has the /etc/felhom-bootstrap mount)
[INFO] pct config 120:
arch: amd64
cores: 2
features: nesting=1,keyctl=1
hookscript: local:snippets/felhom-guest-hook.sh
hostname: drill-r50
memory: 4096
mp0: local-lvm:vm-120-disk-1,mp=/var/lib/docker,backup=1,size=62G
mp1: local-lvm:vm-120-disk-2,mp=/mnt/sys_drive,backup=1,size=8G
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
mp9: /var/lib/felhom-agent/guests/120/bootstrap,mp=/etc/felhom-bootstrap,ro=1
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:6A:90:26,ip=dhcp,type=veth
net1: name=eth1,bridge=vmbr9,hwaddr=BC:24:11:89:76:A8,ip=169.254.253.2/30,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-120-disk-0,size=32G
swap: 512
unprivileged: 1
Type the vmid (120) to confirm PERMANENT destruction: perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
Logical volume "vm-120-disk-0" successfully removed.
Logical volume "vm-120-disk-1" successfully removed.
Logical volume "vm-120-disk-2" successfully removed.
[OK] guest 120 destroyed
[STEP] host-level removal
Removed '/etc/systemd/system/multi-user.target.wants/felhom-agent.service'.
Removed '/etc/systemd/system/timers.target.wants/felhom-mgmt-watchdog.timer'.
Removed '/etc/systemd/system/multi-user.target.wants/felhom-sshd.service'.
Removed '/etc/systemd/system/multi-user.target.wants/felhom-oob-nft.service'.
The unit files have no installation config (WantedBy=, RequiredBy=, UpheldBy=,
Also=, or Alias= settings in the [Install] section, and DefaultInstance= for
template units). This means they are not meant to be enabled or disabled using systemctl.
Possible reasons for having these kinds of units are:
• A unit may be statically enabled by being symlinked from another unit's
.wants/, .requires/, or .upholds/ directory.
• A unit's purpose may be to act as a helper for some other unit which has
a requirement dependency on it.
• A unit may be started when needed via activation (socket, path, timer,
D-Bus, udev, scripted systemctl call, ...).
• In case of template units, the unit is meant to be enabled with some
instance name specified.
Removed '/etc/systemd/system/multi-user.target.wants/felhom-shared-parent.service'.
Removed '/etc/systemd/system/pve-guests.service.wants/felhom-shared-parent.service'.
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[SKIP] old broad role FelhomAgent already absent
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[SKIP] golden vzdump left in place (pass --remove-golden to remove)
[STEP] kept vs wiped — read before pulling drives or closing the customer out
WIPED (this run):
- guest 120 (container + its OS/Docker/user-data volumes)
- the felhom-agent: binary, unit, sudoers, config (+ its .bak backups), state dir, service user
- self-update artifacts: guarded wrapper, A/B slots (.prev/.new.*), rollback unit, start-limit drop-in
- break-glass watchdog + OOB artifacts (where present); guest-hook snippet; dnsmasq snippets; the mkfs + pbs-apply wrappers
- pveum: the Felhom roles/user/token/scoped ACL; the emptied felhom pool
- the install state file
KEPT (lives on deliberately — remove/rotate these out-of-band if the customer is leaving):
- the enrolled drives + ALL data under /mnt/felhom-drives — unmounted only, NEVER wiped;
the drives are physically removable now.
- the hub host/customer record + report history (operator UI / DB)
- the escrow blob in the hub, if one exists (operator UI)
- the hub-vaulted root@pam recovery credential — the box KEEPS the password step 4b set; rotate it if the box leaves Felhom management
[OK] UNINSTALL complete — removed: guest 120, the felhom-agent (unit/sudoers/binary/state/config+baks/user + selfupdate-artifacts/shared-parent/mkfs-wrapper/pbs-apply-wrapper/hook-snippet/dnsmasq-snippets), the pveum role/user/token/ACL, the felhom pool, and /var/lib/felhom-install/state.json.
[INFO] golden vzdump: left in place (--remove-golden to remove).
[INFO] NOTE: the 'sudo' and 'dnsmasq' packages were left installed (system packages); the host record still exists in the hub — remove it there if desired.
@@ -0,0 +1,767 @@
[INFO] felhom-host-install v1.27.0 — mode=appliance customer=drill-r50 vmid=120
[STEP] 1/8 pre-flight
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
[INFO] node: drill-pve (auto)
[INFO] agent config: /etc/felhom-agent/agent.json
[INFO] agent: not installed yet — will be fetched + installed in step 5/8
[INFO] local-lvm free: ~75 GiB
[WARN] local-lvm free ~75 GiB < hard min 120 GiB
[INFO] free RAM: ~6119 MiB
[INFO] existing guests on this host: 0 (pct+qm)
[INFO] archive storage 'local' present
[INFO] acl storage 'felhom-pbs' not present yet — expected: the PBS-DR tier creates it; the grant is pre-positioned deliberately
[INFO] dnsmasq: already installed BEFORE Felhom — recorded; uninstall will not touch it
[INFO] hub reachable (https://hub.felhom.eu)
[OK] customer 'drill-r50' exists + passphrase valid
[INFO] golden (local): local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
[OK] pre-flight passed
[STEP] 2/8 Proxmox API token
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[OK] token minted (secret captured, not logged)
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[OK] scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm felhom-pbs])
[SKIP] old broad role FelhomAgent already absent
[STEP] 3/8 compute volume grows
[INFO] auto-computed from ~75 GiB free (ONE volume since R-165)
[INFO] grows: rootfs +0G (->32G), data +46G (->70G, ONE volume)
[STEP] 4/8 host enrollment (POST /host-enroll)
[OK] host REUSED (idempotent — existing credential)
[INFO] host_id: drill-r50-0a4f9a (api_key captured, not logged)
[STEP] 4b/8 break-glass credential (root@pam console password → hub vault)
[OK] root@pam password set + vaulted to the hub (retrieve via the operator /admin path; never logged here)
[WARN] NOTE: the root@pam password just CHANGED — the old one now fails at the PVE web GUI (:8006).
[WARN] Retrieve the new one at hub → host page (vaulted recovery credential).
[STEP] 5/8 agent install (fetch + verify + install)
[OK] apt repos aligned to no-subscription (already aligned; apt-get update OK)
[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
[INFO] manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.213.0
[INFO] fetching agent binary v0.128.0 from Gitea …
[OK] verified sha256 c6eba73bf9b9ad69… matches the hub manifest
[OK] installed /usr/local/bin/felhom-agent (felhom-agent 0.128.0)
[OK] created service user felhom-agent
[OK] added felhom-agent to systemd-journal (unprivileged journal read for NAS verify)
[OK] installed /usr/local/sbin/felhom-mkfs-guarded (0755, the guarded mkfs path)
[OK] installed /usr/local/sbin/felhom-selfupdate-guarded (0755, the guarded A/B binary-swap path)
[OK] installed /usr/local/sbin/felhom-pbs-apply (0755, the guarded PBS-DR apply path)
[OK] installed /usr/local/sbin/felhom-backup-target-apply (0755, the guarded backup-target path)
[OK] installed /etc/sudoers.d/felhom-agent (0440, visudo-validated)
[OK] installed /etc/systemd/system/felhom-agent.service + enabled (started in step 6 after config)
[OK] installed self-update rollback unit + start-limit drop-in (auto-rollback armed)
[OK] installed break-glass layers 1+2 (tmpfiles /run/sshd + agent-independent watchdog timer)
[OK] installed OOB felhom-sshd instance + static belt (agent renders config + fills sets once oob.enabled)
[STEP] 6/8 agent config + service
[WARN] backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive.
[WARN] It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard.
[INFO] island bridge vmbr9 already present — leaving it
[INFO] R-50 island ON: local_api=169.254.253.1:8443 (vmbr9); guest net1=169.254.253.2/30; lan_resolver.host_ip=10.0.2.15
[INFO] node=drill-pve local_api=169.254.253.1:8443 tls_fp=F7:CC:33:E6:CA:F5…
[OK] wrote /etc/felhom-agent/agent.json (0600 felhom-agent)
[OK] agent --selftest (read-only) passed
[OK] felhom-agent service active (non-root felhom-agent reads the config OK)
[STEP] 7/8 golden archive
[ERROR] refusing the golden you named (local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst): it is controller 0.98.3, but the vouched golden is 0.213.0.
The vouched golden is 0.213.0. Either pass the archive that matches it,
or re-run with --force-gitea-golden to fetch the vouched one from Gitea.
@@ -0,0 +1,114 @@
[INFO] felhom-host-install v1.27.0 — mode=appliance customer=drill-r50 vmid=120
[STEP] 1/8 pre-flight
[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
[INFO] node: drill-pve (auto)
[INFO] agent config: /etc/felhom-agent/agent.json
[INFO] agent (existing): felhom-agent 0.128.0
[INFO] local-lvm free: ~75 GiB
[WARN] local-lvm free ~75 GiB < hard min 120 GiB
[INFO] free RAM: ~6231 MiB
[INFO] existing guests on this host: 0 (pct+qm)
[INFO] archive storage 'local' present
[INFO] acl storage 'felhom-pbs' not present yet — expected: the PBS-DR tier creates it; the grant is pre-positioned deliberately
[INFO] dnsmasq: already installed BEFORE Felhom — recorded; uninstall will not touch it
[INFO] hub reachable (https://hub.felhom.eu)
[OK] customer 'drill-r50' exists + passphrase valid
[INFO] golden (local): local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
[OK] pre-flight passed
[STEP] 2/8 Proxmox API token
[SKIP] pool felhom already exists
[INFO] user felhom-agent@pve exists
[INFO] existing agent config has a token — testing it (read-only --selftest)
[OK] existing token authenticates — REUSING (no rotation)
[INFO] role FelhomAgentBase exists — ensuring exact privileges
[INFO] role FelhomAgentGuest exists — ensuring exact privileges
[INFO] role FelhomAgentStore exists — ensuring exact privileges
[OK] scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm felhom-pbs])
[SKIP] old broad role FelhomAgent already absent
[STEP] 3/8 compute volume grows
[INFO] auto-computed from ~75 GiB free (ONE volume since R-165)
[INFO] grows: rootfs +0G (->32G), data +46G (->70G, ONE volume)
[STEP] 4/8 host enrollment (POST /host-enroll)
[OK] host REUSED (idempotent — existing credential)
[INFO] host_id: drill-r50-0a4f9a (api_key captured, not logged)
[STEP] 4b/8 break-glass credential (root@pam console password → hub vault)
[SKIP] recovery credential already vaulted (use --rotate-recovery to regenerate)
[STEP] 5/8 agent install (fetch + verify + install)
[OK] apt repos aligned to no-subscription (already aligned; apt-get update OK)
[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
[INFO] manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.213.0
[SKIP] agent v0.128.0 already installed + service active — skipping binary install
[INFO] service user felhom-agent exists
[INFO] felhom-agent already in systemd-journal
[OK] installed /usr/local/sbin/felhom-mkfs-guarded (0755, the guarded mkfs path)
[OK] installed /usr/local/sbin/felhom-selfupdate-guarded (0755, the guarded A/B binary-swap path)
[OK] installed /usr/local/sbin/felhom-pbs-apply (0755, the guarded PBS-DR apply path)
[OK] installed /usr/local/sbin/felhom-backup-target-apply (0755, the guarded backup-target path)
[OK] installed /etc/sudoers.d/felhom-agent (0440, visudo-validated)
[OK] installed /etc/systemd/system/felhom-agent.service + enabled (started in step 6 after config)
[OK] installed self-update rollback unit + start-limit drop-in (auto-rollback armed)
[OK] installed break-glass layers 1+2 (tmpfiles /run/sshd + agent-independent watchdog timer)
[OK] installed OOB felhom-sshd instance + static belt (agent renders config + fills sets once oob.enabled)
[STEP] 6/8 agent config + service
[WARN] backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive.
[WARN] It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard.
[INFO] island bridge vmbr9 already present — leaving it
[INFO] R-50 island ON: local_api=169.254.253.1:8443 (vmbr9); guest net1=169.254.253.2/30; lan_resolver.host_ip=10.0.2.15
[INFO] node=drill-pve local_api=169.254.253.1:8443 tls_fp=F7:CC:33:E6:CA:F5…
[OK] wrote /etc/felhom-agent/agent.json (0600 felhom-agent)
[OK] agent --selftest (read-only) passed
[OK] felhom-agent service active (non-root felhom-agent reads the config OK)
[STEP] 7/8 golden archive
[WARN] ignoring the local golden local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst — it is controller 0.98.3, but the vouched golden is 0.213.0
[WARN] fetching the vouched golden instead (this is what the manifest is for)
[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
[INFO] fetching golden v0.213.0 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_08_12-17_31_32.tar.zst
[OK] verified sha256 77429bfc7e39ffc8… matches the hub manifest
[OK] golden imported + verified: local:backup/vzdump-lxc-9100-2026_08_12-17_31_32.tar.zst
[STEP] 8/8 provision guest 120
[SKIP] pool felhom already exists
=== felhom-agent 0.128.0 selftest=provision (vmid=120 customer=drill-r50 hostname=drill-r50) ===
--- front half: bring-up (provision) local:backup/vzdump-lxc-9100-2026_08_12-17_31_32.tar.zst → vmid 120 ---
time=2026-08-12T17:32:57.812+02:00 level=INFO msg="bring-up: pool membership re-asserted" vmid=120 pool=felhom
[OK] front half: vmid 120 up (boot+running) in 1m10s; MAC=BC:24:11:D3:D7:8C
--- back half: mint per-guest token + populate bootstrap config mount ---
time=2026-08-12T17:33:03.626+02:00 level=INFO msg="provision: back-half complete" vmid=120 mount=mp9 guest_path=/etc/felhom-bootstrap endpoint=169.254.253.1:8443
[OK] back half: bootstrap mount mp9 → /etc/felhom-bootstrap on vmid 120 (host dir /var/lib/felhom-agent/guests/120/bootstrap)
local-api endpoint 169.254.253.1:8443 · leaf fp 0c7b12eec0ab8c4f8e879dba1af0f7203ef82aea92b023d543b7632b865ead58 · token: minted (not printed)
=== selftest=provision OK — guest 120 provisioned + bootstrap-mounted (KEPT) ===
next: reboot the guest → the golden's baked controller-bootstrap unit deploys the controller,
which PULLS its controller.yaml from the hub (retrieval passphrase) and merges in this local_api.
[OK] provision completed
[INFO] rebooting guest 120 so the baked controller-bootstrap unit picks up the mount
[STEP] verify
[OK] pct status: running
[OK] onboot: 1
mp0: local-lvm:vm-120-disk-1,mp=/var/lib/felhom,backup=1,size=70G
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
rootfs: local-lvm:vm-120-disk-0,size=32G
[OK] pool: guest 120 is a member of felhom
[OK] acl: FelhomAgentBase@/ present (user+token)
[OK] acl: FelhomAgentGuest@/pool/felhom present (user+token)
[OK] acl: FelhomAgentStore@/storage/local present (user+token)
[OK] acl: FelhomAgentStore@/storage/local-lvm present (user+token)
[OK] acl: FelhomAgentStore@/storage/felhom-pbs present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990000 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990001 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990002 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990003 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990004 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990005 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990006 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990007 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990008 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990009 present (user+token)
[OK] authz signers: 2 (operator-signed self-update armed)
[OK] controller: Up 24 seconds (healthy) (after ~0s)
[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.213.0
[WARN] cloudflared not visible yet
[INFO] (confirm in the hub UI that host drill-r50-0a4f9a reports guest 120)
[OK] Day-0 provision SUCCESS — vmid=120 host_id=drill-r50-0a4f9a customer=drill-r50 golden=local:backup/vzdump-lxc-9100-2026_08_12-17_31_32.tar.zst
[INFO] root@pam was rotated + vaulted at step 4b — retrieve at hub → host page (the old GUI password no longer works).
@@ -0,0 +1,15 @@
=== final drill-r50 state 2026-08-12T15:39:28Z ===
VMID Status Lock Name
120 running drill-r50
--- controller ---
gitea.dooplex.hu/admin/felhom-controller:0.213.0
--- goldens on the box ---
Volid Format Type Size VMID
local:backup/vzdump-lxc-120-2026_08_12-17_24_19.tar.zst tar.zst backup 612587653 120
local:backup/vzdump-lxc-9100-2026_08_12-17_31_32.tar.zst tar.zst backup 656657008 9100
local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst tar.zst backup 612002238 9100
--- dnsmasq ---
enabled
active
udp UNCONN 0 0 127.0.0.1:53 0.0.0.0:* users:(("dnsmasq",pid=15124,fd=6))
udp UNCONN 0 0 10.0.2.15:53 0.0.0.0:* users:(("dnsmasq",pid=15124,fd=4))
+9 -2
View File
@@ -597,10 +597,17 @@ unrelated finding, which is why the sweep's proposals were renumbered to R-159
class (an image `VOLUME` at an unmounted path) is still live — `immich-server` has one today.
| **R-295** | **One name per secret — CONTROLLER HALF SHIPPED.** The claim page called the SAME three-word dashboard code „Beállító kód" on the first-time branch and „Visszaállító kód" on the reset branch, while the TEN-word escrow code is „Helyreállítási kód". Two near-homographs for two different secrets; the collision cost a real code. „Visszaállító kód" is **retired** in the controller (`claim.html` label/subtitle/button, `claim.go` print-reset-code + lockout strings); the name is now constant and the SENTENCE changes. Naming only — pinned by `TestResetCode_StillAcceptedOnTheSetupPage`. **HUB HALF NOT DONE (Part 4a, dropped per the session's own drop order):** the hub's send button „Visszaállító kód küldése", the mail subject „Jelszó-visszaállítási kód", its body „Visszaállító kód:", and the mail sending the customer to an „Elfelejtett jelszó" page while a rebuilt box actually serves „A szerver beállítása" | **PARTIAL — controller shipped v0.211.0; hub half OPEN (S)** | R-294 | Apply the same ruling in `felhom.eu/hub`, and make the mail name the page the machine is actually showing | CC |
| **R-296** | **The orphan card's OTHER sentence makes the same promise, and the spec says it is fine.** `SPEC-orphan-card-copy-2026-08-10.md` §1 lists `backups_remote.html:98` as *"Accurate; keep"*. It is not: it ends *„A meglévő mentések nem sérültek — a hozzájuk tartozó helyreállítási kóddal később **visszaállíthatók lehetnek**."* — the identical unevaluable promise in a different conjugation, which the spec's own regression guard (`visszaállítható lehet`, singular) does not match. Line 101 was fixed; line 98 was left **deliberately unchanged**: the instruction was not to improvise Hungarian at the customer | **CLOSED — shipped in controller v0.212.0 (R-299); verified: the sentence at backups_remote.html:98 was replaced and the stem guard covers it** | R-294 | Needs a wording decision, then the same treatment as line 101 and an assertion covering BOTH conjugations | operator + CC |
| **R-297** | **An install took whatever golden was lying around.** Step 7 of the host installer short-circuited on any local archive with no version compare, no digest and no warning, so the manifest's sha256 — vouching from a different trust root than the code host — was consulted only on the fetch path. Discovery is newest-by-filename: correct by recency, never by verification. A box reinstalled from a stale archive can come back below the version where the off-site recovery screen exists, and below the update floor. Three goldens on `demo-hp` carry controller 0.153.0 and 0.210.0; nothing compared either to the vouched version. **FIXED in `felhom-host-install.sh` v1.26.0** (digest first, then the baked `/etc/felhom-controller-image` tag; auto-discovered mismatch re-fetches, an operator-named mismatch REFUSES; an unreadable manifest refuses). **NOT PUBLISHED and NOT YET OBSERVED FAILING** — the pre-fix stale install was never watched on `drill-r50`, so the `installer-v1.26.0` tag is deliberately not cut | **SHIPPED TO main, UNPUBLISHED (M)** | — | Observe a fresh install taking a stale local golden on `drill-r50` BEFORE publishing; then cut `installer-v1.26.0` and bump both `--ref`s in `webpage.yaml` | CC |
| **R-297** | **An install took whatever golden was lying around.** Step 7 of the host installer short-circuited on any local archive with no version compare, no digest and no warning, so the manifest's sha256 — vouching from a different trust root than the code host — was consulted only on the fetch path. Discovery is newest-by-filename: correct by recency, never by verification. A box reinstalled from a stale archive can come back below the version where the off-site recovery screen exists, and below the update floor. Three goldens on `demo-hp` carry controller 0.153.0 and 0.210.0; nothing compared either to the vouched version. **FIXED in `felhom-host-install.sh` v1.26.0** (digest first, then the baked `/etc/felhom-controller-image` tag; auto-discovered mismatch re-fetches, an operator-named mismatch REFUSES; an unreadable manifest refuses). **OBSERVED LIVE 2026-08-12** on `drill-r50` from `virgin`: a genuinely stale `golden-0.98.3.tar.zst` planted as the newest-by-filename `vzdump-lxc-9100-…-23_59_59.tar.zst`; v1.25.0 logged `manifest: … golden v0.213.0` at step 5 and `[SKIP] using local golden` at step 7, and **the box came up on controller 0.98.3** — confirmed from both `docker ps` and the in-guest `/etc/felhom-controller-image`. That is below the floor (0.213.0) AND below v0.206.0, where the off-site recovery screen exists: the row's sharp end was real. Both fixed shapes then watched — auto-discovered → `ignoring the local golden … it is controller 0.98.3, but the vouched golden is 0.213.0` → re-fetched and sha-verified → landed **0.213.0**; operator-named → **refused**, exit 1 | **CLOSED — observed live + PUBLISHED as `installer-v1.27.0` (both refs bumped)** | — | — | CC |
| **R-298** | **The `/storage` page's unregistered list is filtered by `role==='user-data'`, so a drive that is also the backup target can never be registered from it.** `storage.html:363` routes anything not `user-data` into the read-only protected group with NO actions. On the rebuilt `demo-hp` the NVMe is deliberately BOTH the user-data drive and the `felhom-backup` target (`/etc/pve/storage.cfg`: `dir: felhom-backup``/mnt/nvme-1tb`), so it renders locked. **This is the SECOND reason that page was empty** during the reinstall rehearsal, independent of R-280's candidate-source defect, and R-280's fix does not touch it — attaching is non-destructive, so the format-wizard protection is the wrong gate for a REGISTER action | **READY (S) — NEW 2026-08-10** | R-280 | Split the role gate: `user-data` keeps destructive actions; any mounted role may be REGISTERED | CC |
| **R-299** | **The orphan card's OTHER sentence made the same unevaluable promise, and the spec called it accurate.** `backups_remote.html` line 98 — the **always-visible** half of the card, where the paragraph fixed in v0.211.0 renders only after a click — ended *„a hozzájuk tartozó helyreállítási kóddal később **visszaállíthatók lehetnek**"*. It survived for two independent reasons: `SPEC-orphan-card-copy-2026-08-10.md` §1 listed it as *"Accurate; keep"*, and **the regression guard matched one INFLECTION** (`visszaállítható lehet`, singular) which the plural does not contain. **A guard matching one inflection of a Hungarian verb guards one sentence, not the claim.** Guard broadened to the stem `visszaállíthat`; proven by planting the shipped plural (stem guard convicts, old singular guard does not match it at all). Spec corrected in both places. The two accurate halves kept | **CLOSED — controller v0.212.0** | R-294 | — | CC |
| **R-300** | **Our own uninstall left the thing that makes our own reinstall refuse.** `--uninstall` removed the dnsmasq snippet and RESTARTED the daemon, leaving it enabled and unconstrained on `0.0.0.0:53`; the next byo install's preflight then hard-refuses with *"a resolver is already bound to :53"* and **the customer goes debugging a home network that was never at fault**. Ownership is now RECORDED at preflight (`dnsmasq_preexisting`) — the only moment it is a fact, since the package is installed by the AGENT (`lanresolver.go:107`), not the installer — and honoured at removal: Felhom's → stop+disable; the owner's → restart only; **no record (every box already in the field) → restart only, fail-safe, with the reason and the exact command logged**. The preflight refusal keeps its two routes and its promise not to touch DNS on a host we do not own, and gains the missing line: when the bound resolver is dnsmasq, it says it looks like ours and names the command. **NOT OBSERVED LIVE — the install→uninstall→install cycle on `drill-r50` was not run**, so `installer-v1.27.0` is NOT cut | **SHIPPED TO main, UNPUBLISHED + UNVERIFIED LIVE (M)** | — | Run the cycle on `drill-r50` (quote the second install refusing before the fix, passing after), together with R-297's stale-golden observation — one drill session covers both; only then publish | CC |
| **R-300** | **Our own uninstall left the thing that makes our own reinstall refuse.** `--uninstall` removed the dnsmasq snippet and RESTARTED the daemon, leaving it enabled and unconstrained on `0.0.0.0:53`; the next byo install's preflight then hard-refuses with *"a resolver is already bound to :53"* and **the customer goes debugging a home network that was never at fault**. Ownership is now RECORDED at preflight (`dnsmasq_preexisting`) — the only moment it is a fact, since the package is installed by the AGENT (`lanresolver.go:107`), not the installer — and honoured at removal: Felhom's → stop+disable; the owner's → restart only; **no record (every box already in the field) → restart only, fail-safe, with the reason and the exact command logged**. The preflight refusal keeps its two routes and its promise not to touch DNS on a host we do not own, and gains the missing line: when the bound resolver is dnsmasq, it says it looks like ours and names the command. **OBSERVED LIVE 2026-08-12** on `drill-r50`. Chain re-established from `virgin` (dnsmasq **not-installed**, `:53` **free**) → appliance install → the **agent** installed it, snippet-constrained to `10.0.2.15:53`+`127.0.0.1:53` → pre-fix uninstall → `enabled`/`active` on `0.0.0.0:53`+`[::]:53` → pre-fix byo install **refused, exit 1**. Fixed path from `virgin`: recorded `not present before Felhom` → uninstall logged `stopping + disabling it``:53 FREE` → byo preflight **PASS**. Owner's side also proven: with dnsmasq pre-installed the record read `yes` and the uninstall left it running. **But see R-305 — the cleanup fires only once per machine** | **CLOSED — observed live + PUBLISHED as `installer-v1.27.0` (both refs bumped)** | — | — | CC |
| **R-301** | **The abandon countdown banner makes the retired promise a third time, and as a flat statement.** `controller/internal/web/templates/layout.html:143`: *"A korábbi távoli mentéseidet N nap múlva véglegesen töröljük, a kérésed szerint. **Addig még visszaszerezheted őket a helyreállítási kóddal.**"* Unlike the orphan card this one is not hedged at all. **It is probably TRUE where it renders** — it is gated on `backupMgr.AbandonStatus().Active`, i.e. the customer chose to abandon a recovery offer that exists — which is why it was NOT changed (this session was fenced to the orphan card). **But it renders in `layout.html`, i.e. on EVERY page, and a rebuilt box can have an active abandon countdown WHILE its remote store is orphaned**, so a customer can read *"we cannot determine whether these can be opened"* on the backups page and *"until then you can still retrieve them with your recovery code"* in the banner above it, about copies they have no way to tell apart. **Not established:** whether the two sentences refer to the same bytes. The spec anticipated this in its §4 note | **CLOSED — premise CONFIRMED and fixed in controller v0.213.0 (R-302)** | R-299 | Establish whether the two can co-render for one customer and whether they name the same copies; if so, make the banner name WHICH copy | operator + CC |
| **R-302** | **The abandon banner promised retrieval it could not see was still true — fixed by PINNING a fingerprint at the decision.** The countdown strip said *„Addig még visszaszerezheted őket a helyreállítási kóddal"* unconditionally, on every page, and R-301 established it can co-render with the orphan card correctly saying we cannot tell. **THE OBVIOUS CONDITION WAS REJECTED AND HERE IS WHY, so nobody re-proposes it:** the proxy *"does the hub hold a key different from the one I use?"* asks about the WRONG key — the set-aside copies were written under an OLDER key the box no longer has, which is why they were set aside — so on a twice-rebuilt box it answers "promise it" about copies no key on file can open. **Demonstrated, not argued:** under the proxy both Scenario B (package replaced) and Scenario D (legacy countdown) flip back to promising. Instead `startAbandonCountdown` pins `AbandonPinnedEscrowKeySHA256` — the hub's escrow key fingerprint AS CACHED AT THE DECISION — written once, never refreshed; the banner then asks only *"is the hub still holding that same package?"*. **⚠ THE PIN IS A RECORDED ASSUMPTION:** nothing on the box records which key wrote the set-aside copies, so it presumes the package held at the decision is that one — true in the ordinary rebuilt-box story, NOT provable, wrong on a twice-rebuilt box. Recorded in the field comment so it can be narrowed rather than hardening into a fact. Empty is not a match on either side; a countdown started before v0.213.0 carries no pin and takes the cautious branch (deliberately NOT backfilled). **A template sweep found a FOURTH instance** (the backups-page abandon block — same condition applied, since fixing the strip and not the page would leave them contradicting) **and a FIFTH** (the abandon confirmation screen, deliberately LEFT: it renders at the moment of the decision, where the promise is true by construction because that is the package about to be pinned) | **CLOSED — controller v0.213.0** | R-301 | — | CC |
| **R-303** | **`markOrphaned` has no guard against an active abandon countdown — the co-render is made HARMLESS, not IMPOSSIBLE.** `ensureOffboxRepo` calls `markOrphaned()` for a claimed box (`offbox.go:804`) with no check on `AbandonAt`, so a later run finding the FRESH store unopenable re-raises the orphan card while the countdown runs. R-302 ensures the two surfaces no longer contradict each other in that state, but the state itself is still reachable and is arguably incoherent: a box counting down to deleting its old history while simultaneously reporting its NEW history is unopenable is in trouble in two ways at once and says so in two separate cards. **Ranked LOW deliberately** — it is a coherence question, not a correctness one, and the wrong fix (suppressing the orphan card during a countdown) would hide a real second fault | **READY (S) — NEW 2026-08-12** | R-302 | Decide whether a countdown should suppress, defer or annotate the orphan card — and beware hiding a genuine second failure | operator + CC |
| **R-304** | **The retained escrow key works, and the customer is told their correct code is wrong.** DRILL 2026-08-12 answered the three questions separately, on `demo-felhom`, with planted data. **(a) retention: WORKS** — the first retained row in fleet history to carry material (`host_escrow_superseded` id 11, `identity_blob` 572 B), byte-identical (`sha256 a10032341c8584ed…`) to the pre-supersession `host_escrow` row. **(b) the material opens the old store: YES** — unsealed with the OLD recovery code it yielded a password byte-identical to the pre-change one (`sha c60c8bc737a6b7c6…`), and restored three planted files **byte-identical** from a store the box itself could no longer open (negative control first: `Fatal: wrong password or no key found`), **including a Hungarian accented filename verified as raw bytes**. **(c) the customer's route: DOES NOT EXIST, and misinforms.** `ListSupersededEscrow` (`store.go:2841`) is the only reader of a retained `identity_blob` and has **zero production callers** — five call sites, all `_test.go`; the product path (`POST /escrow/recover-offsite-password``FetchIdentityEscrow``GetHostDRBundle`, `store.go:3152`) selects `FROM host_escrow` — the CURRENT row only. Asked for the old password with the code that demonstrably opens the retained row, the product answered **"the recovery code did not open the sealed bundle — nothing was written"**. **This is the R-224 class again**: there an unreachable hub was reported as a bad code; here a VALID code for retained history is reported as a bad code, and the customer's attempt ends there. **Consequence:** the census answer stands (it was about retention); the countdown banner's promise is true in substance and false in practice; **any capability-map claim that the customer can recover the old history with their recovery code is false today and must move** | **READY (L) — NEW 2026-08-12, RANK 1** | R-198, R-199, R-224, R-241 | Decide the shape: serve retained rows on the recovery path (needs a "which package?" choice — a customer may have several), or stop promising retrieval anywhere the customer cannot perform it. **Until one of those, the honest position is that retention is an operator-only capability.** At minimum, the refusal must stop asserting the code is wrong when the hub simply never looked | operator + CC |
| **R-305** | **The R-300 cleanup fires exactly once per machine, and the second reinstall hits the original wall.** PROVEN on `drill-r50` 2026-08-12, not deduced. The uninstall stops+disables dnsmasq but deliberately never **purges** the package, so on every later install `dpkg-query` reports it installed, preflight records `dnsmasq_preexisting=yes`, the agent's `EnsureDnsmasq` re-enables it, and the uninstall then only *restarts* it — leaving it `enabled`/`active` on `0.0.0.0:53` exactly as before the fix. Cycle 1: `no` → disabled → `:53 FREE` → next install PASSES. Cycle 2: `yes` → left running → **cycle-3 byo preflight FAILED, exit 1**. The improved refusal does fire and names the command, so the customer is not stranded — but they must run by hand a cleanup Felhom could have run itself, on a box Felhom does own | **READY (M) — NEW 2026-08-12, RANK 2** | R-300 | Distinguish "the package pre-dates Felhom" from "a previous **Felhom** install left it" — e.g. keep the ownership record outside `state.json` (which the uninstall deletes), or record Felhom-installed-ness where the agent installs it. **Do not fix by purging on hosts we may not own** | CC |
| **R-306** | **`--preflight-only` says "no state written" and writes state — with an answer that can be wrong.** `_state_put` short-circuits on `DRY_RUN` only (`felhom-host-install.sh:418`), so a preflight-only run creates `/var/lib/felhom-install/state.json`. Observed live: after a run whose banner read `PRE-FLIGHT PASS (mode=byo) — no state written, no install step executed`, the file existed containing `{"completed": [], "dnsmasq_preexisting": "yes"}`. Both the banner and the flag's own comment at line 226 assert the opposite. **The harm is not the file, it is the value**: the runbook recommends preflight-only first, then the same command without the flag, so on a box carrying a Felhom leftover the wrong ownership answer is baked in before the real install begins | **READY (S) — NEW 2026-08-12, RANK 3** | R-300, R-305 | Either make `_state_put` a no-op under `PREFLIGHT_ONLY` (and record ownership at install instead), or correct both claims. A comment asserting an invariant needs a test pinning it | CC |
| **R-307** | **`demo-felhom` carries a LIVE abandon countdown that this drill did not start — and the end state says there should be none.** `settings.json`: `abandon_started_at 2026-08-10T08:06:31Z`, `abandon_at 2026-08-24T08:06:31Z`, `abandon_repo_path /home/felhom-repo.orphaned-20260810`. The drill's fence forbade starting, shortening or triggering a countdown, and none was; but its required end state was *"no abandon countdown anywhere"*, and one exists. **The two ways to satisfy that are not equivalent and the choice is the operator's:** cancel it (the orphaned repository is kept indefinitely — storage cost, no data risk) or let it run (on **2026-08-24** that repository is deleted, **irreversibly**). **Doing nothing selects deletion by default.** Deliberately not resolved by CC | **BLOCKED ON OPERATOR — NEW 2026-08-12** | R-301, R-302, R-303 | Decide before 2026-08-24 | operator |
| **R-308** | **The stored controller password no longer opens `demo-felhom`.** `PASSWORD` in `~/.config/credentials` on DooPlex returns `Hibás jelszó` against `https://127.0.0.1:443/login` with `Host: felhom.demo-felhom.eu`. **NOT the known quoting trap** — the surrounding quotes were stripped before use and the stored value is unquoted (16 bytes for 15 chars), so this is a genuinely stale credential, not a mis-read. Cost during the drill: the dashboard was unreachable headlessly, so the offsite push had to be made with `restic` directly instead of the page's own button, which left the app-backup→unit→offsite chain unexercised. `R_DEMO-FELHOM` **was** correct and has since been rotated by this session's final ceremony (updated in place; prior file backed up alongside) | **READY (S) — NEW 2026-08-12, RANK 2** | — | Recover or reset the demo-felhom dashboard password and re-store it; note which boxes share one and which do not | operator |
| **R-309** | **The day-0 runbook says pushing the installer publishes it. It has not since R-110.** `documentation/runbooks/day0-install.md` §C.1: *"it is always the current `main` script. There is no release tag, no staging copy and no version selector; pushing `scripts/felhom-host-install.sh` publishes it."* R-110 moved `/scripts/` to the **installer tag's** tree — `manifests/webpage.yaml` carries **two** `--ref=installer-v…` pins (lines 327, 372) and `hostinstall_gates.py` gate 6 enforces that they name a tag. Confirmed live 2026-08-12: `https://felhom.eu/scripts/felhom-host-install.sh` served `SCRIPT_VERSION="1.25.0"` while `main` had `1.27.0`, three and a half hours after the push. **The runbook's version is the dangerous one to believe in both directions** — it would have an operator think a pushed fix is live when it is not, and think a pushed mistake is live when it is not | **READY (S) — NEW 2026-08-12, RANK 3** | R-110 | Correct §C.1 to name the tag-and-manifest publication step (as performed today) | CC |
| **R-310** | **Two small edges on the installer, neither costing more than a moment.** (1) The R-297 operator-named refusal states the vouched version twice in consecutive sentences (*"…but the vouched golden is 0.213.0. The vouched golden is 0.213.0."*). (2) `--uninstall` reads its typed vmid confirmation from `/dev/tty` and `--force` deliberately does **not** bypass it, so teardown cannot be scripted without a pty — correct for an irreversible destroy, but undocumented; it surfaces as `line 891: /dev/tty: No such device or address` and an rc=1 that looks like a failure rather than a refusal to proceed unattended | **READY (S) — NEW 2026-08-12, RANK 4** | R-297 | Drop the duplicated sentence; add one runbook line naming the pty requirement | CC |