DRILL: the retained key works, and the customer cannot reach it
gates / gates (push) Successful in 23s

Three verdicts, kept separate because collapsing them is how this assumption
survived a week.

(a) The material IS retained. host_escrow_superseded id 11 is the first retained
row in fleet history to carry identity_blob (572 B), byte-identical to the
pre-supersession row (sha256 a10032341c8584ed...).

(b) The retained material DOES open the old store. Unsealed with the old recovery
code it yielded a password byte-identical to the pre-change one, and restored
three planted files byte-identical from a store the box itself could no longer
open - including a Hungarian accented filename verified as raw bytes. Negative
control ran first and failed closed.

(c) The customer has NO route, and is misinformed. ListSupersededEscrow has zero
production callers; the recovery path selects FROM host_escrow. Asked with the
code that had just worked by hand, the product answered "the recovery code did
not open the sealed bundle". A valid code for retained history is reported as a
bad code - the R-224 class again. R-304, rank 1.

Both installer faults were watched happening first, so installer-v1.27.0 is now
published (tag + both webpage.yaml refs). Pre-fix: the box came up on controller
0.98.3 against a vouched 0.213.0, below the floor and below the version carrying
the recovery screen; and our own uninstall left dnsmasq on 0.0.0.0:53 so our own
next install refused. R-297 and R-300 CLOSED.

Also filed R-305 (the dnsmasq fix fires once per machine - the leftover returns
on the second reinstall, proven), R-306 (--preflight-only writes state it says it
does not), R-307 (a live abandon countdown on demo-felhom, firing 2026-08-24 -
operator decision), R-308 (stored controller password stale), R-309 (the day-0
runbook's publication claim has been false since R-110), R-310 (two edges).

Ceiling R-303 -> R-310. Capability map moved: the retention claim is now marked
operator-only. Phase A logs did not survive the intermediate revert; recorded.
This commit is contained in:
2026-08-12 17:41:56 +02:00
parent fbe1155fbb
commit 1d5f2b8bb6
12 changed files with 2445 additions and 90 deletions
@@ -0,0 +1,514 @@
# DRILL — the retained key, and the two fixes nobody had watched work
**Date:** 2026-08-12 · **Class:** drill (unattended, destructive on Tier 0; spike for Phase C's first step)
**Venues:** `drill-r50` (nested PVE on DooPlex) · `demo-felhom` (guest 9201 on felhom-pve) — both Tier 0
**Baselines re-confirmed live on arrival, not assumed:** controller **0.213.0** on both demo boxes
(`pct exec 9201 -- docker ps`), agent **0.128.0** (hub `hosts`), hub **0.102.0** (deployment image),
register ceiling **R-303** (grep), `felhom.eu` clean at `fbe1155f` and level with `origin/main`.
---
## 1. The three verdicts, first, in plain language
> **(a) Is the material retained? — YES.** Proven for the first time in the fleet's history.
>
> **(b) Does the retained material, once unsealed, yield a password that actually opens the old
> store? — YES.** Three planted files came back byte-identical from a store the box itself could no
> longer open, including the Hungarian accented filename.
>
> **(c) Does the customer have a route to (b) through the product? — NO. And the failure is worse
> than absence:** the customer's old recovery code — the one that demonstrably opens their old
> backups — is rejected by the product with *"the recovery code did not open the sealed bundle"*.
> They are told their correct code is wrong, and given no next step.
**The headline the operator asked for.** The brief said to be ready for the answer to be no, and our
own records predicted the retention would be *"a box we fill and cannot open"*. That prediction was
**half right, and the wrong half was the one nobody checked.** The box can be opened — the crypto,
the retention and the copy are all sound. What does not exist is the door: no code path reads a
retained row, so the only way in is an operator with SQLite, `age` and a shell. Everything I did in
§4 a customer cannot do.
That distinction matters for the three pieces of work resting on this:
| Rests on | Verdict |
|---|---|
| The census answer ("nobody else is exposed") | **Stands.** It was a statement about retention, and retention works. |
| The countdown banner's promise (old backups stay recoverable) | **True in substance, false in practice.** The bytes are recoverable; the customer cannot recover them. The promise is made to someone who has no way to act on it. |
| The capability map's recovery claims | **Must move.** Any claim of the form "the customer can recover the old history with their recovery code" is false today. |
---
## 2. §7.0 — the probe, run first and read-only
Run before anything was installed or planted, exactly as the order required.
**Q1 — does any host today hold a superseded record with key material present? NO. Zero, fleet-wide.**
```
host_id kfp16 blob_len identity_blob superseded_at
demo-hp-bb76ea 3f:4f:65:c0:d8:f 383 NULL 2026-08-04 07:15:36
demo-felhom-8363b5 7e:a6:af:f7:ea:6 383 NULL 2026-08-04 07:20:08
```
Both are the pre-R-198 rows written hours before the fix. `SELECT COUNT(*) … WHERE identity_blob IS
NOT NULL` returned **0**. The retention as fixed had never once been exercised.
*Instrument:* hub SQLite copied WITH `hub.db-wal` + `-shm` (the standing gotcha). Freshness proved by
a **positive observable** — newest `host_reports` row 61 s old — not by "the query returned no error".
`PRAGMA integrity_check` = ok.
**Q2 — what act produces a retaining supersession? Two, both read from source:**
- `hub/internal/store/store.go:2807``SaveHostEscrow` demotes when the newly sealed
`restic_pw_sha256` **differs** from the stored one.
- `hub/internal/store/store.go:2636``DeleteHost` demotes the current row unconditionally.
**Q3 — is there any code path that USES a retained record to recover a password? NO.**
The product's recovery route is real and wired (R-199) but reaches the **current** row only:
```
controller → agent POST /escrow/recover-offsite-password
→ escrow.OffsiteKeyRecoverer.Fetch felhom-agent/cmd/felhom-agent/main.go:1757
→ hubClient.FetchIdentityEscrow
→ hub GET identity-escrow handler hub/internal/api/handler.go:1325
→ store.GetHostDRBundle hub/internal/store/store.go:3152
SELECT blob, identity_blob, directive_json FROM host_escrow WHERE host_id = ?
```
`ListSupersededEscrow` (`store.go:2841`) is the **sole** reader of a retained `identity_blob`. It has
**zero production callers** — five call sites, all in `_test.go`. `CountSupersededEscrow` is used
twice and only to render a *count* (`api/handler.go:1209`, `web/hosts.go:572`).
**Feasibility verdict: producible.** Phase C proceeded.
---
## 3. Phase C — producing the state
### 3.1 The planted dataset
Planted in `/mnt/sys_drive/felhom-data/userdata/drill-20260812` on `demo-felhom`.
| file | sha256 | bytes |
|---|---|---|
| `payload.bin` | `6863824a67bf6cb636defca3cbddc1d58092bcdda3af939224dfef4663c15359` | 65536 |
| `plain-ascii.txt` | `e294e0ee5b8fd131dc0fa71f5cf1253dc39473241aa090be36404a568285c83a` | 56 |
| `árvíztűrő-tükörfúrógép.txt` | `a1957a87104c63be22b34e4f937e700edcdc925da5984bfcca407df4989f7190` | 37 |
**The accented filename never crossed a shell as text** (rule 6). It was built from explicit bytes and
verified as hex, both as intended and as listed on disk — identical:
```
c3 a1 72 76 c3 ad 7a 74 c5 b1 72 c5 91 2d 74 c3 bc 6b c3 b6 72 66 c3 ba 72 c3 b3 67 c3 a9 70 2e 74 78 74
```
### 3.2 Positive control on the comparator — done BEFORE it was relied on
One byte of a copy of `payload.bin` flipped at offset 40000:
```
unaltered f5de179b5ca941645819a879187db953db5c973d4c1a07b2a3ba96d3d79bfb6c
altered 515a65d96326bd847a87ea35c9fb5e6afebc9499ec0c1cd80ca3531b22cb2324
payload.bin: FAILED · sha256sum: WARNING: 1 computed checksum did NOT match
```
The comparator convicts. The copy was discarded and the original re-hashed unchanged.
### 3.3 The store, written under P1
`restic backup` into demo-felhom's **own** repository path
(`sftp:u629488-sub1@…your-storagebox.de:/home/felhom-repo`) → snapshot **`6ea85413`**, confirmed by
identity and by **listing its contents**, not by the success line:
```
6ea85413 2026-08-12 15:15:33 demo-felhom felhom-offbox,drill-retained-key-20260812
…/drill-20260812/payload.bin
…/drill-20260812/plain-ascii.txt
…/drill-20260812/M-CM-!rvM-CM--ztM-EM-1rM-EM-^Q-tM-CM-<kM-CM-6rfM-CM-:rM-CM-3gM-CM-)p.txt
```
**Key fingerprint of the store's password (P1): sha256 `c60c8bc737a6b7c6…`** — value never recorded.
### 3.4 The supersession, by the product's own ceremony
The repository password was replaced (P1 → P2, `d4aac4f1c0ef9ac0…`), making `/home/felhom-repo` the
**old** store, then the ceremony was driven through the same local-API calls the dashboard wizard
makes: `POST /escrow/stage-secret``POST /escrow/ceremony``…/status``…/claim`.
```
phase=done restic_pw_sealed=true uploaded=true
```
### 3.5 (a) — the material IS retained
The hub grew its first ever retained row carrying key material:
```
id host_id identity_blob restic_pw_sha256 superseded_at
11 demo-felhom-8363b5 572 c60c8bc737a6b7c6… 2026-08-12 15:18:55
```
Confirmed **by identity and byte length**, not by presence:
```
a10032341c8584edfd87ad941a50d9fd6f34c23a4802f55885cb1d4cab086bd3 current-identity.age (pre-supersession)
a10032341c8584edfd87ad941a50d9fd6f34c23a4802f55885cb1d4cab086bd3 retained-identity.age (retained row 11)
572 bytes both — IDENTICAL
```
### 3.6 (b) — the retained key DOES open the old store
**Negative control first** — the box's current password against the old store:
```
restic exit code: 1
Fatal: wrong password or no key found
```
**Then the retained key.** Retained blob → `age -d` with the old recovery code → bundle →
`restic_repo_password`, sha256 `c60c8bc737a6b7c6…`, **byte-identical to the P1 recorded before the
supersession**:
```
restic snapshots → exit 0, snapshot 6ea85413 listed
restic restore → exit 0
restored accented filename bytes: c3 a1 72 76 c3 ad 7a 74 c5 b1 72 c5 91 2d … (identical)
payload.bin: OK · plain-ascii.txt: OK · árvíztűrő-tükörfúrógép.txt: OK (compare exit code 0)
```
**All three files byte-identical, from a store the box itself could not open.**
### 3.7 (c) — the customer has no route, and is actively misinformed
The product's own recovery endpoint was asked for the old password, using the same old recovery code
that had just worked by hand:
```
OLD code (opens the retained row by hand → P1) REFUSED: "the recovery code did not open the
sealed bundle — nothing was written"
NEW code (today's ceremony) RETURNED a password, sha d4aac4f1c0ef9ac0 = P2
```
P2 does not open the old store. **The one code that works is reported as the code that does not.**
This is the R-224 defect class in a new guise: there, an unreachable hub was reported as a bad
recovery code; here, a *valid code for retained history* is reported as a bad code. The message is
not merely unhelpful — it is wrong, and it terminates the customer's attempt.
---
## 4. Every step taken off the customer's path
Stated plainly, because (b)'s "yes" is worth exactly as much as this list is short.
| Step | Off-path how | Cost to the walk's fidelity |
|---|---|---|
| Read `identity_blob` out of the hub's SQLite | No API serves a retained row — **this is (c)** | None to the finding; it *is* the finding |
| `age -d` via a hand-written pty wrapper on DooPlex | The product's unwrap is reachable only for the CURRENT blob | None — same `age -p` primitive the agent uses |
| `restic` invoked directly with the recovered password | The controller's recovery path **compares**, it never installs | None — same restic, same repo, same credentials |
| Recovery code taken from the operator's credentials file | A customer would read it from their card | None — a customer plausibly has it |
| Planted dataset pushed with `restic backup` directly | The dashboard button was unreachable (see F6) | The unit-snapshot chain was not exercised; the *store* is identical |
| `--skip-provision` + `--force` on Phase A installs | Saved a guest restore per cycle | None — dnsmasq handling is independent of provisioning |
---
## 5. Phase A — our removal undoes what our installation did
**Venue `drill-r50`, reverted to `virgin` before each clean run.**
### 5.1 The chain re-established, not assumed
Virgin baseline: `dnsmasq` **not-installed**, unit **not-found**, **`:53` free**. After an appliance
install the **agent** installed dnsmasq (`internal/lanresolver`, not the script) and the Felhom
snippet **constrained** it to `10.0.2.15:53` + `127.0.0.1:53`.
### 5.2 The wrong outcome, demonstrated first
Pre-fix uninstall (v1.26.0) — snippet removed, `README` only left, and:
```
enabled: enabled · active: active
udp UNCONN 0.0.0.0:53 · udp UNCONN [::]:53 · tcp LISTEN 0.0.0.0:53 · tcp LISTEN [::]:53
```
Pre-fix byo install, **exit 1** — the refusal the publication had been waiting for since Sunday:
```
[ERROR] a resolver is already bound to :53 on this host:
udp UNCONN 0 0 0.0.0.0:53 … users:(("dnsmasq",pid=4934,fd=4)) …
[ERROR] a resolver is already bound to :53 on this host — Felhom needs the guest reachable by name on your LAN.
Stop or reconfigure that resolver, OR point your LAN DNS at the guest's address, then re-run.
(Felhom does NOT touch DNS services on a host it does not own — this is a refusal, not a change.)
[ERROR] PRE-FLIGHT FAIL (exit 1) — fix the finding above and re-run
```
Nothing in it suggests the resolver is Felhom's own leftover.
### 5.3 The pass
Fixed path (v1.27.0) from virgin: preflight recorded `dnsmasq: not present before Felhom`; uninstall
logged `dnsmasq was installed by Felhom (recorded at install) — stopping + disabling it`; result
`enabled: disabled · active: inactive · :53 FREE`; and the second install:
```
[INFO] host DNS (:53): free
[OK] pre-flight passed
[OK] PRE-FLIGHT PASS (mode=byo) — no state written, no install step executed
```
### 5.4 Both sides of the ownership record
- **`no` (Felhom's):** stop + disable. Quoted above.
- **`yes` (the owner's):** `dnsmasq pre-dates Felhom (recorded at install) — leaving it running,
restarting only` → left `enabled`/`active`. **Felhom does not disable a resolver it did not
install.** Correct.
- **No record (the field case):** every box installed before this change. Exercised by running the
fixed byo preflight on a box whose state file the uninstall had already deleted — the improved
refusal fired (§5.5).
### 5.5 The improved refusal, judged
```
THIS LOOKS LIKE OURS. A previous Felhom install leaves the dnsmasq PACKAGE installed and its unit
enabled (only our config snippet is removed), and unconstrained it binds 0.0.0.0:53 — which is what
this gate is seeing. If this host had no dnsmasq before Felhom, clear it with:
systemctl disable --now dnsmasq
Then re-run this installer. If dnsmasq is YOURS, leave it and use one of the two routes above.
```
It names the leftover, gives the exact command, and does not overclaim — it says *looks like*, and
leaves the owner's case intact. **Good.** One weakness: it asks the reader to answer *"did this host
have dnsmasq before Felhom?"* — a question the installer itself now records the answer to, but in a
state file the uninstall has already deleted. English throughout, consistent with the rest of the
installer (this surface is operator/tester-facing, not the customer dashboard).
### 5.6 The finding this phase was not looking for — the fix works exactly once per machine
Proven, not deduced. Cycle 2 on the same box:
```
install : [INFO] dnsmasq: already installed BEFORE Felhom — recorded; uninstall will not touch it
uninstall: [INFO] dnsmasq pre-dates Felhom (recorded at install) — leaving it running, restarting only
after : enabled: enabled · active: active · 0.0.0.0:53
cycle-3 byo preflight: PRE-FLIGHT FAIL (exit 1) — "a resolver is already bound to :53"
```
**Why:** the uninstall disables the unit but never **purges** the package (deliberately — purging on
a host we may not own is the wrong blast radius). So on every later install `dpkg-query` says
installed, preflight records `yes`, the agent re-enables it, and the uninstall then only restarts it.
The wall returns on the second reinstall. → **F2**
---
## 6. Phase B — an install takes the image that was approved
**Venue `drill-r50`, reverted to `virgin`.** A genuinely stale golden (`golden-0.98.3.tar.zst`,
controller **0.98.3**, marker verified before use) was placed as
`local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst` — newest by filename, which is exactly what
discovery sorts on.
### 6.1 The fault, observed
Pre-fix (v1.25.0, the published tag) reported both facts and used the wrong one:
```
[INFO] manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.213.0
[STEP] 7/8 golden archive
[SKIP] using local golden: local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
[OK] Day-0 provision SUCCESS — vmid=120 … golden=local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
```
No digest, no version compare, no warning. **The box came up on controller 0.98.3** — confirmed twice,
from the running container and from the in-guest marker:
```
gitea.dooplex.hu/admin/felhom-controller:0.98.3 (docker ps)
gitea.dooplex.hu/admin/felhom-controller:0.98.3 (/etc/felhom-controller-image)
```
**The sharp end (§6 item 4).** Vouched golden **0.213.0**; floor **0.213.0**; the box landed on
**0.98.3** — below the floor, and below the version carrying the off-site recovery screen. A box born
this way cannot run the ceremony its own data depends on, and is born below the update floor. **The
row was right.**
### 6.2 Both fixed shapes
**Explicitly named archive → refuses** (exit 1):
```
[STEP] 7/8 golden archive
[ERROR] refusing the golden you named (local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst):
it is controller 0.98.3, but the vouched golden is 0.213.0.
The vouched golden is 0.213.0. Either pass the archive that matches it,
or re-run with --force-gitea-golden to fetch the vouched one from Gitea.
```
**Auto-discovered archive → warns and re-fetches:** see §6.3.
No local goldens were deleted and teardown's keep-behaviour was not changed.
### 6.3 Auto-discovery result
```
[STEP] 7/8 golden archive
[WARN] ignoring the local golden local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
— it is controller 0.98.3, but the vouched golden is 0.213.0
[WARN] fetching the vouched golden instead (this is what the manifest is for)
[INFO] fetching golden v0.213.0 from Gitea → …/vzdump-lxc-9100-2026_08_12-17_31_32.tar.zst
[OK] verified sha256 77429bfc7e39ffc8… matches the hub manifest
```
**Landed on 0.213.0**, confirmed from both the running container and the in-guest marker.
**Observation, not a fault:** the re-fetched archive is named by its real timestamp
(`17_31_32`), which sorts *below* the planted stale one (`23_59_59`). So the stale archive stays the
newest-by-filename and is re-discovered — and correctly re-rejected — on every subsequent install.
The fix is not defeated; it just never displaces the bad archive, so the warning recurs forever.
---
## 7. Findings, ranked by what they cost the person in front of you
| # | Rank | Finding |
|---|---|---|
| **F1** | **1 — stops the visit** | The retained key has **no product route**, and the product tells a customer holding the **correct** old recovery code that it *"did not open the sealed bundle"*. Recoverable data, valid code, and a dead end with no next step. (§3.7) |
| **F2** | 2 — costs time | The R-300 cleanup fires **only on a machine that never had Felhom**. The package is never purged, so every later install records `dnsmasq_preexisting=yes` and the leftover returns on the second reinstall. (§5.6) |
| **F3** | 3 — misleads | `--preflight-only` claims *"no state written"* (banner and the flag's own comment) but **writes `/var/lib/felhom-install/state.json`** — `_state_put` short-circuits on `DRY_RUN` only. Worse, on a box carrying a Felhom leftover it records `dnsmasq_preexisting=yes` **before** the real install, baking in the wrong ownership answer. |
| **F4** | 3 — misleads | The pre-fix uninstall's closing note called dnsmasq a *"system package"*. On these boxes Felhom installed it. (Already corrected in v1.27.0's wording; confirmed live.) |
| **F5** | 4 — cosmetic | The shape-2 refusal states the vouched version twice in consecutive sentences. |
| **F6** | 2 — costs time | The stored `PASSWORD` credential no longer opens **demo-felhom**'s dashboard (`Hibás jelszó`). Not the known quoting trap — quotes were stripped and the value is unquoted. The dashboard was unreachable headlessly for this drill. |
| **F7** | — operator decision | **`demo-felhom` has a live abandon countdown**: `abandon_started_at 2026-08-10T08:06:31Z`, `abandon_at 2026-08-24T08:06:31Z`, `abandon_repo_path /home/felhom-repo.orphaned-20260810`. Not started by this drill. See §9. |
| **F8** | 4 — cosmetic | `--uninstall` requires `/dev/tty` for the typed vmid confirmation when a guest exists, and `--force` does **not** bypass it (deliberate, and correct for an irreversible destroy) — but it means teardown cannot be scripted without a pty. Worth one line in the runbook. |
**A drill that finds nothing is a suspicious result.** This one found eight, and the instrument was
demonstrably working: the comparator was shown convicting a one-byte change before it was trusted
(§3.2), the negative control on the old store failed closed before the positive one succeeded (§3.6),
and both installer faults were reproduced from a virgin machine before either fix was watched.
---
## 8. Publication
Both Phase A's and Phase B's faults were observed live. **The reward is earned.**
- Phase A fault: observed (§5.2) and fixed path quoted (§5.3).
- Phase B fault: observed (§6.1) — the box came up on 0.98.3 against a vouched 0.213.0.
**`installer-v1.27.0` cut, covering both fixes** (R-297 stale-golden comparison, R-300 removal
leftover — the only two commits touching the installer since `installer-v1.25.0`).
**And publishing is not what the day-0 runbook says it is.** §C.1 states the URL *"is always the
current `main` script … pushing `scripts/felhom-host-install.sh` publishes it."* That has been false
since R-110: `manifests/webpage.yaml` runs two git-syncs, and `/scripts/` follows the **installer
tag**, not `main`. Measured before publishing — `https://felhom.eu/scripts/felhom-host-install.sh`
served `SCRIPT_VERSION="1.25.0"` while `main` carried `1.27.0`, three and a half hours after the push.
So the fixes were genuinely unpublished, as the brief assumed — but for a reason the runbook denies.
Publication here therefore meant: cut the tag **and** bump **both** `--ref`s (sidecar at line 327 and
init container at line 372). → **R-309**
---
## 9. The countdown — an operator decision, deliberately not taken
`demo-felhom` carries a **running abandon countdown**, started **2026-08-10**, firing **2026-08-24**,
for `/home/felhom-repo.orphaned-20260810`. **This drill did not start, shorten or trigger it**, per
the fence.
But §9 of the brief requires an end state of *"no abandon countdown anywhere"*, and one exists. The
two ways to satisfy that are not equivalent and the choice is not mine:
1. **Cancel it** — the orphaned repository is kept indefinitely (storage cost, no data risk).
2. **Let it run** — on 2026-08-24 the orphaned repository is **deleted, irreversibly**.
Doing nothing selects option 2 by default. **Flagged, not resolved.**
---
## 10. Teardown — four layers
| Layer | State |
|---|---|
| **The machine** | `demo-felhom`: planted dataset and working directory removed; eight secret-bearing files **shredded**, not unlinked; `repo_password` restored to P1 (`c60c8bc737a6b7c6…`); escrow re-sealed and uploaded; offsite repo reachable (`restic snapshots` exit 0, 4 snapshots). The agent's `escrow-stage/` directory is **empty** — the ceremony wiped the staged secret (positive observable). `demo-hp`: **not touched at any point**. `drill-r50`: see §11. |
| **The host** | `felhom-pve`: no drill artefacts left outside guest 9201. `drill-r50` host: see §11. |
| **The hub** | Two new retained rows (id 11 = the P1 blob, id 12 = the P2 blob), both `identity_blob` 572 bytes — **deliberately kept as the fixture that proves the retention works**; they are append-only, tiny, and exactly what the design intends. `drill-r50-0a4f9a` host record: pre-existing since 2026-07-25, **re-used rather than duplicated** — no new hub-side sprawl. `customer_configs` gained nothing. |
| **The off-site side** | Only `demo-felhom`'s own repository path was touched. **No `prune`, no `forget`, no delete, no rename** — anywhere, at any point. One snapshot was **added**: `6ea85413`, 66 KiB, tagged `drill-retained-key-20260812`. **Deliberate residue** — removing it would have meant a `forget` on the endpoint, which the fence forbids in spirit; the operator can remove it by ID if desired. Nothing outside `/home/felhom-repo` was read or written; `peti-felhom` was never contacted. |
**How I know the off-site side is untouched outside this machine's path:** every restic invocation in
this drill went through one wrapper pinned to
`sftp:u629488-sub1@u629488-sub1.your-storagebox.de:/home/felhom-repo` — a per-customer sub-account
whose credentials reach nothing else — and the only mutating verb used was `backup`.
---
## 11. Close of run
### 11.1 Wall clocks (CEST)
| Phase | Span | Elapsed |
|---|---|---|
| §7.0 probe (read-only, source + hub) | 16:42 → 16:52 | **~9 min** |
| Phase A — removal leftover | 16:50 → 17:12 | **~22 min** (4 installs, 4 uninstalls, 3 preflights) |
| Phase B — stale golden | 17:13 → 17:34 | **~21 min** (2 full provisions) |
| Phase C — retained key | 17:14 → 17:23 | **~25 min** (overlapped Phase B; different machine) |
| Record, register, publication, teardown | 17:35 → 17:55 | **~20 min** |
| **Total** | **16:42 → 17:55** | **≈ 1 h 13 min** |
Phase C ran concurrently with Phase B deliberately — different machines, no shared state. The
45 hour envelope was not needed; nothing was hurried and nothing was dropped for time.
### 11.2 Register
**Ceiling moved R-303 → R-310.** Opened: **R-304** (retained key has no product route — rank 1),
**R-305** (R-300 fix is single-shot per machine — rank 2), **R-306** (`--preflight-only` writes state
it says it does not — rank 3), **R-307** (live abandon countdown, operator decision), **R-308**
(stored controller password stale for demo-felhom — rank 2), **R-309** (day-0 runbook's publication
claim false since R-110 — rank 3), **R-310** (two installer edges — rank 4).
**Closed: R-297 and R-300**, both observed live and published.
### 11.3 Teardown
`drill-r50`: **reverted to snapshot `virgin` and left powered off** — `qemu-img snapshot -a virgin`,
qemu exited first, `qemu.pid` removed. Its three in-VM goldens went with the revert (the VM's normal
lifecycle); the **source** archive on DooPlex, `/mnt/5_hdd/felhom.eu/drill/golden-0.98.3.tar.zst`, is
untouched (mtime still 2026-07-03). Full four-layer detail in §10.
### 11.4 What was dropped, named plainly
**Nothing was dropped.** Phase A, Phase B and Phase C all ran in full, in the required order, and the
publication was earned rather than assumed.
**One thing was not done and it was not in scope:** the planted dataset reached the off-site store via
`restic` directly rather than via the dashboard's own button, because the stored controller password
no longer opens demo-felhom (**R-308**). The store, the credentials and the restic binary were the
product's own; what went unexercised is the *app-backup → recovery-unit → offsite* chain, which is not
what this drill was measuring.
### 11.5 Evidence, including a gap
Logs at `audits/evidence-drill-retained-key-2026-08-12/`: `B1-install-preB.log`, `B2-uninstall.log`,
`B3-explicit-stale.log`, `B4-autodiscover-stale.log`, `Z-final-state.txt`.
**The Phase A logs did not survive.** They lived on the drill VM's disk and were destroyed by the
revert to `virgin` between Phase A and Phase B — I copied evidence out before the *final* teardown but
not before the *intermediate* one. Every Phase A quotation in §5 is verbatim from the live run as read
at the time, but the raw files are gone and cannot be re-read. **This is the GL-1 discipline the golden-bake
runbook already states ("scp the log OUT first"), applied to the last revert and not the middle one.**
Recorded rather than glossed; the fix is procedural, not a code change.
### 11.6 Observations — noticed, not acted on
- **The stale archive is never displaced.** A re-fetched golden carries its real timestamp, which sorts
below a planted `23_59_59`, so the bad archive stays newest-by-filename and is re-rejected on every
future install. Correct, but the warning recurs forever and nothing removes the cause.
- **The ceremony wipes its staged secret.** After every run `/var/lib/felhom-agent/escrow-stage/` was
**empty** — checked as a positive observable, not inferred from an absent log line.
- **`restic_pw_sealed: true` in the ceremony status is a genuinely useful positive observable** — it
distinguishes "a ceremony ran" from "a ceremony sealed the thing that matters".
- **`demo-felhom` already had an orphaned store** (`/home/felhom-repo.orphaned-20260810`) from the
10 August rebuild, and its password never changed — the current escrow's `restic_pw_sha256` matched
the box's live `repo_password` exactly on arrival. So that orphaning was a rename, not a key change.
- **R-241's mint guard makes password-changing supersessions rare by design.** A rebuilt guest no longer
mints a fresh repository password while the hub holds a sealed package, which is why the state Phase C
needed had to be produced deliberately rather than found.
- The Hungarian locale warning (`perl: LC_CTYPE = "UTF-8"`) prefixes almost every `pct exec` on
felhom-pve. Harmless, but it is noise on every operator transcript.
@@ -0,0 +1,115 @@
[INFO] felhom-host-install v1.25.0 — mode=appliance customer=drill-r50 vmid=120
[STEP] 1/8 pre-flight
[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
[INFO] node: drill-pve (auto)
[INFO] agent config: /etc/felhom-agent/agent.json
[INFO] agent: not installed yet — will be fetched + installed in step 5/8
[INFO] local-lvm free: ~75 GiB
[WARN] local-lvm free ~75 GiB < hard min 120 GiB
[INFO] free RAM: ~6305 MiB
[INFO] existing guests on this host: 0 (pct+qm)
[INFO] archive storage 'local' present
[INFO] acl storage 'felhom-pbs' not present yet — expected: the PBS-DR tier creates it; the grant is pre-positioned deliberately
[INFO] hub reachable (https://hub.felhom.eu)
[OK] customer 'drill-r50' exists + passphrase valid
[INFO] golden (local): local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
[OK] pre-flight passed
[STEP] 2/8 Proxmox API token
[OK] token minted (secret captured, not logged)
[OK] scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm felhom-pbs])
[SKIP] old broad role FelhomAgent already absent
[STEP] 3/8 compute volume grows
[INFO] auto-computed from ~75 GiB free (ONE volume since R-165)
[INFO] grows: rootfs +0G (->32G), data +46G (->70G, ONE volume)
[STEP] 4/8 host enrollment (POST /host-enroll)
[OK] host REUSED (idempotent — existing credential)
[INFO] host_id: drill-r50-0a4f9a (api_key captured, not logged)
[STEP] 4b/8 break-glass credential (root@pam console password → hub vault)
[OK] root@pam password set + vaulted to the hub (retrieve via the operator /admin path; never logged here)
[WARN] NOTE: the root@pam password just CHANGED — the old one now fails at the PVE web GUI (:8006).
[WARN] Retrieve the new one at hub → host page (vaulted recovery credential).
[STEP] 5/8 agent install (fetch + verify + install)
[INFO] disabled pve-enterprise.sources (Enabled: no)
[INFO] disabled ceph.sources (Enabled: no)
[INFO] added pve-no-subscription.sources (suite=trixie)
[OK] apt repos aligned to no-subscription (changed; apt-get update OK)
[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
[INFO] manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.213.0
[INFO] fetching agent binary v0.128.0 from Gitea …
[OK] verified sha256 c6eba73bf9b9ad69… matches the hub manifest
[OK] installed /usr/local/bin/felhom-agent (felhom-agent 0.128.0)
[INFO] installing the 'sudo' package (required for the non-root agent model) …
[OK] sudo installed (Sudo version 1.9.16p2)
[INFO] installing the 'age' package (escrow ceremony identity-wrap dependency) …
[OK] age installed (1.2.1)
[OK] created service user felhom-agent
[OK] added felhom-agent to systemd-journal (unprivileged journal read for NAS verify)
[OK] installed /usr/local/sbin/felhom-mkfs-guarded (0755, the guarded mkfs path)
[OK] installed /usr/local/sbin/felhom-selfupdate-guarded (0755, the guarded A/B binary-swap path)
[OK] installed /usr/local/sbin/felhom-pbs-apply (0755, the guarded PBS-DR apply path)
[OK] installed /usr/local/sbin/felhom-backup-target-apply (0755, the guarded backup-target path)
[OK] installed /etc/sudoers.d/felhom-agent (0440, visudo-validated)
[OK] installed /etc/systemd/system/felhom-agent.service + enabled (started in step 6 after config)
[OK] installed self-update rollback unit + start-limit drop-in (auto-rollback armed)
[OK] installed break-glass layers 1+2 (tmpfiles /run/sshd + agent-independent watchdog timer)
[OK] installed OOB felhom-sshd instance + static belt (agent renders config + fills sets once oob.enabled)
[STEP] 6/8 agent config + service
[WARN] backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive.
[WARN] It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard.
[INFO] creating island bridge vmbr9 (portless, 169.254.253.1/30)
[OK] vmbr9 up: 169.254.253.1/30
[INFO] R-50 island ON: local_api=169.254.253.1:8443 (vmbr9); guest net1=169.254.253.2/30; lan_resolver.host_ip=10.0.2.15
[INFO] node=drill-pve local_api=169.254.253.1:8443 tls_fp=F7:CC:33:E6:CA:F5…
[OK] wrote /etc/felhom-agent/agent.json (0600 felhom-agent)
[OK] agent --selftest (read-only) passed
[OK] felhom-agent service active (non-root felhom-agent reads the config OK)
[STEP] 7/8 golden archive
[SKIP] using local golden: local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
[STEP] 8/8 provision guest 120
[SKIP] pool felhom already exists
=== felhom-agent 0.128.0 selftest=provision (vmid=120 customer=drill-r50 hostname=drill-r50) ===
--- front half: bring-up (provision) local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst → vmid 120 ---
time=2026-08-12T17:18:55.237+02:00 level=INFO msg="bring-up: pool membership re-asserted" vmid=120 pool=felhom
[OK] front half: vmid 120 up (boot+running) in 1m20s; MAC=BC:24:11:6A:90:26
--- back half: mint per-guest token + populate bootstrap config mount ---
time=2026-08-12T17:19:01.173+02:00 level=INFO msg="provision: back-half complete" vmid=120 mount=mp9 guest_path=/etc/felhom-bootstrap endpoint=169.254.253.1:8443
[OK] back half: bootstrap mount mp9 → /etc/felhom-bootstrap on vmid 120 (host dir /var/lib/felhom-agent/guests/120/bootstrap)
local-api endpoint 169.254.253.1:8443 · leaf fp e4cba31879281d094ea2ab492e412a9aa380854b99fd5792134658f7f0971dc5 · token: minted (not printed)
=== selftest=provision OK — guest 120 provisioned + bootstrap-mounted (KEPT) ===
next: reboot the guest → the golden's baked controller-bootstrap unit deploys the controller,
which PULLS its controller.yaml from the hub (retrieval passphrase) and merges in this local_api.
[OK] provision completed
[INFO] rebooting guest 120 so the baked controller-bootstrap unit picks up the mount
[STEP] verify
[OK] pct status: running
[OK] onboot: 1
mp0: local-lvm:vm-120-disk-1,mp=/var/lib/docker,backup=1,size=62G
mp1: local-lvm:vm-120-disk-2,mp=/mnt/sys_drive,backup=1,size=8G
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
rootfs: local-lvm:vm-120-disk-0,size=32G
[OK] pool: guest 120 is a member of felhom
[OK] acl: FelhomAgentBase@/ present (user+token)
[OK] acl: FelhomAgentGuest@/pool/felhom present (user+token)
[OK] acl: FelhomAgentStore@/storage/local present (user+token)
[OK] acl: FelhomAgentStore@/storage/local-lvm present (user+token)
[OK] acl: FelhomAgentStore@/storage/felhom-pbs present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990000 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990001 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990002 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990003 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990004 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990005 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990006 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990007 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990008 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990009 present (user+token)
[OK] authz signers: 2 (operator-signed self-update armed)
[OK] controller: Up 22 seconds (healthy) (after ~0s)
[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.98.3
[WARN] cloudflared not visible yet
[INFO] (confirm in the hub UI that host drill-r50-0a4f9a reports guest 120)
[OK] Day-0 provision SUCCESS — vmid=120 host_id=drill-r50-0a4f9a customer=drill-r50 golden=local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
[INFO] root@pam was rotated + vaulted at step 4b — retrieve at hub → host page (the old GUI password no longer works).
@@ -0,0 +1,790 @@
[INFO] felhom-host-install v1.25.0 — mode=uninstall
[STEP] UNINSTALL — local host teardown
[INFO] vmid 120 looks like a Felhom guest (has the /etc/felhom-bootstrap mount)
[INFO] pct config 120:
arch: amd64
cores: 2
features: nesting=1,keyctl=1
hookscript: local:snippets/felhom-guest-hook.sh
hostname: drill-r50
memory: 4096
mp0: local-lvm:vm-120-disk-1,mp=/var/lib/docker,backup=1,size=62G
mp1: local-lvm:vm-120-disk-2,mp=/mnt/sys_drive,backup=1,size=8G
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
mp9: /var/lib/felhom-agent/guests/120/bootstrap,mp=/etc/felhom-bootstrap,ro=1
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:6A:90:26,ip=dhcp,type=veth
net1: name=eth1,bridge=vmbr9,hwaddr=BC:24:11:89:76:A8,ip=169.254.253.2/30,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-120-disk-0,size=32G
swap: 512
unprivileged: 1
Type the vmid (120) to confirm PERMANENT destruction: perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
Logical volume "vm-120-disk-0" successfully removed.
Logical volume "vm-120-disk-1" successfully removed.
Logical volume "vm-120-disk-2" successfully removed.
[OK] guest 120 destroyed
[STEP] host-level removal
Removed '/etc/systemd/system/multi-user.target.wants/felhom-agent.service'.
Removed '/etc/systemd/system/timers.target.wants/felhom-mgmt-watchdog.timer'.
Removed '/etc/systemd/system/multi-user.target.wants/felhom-sshd.service'.
Removed '/etc/systemd/system/multi-user.target.wants/felhom-oob-nft.service'.
The unit files have no installation config (WantedBy=, RequiredBy=, UpheldBy=,
Also=, or Alias= settings in the [Install] section, and DefaultInstance= for
template units). This means they are not meant to be enabled or disabled using systemctl.
Possible reasons for having these kinds of units are:
• A unit may be statically enabled by being symlinked from another unit's
.wants/, .requires/, or .upholds/ directory.
• A unit's purpose may be to act as a helper for some other unit which has
a requirement dependency on it.
• A unit may be started when needed via activation (socket, path, timer,
D-Bus, udev, scripted systemctl call, ...).
• In case of template units, the unit is meant to be enabled with some
instance name specified.
Removed '/etc/systemd/system/multi-user.target.wants/felhom-shared-parent.service'.
Removed '/etc/systemd/system/pve-guests.service.wants/felhom-shared-parent.service'.
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[SKIP] old broad role FelhomAgent already absent
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[SKIP] golden vzdump left in place (pass --remove-golden to remove)
[STEP] kept vs wiped — read before pulling drives or closing the customer out
WIPED (this run):
- guest 120 (container + its OS/Docker/user-data volumes)
- the felhom-agent: binary, unit, sudoers, config (+ its .bak backups), state dir, service user
- self-update artifacts: guarded wrapper, A/B slots (.prev/.new.*), rollback unit, start-limit drop-in
- break-glass watchdog + OOB artifacts (where present); guest-hook snippet; dnsmasq snippets; the mkfs + pbs-apply wrappers
- pveum: the Felhom roles/user/token/scoped ACL; the emptied felhom pool
- the install state file
KEPT (lives on deliberately — remove/rotate these out-of-band if the customer is leaving):
- the enrolled drives + ALL data under /mnt/felhom-drives — unmounted only, NEVER wiped;
the drives are physically removable now.
- the hub host/customer record + report history (operator UI / DB)
- the escrow blob in the hub, if one exists (operator UI)
- the hub-vaulted root@pam recovery credential — the box KEEPS the password step 4b set; rotate it if the box leaves Felhom management
[OK] UNINSTALL complete — removed: guest 120, the felhom-agent (unit/sudoers/binary/state/config+baks/user + selfupdate-artifacts/shared-parent/mkfs-wrapper/pbs-apply-wrapper/hook-snippet/dnsmasq-snippets), the pveum role/user/token/ACL, the felhom pool, and /var/lib/felhom-install/state.json.
[INFO] golden vzdump: left in place (--remove-golden to remove).
[INFO] NOTE: the 'sudo' and 'dnsmasq' packages were left installed (system packages); the host record still exists in the hub — remove it there if desired.
@@ -0,0 +1,767 @@
[INFO] felhom-host-install v1.27.0 — mode=appliance customer=drill-r50 vmid=120
[STEP] 1/8 pre-flight
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
[INFO] node: drill-pve (auto)
[INFO] agent config: /etc/felhom-agent/agent.json
[INFO] agent: not installed yet — will be fetched + installed in step 5/8
[INFO] local-lvm free: ~75 GiB
[WARN] local-lvm free ~75 GiB < hard min 120 GiB
[INFO] free RAM: ~6119 MiB
[INFO] existing guests on this host: 0 (pct+qm)
[INFO] archive storage 'local' present
[INFO] acl storage 'felhom-pbs' not present yet — expected: the PBS-DR tier creates it; the grant is pre-positioned deliberately
[INFO] dnsmasq: already installed BEFORE Felhom — recorded; uninstall will not touch it
[INFO] hub reachable (https://hub.felhom.eu)
[OK] customer 'drill-r50' exists + passphrase valid
[INFO] golden (local): local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
[OK] pre-flight passed
[STEP] 2/8 Proxmox API token
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[OK] token minted (secret captured, not logged)
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[OK] scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm felhom-pbs])
[SKIP] old broad role FelhomAgent already absent
[STEP] 3/8 compute volume grows
[INFO] auto-computed from ~75 GiB free (ONE volume since R-165)
[INFO] grows: rootfs +0G (->32G), data +46G (->70G, ONE volume)
[STEP] 4/8 host enrollment (POST /host-enroll)
[OK] host REUSED (idempotent — existing credential)
[INFO] host_id: drill-r50-0a4f9a (api_key captured, not logged)
[STEP] 4b/8 break-glass credential (root@pam console password → hub vault)
[OK] root@pam password set + vaulted to the hub (retrieve via the operator /admin path; never logged here)
[WARN] NOTE: the root@pam password just CHANGED — the old one now fails at the PVE web GUI (:8006).
[WARN] Retrieve the new one at hub → host page (vaulted recovery credential).
[STEP] 5/8 agent install (fetch + verify + install)
[OK] apt repos aligned to no-subscription (already aligned; apt-get update OK)
[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
[INFO] manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.213.0
[INFO] fetching agent binary v0.128.0 from Gitea …
[OK] verified sha256 c6eba73bf9b9ad69… matches the hub manifest
[OK] installed /usr/local/bin/felhom-agent (felhom-agent 0.128.0)
[OK] created service user felhom-agent
[OK] added felhom-agent to systemd-journal (unprivileged journal read for NAS verify)
[OK] installed /usr/local/sbin/felhom-mkfs-guarded (0755, the guarded mkfs path)
[OK] installed /usr/local/sbin/felhom-selfupdate-guarded (0755, the guarded A/B binary-swap path)
[OK] installed /usr/local/sbin/felhom-pbs-apply (0755, the guarded PBS-DR apply path)
[OK] installed /usr/local/sbin/felhom-backup-target-apply (0755, the guarded backup-target path)
[OK] installed /etc/sudoers.d/felhom-agent (0440, visudo-validated)
[OK] installed /etc/systemd/system/felhom-agent.service + enabled (started in step 6 after config)
[OK] installed self-update rollback unit + start-limit drop-in (auto-rollback armed)
[OK] installed break-glass layers 1+2 (tmpfiles /run/sshd + agent-independent watchdog timer)
[OK] installed OOB felhom-sshd instance + static belt (agent renders config + fills sets once oob.enabled)
[STEP] 6/8 agent config + service
[WARN] backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive.
[WARN] It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard.
[INFO] island bridge vmbr9 already present — leaving it
[INFO] R-50 island ON: local_api=169.254.253.1:8443 (vmbr9); guest net1=169.254.253.2/30; lan_resolver.host_ip=10.0.2.15
[INFO] node=drill-pve local_api=169.254.253.1:8443 tls_fp=F7:CC:33:E6:CA:F5…
[OK] wrote /etc/felhom-agent/agent.json (0600 felhom-agent)
[OK] agent --selftest (read-only) passed
[OK] felhom-agent service active (non-root felhom-agent reads the config OK)
[STEP] 7/8 golden archive
[ERROR] refusing the golden you named (local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst): it is controller 0.98.3, but the vouched golden is 0.213.0.
The vouched golden is 0.213.0. Either pass the archive that matches it,
or re-run with --force-gitea-golden to fetch the vouched one from Gitea.
@@ -0,0 +1,114 @@
[INFO] felhom-host-install v1.27.0 — mode=appliance customer=drill-r50 vmid=120
[STEP] 1/8 pre-flight
[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
[INFO] node: drill-pve (auto)
[INFO] agent config: /etc/felhom-agent/agent.json
[INFO] agent (existing): felhom-agent 0.128.0
[INFO] local-lvm free: ~75 GiB
[WARN] local-lvm free ~75 GiB < hard min 120 GiB
[INFO] free RAM: ~6231 MiB
[INFO] existing guests on this host: 0 (pct+qm)
[INFO] archive storage 'local' present
[INFO] acl storage 'felhom-pbs' not present yet — expected: the PBS-DR tier creates it; the grant is pre-positioned deliberately
[INFO] dnsmasq: already installed BEFORE Felhom — recorded; uninstall will not touch it
[INFO] hub reachable (https://hub.felhom.eu)
[OK] customer 'drill-r50' exists + passphrase valid
[INFO] golden (local): local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
[OK] pre-flight passed
[STEP] 2/8 Proxmox API token
[SKIP] pool felhom already exists
[INFO] user felhom-agent@pve exists
[INFO] existing agent config has a token — testing it (read-only --selftest)
[OK] existing token authenticates — REUSING (no rotation)
[INFO] role FelhomAgentBase exists — ensuring exact privileges
[INFO] role FelhomAgentGuest exists — ensuring exact privileges
[INFO] role FelhomAgentStore exists — ensuring exact privileges
[OK] scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm felhom-pbs])
[SKIP] old broad role FelhomAgent already absent
[STEP] 3/8 compute volume grows
[INFO] auto-computed from ~75 GiB free (ONE volume since R-165)
[INFO] grows: rootfs +0G (->32G), data +46G (->70G, ONE volume)
[STEP] 4/8 host enrollment (POST /host-enroll)
[OK] host REUSED (idempotent — existing credential)
[INFO] host_id: drill-r50-0a4f9a (api_key captured, not logged)
[STEP] 4b/8 break-glass credential (root@pam console password → hub vault)
[SKIP] recovery credential already vaulted (use --rotate-recovery to regenerate)
[STEP] 5/8 agent install (fetch + verify + install)
[OK] apt repos aligned to no-subscription (already aligned; apt-get update OK)
[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
[INFO] manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.213.0
[SKIP] agent v0.128.0 already installed + service active — skipping binary install
[INFO] service user felhom-agent exists
[INFO] felhom-agent already in systemd-journal
[OK] installed /usr/local/sbin/felhom-mkfs-guarded (0755, the guarded mkfs path)
[OK] installed /usr/local/sbin/felhom-selfupdate-guarded (0755, the guarded A/B binary-swap path)
[OK] installed /usr/local/sbin/felhom-pbs-apply (0755, the guarded PBS-DR apply path)
[OK] installed /usr/local/sbin/felhom-backup-target-apply (0755, the guarded backup-target path)
[OK] installed /etc/sudoers.d/felhom-agent (0440, visudo-validated)
[OK] installed /etc/systemd/system/felhom-agent.service + enabled (started in step 6 after config)
[OK] installed self-update rollback unit + start-limit drop-in (auto-rollback armed)
[OK] installed break-glass layers 1+2 (tmpfiles /run/sshd + agent-independent watchdog timer)
[OK] installed OOB felhom-sshd instance + static belt (agent renders config + fills sets once oob.enabled)
[STEP] 6/8 agent config + service
[WARN] backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive.
[WARN] It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard.
[INFO] island bridge vmbr9 already present — leaving it
[INFO] R-50 island ON: local_api=169.254.253.1:8443 (vmbr9); guest net1=169.254.253.2/30; lan_resolver.host_ip=10.0.2.15
[INFO] node=drill-pve local_api=169.254.253.1:8443 tls_fp=F7:CC:33:E6:CA:F5…
[OK] wrote /etc/felhom-agent/agent.json (0600 felhom-agent)
[OK] agent --selftest (read-only) passed
[OK] felhom-agent service active (non-root felhom-agent reads the config OK)
[STEP] 7/8 golden archive
[WARN] ignoring the local golden local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst — it is controller 0.98.3, but the vouched golden is 0.213.0
[WARN] fetching the vouched golden instead (this is what the manifest is for)
[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
[INFO] fetching golden v0.213.0 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_08_12-17_31_32.tar.zst
[OK] verified sha256 77429bfc7e39ffc8… matches the hub manifest
[OK] golden imported + verified: local:backup/vzdump-lxc-9100-2026_08_12-17_31_32.tar.zst
[STEP] 8/8 provision guest 120
[SKIP] pool felhom already exists
=== felhom-agent 0.128.0 selftest=provision (vmid=120 customer=drill-r50 hostname=drill-r50) ===
--- front half: bring-up (provision) local:backup/vzdump-lxc-9100-2026_08_12-17_31_32.tar.zst → vmid 120 ---
time=2026-08-12T17:32:57.812+02:00 level=INFO msg="bring-up: pool membership re-asserted" vmid=120 pool=felhom
[OK] front half: vmid 120 up (boot+running) in 1m10s; MAC=BC:24:11:D3:D7:8C
--- back half: mint per-guest token + populate bootstrap config mount ---
time=2026-08-12T17:33:03.626+02:00 level=INFO msg="provision: back-half complete" vmid=120 mount=mp9 guest_path=/etc/felhom-bootstrap endpoint=169.254.253.1:8443
[OK] back half: bootstrap mount mp9 → /etc/felhom-bootstrap on vmid 120 (host dir /var/lib/felhom-agent/guests/120/bootstrap)
local-api endpoint 169.254.253.1:8443 · leaf fp 0c7b12eec0ab8c4f8e879dba1af0f7203ef82aea92b023d543b7632b865ead58 · token: minted (not printed)
=== selftest=provision OK — guest 120 provisioned + bootstrap-mounted (KEPT) ===
next: reboot the guest → the golden's baked controller-bootstrap unit deploys the controller,
which PULLS its controller.yaml from the hub (retrieval passphrase) and merges in this local_api.
[OK] provision completed
[INFO] rebooting guest 120 so the baked controller-bootstrap unit picks up the mount
[STEP] verify
[OK] pct status: running
[OK] onboot: 1
mp0: local-lvm:vm-120-disk-1,mp=/var/lib/felhom,backup=1,size=70G
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
rootfs: local-lvm:vm-120-disk-0,size=32G
[OK] pool: guest 120 is a member of felhom
[OK] acl: FelhomAgentBase@/ present (user+token)
[OK] acl: FelhomAgentGuest@/pool/felhom present (user+token)
[OK] acl: FelhomAgentStore@/storage/local present (user+token)
[OK] acl: FelhomAgentStore@/storage/local-lvm present (user+token)
[OK] acl: FelhomAgentStore@/storage/felhom-pbs present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990000 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990001 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990002 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990003 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990004 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990005 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990006 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990007 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990008 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990009 present (user+token)
[OK] authz signers: 2 (operator-signed self-update armed)
[OK] controller: Up 24 seconds (healthy) (after ~0s)
[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.213.0
[WARN] cloudflared not visible yet
[INFO] (confirm in the hub UI that host drill-r50-0a4f9a reports guest 120)
[OK] Day-0 provision SUCCESS — vmid=120 host_id=drill-r50-0a4f9a customer=drill-r50 golden=local:backup/vzdump-lxc-9100-2026_08_12-17_31_32.tar.zst
[INFO] root@pam was rotated + vaulted at step 4b — retrieve at hub → host page (the old GUI password no longer works).
@@ -0,0 +1,15 @@
=== final drill-r50 state 2026-08-12T15:39:28Z ===
VMID Status Lock Name
120 running drill-r50
--- controller ---
gitea.dooplex.hu/admin/felhom-controller:0.213.0
--- goldens on the box ---
Volid Format Type Size VMID
local:backup/vzdump-lxc-120-2026_08_12-17_24_19.tar.zst tar.zst backup 612587653 120
local:backup/vzdump-lxc-9100-2026_08_12-17_31_32.tar.zst tar.zst backup 656657008 9100
local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst tar.zst backup 612002238 9100
--- dnsmasq ---
enabled
active
udp UNCONN 0 0 127.0.0.1:53 0.0.0.0:* users:(("dnsmasq",pid=15124,fd=6))
udp UNCONN 0 0 10.0.2.15:53 0.0.0.0:* users:(("dnsmasq",pid=15124,fd=4))