Rehearsal 2026-08-09: the walk stops at P3 — R-273 blocks every install fleet-wide
gates / gates (push) Successful in 24s

P1 uninstall, P2 preflight, P3 install. The install FAILED at step 5/8 in 44s,
and the two rank-1 findings are both on the setting-up path a tester's visit is
made of. Eleven register rows minted (R-269..R-279); ceiling moves 268 -> 279.

R-273 (RANK 1) — the hub vouches agent 0.128.0; that version was published as a
Gitea PACKAGE but never git-tagged. Since R-183 the installer correctly pins its
config fetches to raw/tag/v<vouched>, so every fresh install and every reinstall
now 404s as root, mid-install. Measured: main 200, v0.127.0 200, v0.128.0 404.
This is R-184 arriving; release-agent.sh:23 already documents the exact hazard.
Existing boxes are fine (self-update takes the binary from the registry).
NOT fixed here — publishing a release tag is outward-facing and the runbook says
stop and report. One command unblocks it; it is in STATUS.md.

R-272 (RANK 1) — Felhom's own uninstall leaves the condition that makes Felhom's
own reinstall refuse. It installs dnsmasq at day-0, then on teardown removes the
snippet and RESTARTS the daemon unconstrained (process start time lands inside
the uninstall window), which grabs 0.0.0.0:53; the next preflight then refuses,
and the message reads as though the owner's LAN DNS is at fault.

R-274 — a local golden is adopted with no version and no sha check; the manifest
vouch is consulted only on the fetch path. demo-hp's local copy is controller
0.192.0 against a vouched 0.210.0, and below the 0.200.0 where the recovery
screen shipped. Not yet observed end-to-end (R-273 killed step 5 first).

Also: R-275 orphaned credential backups + uid reuse, R-276 the wg tunnel outlives
the uninstall, R-269/270/271 from the token rotation, R-277 three hub surfaces
misreport a healthy off-site tier, R-278 demo-felhom six days unprotected,
R-279 no operator-triggerable off-site run.

Two hypotheses of mine were tested and REFUTED rather than shipped as findings:
the leftover sudoers file is inert (sudo skips dotted filenames), and demo-hp's
off-site tier was healthy all along - I had misread the hub and said so.

STATUS.md records the three rulings §8.3 asked for, with the floor CORRECTED to
its live value 0.200.0 and the count corrected to twenty.
This commit is contained in:
2026-08-09 10:59:44 +02:00
parent 34646295dc
commit b1afbb8a4d
9 changed files with 2280 additions and 41 deletions
+136
View File
@@ -0,0 +1,136 @@
# REPORT — BYO reinstall rehearsal, 2026-08-09 (session report)
*Written as `REPORT-<topic>.md` rather than `REPORT.md` per the repo's parallel-session rule.*
## The answer to the runbook's question, first
**Not answerable today, and the reason is the answer.** *"If a person's machine is set up again from
scratch, does their data come back through the product, in one sitting, with no command line inside
the machine?"* — the walk never reached the data. **A machine wiped today cannot be set up again at
all**: the install fails at step 5 of 8, on every host, because the agent version the hub vouches was
never given a git tag (**R-273**). The step before that had already refused, because removing Felhom
leaves behind the very thing that blocks reinstalling it (**R-272**).
The data was never at risk and is not now: the planted dataset sits in off-site snapshot `41c830db`
with its fingerprints recorded, plus a PBS snapshot and four local vzdump archives.
## Venue
**demo-hp (t740)**, operator-approved at STOP 1. It won every fidelity criterion that distinguishes
the two demo boxes: three customer apps against one, a registered storage path against none, and a
Secure-Boot `shim` install — the customer shape — against demo-felhom's SB-off `mkimage` firmware
workaround. `drill-r50` (VM 300) was verified not at risk before proceeding: it is outside the
`felhom` pool, on `local-lvm`, and `--uninstall` removes no storage and no non-pool guest.
## Findings, ranked by what they cost the person in front of you
**1 — stops the visit**
- **R-273** · the vouched agent (0.128.0) has no git tag; every install and reinstall dies at 5/8.
- **R-272** · Felhom's uninstall restarts its own dnsmasq unconstrained; it grabs `:53`; the next
install refuses and appears to blame the owner's network.
**2 — costs the visit**
- **R-274** · a local golden is adopted with no version and no checksum check. The copy on demo-hp is
controller **0.192.0** against a vouched **0.210.0** — and below 0.200.0, where the recovery screen
the customer needs actually shipped.
- **R-276** · an uninstalled box keeps a live WireGuard tunnel into the off-site endpoint; declared
in neither the KEPT nor the WIPED list.
**3 — misleads**
- **R-269** · a rotated-out local-API token still authorises until an unrelated lookup forces a
reload. The shipped test passes only because of its lookup order.
- **R-270** · R-268's own rotation recipe is a step short; the controller never re-reads the mount.
- **R-271** · the `agent_channel_unauthorized` alarm can never close — its own advice silences the
all-clear.
- **R-277** · three hub surfaces present a healthy off-site tier as absent. **This one caught me.**
- **R-278** · demo-felhom has had no off-site backup for six days, waiting on a ceremony nobody ran.
**4 — cosmetic / hygiene**
- **R-275** · five orphaned 0600 credential backups survive, and uid reuse hands them to the new
service account. Superseded keys here; live if the backups were recent.
- **R-279** · no operator-triggerable off-site backup exists.
## What I got wrong, and corrected
I reported to the operator that the off-site tier had not run on **either** box since 2026-08-03.
That was true of demo-felhom and **false of demo-hp**, which had 18 unbroken daily snapshots. I had
read three hub surfaces that agreed with each other and none of which said what I took them to say
(now **R-277**). I corrected it before it changed any decision, and the operator's "repair off-site
first" ruling turned out to be unnecessary for the chosen venue.
I also raised **R-275**'s sudoers half as a likely privilege-escalation on reinstall, then **tested it
and refuted my own hypothesis**: sudo skips filenames containing dots, so the leftover file is inert.
`visudo -c -f` parsing a file OK is not evidence that sudo loads it.
## Integrity verdict
**Not reached** — P6 never ran. The instrument is proven and waiting: the comparator was watched
**failing** three ways, including an NFC→NFD rename that renders identically to the eye and would
have passed a rendered-text comparison. Four files, two with Hungarian accented names
(`árvíztűrő-tükörfúrógép.txt`, `nested/őszibarack.md`), one 3 MB binary; manifest keyed on raw name
bytes, committed as `evidence-rehearsal-2026-08-09/GATE0-before-manifest.json`; all four confirmed
present in snapshot `41c830db`.
## Wall clocks
| phase | duration |
|---|---|
| Pre-phase (R-268 rotation, proved both ways) | ~25 min |
| Gate 0 (venue, dataset, positive control, off-site run, capture) | ~55 min |
| P1 uninstall | **60 s** (08:37:23 → 08:38:23 UTC) |
| P1 leave-behind measurement | ~12 min |
| P2 preflight (3 runs: 2 refusals, 1 pass) | ~6 min |
| **P3 install — FAILED** | **44 s** (08:51:33 → 08:52:17 UTC) |
| bare machine → visible files | **not reached** |
## Steps taken off-path, and what they cost
1. **R-268 rotation on demo-felhom** — required by the runbook's pre-phase; not the venue.
2. **demo-hp's dashboard password re-set to the credentials-file value**, on operator instruction.
The customer-owned password was unknown to this session and no operator route to the off-site
button exists (R-279). Prior hash preserved in-guest; destroyed with the guest at P1. Cost: none —
P1 wiped it and P4 re-claims.
3. **The off-site run was started by a script pressing the dashboard's own endpoint** with a real
session and CSRF token, not by a person clicking. Identical server path; only the click synthetic.
4. **`--passphrase-file` instead of the no-echo prompt** — a first-class documented option with a
permission check, so the secret still never touched argv. A person would type it.
5. **`systemctl stop dnsmasq && systemctl disable dnsmasq`** — the action the refusal message tells
the owner to take, used as the counterfactual that confirmed R-272.
Everything after the install command returned was read-only. No repair was attempted on the
half-installed box.
## Teardown — all four layers
1. **The machine** — nothing created beyond the half-install itself, which is **left in place
deliberately** for inspection and resumption (`state.json` completed: preflight, token, grows,
enroll). demo-hp is **not serving** right now: no guest, agent installed but no unit.
2. **The host**`local-lvm` 20 904 790 → 12 355 143 KiB (**≈8.5 GiB returned**); `local` ≈64 MiB;
NVMe unchanged, backups deliberately kept. Pre-existing leftovers found and **not** removed
(not this run's, recorded instead): storage `c11-scratch`, the orphaned `vzdump-lxc-9100` archive,
and `/root/.dpw`, `.h`, `.sec.html` in the old guest (now destroyed with it).
3. **The hub****no customer or appliance record was created**; the `demo-hp` customer is retained
deliberately, as the runbook requires. Nothing to delete.
4. **The off-site side****one write, and it was the intended one**: the Gate 0 backup that created
snapshots `41c830db`, `9e38b84c`, `78b93f04`. **No prune, no forget, no delete.** How I know: every
restic call was `snapshots`, `ls`, or the product's own `POST /backup/offbox/run`; retention runs
inside that product path and is ep0's server-side job (R-89, boxes keep `keep_last: 0`).
## Secrets handling
No secret reached stdout. Token values, the retrieval passphrase, the controller password and the hub
DB copy were handled file→file at 0600 and shredded; the hub DB copy (which carries every host's
break-glass credential) was shredded immediately after the one hash comparison it was taken for.
Credential comparisons were done by sha256 prefix, never by value.
## State for whoever resumes
- **Blocked on one command** — `git tag v0.128.0 28ba8593b8 && git push origin v0.128.0`.
- **Nothing in Gate 0 needs redoing.** Snapshot `41c830db` holds the dataset; the before-manifest is
committed; the comparator is proven.
- **First thing to record at step 7/8** when the install resumes: which golden it takes (R-274 is the
one finding still owed an end-to-end observation).
- **P4 will need you** — the claim code is emailed to `doodoo21@freemail.hu` and there is no
operator-side route to it (R-119).
Full account: `documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md`.
+79 -37
View File
@@ -1,6 +1,6 @@
# STATUS — what works, what's broken, what's next # STATUS — what works, what's broken, what's next
**Updated 2026-08-08.** **Updated 2026-08-09.**
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority; this page restates > **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority; this page restates
> part of it in plain words, and **nothing may exist only here**. Not `CONTEXT.md`, which is technical > part of it in plain words, and **nothing may exist only here**. Not `CONTEXT.md`, which is technical
@@ -23,9 +23,25 @@ destroyed on purpose and its files came back byte for byte identical — four ti
their recovery code got everything back with **no command line inside the machine at any point**, their recovery code got everything back with **no command line inside the machine at any point**,
in 72 seconds. The two rough edges that walk found are also gone. *(R-201, R-252, R-253 — closed.)* in 72 seconds. The two rough edges that walk found are also gone. *(R-201, R-252, R-253 — closed.)*
## Broken right now — and it stops every new machine
- **No machine can be set up from scratch today.** The install dies four steps in, on every host, and
it needs one command from you to unblock (below). Nothing already running is affected — the two demo
machines and the tester's box are fine. It is only the *setting-up* path, which is the one his visit
is made of. *(R-273)*
- **A rebuilt machine may come back on software from last week** — the install will happily reuse an
old copy it finds lying on the disk, without checking it against the version you approved. The copy
on the demo machine is old enough to predate the recovery screen. *(R-274)*
- **Taking Felhom off a machine leaves the one thing that stops it going back on.** We install a small
network service at setup; removing Felhom restarts it without its settings, it takes over the port
the next install needs, and the next install then refuses — blaming the owner's network. *(R-272)*
- **A machine we removed keeps its private line to us open.** *(R-276)*
- **demo-felhom has not had an off-site backup in six days** and is waiting for a recovery code nobody
has entered. demo-hp, rebuilt the same day, recovered by itself. *(R-278)*
## What's broken ## What's broken
- **Nothing new is broken.** The *check* against a fourth secret-in-a-page covers 4 pages of 27, and - **Nothing else new is broken.** The *check* against a fourth secret-in-a-page covers 4 pages of 27, and
the cheap one covering all of them is blind to the shape that shipped. *(R-255)* the cheap one covering all of them is blind to the shape that shipped. *(R-255)*
- **An already-paired box is still told to pair itself**, 25 minutes on. *(R-214, R-235)* - **An already-paired box is still told to pair itself**, 25 minutes on. *(R-214, R-235)*
- **A backup that covered nothing still calls itself „Sikeres".** *(R-240)* - **A backup that covered nothing still calls itself „Sikeres".** *(R-240)*
@@ -35,48 +51,77 @@ in 72 seconds. The two rough edges that walk found are also gone. *(R-201, R-252
accumulates. *(R-244)* accumulates. *(R-244)*
- **Putting restored files back where they belong is still manual.** *(R-213)* - **Putting restored files back where they belong is still manual.** *(R-213)*
## Fixed today — four things the machine knew and did not say ## The reinstall rehearsal — what one afternoon on one machine found
All one family: something the box already knows, thrown away or drawn as its opposite. **We took a working demo machine, removed Felhom, and tried to put it back the way a person with their
own hardware would.** It got three steps in. That is the finding: **nine defects, two of which stop a
visit dead**, and none of them were visible from reading the code — the last three sessions of review
did not see any of them.
- **A rebuilt machine can set up its own recovery again.** The one fact the setup needs was written - It stopped at the install, on a missing version label *(R-273)* — and the step before that had
only the first time, and a rebuild replaced the configuration while leaving the note saying already refused, because removing Felhom leaves behind the thing that blocks reinstalling it
"already done". It is now checked and re-written every minute instead of remembered once, so a *(R-272)*.
hand-edited or restored configuration heals too. **This was the last item blocking a customer from - Removing Felhom also leaves five files holding old keys *(R-275)* and a live private line to us
something we promise them.** *(R-221 — agent 0.128.0.)* *(R-276)*.
- **A disk we failed to read is no longer drawn as a healthy empty one.** No figures, no bar, and it - Along the way: rotating a leaked key does not actually revoke the old one until the service is
says so: „A tárhely mérete most nem olvasható ki." *(R-259 — controller 0.210.0.)* restarted *(R-269)*, the written-down recipe for that rotation is missing a step *(R-270)*, and the
- **A backup tick now answers about that app.** It went green because *some* backup file existed and alarm it raises can never be closed because the fix it recommends is what silences the all-clear
*some other* app's database dump had succeeded most recently. Now: that app's own result, and *(R-271)*.
**no mark at all** when we have none. *(R-258 — controller 0.210.0.)* - **It also caught me being wrong.** Three separate hub screens agreed that demo-hp had no off-site
- **Our own alarm no longer points at a page that may not exist.** A check run now gives up after backup. It had eighteen. I told you the fleet's backups were down and had to take it back an hour
five minutes rather than hanging until something else kills it, and the mail says how long it ran. later — the screens were misleading, but I should have asked the machine before telling you. *(R-277)*
*(R-265.)*
**Not fixed, and said rather than glossed:** that failed disk reading still reaches us as "0 of **The data was never at risk** and is not now: the test files sit in an off-site snapshot with their
0 GB". It is the quiet direction — it can only miss a true alarm, never raise a false one. *(R-266)* fingerprints recorded, plus two local copies. Full account:
`documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md`.
**And R-221 is now proven on hardware, not just in tests** — on a demo machine we removed the one ## Three rulings, written down so they stop living in a conversation
line, watched the setup screen refuse, waited one minute, and watched it go green by itself with
nothing restarted. Every other line of that file came back identical. - **The managed-update floor.** It is deliberately parked, and the trigger to raise it is **the first
machine that is not ours**; after that it moves with the publish train. Worth knowing alongside it:
the updater always aims at the floor, never at the newest, so a machine at or above the floor
updates to nothing. **Correction to the number that was going round: the floor is live at 0.200.0,
not 0.156.0** — checked twice today, on the hub page and in both machines' own logs.
- **R-264 is decided.** Build a reader for guest-network health, the staged-update pair, the
restore-test depth pair, and the two backup-integrity timestamps. Record a stated "no reader
wanted" for the repaired-recently flag, the tier-applied timestamp, the config fingerprint, the
per-stack object and the drive-migration marker. Decide the reporting-disabled flag on its own
merits — if that is a state we support, it must be visible or a staleness alarm will one day fire
on a machine that is fine. A "no" ends by changing the allowlist reason from *arguably owed* to
*deliberately not consumed* — not by ripping out an emitter, which is a two-repo change that also
breaks a shared fixture. **The implementation is its own session.** And **it is twenty facts, not
twenty-one**.
- **R-268 is closed** — the leaked key is rotated, and the rotation is proved in both directions
rather than assumed.
## Fixed 2026-08-08 — four things the machine knew and did not say
A rebuilt machine can set up its own recovery again *(R-221, agent 0.128.0 — proved on hardware)*; an
unreadable disk is no longer drawn as a healthy empty one *(R-259)*; a backup tick now answers about
*that* app *(R-258)*; our own alarm no longer points at a log that may not exist *(R-265)*. Still true
and not glossed: a failed disk reading still reaches us as "0 of 0 GB" — the quiet direction, it can
only miss a true alarm, never raise a false one *(R-266)*.
## What we're working on ## What we're working on
- **Widening the check** so a fourth secret-in-a-page is caught by a machine. *(R-255)* · **Deciding - **Widening the check** so a fourth secret-in-a-page is caught by a machine. *(R-255)* · **R-264 is
the twenty-one** — each gets a reader, or stops being sent. *(R-264)* now decided** (above); building the readers is a session of its own.
- **Proving the hub really keeps the old sealed key** when a machine re-seals. *(R-198)* · Still open, - **Proving the hub really keeps the old sealed key** when a machine re-seals. *(R-198)* · Still open,
none urgent: *(R-256, R-257, R-261…R-263, R-266)* none urgent: *(R-256, R-257, R-261…R-263, R-266)*
## Waiting on you ## Waiting on you
- **A watching moment, five minutes.** Today's recovery fix is proved by removing one line from a - **One command, and it unblocks every install.** The version you approved yesterday was published as
demo machine's config — backed up first, disposable machine, no customer data near it — and a download but never given its version label in the source repository, and the installer looks it up
watching the setup screen go green on its own. Nothing is destroyed. Say when. by that label. The code is right and already shipped; only the label is missing.
- **One approval, three values this time.** Hub → Configuration → Day-0 artifacts: Golden `cd felhom-agent && git tag v0.128.0 28ba8593b8 && git push origin v0.128.0`. I did not do it
**0.210.0**, Agent **0.128.0**, minimum agent **0.127.0** (unchanged) → Save. Each was checked to myself: publishing a release label is not a thing to do quietly in the middle of a measurement.
be downloadable and selectable before being written here. **Agent 0.128.0 is the one that carries *(R-273)*
today's recovery fix**, so a new machine needs both, not just the image. It supersedes the 0.209.0 - **Then say go, and the rehearsal resumes** where it stopped. Nothing has to be redone — the test
approval you already gave, and it is reversible. *(R-242)* files are already in an off-site backup and their fingerprints are recorded.
*(Both items previously here are done: the approval is saved — golden 0.210.0, agent 0.128.0, minimum
0.127.0 — and the recovery fix was proved on hardware.)*
## DooPlex infrastructure — separate from the product ## DooPlex infrastructure — separate from the product
@@ -97,11 +142,8 @@ being readable.*
connections are kept, the answer is held for a minute, and the old artifacts are gone. Worst case connections are kept, the answer is held for a minute, and the old artifacts are gone. Worst case
is 5 s, once a minute at most. **Your instinct to prune was right and my measurement said is 5 s, once a minute at most. **Your instinct to prune was right and my measurement said
otherwise** — trimming to ten of each halved the slow path. *(R-267 — closed.)* otherwise** — trimming to ten of each halved the slow path. *(R-267 — closed.)*
- **I printed a live access token into a session log** while setting up today's drill, and I am - **The access token I printed into a log yesterday is rotated**, and I checked it both ways: the old
telling you rather than quietly rotating it. It only opens the agent's private channel to one one is refused, the new one works, and the machine's own channel is back up. *(R-268 — closed.)*
demo guest, on a wire that exists solely between that host and that guest — not reachable from
your network or the internet, on a disposable machine with no customer data. Rotating it also
means updating the guest, so it is a deliberate act, not a background one. *(R-268)*
- **Our build-check alarm has one gap left.** A run that hangs is now cut off after five minutes and - **Our build-check alarm has one gap left.** A run that hangs is now cut off after five minutes and
the mail says how long it took — but **whether the alarm fires at all when the machinery kills a the mail says how long it took — but **whether the alarm fires at all when the machinery kills a
run outright is still unverified**, and we have not claimed otherwise. *(R-265)* run outright is still unverified**, and we have not claimed otherwise. *(R-265)*
@@ -250,7 +250,276 @@ are Gate 0 preparation acts by the operator, recorded because they are mutations
--- ---
## 7. What happens next ## 7. P1 — removing Felhom from the host
P1 — remove Felhom from demo-hp with `--uninstall`, then measure what it leaves behind. Nothing in `--uninstall`, the product's own path, dry-run first. **60 seconds** (08:37:23 → 08:38:23 UTC), exit 0.
§1–§4 is repeated after that point; the "before" is fixed as of this file. The typed-vmid confirmation was satisfied, not bypassed.
### 7.1 The reversal, proved rather than trusted
| component | before | after |
|---|---|---|
| agent binary | `/usr/local/bin/felhom-agent` 0.127.0 | **ABSENT** |
| `felhom-agent` service user | present | **ABSENT** |
| `felhom-op` (OOB) user | present | **ABSENT** |
| `/etc/felhom-sshd` (OOB belt) | present | **ABSENT** |
| systemd units | agent, mgmt-watchdog, felhom-sshd, oob-nft, shared-parent | **all removed** |
| pveum roles | `FelhomAgentBase/Guest/Store` | **none** |
| pveum user + token | `felhom-agent@pve!agent` | **gone**; user list is `root@pam` only |
| pveum ACL | pool + 9 scratch-vmid + 3 storage grants | **empty** |
| `felhom` pool | members `[9201]` | **`[]` — pool removed** |
| guest 9201 | running | **destroyed** |
| guest-hook snippet | present | **removed** |
| dnsmasq snippets | present | **removed** |
| `/var/lib/felhom-agent`, `/var/lib/felhom-install` | present | **removed** |
| **KEPT as declared** | | PBS snapshots (2), local vzdump (4), `/mnt/felhom-drives/Felhom-Share`, storage defs |
Space returned: `local-lvm` 20 904 790 → 12 355 143 KiB (**≈8.5 GiB**); `local` ≈64 MiB; the NVMe
unchanged (backups deliberately kept). `drill-r50` (VM 300) untouched, as predicted.
### 7.2 What it left behind that it did not declare
**F-8 · Five orphaned `agent.json.*` backups survive, each carrying a hub API key and a Proxmox
token.** `/etc/felhom-agent/` remains with `agent.json.campaign8-before`, `.campaign9-before`,
`.campaign9-prev`, `.pre-e-target-move`, `.pre-prunegate.bak` — all 0600, three owned by the **now-deleted
service user's uid 999**, each holding a 64-char `hub.api_key` and a 59-char `proxmox.token`.
The uninstall's own WIPED statement claims it removes *"config (+ its .bak backups)"*, and
`scripts/CHANGELOG` records F1 as *"uninstall now purges the agent config's `.bak*` siblings (one held a
live hub api_key)"*. **That fix does not match the filenames in use** — and note it misses
`agent.json.pre-prunegate.bak`, a file that literally ends in `.bak`.
**Assessed rather than assumed: these particular secrets are SUPERSEDED, not live.** The orphaned
`hub.api_key` hashes to `a5d2222a…`; the hub's current key for demo-hp is `8c59d1b6…`. The Proxmox token
was deleted by this same uninstall. **The exposure is timing-dependent, not absent** — a backup taken
shortly before an uninstall would hold a current key, which is exactly what R-179's precedent records.
The broken thing is the stated guarantee.
**And the reinstall hands them to the new service account.** The rebuilt box recreates `felhom-agent` at
**uid 999 — the same uid the deleted one had** — so the three backups owned by uid 999 become the new
account's files. Verified: `su -s /bin/sh felhom-agent -c 'test -r …'` succeeds on
`agent.json.campaign8-before` and `agent.json.pre-prunegate.bak`. A fresh install's service account
inherits read access to the previous install's credentials. Superseded here; live if the backups are recent.
**F-9 · A stale sudoers file survives — and it is INERT. Hypothesis raised, then refuted by test.**
`/etc/sudoers.d/felhom-agent.bak-pre-e2a` (21 833 bytes, 2026-07-28, 0440 root:root) survives the
uninstall and grants `felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, …`,
and `visudo -c -f` on it says **`parsed OK`**.
The obvious worry — that the reinstall recreates the `felhom-agent` user and the box silently regains a
pre-E-2a privilege set — **was tested against the rebuilt box and is FALSE.** With the user recreated
(uid 999) and this as the *only* felhom file in `/etc/sudoers.d`, `sudo -l -U felhom-agent` returns
**"User felhom-agent is not allowed to run sudo on felhom-host."** sudo ignores files in `sudoers.d`
whose names contain a dot, so `…​.bak-pre-e2a` is never loaded. That is a robust reason, not luck —
`visudo -c -f <file>` checks a file explicitly and therefore does **not** tell you whether sudo loads it.
**What is left is rank 4:** 21 KB of the privilege grammar lingering on a host Felhom was told to leave,
inside a teardown that claims to have removed the sudoers file.
**F-10 · The WireGuard tunnel to Felhom's off-site endpoint stays up.** After an uninstall,
`wg-quick@wg-felhom` is **enabled and active**, `/etc/wireguard/wg-felhom.conf` present, with a live
handshake to `167.233.158.164:443` 52 seconds old and 5.86 GiB received / 2.48 GiB sent on the counter.
It appears in **neither** the WIPED nor the KEPT list. A box told to leave Felhom keeps a live inbound
route into Felhom's infrastructure, and the operator is not told.
**F-11 · Smaller undeclared residue:** `/etc/felhom/` (`.bootstrap-done`, `appliance-pairing-code` —
the console pairing code); `/etc/systemd/system/felhom-bootstrap.service` + `/usr/local/sbin/felhom-bootstrap.sh`;
the `vmbr9` island-bridge stanza (169.254.253.1/30) still in `/etc/network/interfaces`.
**F-12 · A teardown-ordering warning:** `user config - ignore invalid acl role 'FelhomAgentStore'` —
the role is deleted before an ACL that still references it. Harmless here; noise that would mask a real one.
---
## 8. P2 — the preflight, as the owner runs it
### 8.1 The refusals, judged on their messages
**First run, no caps:** `byo mode requires explicit --cores and --memory (noisy-neighbor caps on a host
you do not own)`. **Good** — names the missing thing and the reason.
**Second run:** it refused again —
```
[ERROR] a resolver is already bound to :53 on this host — Felhom needs the guest reachable by name on your LAN.
Stop or reconfigure that resolver, OR point your LAN DNS at the guest's address, then re-run.
(Felhom does NOT touch DNS services on a host it does not own — this is a refusal, not a change.)
```
As a *message* this is excellent: the finding, the evidence, two routes out, and an explicit promise
about what it will not do.
### 8.2 F-13 (RANK 1 — STOPS THE VISIT) · Felhom's uninstall leaves the exact condition that makes Felhom's reinstall refuse
The resolver it is refusing over **is Felhom's own**, and the chain is fully evidenced:
| fact | evidence |
|---|---|
| Felhom installed `dnsmasq` | `/var/lib/dpkg/info/dnsmasq.list` dated **2026-07-21 18:24 CEST** — demo-hp's Felhom day-0 |
| it was constrained by a Felhom snippet | `/etc/dnsmasq.d/` held Felhom snippets before; only `README` after |
| **the uninstall restarted it unconstrained** | the running process started **2026-08-09 10:37:39 CEST — inside the uninstall window (10:37:2310:38:23)** |
| unconstrained, it grabs the wildcard | `0.0.0.0:53` and `[::]:53`, `--local-service`, empty `dnsmasq.conf` |
| it is not PVE's | `/etc/pve/sdn/` empty; unit is stock `/usr/lib/systemd/system/dnsmasq.service`, **enabled** |
The uninstall mentions it only as *"the 'sudo' and 'dnsmasq' packages were left installed (**system
packages**)"* — but dnsmasq is not a system package here; Felhom installed it, and it left it **enabled
and running as a wildcard resolver**.
**What a customer would do next:** read a message that says a resolver is in the way, conclude their own
network setup is at fault, and start debugging DNS they never configured. On a BYO box with no shell
guidance, this is where the visit stops.
**Confirmed by counterfactual:** `systemctl stop dnsmasq && systemctl disable dnsmasq` → `host DNS (:53):
free` → **PRE-FLIGHT PASS**. Nothing else changed. *(Recorded as an intervention — it is the action the
message tells the owner to take.)*
### 8.3 F-14 (RANK 2) · A local golden is adopted with NO version and NO checksum check
Preflight reported `golden (local): local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst`. Step 7 is:
```sh
if [[ -n "$GOLDEN_VOLID" ]] && ! $FORCE_GITEA_GOLDEN; then
log_skip " using local golden: $GOLDEN_VOLID"
_state_mark golden; return 0
fi
```
**The hub manifest's `golden.sha256` — the whole point of the hub vouching a different trust root than
Gitea — is consulted only on the fetch path.** A locally-present archive bypasses the vouch entirely: no
version comparison, no digest, no warning.
**What that archive actually contains, read out of the tar:**
`gitea.dooplex.hu/admin/felhom-controller:`**`0.192.0`**, against a vouched golden of **0.210.0**.
Two consequences, and the second is the serious one:
1. The rebuilt box comes up **18 controller releases stale**, reporting success.
2. **0.192.0 is below 0.200.0 — the release where R-193's off-site recovery SCREEN shipped.** So a
customer reinstalled today returns on a controller that cannot run the recovery ceremony their data
depends on. It is also born below the managed-update floor (0.200.0), and the updater's auto-target
is the floor, never the newest — so it converges to 0.200.0, still not the vouched build.
**This is the R-111/R-115/R-120 drift family, one layer down:** the R-120 gate guards what the operator
may *vouch*; nothing guards what an install actually *takes*. It also compounds with P1 — the uninstall
deliberately keeps the old golden (*"golden vzdump left in place"*), and the reinstall then adopts it.
**The walk continues unflagged** (`--force-gitea-golden` NOT passed), because measuring the path a real
reinstall takes is the point. §9 records which controller version actually came up.
### 8.4 The BYO mutation disclosure — read as a customer
Comprehensive and honest: every pveum artifact, the sudoers file, the state and config paths, the
packages it will install, the guest's capped size, the self-update authority, and — creditably — the
**outbound WireGuard tunnel**, in plain words, with the note that the operator can disable it per
customer. It closes with `NOT touched in byo mode: root@pam (no break-glass), host DNS (:53), OOB sshd.`
**One material omission, given §8.3:** it does not say that a pre-existing local golden may be used
instead of the vouched one. The disclosure lists everything the install will *create* and nothing about
what it may *reuse*.
---
## 9. P3 — the install. IT FAILED, AND THE WALK STOPS HERE
Started 08:51:33 UTC, **failed 08:52:17 UTC (44 s), exit 1, at step 5/8.** Run unflagged, as a real
owner would. The typed hostname acknowledgement was given, not bypassed.
```
[INFO] manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.210.0
[OK] verified sha256 c6eba73bf9b9ad69… matches the hub manifest
[OK] installed /usr/local/bin/felhom-agent (felhom-agent 0.128.0)
[OK] created service user felhom-agent
curl: (22) The requested URL returned error: 404
[ERROR] raw fetch failed: configs/felhom-mkfs-guarded.sh (agent tag v0.128.0 — is that version tagged in felhom-agent?)
```
### 9.1 F-15 (RANK 1 — STOPS EVERY INSTALL, FLEET-WIDE) · The vouched agent version was never tagged
**`v0.128.0` does not exist as a git tag** — not locally, not on Gitea. The newest tag is `v0.127.0`.
The *binary* was published to the Gitea package registry (which is why it is selectable in the hub's
artifact dropdown and why its sha256 verified), but the release's tag step never happened.
| ref | HTTP |
|---|---|
| `raw/branch/main/configs/felhom-mkfs-guarded.sh` | **200** |
| `raw/tag/v0.127.0/configs/felhom-mkfs-guarded.sh` | **200** |
| **`raw/tag/v0.128.0/configs/felhom-mkfs-guarded.sh`** | **404** |
The commit that *is* 0.128.0 exists and is on `main`: `28ba8593b8 — v0.128.0 — R-221: the escrow seed is
asserted every tick, not remembered once`. Only the tag is missing.
**This is R-184 happening.** R-184 reads: *"Nothing prevents the hub from vouching an agent version that
was never released. The R-115 gate proves every RELEASED version is installable, but it works from git
tags."* It was filed as a gap. It is now a live outage.
**It is the collision of a good fix with an open gap.** R-183 correctly hardened the installer to pin
config fetches to the vouched tag (`hostinstall_gates.py` even asserts the pin exists —
`ok: fetch_raw pins the agent configs to the vouched agent version`). Nothing asserts that the pinned ref
**resolves**. So the first vouch naming an untagged version fails every install closed.
**The release script already knows.** `felhom-agent/scripts/release-agent.sh:23` says in as many words:
*"`raw/tag/v<version>`, a released version without a git tag 404s a box mid-install, as root."* The
hazard was understood, documented, and the tag step was still missed.
**Blast radius — stated precisely, not inflated:**
- **Every fresh install and every reinstall, on every host, fails at step 5/8** for as long as the Day-0
manifest vouches agent 0.128.0. That manifest was approved on 2026-08-09; the approval is what armed it.
- **Existing boxes are unaffected.** demo-felhom runs 0.128.0 happily — the self-update path takes the
binary from the package registry, not from a tag.
- So the damage is confined to the install path. That is also the only path a new customer, or a
reinstalling tester, ever walks.
**What a customer would do next:** nothing. `curl: (22) … 404` and a question about a git tag is not a
message a household can act on. The visit ends here and a developer is required.
### 9.2 The box's state, left as found for inspection
`--uninstall` is not re-run and no manual repair is attempted (rule 1). demo-hp is mid-install:
```
state.json completed: [preflight, token, grows, enroll] mode: byo
agent 0.128.0 installed, unit NOT created (not-found), inactive
user felhom-agent uid=999 created
pveum 2 users, 3 roles, 63 ACL lines, pool 'felhom' recreated (empty)
guest none
```
The documented `--resume` path exists and **was deliberately not used: it cannot help.** Resume replays
from `enroll`, hits the same 404, and fails identically — the missing tag is not a transient.
### 9.3 The one-line unblock, for the operator to decide
```sh
cd /mnt/5_hdd/felhom.eu/git/felhom-agent
git tag v0.128.0 28ba8593b8 && git push origin v0.128.0
```
This publishes the tag for a commit that is already on `main` and already released as a package — it
completes a half-finished release rather than changing any code. The alternative, rolling the manifest
back to agent 0.127.0, costs R-221 — the fix `STATUS.md` calls *"the last item blocking a customer from
something we promise them"* — so it is the worse option.
**Not done in this session.** Publishing a release tag is an outward-facing act and the runbook is
explicit: *"If a defect is so severe that the run cannot continue, stop and report; do not fix it here."*
### 9.4 What F-14 still owes
Step 7 never ran, so **whether the install would actually have adopted the stale 0.192.0 golden is not
yet observed** — F-14 rests on the code path plus the preflight's `golden (local): …9100…2026_08_03…`
line and the archive's own `felhom-controller:0.192.0` marker. That is strong, but it is not the
observation. **When the tag is published and the walk resumes, the first thing to record at step 7/8 is
which golden it takes.**
---
## 10. Where the run stands
**Reached: P1 ✓, P2 ✓, P3 ✗ (blocked).** P4P7 not attempted.
**The honest answer to §2's question is: NOT YET ANSWERABLE, and the reason is itself the answer for
today.** A machine wiped today cannot be set up again at all — not because the data is gone (it is
safe, in snapshot `41c830db` and in two local tiers), but because the install refuses at step 5 of 8 on
a missing git tag. The walk found a hard stop before it ever reached the question about data.
**The dataset is intact and waiting.** Nothing about Gate 0 needs redoing when the walk resumes:
snapshot `41c830db` holds all four files, the before-manifest is committed, and the comparator is proven.
@@ -0,0 +1,167 @@
=== captured 2026-08-09T08:38:48Z UTC ===
### agent binary
ABSENT
### agent unit
felhom-bootstrap.service disabled enabled
### service user
id: 'felhom-agent': no such user
ABSENT
### felhom-op user (OOB)
id: 'felhom-op': no such user
ABSENT
### sudoers
total 40
drwxr-xr-x 2 root root 4096 Aug 9 10:37 .
drwxr-xr-x 101 root root 4096 Aug 9 10:37 ..
-r--r----- 1 root root 1068 Apr 11 14:21 README
-r--r----- 1 root root 21833 Jul 28 11:29 felhom-agent.bak-pre-e2a
-r--r----- 1 root root 666 May 20 2023 zfs
/etc/sudoers.d/felhom-agent.bak-pre-e2a
### pct list
### qm list
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
300 drill-r50 stopped 8192 32.00 0
### pools
[]
### pveum users/roles/acl
+----------+---------+-------------------+--------+--------+-----------+--------+------+----------+------------+------------------+--------+-------------+
| userid | comment | email | enable | expire | firstname | groups | keys | lastname | realm-type | tfa-locked-until | tokens | totp-locked |
+==========+=========+===================+========+========+===========+========+======+==========+============+==================+========+=============+
| root@pam | | noreply@felhom.eu | 1 | 0 | | | | | pam | | | |
+----------+---------+-------------------+--------+--------+-----------+--------+------+----------+------------+------------------+--------+-------------+
--roles--
no felhom roles
--acl--
### pvesm status
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
c11-scratch dir active 983379700 6940296 926412792 0.71%
felhom-backup dir active 983379700 6940296 926412792 0.71%
felhom-pbs pbs active 0 0 0 0.00%
local dir active 40516856 19063736 19362728 47.05%
local-lvm lvmthin active 56545280 12355143 44190136 21.85%
### storage.cfg
dir: local
path /var/lib/vz
content iso,import,vztmpl,backup
lvmthin: local-lvm
thinpool data
vgname pve
content rootdir,images
pbs: felhom-pbs
datastore felhom-offsite
server 10.77.0.1
content backup
encryption-key 3f:4f:65:c0:d8:f3:9f:3c:43:29:8c:c8:61:3c:89:40:e6:7a:c8:ae:b1:44:70:09:0e:f1:f7:1b:60:a0:c4:99
fingerprint c6:07:28:3f:5b:7b:5a:41:90:28:d7:ca:4f:37:14:70:56:39:2e:2f:0b:71:e8:06:ca:60:4a:d5:56:5f:3c:fd
namespace demo-hp
username felhom@pbs!demo-hp
dir: felhom-backup
path /mnt/nvme-1tb
content backup
is_mountpoint 1
dir: c11-scratch
path /mnt/nvme-1tb
content images
### df
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-root 39G 19G 19G 50% /
/dev/nvme0n1 938G 6.7G 884G 1% /mnt/nvme-1tb
### mounts under /mnt
/mnt/nvme-1tb /dev/nvme0n1
### felhom-drives tree
/mnt/felhom-drives/:
total 12
drwxr-xr-x 3 root root 4096 Aug 3 08:05 .
drwxr-xr-x 6 root root 4096 Aug 6 14:39 ..
drwxr-xr-x 2 root root 4096 Jul 22 13:53 Felhom-Share
/mnt/felhom-drives/Felhom-Share:
total 8
drwxr-xr-x 2 root root 4096 Jul 22 13:53 .
drwxr-xr-x 3 root root 4096 Aug 3 08:05 ..
### leftover felhom paths
/etc/felhom
/etc/felhom-agent
/usr/local/sbin/felhom-bootstrap.sh
none of the usual paths remain
### systemd leftovers on disk
-rw-r--r-- 1 root root 1662 Jul 21 18:20 felhom-bootstrap.service
### guest hook snippets
total 8
drwxr-xr-x 2 root root 4096 Aug 9 10:37 .
drwxr-xr-x 7 root root 4096 Jul 30 12:36 ..
### dnsmasq snippets
no felhom dnsmasq snippets
### wg tunnel
interface: wg-felhom
public key: KEYSzMyTKEZVDqDbikDXnadyeX9yJUjy8T58q0f4/1U=
private key: (hidden)
listening port: 59688
peer: f3d1ZI7/2/r+FWz8eo8Wl2HVygTZK9wBIN2M3dNHsgA=
endpoint: 167.233.158.164:443
allowed ips: 10.77.0.1/32, 10.77.0.250/32
latest handshake: 52 seconds ago
transfer: 5.86 GiB received, 2.48 GiB sent
vmbr9 DOWN 00:00:00:00:00:00 <NO-CARRIER,BROADCAST,MULTICAST,UP>
wg-felhom UNKNOWN <POINTOPOINT,NOARP,UP,LOWER_UP>
### network bridges
auto vmbr9
iface vmbr9 inet static
address 169.254.253.1/30
bridge-ports none
bridge-stp off
bridge-fd 0
### listening ports
State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0 4096 100.76.96.79:39440 0.0.0.0:* users:(("tailscaled",pid=1152166,fd=24))
LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=1052,fd=6))
LISTEN 0 32 0.0.0.0:53 0.0.0.0:* users:(("dnsmasq",pid=1291306,fd=5))
LISTEN 0 4096 0.0.0.0:111 0.0.0.0:* users:(("rpcbind",pid=701,fd=4),("systemd",pid=1,fd=97))
LISTEN 0 100 127.0.0.1:25 0.0.0.0:* users:(("master",pid=1230,fd=13))
LISTEN 0 4096 127.0.0.1:85 0.0.0.0:* users:(("pvedaemon worke",pid=1179410,fd=6),("pvedaemon worke",pid=1167090,fd=6),("pvedaemon worke",pid=1015859,fd=6),("pvedaemon",pid=1283,fd=6))
LISTEN 0 100 [::1]:25 [::]:* users:(("master",pid=1230,fd=14))
LISTEN 0 4096 *:3128 *:* users:(("spiceproxy work",pid=681975,fd=6),("spiceproxy",pid=1370,fd=6))
LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=1052,fd=7))
LISTEN 0 32 [::]:53 [::]:* users:(("dnsmasq",pid=1291306,fd=7))
LISTEN 0 4096 [::]:111 [::]:* users:(("rpcbind",pid=701,fd=6),("systemd",pid=1,fd=99))
LISTEN 0 4096 [fd7a:115c:a1e0::ce36:6051]:51957 [::]:* users:(("tailscaled",pid=1152166,fd=25))
LISTEN 0 4096 *:8006 *:* users:(("pveproxy worker",pid=1245509,fd=6),("pveproxy worker",pid=1244391,fd=6),("pveproxy worker",pid=1241021,fd=6),("pveproxy",pid=1295,fd=6))
### PBS snapshots (must be KEPT)
Volid Format Type Size VMID
felhom-pbs:backup/ct/9201/2026-07-28T19:19:45Z pbs-ct backup 6264034049 9201
felhom-pbs:backup/ct/9201/2026-08-04T19:24:16Z pbs-ct backup 4637840513 9201
### local vzdump (must be KEPT)
Volid Format Type Size VMID
felhom-backup:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst tar.zst backup 649547835 9100
felhom-backup:backup/vzdump-lxc-9201-2026_08_07-04_34_58.tar.zst tar.zst backup 2073915792 9201
felhom-backup:backup/vzdump-lxc-9201-2026_08_08-04_37_15.tar.zst tar.zst backup 2142232544 9201
felhom-backup:backup/vzdump-lxc-9201-2026_08_09-04_42_15.tar.zst tar.zst backup 2143026831 9201
@@ -0,0 +1,755 @@
[INFO] felhom-host-install v1.25.0 — mode=uninstall
[STEP] UNINSTALL — local host teardown
9201
[INFO] no --vmid given; using recorded provisioned_vmid=9201 from /var/lib/felhom-install/state.json
[INFO] vmid 9201 looks like a Felhom guest (has the /etc/felhom-bootstrap mount)
[INFO] pct config 9201:
arch: amd64
cores: 7
features: nesting=1,keyctl=1
hookscript: local:snippets/felhom-guest-hook.sh
hostname: demo-hp
memory: 26906
mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:F6:C5:F5,ip=dhcp,type=veth
net1: name=eth1,bridge=vmbr9,hwaddr=BC:24:11:CC:43:3E,ip=169.254.253.2/30,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-9201-disk-0,size=32G
swap: 512
unprivileged: 1
Type the vmid (9201) to confirm PERMANENT destruction: perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
Logical volume "vm-9201-disk-0" successfully removed.
Logical volume "vm-9201-disk-1" successfully removed.
[OK] guest 9201 destroyed
[STEP] host-level removal
Removed '/etc/systemd/system/multi-user.target.wants/felhom-agent.service'.
Removed '/etc/systemd/system/timers.target.wants/felhom-mgmt-watchdog.timer'.
Removed '/etc/systemd/system/multi-user.target.wants/felhom-sshd.service'.
Removed '/etc/systemd/system/multi-user.target.wants/felhom-oob-nft.service'.
The unit files have no installation config (WantedBy=, RequiredBy=, UpheldBy=,
Also=, or Alias= settings in the [Install] section, and DefaultInstance= for
template units). This means they are not meant to be enabled or disabled using systemctl.
 
Possible reasons for having these kinds of units are:
• A unit may be statically enabled by being symlinked from another unit's
 .wants/, .requires/, or .upholds/ directory.
• A unit's purpose may be to act as a helper for some other unit which has
 a requirement dependency on it.
• A unit may be started when needed via activation (socket, path, timer,
 D-Bus, udev, scripted systemctl call, ...).
• In case of template units, the unit is meant to be enabled with some
 instance name specified.
Removed '/etc/systemd/system/pve-guests.service.wants/felhom-shared-parent.service'.
Removed '/etc/systemd/system/multi-user.target.wants/felhom-shared-parent.service'.
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
user config - ignore invalid acl role 'FelhomAgentStore'
[SKIP] old broad role FelhomAgent already absent
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[SKIP] golden vzdump left in place (pass --remove-golden to remove)
[STEP] kept vs wiped — read before pulling drives or closing the customer out
@@ -0,0 +1,63 @@
[INFO] felhom-host-install v1.25.0 — mode=byo customer=demo-hp vmid=9201
[STEP] 1/8 pre-flight
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
[INFO] node: felhom-host (auto)
[INFO] agent config: /etc/felhom-agent/agent.json
[INFO] agent: not installed yet — will be fetched + installed in step 5/8
[INFO] local-lvm free: ~42 GiB
[WARN] local-lvm free ~42 GiB < hard min 120 GiB
[INFO] free RAM: ~26457 MiB
[INFO] existing guests on this host: 1 (pct+qm)
[INFO] archive storage 'local' present
@@ -0,0 +1,28 @@
=== half-installed state 2026-08-09T08:54:00Z ===
### agent
felhom-agent 0.128.0
inactive
not-found
### install state file
{
"completed": [
"preflight",
"token",
"grows",
"enroll"
],
"customer_id": "demo-hp",
"mode": "byo"
}### service user
uid=999(felhom-agent) gid=990(felhom-agent) groups=990(felhom-agent),999(systemd-journal)
### sudoers now present
-r--r----- 1 root root 21833 Jul 28 11:29 felhom-agent.bak-pre-e2a
### pveum
2
3
63
### pool
[{"comment":"Felhom-managed guests","poolid":"felhom"}]
### guest
### wrappers
-rwxr-xr-x 1 root root 15730 Jul 21 18:20 /usr/local/sbin/felhom-bootstrap.sh
@@ -0,0 +1,768 @@
[INFO] felhom-host-install v1.25.0 — mode=byo customer=demo-hp vmid=9201
[STEP] 1/8 pre-flight
felhom-host
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
[INFO] node: felhom-host (auto)
[INFO] agent config: /etc/felhom-agent/agent.json
[INFO] agent: not installed yet — will be fetched + installed in step 5/8
[INFO] local-lvm free: ~42 GiB
[WARN] local-lvm free ~42 GiB < hard min 120 GiB
[INFO] free RAM: ~26440 MiB
[INFO] existing guests on this host: 1 (pct+qm)
[INFO] archive storage 'local' present
[INFO] acl storages all present or pre-positioned: local local-lvm felhom-pbs
[INFO] host DNS (:53): free
[INFO] hub reachable (https://hub.felhom.eu)
[OK] customer 'demo-hp' exists + passphrase valid
[INFO] golden (local): local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst
[STEP] BYO disclosure — installing Felhom on a host the operator does NOT own will create:
pveum: roles FelhomAgentBase/FelhomAgentGuest/FelhomAgentStore + user felhom-agent@pve + token felhom-agent@pve!agent
+ scoped ACL grants (Base@/, Guest@/pool/felhom, Store@[local local-lvm felhom-pbs]) + the 'felhom' pool
agent: service user felhom-agent + /usr/local/bin/felhom-agent (+ .bak-* backups; A/B slots via the guarded self-update)
+ /etc/systemd/system/felhom-agent.service (+ felhom-agent-rollback.service + start-limit drop-in)
+ /etc/sudoers.d/felhom-agent + state dir /var/lib/felhom-agent + config /etc/felhom-agent/agent.json (0600)
host: /usr/local/sbin/felhom-mkfs-guarded + /usr/local/sbin/felhom-selfupdate-guarded
+ /usr/local/sbin/felhom-pbs-apply (PBS-DR apply wrapper — DR capability is baked
on every install; ACTIVATION stays a hub flag, off = zero effect on this host)
+ felhom-mgmt-watchdog service+timer+script + /etc/tmpfiles.d/felhom-privsep.conf
+ guest-hook snippet under /var/lib/vz/snippets/ (agent-installed at runtime)
+ the 'sudo' and 'age' packages if absent + install state dir /var/lib/felhom-install
wg: an OUTBOUND WireGuard tunnel to the Felhom hub (wg_tunnel.enabled=true — base
infrastructure like the cloudflared tunnel; hands-free peer registration; the
operator can disable it per-customer hub-side on request)
guest: the provisioned Felhom LXC (vmid 9201, capped 4 cores / 8192 MiB) + its volumes
+ the golden vzdump imported onto storage 'local'
update: operator-signed self-update authority: felhom-op-1
NOT touched in byo mode: root@pam (no break-glass), host DNS (:53), OOB sshd.
Type this host's short hostname (felhom-host) to acknowledge the byo install: [OK] byo install acknowledged for host felhom-host
[OK] pre-flight passed
[STEP] 2/8 Proxmox API token
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[OK] token minted (secret captured, not logged)
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[OK] scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm felhom-pbs])
[SKIP] old broad role FelhomAgent already absent
[STEP] 3/8 compute volume grows
[INFO] auto-computed from ~42 GiB free (ONE volume since R-165)
[INFO] grows: rootfs +0G (->32G), data +46G (->70G, ONE volume)
[STEP] 4/8 host enrollment (POST /host-enroll)
[OK] host REUSED (idempotent — existing credential)
[INFO] host_id: demo-hp-bb76ea (api_key captured, not logged)
[SKIP] break-glass (step 4b) SKIPPED (byo: the host owner manages break-glass; root@pam untouched, nothing vaulted)
[STEP] 5/8 agent install (fetch + verify + install)
[SKIP] apt repo alignment skipped (byo — the owner manages repos)
[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
[INFO] manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.210.0
[INFO] fetching agent binary v0.128.0 from Gitea …
[OK] verified sha256 c6eba73bf9b9ad69… matches the hub manifest
[OK] installed /usr/local/bin/felhom-agent (felhom-agent 0.128.0)
[OK] created service user felhom-agent
[OK] added felhom-agent to systemd-journal (unprivileged journal read for NAS verify)
curl: (22) The requested URL returned error: 404
[ERROR] raw fetch failed: configs/felhom-mkfs-guarded.sh (agent tag v0.128.0 — is that version tagged in felhom-agent?)
Connection to 100.76.96.79 closed.
+12 -1
View File
@@ -496,7 +496,18 @@ applied.** The one that matters: Scenario A **fails against today's tree** with
| **R-267** | **The Configuration page is 2.6× faster and is still ~10 s, and the remaining cost is ONE Gitea call whose latency swings 20× with load.** Reported by the operator as *"almost minutes to load"*, with the reasonable guess that it hashes artifacts on page load. **THAT GUESS DOES NOT HOLD and the code already said so** — Gitea stores each file's sha256 and `gitea.FileSHA256` reads it as metadata (*"the artifact bytes are never downloaded"*). The cost was latency × count. **Fixed in hub v0.100.00.100.2, three legs, each found by refusing to accept a number that did not match the arithmetic:** (1) the per-version sha lookups were **serial** — 2 packages × (1 search + 20 lookups) = 42 sequential calls; now concurrent, bounded 8. (2) The two dropdowns resolved **one after the other**; now side by side. (3) The client used `http.DefaultTransport`, whose **`MaxIdleConnsPerHost` is 2**, so under a 16-way fan-out nearly every call paid a fresh TCP setup *and* a fresh authentication — and authentication is the expensive half (`/api/v1/version` unauthenticated **0.03 s** vs an authenticated package call **0.24 s**). **Measured: 26.2 s → mean 9.85 s over 8 samples (min 5.13, max 18.13).** **THE REMAINING COST IS THE PACKAGE SEARCH** `/api/v1/packages/admin?type=generic&q=…&limit=100`, one per dropdown: **0.203.8 s each depending on load**, and running the two concurrently does not help (3.68 s for both together vs 3.82 s for one alone — Gitea appears to serialise them). By contrast **16 concurrent file-metadata calls take 0.58 s**. **⚠ EVERY NUMBER HERE IS CONTAMINATED and that is stated rather than hidden:** they were taken on DooPlex at load average 711 while this same session was building images, running two Go suites and baking a golden. The same search measured **3.8 s** in-cluster and **0.44 s** from the host ninety seconds later. **Re-measure on an idle box before deciding anything.** **The operator's two proposals, answered on the measurement:** *"reduce the number of artifacts"* — only **50 generic versions exist in total** (33 agent + 17 golden) and `limit=25` costs 0.20 s against `limit=100`'s 0.44 s, so pruning helps **somewhat and sub-linearly**; it is worth doing for its own sake (it pairs with R-210) but it is not the lever. *"hash on creation, store in the DB"***NOT RECOMMENDED, and the reason is a rule this project already holds**: Gitea IS the store, and a copy in `hub_settings` would be a second source of truth that can drift from the registry it describes, while the operator reads exactly that value to confirm what they are about to vouch. `golden_currency_gate.py` records the same reasoning for the vouched version. **The lever that would actually work, and the trade-off that makes it a DECISION rather than an implementation:** cache the version list + shas **in memory** with a short TTL and refresh in the background — the page becomes instant and bounded-stale, at the cost that a just-published artifact does not appear for up to the TTL. That is an operator call about how fresh the dropdown must be, and it is why this row is open rather than closed. **Cheaper interim, also a decision:** drop the dropdown cap from 20 to ~8, which cuts the fan-out but hides older versions from a rollback | **CLOSED 2026-08-08 — hub v0.101.0 + a registry prune.** **Final: cold 5.4 s, warm 0.14 s** (was 26.2 s). Three serialisation legs took it to 9.85 s mean, the 60 s in-memory memo took the warm path to a quarter-second, and the prune halved what remains of the cold path. **⚠ TWO CORRECTIONS TO THIS ROW'S OWN EARLIER TEXT, because both were wrong and both mattered.** **(1) "Only 50 generic versions exist" WAS NOT A COUNT, IT WAS A PAGE LIMIT.** `?type=generic&limit=1000` returns at most 50; the 50 I measured was exactly the cap, and three older agent versions (0.81.0, 0.80.0, 0.79.0) only became visible after the first 30 deletions moved them onto page one. **An unpaginated listing is not evidence of a total** — this repo's own "an empty listing is not evidence of emptiness" rule, walked into while measuring. **(2) THE OPERATOR'S "REDUCE THE NUMBER OF ARTIFACTS" WAS THE BETTER CALL AND MY MEASUREMENT SAID OTHERWISE.** I reported it helps "sub-linearly" and "is not the lever". Measured after: trimming to 10+10 took the COLD load from 13.4 s to 5.4 s — a 2.5× improvement on the path the memo cannot help, because the fan-out is per-version. Recorded rather than quietly dropped (the R-96 standing rule). **Pruned to the newest 10 per package on the operator's rule**, with the live-vouched golden/agent/floor asserted into the KEEP set before a single DELETE was issued; 33 deletions, all HTTP 204, and golden 0.210.0 / agent 0.128.0 / agent 0.127.0 verified still fetchable afterwards. `drill-r50` runs agent 0.113.0, now deleted — flagged to the operator first; it is a disposable nested drill VM and only its re-download path is gone | | **R-267** | **The Configuration page is 2.6× faster and is still ~10 s, and the remaining cost is ONE Gitea call whose latency swings 20× with load.** Reported by the operator as *"almost minutes to load"*, with the reasonable guess that it hashes artifacts on page load. **THAT GUESS DOES NOT HOLD and the code already said so** — Gitea stores each file's sha256 and `gitea.FileSHA256` reads it as metadata (*"the artifact bytes are never downloaded"*). The cost was latency × count. **Fixed in hub v0.100.00.100.2, three legs, each found by refusing to accept a number that did not match the arithmetic:** (1) the per-version sha lookups were **serial** — 2 packages × (1 search + 20 lookups) = 42 sequential calls; now concurrent, bounded 8. (2) The two dropdowns resolved **one after the other**; now side by side. (3) The client used `http.DefaultTransport`, whose **`MaxIdleConnsPerHost` is 2**, so under a 16-way fan-out nearly every call paid a fresh TCP setup *and* a fresh authentication — and authentication is the expensive half (`/api/v1/version` unauthenticated **0.03 s** vs an authenticated package call **0.24 s**). **Measured: 26.2 s → mean 9.85 s over 8 samples (min 5.13, max 18.13).** **THE REMAINING COST IS THE PACKAGE SEARCH** `/api/v1/packages/admin?type=generic&q=…&limit=100`, one per dropdown: **0.203.8 s each depending on load**, and running the two concurrently does not help (3.68 s for both together vs 3.82 s for one alone — Gitea appears to serialise them). By contrast **16 concurrent file-metadata calls take 0.58 s**. **⚠ EVERY NUMBER HERE IS CONTAMINATED and that is stated rather than hidden:** they were taken on DooPlex at load average 711 while this same session was building images, running two Go suites and baking a golden. The same search measured **3.8 s** in-cluster and **0.44 s** from the host ninety seconds later. **Re-measure on an idle box before deciding anything.** **The operator's two proposals, answered on the measurement:** *"reduce the number of artifacts"* — only **50 generic versions exist in total** (33 agent + 17 golden) and `limit=25` costs 0.20 s against `limit=100`'s 0.44 s, so pruning helps **somewhat and sub-linearly**; it is worth doing for its own sake (it pairs with R-210) but it is not the lever. *"hash on creation, store in the DB"***NOT RECOMMENDED, and the reason is a rule this project already holds**: Gitea IS the store, and a copy in `hub_settings` would be a second source of truth that can drift from the registry it describes, while the operator reads exactly that value to confirm what they are about to vouch. `golden_currency_gate.py` records the same reasoning for the vouched version. **The lever that would actually work, and the trade-off that makes it a DECISION rather than an implementation:** cache the version list + shas **in memory** with a short TTL and refresh in the background — the page becomes instant and bounded-stale, at the cost that a just-published artifact does not appear for up to the TTL. That is an operator call about how fresh the dropdown must be, and it is why this row is open rather than closed. **Cheaper interim, also a decision:** drop the dropdown cap from 20 to ~8, which cuts the fan-out but hides older versions from a rollback | **CLOSED 2026-08-08 — hub v0.101.0 + a registry prune.** **Final: cold 5.4 s, warm 0.14 s** (was 26.2 s). Three serialisation legs took it to 9.85 s mean, the 60 s in-memory memo took the warm path to a quarter-second, and the prune halved what remains of the cold path. **⚠ TWO CORRECTIONS TO THIS ROW'S OWN EARLIER TEXT, because both were wrong and both mattered.** **(1) "Only 50 generic versions exist" WAS NOT A COUNT, IT WAS A PAGE LIMIT.** `?type=generic&limit=1000` returns at most 50; the 50 I measured was exactly the cap, and three older agent versions (0.81.0, 0.80.0, 0.79.0) only became visible after the first 30 deletions moved them onto page one. **An unpaginated listing is not evidence of a total** — this repo's own "an empty listing is not evidence of emptiness" rule, walked into while measuring. **(2) THE OPERATOR'S "REDUCE THE NUMBER OF ARTIFACTS" WAS THE BETTER CALL AND MY MEASUREMENT SAID OTHERWISE.** I reported it helps "sub-linearly" and "is not the lever". Measured after: trimming to 10+10 took the COLD load from 13.4 s to 5.4 s — a 2.5× improvement on the path the memo cannot help, because the fan-out is per-version. Recorded rather than quietly dropped (the R-96 standing rule). **Pruned to the newest 10 per package on the operator's rule**, with the live-vouched golden/agent/floor asserted into the KEEP set before a single DELETE was issued; 33 deletions, all HTTP 204, and golden 0.210.0 / agent 0.128.0 / agent 0.127.0 verified still fetchable afterwards. `drill-r50` runs agent 0.113.0, now deleted — flagged to the operator first; it is a disposable nested drill VM and only its re-download path is gone |
| **R-268** | **A live per-guest local-API token was printed into a session transcript.** Done by me on 2026-08-08 while setting up R-221's live drill: a `python3` one-liner meant to list `bootstrap.json`'s keys printed the `local_api` object whole, including its `token`, for guest **9201 on demo-felhom**. **Reported immediately rather than quietly rotated**, because this project's rule is that a secret reaching a transcript is a finding whatever its blast radius. **THE EXPOSURE, ASSESSED RATHER THAN ASSUMED — it is small, and saying so is not the same as excusing it.** The token authorises only the agent's **per-guest** local API on `169.254.253.1:8443`, which listens on the island bridge `vmbr9` — an L2 segment between the PVE host and that one guest. It is not routable from the LAN or the internet, it is self-scoped to guest 9201 (the agent authorises against the token's own guest, never a caller-supplied id), and using it already requires code execution on that host or guest, at which point an attacker has strictly more than the token. The box is **Tier 0, disposable, no customer data**. **ROTATION IS AVAILABLE BUT IS NOT A ONE-LINER, which is why it was not done unannounced:** `localapi.TokenStore.Mint(vmid)` is last-write-wins per VMID (`tokenstore.go:127`, the store keeps only hashes), but the new plaintext must also be written into the guest's `/etc/felhom-bootstrap/bootstrap.json` or the in-guest controller loses its agent access — so it is an operator-timed act, not a background one. **The honest general fix is upstream of the incident:** reading a secret-bearing JSON should go through a helper that prints keys and never values, the same discipline the bake uses for the Gitea token (file → file, `grep -c` on the shape, never the value). Two of this fortnight's findings (R-249, R-132) are the same class from the other direction | **READY** — owner Viktor | | **R-268** | **A live per-guest local-API token was printed into a session transcript.** Done by me on 2026-08-08 while setting up R-221's live drill: a `python3` one-liner meant to list `bootstrap.json`'s keys printed the `local_api` object whole, including its `token`, for guest **9201 on demo-felhom**. **Reported immediately rather than quietly rotated**, because this project's rule is that a secret reaching a transcript is a finding whatever its blast radius. **THE EXPOSURE, ASSESSED RATHER THAN ASSUMED — it is small, and saying so is not the same as excusing it.** The token authorises only the agent's **per-guest** local API on `169.254.253.1:8443`, which listens on the island bridge `vmbr9` — an L2 segment between the PVE host and that one guest. It is not routable from the LAN or the internet, it is self-scoped to guest 9201 (the agent authorises against the token's own guest, never a caller-supplied id), and using it already requires code execution on that host or guest, at which point an attacker has strictly more than the token. The box is **Tier 0, disposable, no customer data**. **ROTATION IS AVAILABLE BUT IS NOT A ONE-LINER, which is why it was not done unannounced:** `localapi.TokenStore.Mint(vmid)` is last-write-wins per VMID (`tokenstore.go:127`, the store keeps only hashes), but the new plaintext must also be written into the guest's `/etc/felhom-bootstrap/bootstrap.json` or the in-guest controller loses its agent access — so it is an operator-timed act, not a background one. **The honest general fix is upstream of the incident:** reading a secret-bearing JSON should go through a helper that prints keys and never values, the same discipline the bake uses for the Gitea token (file → file, `grep -c` on the shape, never the value). Two of this fortnight's findings (R-249, R-132) are the same class from the other direction | **CLOSED — ROTATED + PROVEN LIVE 2026-08-09** (rehearsal pre-phase, `audits/REHEARSAL-byo-reinstall-2026-08-09.md` §3) | — | Old token **HTTP 401**, new token **HTTP 200**, controller channel back up with a positive observable (`local-api: channel up … 3 mount(s) visible`). No value reached stdout; both shredded. **The recipe in this row was INCOMPLETE and rotating it proved so** — writing `bootstrap.json` is not enough (→ **R-270**), and the store's own revocation claim is false across processes (→ **R-269**) | CC |
| **R-269** | **A rotated-out per-guest local-API token still authorises, and the test that appears to pin the opposite passes only because of its lookup ORDER.** `localapi.TokenStore.Mint` documents *"last-write wins — any previous token for this guest is revoked"*. Across processes that is FALSE until something unrelated forces a reload: the long-lived agent serves `Lookup` from an in-memory index and re-reads the store **only on a miss** (the B3 reload-on-miss optimisation, `tokenstore.go`), so a superseded token is a direct map **hit** and returns `(vmid, true)`. **Red-proved twice.** (a) A unit probe — `TestTokenStore_ReloadOnMiss_RemintCoherence` with the two lookups swapped, i.e. present the rotated-out token FIRST — **fails**; the shipped test passes only because it looks up the NEW token first, and that miss is what evicts the old hash. (b) On hardware, 2026-08-09: after the on-disk rotation the old token returned **HTTP 200**, then 401 only once a new-token lookup had forced the reload, and reliably 401 after `systemctl restart felhom-agent`. **This is the `CLAUDE.md` invariant-comment case exactly** — the comment reads as settled and the test that looks like its pin is order-dependent. **Fix options:** pin the reversed order with a test, or make eviction not depend on an unrelated miss. Until then, **an operator rotating a leaked token MUST restart the agent** — the runbook step is not optional | **READY (S) — NEW 2026-08-09** | — | Found by doing R-268's rotation rather than reading about it | CC |
| **R-270** | **R-268's stated rotation recipe is incomplete: the controller never re-reads `bootstrap.json`'s `local_api`, so a rotation leaves the agent channel dead across restarts.** `bootstrap.ensureLocalAPI` returns early when `cfg.LocalAPI.Endpoint != ""` — by design it FILLS an absent block and never refreshes a present one — so the token the controller uses lives in its own `controller.yaml`, not in the mount. Proved live 2026-08-09: two controller restarts after a correct `bootstrap.json` rotation, still `HTTP 401`; the channel came up only once `local_api.token` was written into `controller.yaml`. The neighbouring `DetectEndpointDrift` compares the ENDPOINT and deliberately does not compare the token (*"a token mismatch is a different failure"*), so this shape is knowingly unmodelled. Parent question — which file is authoritative — is **R-78** | **READY (S) — NEW 2026-08-09** | — | Either teach the drift detector the token, or make the rotation path write both files. Correct the R-268 row's recipe either way | CC |
| **R-271** | **The `agent_channel_unauthorized` alarm can never be closed, because its own prescribed remedy is what silences the recovery.** `channelhealth.Checker.Check`'s UP branch notifies only when `prev != "" && prev != "up"`; a controller restart resets `state` to `""`, so an unseeded→up transition is silent by construction. The alert text says *"token stale/rotated (**re-bootstrap**)"* — i.e. restart the controller — so **following the instruction guarantees no recovery event.** Observed live 2026-08-09: two `agent_channel_unauthorized` errors on the hub (one `sent`, one `suppressed` by the 1 h operator cooldown) and **nothing afterwards**, though the channel came up 3 minutes later and stayed up. The down side is deliberately asymmetric (F2: a born-down channel alerts on cycle 1); the up side never got the matching treatment. Customer dashboard is fine — `SetDashboard` reflects current state every cycle. It is the OPERATOR's trail that ends on "down" | **READY (S) — NEW 2026-08-09** | — | Notify on unseeded→up when the previous *persisted* state was down, or seed from the hub's last event | CC |
| **R-272** | **RANK 1 — Felhom's `--uninstall` leaves the exact condition that makes Felhom's own reinstall REFUSE.** Chain, fully evidenced on demo-hp 2026-08-09: Felhom installs `dnsmasq` at day-0 (`/var/lib/dpkg/info/dnsmasq.list` dated **2026-07-21 18:24 CEST**, demo-hp's day-0) and constrains it with a snippet in `/etc/dnsmasq.d/`; `--uninstall` removes the snippet and **restarts the daemon** (running process start time **2026-08-09 10:37:39 CEST — inside the 10:37:2310:38:23 uninstall window**) but leaves the package installed and the unit **enabled**; unconstrained, dnsmasq binds `0.0.0.0:53`; the next install's preflight then hard-refuses with *"a resolver is already bound to :53"*. It is **not** PVE SDN's (`/etc/pve/sdn/` empty; stock unit). The teardown mentions it only as *"the 'sudo' and 'dnsmasq' packages were left installed (**system packages**)"* — **dnsmasq is not a system package here, Felhom installed it.** **What a customer does next:** reads a message blaming a resolver, concludes their own LAN DNS is at fault, and debugs something they never configured. **Counterfactual confirmed:** `systemctl stop dnsmasq && systemctl disable dnsmasq``host DNS (:53): free` → PRE-FLIGHT PASS, nothing else changed. **The refusal MESSAGE is good** (finding, evidence, two routes, and an explicit promise not to touch DNS on a host it does not own) — the defect is that Felhom caused the condition and does not say so | **READY (M) — NEW 2026-08-09** | — | Either stop+disable dnsmasq on uninstall when Felhom installed it, or have the preflight recognise its own leftover and say so | CC |
| **R-273** | **RANK 1 — the hub vouched an agent version that was never git-tagged, and every install fleet-wide now fails at step 5/8.** `v0.128.0` exists as a Gitea **package** (so it is selectable in the artifact manifest and its sha256 verifies) but **not as a git tag** — newest tag is `v0.127.0`. Since R-183, `felhom-host-install.sh` correctly pins config fetches to `raw/tag/v$ART_AGENT_VER`, so the install dies: `curl: (22) … 404``raw fetch failed: configs/felhom-mkfs-guarded.sh (agent tag v0.128.0 — is that version tagged in felhom-agent?)`. Measured: `raw/branch/main/…`**200**, `raw/tag/v0.127.0/…`**200**, `raw/tag/v0.128.0/…`**404**. The commit that IS 0.128.0 is on `main` (`28ba8593b8`); only the tag is missing. **THIS IS R-184 ARRIVING** — that row says nothing prevents the hub vouching a never-released version because the R-115 gate works from tags. **A good fix met an open gap:** `hostinstall_gates.py` asserts the pin EXISTS (`ok: fetch_raw pins the agent configs to the vouched agent version`); nothing asserts the pinned ref RESOLVES. And `felhom-agent/scripts/release-agent.sh:23` already documents the hazard verbatim — *"a released version without a git tag 404s a box mid-install, as root"* — so it was understood and the step was still missed. **Blast radius, precise:** every fresh install and every reinstall is dead while the manifest vouches 0.128.0 (approved 2026-08-09 — the approval armed it); **existing boxes are unaffected** (self-update takes the binary from the package registry, and demo-felhom runs 0.128.0 healthily). **Unblock:** `git tag v0.128.0 28ba8593b8 && git push origin v0.128.0` — publishes a tag for a commit already on `main` and already released as a package. Rolling the manifest back to 0.127.0 instead would cost R-221. **Deliberately NOT done in the rehearsal session** (diagnose-not-fix; publishing a tag is outward-facing). **Owed gates:** (a) refuse a vouch whose tag does not resolve, (b) a release-completeness check that the package and the tag ship together | **WAITING-ON-OPERATOR** — one tag push | — | `audits/REHEARSAL-byo-reinstall-2026-08-09.md` §9 | Viktor |
| **R-274** | **A local golden is adopted with NO version and NO checksum check, so a reinstall can silently come up releases behind.** `felhom-host-install.sh` step 7: `if [[ -n "$GOLDEN_VOLID" ]] && ! $FORCE_GITEA_GOLDEN; then log_skip "using local golden"; return 0; fi`**the hub manifest's `golden.sha256`, whose entire purpose is to vouch from a different trust root than Gitea, is consulted only on the FETCH path.** A locally-present archive bypasses the vouch: no version compare, no digest, no warning. On demo-hp 2026-08-09 the preflight selected `local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst`, whose baked marker reads `felhom-controller:`**`0.192.0`**, against a vouched golden of **0.210.0** — 18 releases stale. **The sharp consequence:** 0.192.0 is **below 0.200.0, where R-193's off-site recovery SCREEN shipped**, so a customer reinstalled today returns on a controller that cannot run the recovery ceremony their data depends on; it is also born below the managed-update floor (0.200.0), and the updater's auto-target is the floor, never the newest. **It compounds with the teardown**, which deliberately keeps the old golden (*"golden vzdump left in place"*). This is the R-111/R-115/R-120 drift family one layer down: the R-120 gate guards what may be VOUCHED, nothing guards what an install TAKES. **NOT YET OBSERVED END-TO-END** — step 7 never ran because R-273 killed step 5, so this rests on the code path + the preflight line + the archive marker. **First thing to record when the walk resumes** | **READY (S) — NEW 2026-08-09** | R-273 | Compare the local golden's version/sha against the manifest and refuse or re-fetch on mismatch; say so in the BYO disclosure, which today lists only what the install CREATES, never what it REUSES | CC |
| **R-275** | **`--uninstall` leaves five 0600 `agent.json.*` credential backups, and the reinstall hands them to the new service account.** `/etc/felhom-agent/` survives with `agent.json.{campaign8-before,campaign9-before,campaign9-prev,pre-e-target-move,pre-prunegate.bak}`, each carrying a 64-char `hub.api_key` and a 59-char `proxmox.token`. The teardown claims to remove *"config (+ its .bak backups)"* and `scripts/CHANGELOG` F1 records *"uninstall now purges the agent config's `.bak*` siblings (one held a live hub api_key)"* — **that fix does not match the filenames in use, and it misses `agent.json.pre-prunegate.bak`, a file that literally ends in `.bak`.** **Exposure assessed, not assumed:** these are SUPERSEDED — the orphaned key hashes to `a5d2222a…`, the hub's current demo-hp key to `8c59d1b6…`, and the Proxmox token was deleted by the same uninstall. **But the reinstall recreates `felhom-agent` at uid 999, the same uid the deleted account had**, so three of the backups become the new account's files — verified readable as `felhom-agent`. A fresh install's service account inherits read access to the prior install's credentials; superseded today, live if the backups were recent (R-179's precedent). **Also left, undeclared:** `/etc/felhom/{.bootstrap-done,appliance-pairing-code}`, `felhom-bootstrap.service` + `/usr/local/sbin/felhom-bootstrap.sh`, the `vmbr9` stanza in `/etc/network/interfaces`, and `/etc/sudoers.d/felhom-agent.bak-pre-e2a` (21 KB — **INERT: sudo skips dotted filenames, verified with `sudo -l -U felhom-agent`; `visudo -c -f` parsing it OK is NOT evidence sudo loads it**) | **READY (S) — NEW 2026-08-09** | — | Purge by directory, not by glob; and do not let a new service account reuse a uid that owns old secrets | CC |
| **R-276** | **RANK 2 — an uninstalled box keeps a live WireGuard tunnel into Felhom's off-site endpoint, and the teardown says nothing.** After `--uninstall` on demo-hp, `wg-quick@wg-felhom` is **enabled and active**, `/etc/wireguard/wg-felhom.conf` present, handshake to `167.233.158.164:443` **52 s old**, counters 5.86 GiB in / 2.48 GiB sent. It appears in **neither** the WIPED nor the KEPT list, though the BYO install disclosure names it prominently on the way in (*"an OUTBOUND WireGuard tunnel to the Felhom hub"*). A host told to leave Felhom retains a live network path into Felhom infrastructure, its hub-side peer registration intact, and nobody is told | **READY (S) — NEW 2026-08-09** | — | Tear the tunnel down and deregister the peer, or list it under KEPT with the reason and the removal command | CC |
| **R-277** | **Three hub surfaces jointly present a HEALTHY off-site tier as an absent one — and it produced a wrong operator statement during this run.** For demo-hp on 2026-08-09 the box was pushing off-site daily without a gap (18 restic snapshots, `last_status: ok`), yet: (a) the customer page's Backup panel read `Snapshots 0 / Repo Size 0 MB / Integrity Unknown` — it renders the **local disk tier**, while the healthy `offsite` object sits **in the same report** unrendered on that panel; (b) the Offsite page read `0.0 GB` — true, but a 162 KB repo rounds to nothing; (c) a stale `offsite_delivery_stuck` event from **2026-08-07 10:19** (not recurring) reads as current state. **Three independent surfaces agreeing on a wrong picture is how a working backup gets "fixed".** It did exactly that here: the rehearsal reported a fleet-wide off-site outage to the operator and had to retract it. **Note the true half:** demo-felhom IS genuinely stuck (`offsite.state=needs_credential`, no run has ever succeeded) → **R-278** | **READY (S) — NEW 2026-08-09** | — | Render the offsite object on the offsite row; show bytes not rounded GB; distinguish a live alarm from event history | CC |
| **R-278** | **demo-felhom's off-site tier has never completed a run and has been stuck for six days.** `offsite.state=needs_credential` since the 2026-08-03 guest rebuild; the hub's own alarm reads *"enabled + escrowed but no run has EVER succeeded"*; the controller's `offsite-credential-retry` job runs every 5 minutes and completes in 0 s, doing nothing. R-193's fix (the recovery SCREEN, controller 0.200.0) is present on the box, so the remedy exists — it just needs the customer-present ceremony that nobody has run, which is R-243's shape (*"a machine waiting for its recovery code can stop backing up off-site without alarming us"*) landing on a real box. **Contrast that makes it a defect and not a chore:** demo-hp, same rebuild, same day, recovered and has 18 snapshots | **READY (S) — NEW 2026-08-09** | — | Either the self-heal reconciler owns this shape end-to-end, or the box must say plainly on the dashboard that it is unprotected pending the recovery code | CC |
| **R-279** | **There is no operator-triggerable off-site backup.** The only route to `POST /backup/offbox/run` is the customer's own dashboard session; `signed_jobs` carries opaque operator-SIGNED blobs and the hub holds no signing key. This cost the rehearsal a stop: preparing the run needed one off-site push and there was no operator path to it. Sibling of **R-177** (no operator-triggerable fill check) | **READY (XS) — NEW 2026-08-09** | — | Same shape as R-177; solve both together | CC |
**Explicitly still open, untouched by this session:** R-246, R-255, R-256, R-257, R-261, R-262, **Explicitly still open, untouched by this session:** R-246, R-255, R-256, R-257, R-261, R-262,
R-263, **R-264** (the twenty-one undecided facts — a design session of its own), R-240, R-243, R-263, **R-264** (the twenty-one undecided facts — a design session of its own), R-240, R-243,