Merge branch 'main' of https://gitea.dooplex.hu/admin/felhom.eu
gates / gates (push) Successful in 24s

This commit is contained in:
2026-08-09 18:04:46 +02:00
14 changed files with 4015 additions and 38 deletions
+183
View File
@@ -0,0 +1,183 @@
# REPORT — BYO reinstall rehearsal, 2026-08-09 (session report)
*Written as `REPORT-<topic>.md` rather than `REPORT.md` per the repo's parallel-session rule.*
## The answer to the runbook's question, first
**Yes, the data comes back byte for byte. No, not in one sitting, and not without a shell.**
All four planted files returned **BYTE-IDENTICAL** — including two Hungarian accented filenames
verified as *raw name bytes*, not as rendered text. The unlock took **21 s**, the restore **13.2 s**.
But the walk completed only because two hard stops were cleared by someone who could open a terminal
and read source. **R-273**: the install died at step 5/8 on an agent version that was published as a
package but never git-tagged — cleared by completing the release. **R-280**: the reinstalled machine
could not re-attach its own data drive through any dashboard route, while the restore page said
„**Ez két kattintás**" and pointed at an empty list — cleared by POSTing an internal path
(`/mnt/sys_drive`) that no household could produce.
Neither is a data-integrity problem. Both stop a household dead. **This is the same shape the R-201
walks kept finding: the data half passes, the journey half fails.**
## Venue
**demo-hp (t740)**, operator-approved at STOP 1. It won every fidelity criterion that distinguishes
the two demo boxes: three customer apps against one, a registered storage path against none, and a
Secure-Boot `shim` install — the customer shape — against demo-felhom's SB-off `mkimage` firmware
workaround. `drill-r50` (VM 300) was verified not at risk before proceeding: it is outside the
`felhom` pool, on `local-lvm`, and `--uninstall` removes no storage and no non-pool guest.
## Findings, ranked by what they cost the person in front of you
**1 — stops the visit**
- **R-273** · the vouched agent (0.128.0) had no git tag; every install died at 5/8. **CLOSED** — tag
pushed on your instruction after an independent sha check; install then succeeded in 3 m 49 s. The
two guards that would prevent a recurrence are still owed.
- **R-280** · a reinstalled machine cannot re-attach its data drive through any route, and the restore
page promises „két kattintás" at an empty list. **The one to fix before the tester's visit.**
- **R-272** · Felhom's uninstall restarts its own dnsmasq unconstrained; it grabs `:53`; the next
install refuses and appears to blame the owner's network.
**2 — costs the visit**
- **R-274** · a local golden is adopted with no version and no checksum check. The copy on demo-hp is
controller **0.192.0** against a vouched **0.210.0** — and below 0.200.0, where the recovery screen
the customer needs actually shipped.
- **R-276** · an uninstalled box keeps a live WireGuard tunnel into the off-site endpoint; declared
in neither the KEPT nor the WIPED list.
**3 — misleads**
- **R-281** · the hub said nothing at all through the whole reinstall, and the tripwire for a
sealed-backup unseal did not fire on a real one.
- **R-282 / R-283** · one code, three names; the mail points at a page the box is not showing; the hub
reads "Claimed 18d ago" while the box serves its setup page.
- **R-269** · a rotated-out local-API token still authorises until an unrelated lookup forces a
reload. The shipped test passes only because of its lookup order.
- **R-270** · R-268's own rotation recipe is a step short; the controller never re-reads the mount.
- **R-271** · the `agent_channel_unauthorized` alarm can never close — its own advice silences the
all-clear.
- **R-277** · three hub surfaces present a healthy off-site tier as absent. **This one caught me.**
- **R-278** · demo-felhom has had no off-site backup for six days, waiting on a ceremony nobody ran.
**4 — cosmetic / hygiene**
- **R-275** · five orphaned 0600 credential backups survive, and uid reuse hands them to the new
service account. Superseded keys here; live if the backups were recent.
- **R-279** · no operator-triggerable off-site backup exists.
## What I got wrong, and corrected
I reported to the operator that the off-site tier had not run on **either** box since 2026-08-03.
That was true of demo-felhom and **false of demo-hp**, which had 18 unbroken daily snapshots. I had
read three hub surfaces that agreed with each other and none of which said what I took them to say
(now **R-277**). I corrected it before it changed any decision, and the operator's "repair off-site
first" ruling turned out to be unnecessary for the chosen venue.
I also raised **R-275**'s sudoers half as a likely privilege-escalation on reinstall, then **tested it
and refuted my own hypothesis**: sudo skips filenames containing dots, so the leftover file is inert.
`visudo -c -f` parsing a file OK is not evidence that sudo loads it.
## Integrity verdict — BYTE-IDENTICAL
```
expected 4 file(s); found 4
VERDICT: BYTE-IDENTICAL
```
Four expected, four restored, zero differences, compared against
`evidence-rehearsal-2026-08-09/GATE0-before-manifest.json` — a manifest keyed on **raw name bytes**.
`árvíztűrő-tükörfúrógép.txt` and `nested/őszibarack.md` came back with their name bytes intact (NFC
preserved, `c3a1…`), which is the discriminator the Gate 0 positive control was built to enforce: the
comparator had been watched **failing** on an NFC→NFD rename that renders identically to the eye.
Restored out of snapshot `41c830db` into the verification folder the product names, with live data
untouched.
## Wall clocks
| phase | duration |
|---|---|
| Pre-phase (R-268 rotation, proved both ways) | ~25 min |
| Gate 0 (venue, dataset, positive control, off-site run, capture) | ~55 min |
| P1 uninstall | **60 s** (08:37:23 → 08:38:23 UTC) |
| P1 leave-behind measurement | ~12 min |
| P2 preflight (3 runs: 2 refusals, 1 pass) | ~6 min |
| P3 install — first attempt, FAILED | 44 s (08:51:33 → 08:52:17 UTC) |
| P3 install — resumed, SUCCESS | **3 m 49 s** |
| P4 first contact (box live on its own URL) | within ~4 min of install |
| STOP 3 unlock | **21 s** |
| app redeploy (calibre-web) | **1 m 36 s** |
| restore prepare + execute | **8 s + 13.2 s** |
| **bare machine → verified files** | **1 h 49 m 22 s** (08:38:23 → 10:27:45 UTC) |
| — of which the product's own work | **≈ 7 m 47 s** |
**Neither figure is the customer number.** The 1 h 49 m is dominated by the R-273 diagnosis and release
fix (~38 min) and two waits on a human. The 7 m 47 s is what the product costs when the operator
already knows every answer. **The honest unaided figure is undefined, because an unaided household does
not finish.**
## Steps taken off-path, and what they cost
1. **R-268 rotation on demo-felhom** — required by the runbook's pre-phase; not the venue.
2. **demo-hp's dashboard password re-set to the credentials-file value**, on operator instruction.
The customer-owned password was unknown to this session and no operator route to the off-site
button exists (R-279). Prior hash preserved in-guest; destroyed with the guest at P1. Cost: none —
P1 wiped it and P4 re-claims.
3. **The off-site run was started by a script pressing the dashboard's own endpoint** with a real
session and CSRF token, not by a person clicking. Identical server path; only the click synthetic.
4. **`--passphrase-file` instead of the no-echo prompt** — a first-class documented option with a
permission check, so the secret still never touched argv. A person would type it.
5. **`systemctl stop dnsmasq && systemctl disable dnsmasq`** — the action the refusal message tells
the owner to take, used as the counterfactual that confirmed R-272.
6. **Pushed the `v0.128.0` git tag** — outward-facing, done on your "proceed", and only after an
independent download proved the published package's sha256 equalled the hub's vouched value. It
completes a half-finished release rather than changing code; the release script's own recovery text
is the same line. **Cost to the walk: the install that followed was a `--resume`, not a fresh run,
which is why R-274 is only half-observed.**
7. **`POST /settings/storage/add` with `/mnt/sys_drive`** — the manual escape hatch, typed. This is the
R-280 wall; a customer could not produce that path. **The biggest fidelity cost of the run.**
8. **SSH into the guest to fingerprint the restored tree.** This is my *instrument*, not a customer
step — the customer's step (the restore) finished at the dashboard. Byte-comparison inherently needs
file access; nothing about the product was driven this way.
Everything else after the install returned was read-only, and no repair was attempted on the box.
## Teardown — all four layers
1. **The machine** — nothing created beyond the half-install itself, which is **left in place
deliberately** for inspection and resumption (`state.json` completed: preflight, token, grows,
enroll). demo-hp is **not serving** right now: no guest, agent installed but no unit.
2. **The host**`local-lvm` 20 904 790 → 12 355 143 KiB (**≈8.5 GiB returned**); `local` ≈64 MiB;
NVMe unchanged, backups deliberately kept. Pre-existing leftovers found and **not** removed
(not this run's, recorded instead): storage `c11-scratch`, the orphaned `vzdump-lxc-9100` archive,
and `/root/.dpw`, `.h`, `.sec.html` in the old guest (now destroyed with it).
3. **The hub****no customer or appliance record was created**; the `demo-hp` customer is retained
deliberately, as the runbook requires. Nothing to delete.
4. **The off-site side****one write, and it was the intended one**: the Gate 0 backup that created
snapshots `41c830db`, `9e38b84c`, `78b93f04`. **No prune, no forget, no delete.** How I know: every
restic call was `snapshots`, `ls`, or the product's own `POST /backup/offbox/run`; retention runs
inside that product path and is ep0's server-side job (R-89, boxes keep `keep_last: 0`).
## Secrets handling
No secret reached stdout. Token values, the retrieval passphrase, the controller password and the hub
DB copy were handled file→file at 0600 and shredded; the hub DB copy (which carries every host's
break-glass credential) was shredded immediately after the one hash comparison it was taken for.
Credential comparisons were done by sha256 prefix, never by value.
## State demo-hp was left in
**Back in service and healthy** — agent 0.128.0, controller 0.210.0, guest 9201 running and onboot,
claimed, storage path registered, `calibre-web` deployed, off-site repository unlocked and intact at
18 snapshots. `drill-r50` (VM 300) untouched throughout.
**Deliberately left alone, and named rather than tidied:** the pre-existing `c11-scratch` storage and
the three `vzdump-lxc-9100` golden archives on `local` (the teardown keeps goldens by design, and they
now number three). The restored files sit in the product's verification folder, not back in place —
that is R-213 and the product says so.
## Still owed
- **R-280** — the drive wall. The one finding that would stop the tester's visit outright.
- **R-273's two guards** — refuse a vouch whose tag does not resolve; check that a package and its tag
ship together. The tag push fixed one box, not the class.
- **R-274's missing observation** — a *fresh* (non-resume) install taking a stale local golden.
Full account: `documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md`.
+113 -37
View File
@@ -1,6 +1,6 @@
# STATUS — what works, what's broken, what's next
**Updated 2026-08-08.**
**Updated 2026-08-09.**
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority; this page restates
> part of it in plain words, and **nothing may exist only here**. Not `CONTEXT.md`, which is technical
@@ -10,6 +10,25 @@
> *Rebuilt from the register on 2026-08-07, from 258 lines. The old "what shipped recently" log is what
> the per-repo `CHANGELOG.md` files and the register are for, and is not restated here.*
## ⚠ BOTH DEMO MACHINES ARE OFF AND MUTED — unmute them when they are home
**Powered down 2026-08-09 14:08 CEST** for the move back from the vacation home. Guests stopped
cleanly first (no vzdump was running, no locks), then the hosts. Confirmed off at the fabric, not
merely unreachable: the tailnet is healthy and both peers report *"offline, last seen 1m ago"*.
**Both customers are BLOCKED on the hub, deliberately, to stop four false alarms an hour into the
drive.** Blocking gates every monitor and the notification intake; it does **not** gate config pull or
report intake, so the boxes come back normally on power-up.
> **THE TAIL, and it is the reason this banner exists: while they are blocked, a box that FAILS to
> come back up is also silent.** When the machines are home and powered on, unblock them and confirm
> both report:
>
> Hub → Customers → **demo-hp** → Unblock, and **demo-felhom** → Unblock.
>
> Then check both read ONLINE on Hosts. **Until that is done, the hub cannot tell you either box is
> in trouble.**
## What works
A blank machine boots the Felhom disc, installs itself unattended, and is claimed by the customer, who
@@ -23,9 +42,46 @@ destroyed on purpose and its files came back byte for byte identical — four ti
their recovery code got everything back with **no command line inside the machine at any point**,
in 72 seconds. The two rough edges that walk found are also gone. *(R-201, R-252, R-253 — closed.)*
## The rehearsal finished. The data came back byte for byte; the journey did not.
**We wiped a working demo machine and put it back. All four test files returned identical — including
the two with Hungarian accents, checked as raw bytes, not as text on screen.** The unlock took 21
seconds and the restore 13. **But it only finished because I could open a terminal twice.** A
household would have stopped, twice, and the second time the screen would have told them it was easy.
**The two walls, both fixed-or-fixable, neither about the data:**
- **The install died four steps in** — the agent version you approved had been published as a download
but never given its version label, and the installer looks it up by that label. **Now unblocked**
I pushed the label after checking the published file matched what you vouched. *(R-273 — closed. The
two guards that would stop it recurring are still owed.)*
- **A reinstalled machine cannot re-attach its own data drive.** Every route is a dead end, and the
restore page cheerfully says „**Ez két kattintás**" while pointing at an empty list. The drive is
fine and the machine can see it — it just is not offered, because the same drive is also the backup
target. I got past it by typing an internal path no customer could know. **This is the one to fix
before the tester's visit.** *(R-280)*
**Also broken, found on the way:**
- **Taking Felhom off a machine leaves the one thing that stops it going back on.** We install a small
network service at setup; removing Felhom restarts it without its settings, it seizes the port the
next install needs, and the next install then refuses — appearing to blame the owner's network.
*(R-272)*
- **A machine we removed keeps its private line to us open.** *(R-276)*
- **The hub said nothing at all** while a machine was wiped, rebuilt, re-claimed and had its sealed
backups opened. No false alarm — but also no word, and the alarm that exists for "someone is opening
this customer's backups" stayed silent through a real one. *(R-281)*
- **A rebuilt machine may still come back on software from last week** — narrower than I first wrote:
the resumed install fetched the right version, but a fresh one takes whatever copy is newest on the
disk without checking it against what you approved. *(R-274)*
- **demo-felhom has not had an off-site backup in six days** and is waiting for a recovery code nobody
has entered. demo-hp, rebuilt the same day, recovered by itself. *(R-278)*
- **One code, three different names**, and the email points at a page the machine is not showing —
this cost us a wasted code today. *(R-282, R-283)*
## What's broken
- **Nothing new is broken.** The *check* against a fourth secret-in-a-page covers 4 pages of 27, and
- **Nothing else new is broken.** The *check* against a fourth secret-in-a-page covers 4 pages of 27, and
the cheap one covering all of them is blind to the shape that shipped. *(R-255)*
- **An already-paired box is still told to pair itself**, 25 minutes on. *(R-214, R-235)*
- **A backup that covered nothing still calls itself „Sikeres".** *(R-240)*
@@ -35,48 +91,71 @@ in 72 seconds. The two rough edges that walk found are also gone. *(R-201, R-252
accumulates. *(R-244)*
- **Putting restored files back where they belong is still manual.** *(R-213)*
## Fixed today — four things the machine knew and did not say
## The rest of what the rehearsal found
All one family: something the box already knows, thrown away or drawn as its opposite.
Sixteen findings in one afternoon, **none of them visible from reading the code** — three sessions of
review had not seen any.
- **A rebuilt machine can set up its own recovery again.** The one fact the setup needs was written
only the first time, and a rebuild replaced the configuration while leaving the note saying
"already done". It is now checked and re-written every minute instead of remembered once, so a
hand-edited or restored configuration heals too. **This was the last item blocking a customer from
something we promise them.** *(R-221 — agent 0.128.0.)*
- **A disk we failed to read is no longer drawn as a healthy empty one.** No figures, no bar, and it
says so: „A tárhely mérete most nem olvasható ki." *(R-259 — controller 0.210.0.)*
- **A backup tick now answers about that app.** It went green because *some* backup file existed and
*some other* app's database dump had succeeded most recently. Now: that app's own result, and
**no mark at all** when we have none. *(R-258 — controller 0.210.0.)*
- **Our own alarm no longer points at a page that may not exist.** A check run now gives up after
five minutes rather than hanging until something else kills it, and the mail says how long it ran.
*(R-265.)*
- Removing Felhom leaves five files holding old keys *(R-275)*, and rotating a leaked key does not
revoke the old one until the service restarts *(R-269)* — the written recipe for it is a step short
*(R-270)*, and the alarm it raises can never be closed because the fix it recommends is what
silences the all-clear *(R-271)*.
- **It caught me being wrong twice, and that matters more than the count.** I told you the fleet's
off-site backups were down; demo-hp had eighteen snapshots and I had read three misleading screens
instead of asking the machine *(R-277)*. And I raised a leftover permissions file as a security
hole, then tested it and refuted myself — it is inert.
**Not fixed, and said rather than glossed:** that failed disk reading still reaches us as "0 of
0 GB". It is the quiet direction — it can only miss a true alarm, never raise a false one. *(R-266)*
**What worked, and should not be lost in the count:** the machine came up on its own at the approved
version; the setup page appeared unprompted, in Hungarian, naming the customer; **the recovery screen
appeared without being looked for** and said plainly that unlocking changes nothing; the restore told
the truth about putting files in a checking folder rather than back in place; and no false alarm fired.
**And R-221 is now proven on hardware, not just in tests** — on a demo machine we removed the one
line, watched the setup screen refuse, waited one minute, and watched it go green by itself with
nothing restarted. Every other line of that file came back identical.
Full account: `documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md`.
## Three rulings, written down so they stop living in a conversation
- **The managed-update floor.** It is deliberately parked, and the trigger to raise it is **the first
machine that is not ours**; after that it moves with the publish train. Worth knowing alongside it:
the updater always aims at the floor, never at the newest, so a machine at or above the floor
updates to nothing. **Correction to the number that was going round: the floor is live at 0.200.0,
not 0.156.0** — checked twice today, on the hub page and in both machines' own logs.
- **R-264 is decided.** Build a reader for guest-network health, the staged-update pair, the
restore-test depth pair, and the two backup-integrity timestamps. Record a stated "no reader
wanted" for the repaired-recently flag, the tier-applied timestamp, the config fingerprint, the
per-stack object and the drive-migration marker. Decide the reporting-disabled flag on its own
merits — if that is a state we support, it must be visible or a staleness alarm will one day fire
on a machine that is fine. A "no" ends by changing the allowlist reason from *arguably owed* to
*deliberately not consumed* — not by ripping out an emitter, which is a two-repo change that also
breaks a shared fixture. **The implementation is its own session.** And **it is twenty facts, not
twenty-one**.
- **R-268 is closed** — the leaked key is rotated, and the rotation is proved in both directions
rather than assumed.
## Fixed 2026-08-08 — four things the machine knew and did not say
A rebuilt machine can set up its own recovery again *(R-221, agent 0.128.0 — proved on hardware)*; an
unreadable disk is no longer drawn as a healthy empty one *(R-259)*; a backup tick now answers about
*that* app *(R-258)*; our own alarm no longer points at a log that may not exist *(R-265)*. Still true
and not glossed: a failed disk reading still reaches us as "0 of 0 GB" — the quiet direction, it can
only miss a true alarm, never raise a false one *(R-266)*.
## What we're working on
- **Widening the check** so a fourth secret-in-a-page is caught by a machine. *(R-255)* · **Deciding
the twenty-one** — each gets a reader, or stops being sent. *(R-264)*
- **Widening the check** so a fourth secret-in-a-page is caught by a machine. *(R-255)* · **R-264 is
now decided** (above); building the readers is a session of its own.
- **Proving the hub really keeps the old sealed key** when a machine re-seals. *(R-198)* · Still open,
none urgent: *(R-256, R-257, R-261…R-263, R-266)*
## Waiting on you
- **A watching moment, five minutes.** Today's recovery fix is proved by removing one line from a
demo machine's config — backed up first, disposable machine, no customer data near it — and
watching the setup screen go green on its own. Nothing is destroyed. Say when.
- **One approval, three values this time.** Hub → Configuration → Day-0 artifacts: Golden
**0.210.0**, Agent **0.128.0**, minimum agent **0.127.0** (unchanged) → Save. Each was checked to
be downloadable and selectable before being written here. **Agent 0.128.0 is the one that carries
today's recovery fix**, so a new machine needs both, not just the image. It supersedes the 0.209.0
approval you already gave, and it is reversible. *(R-242)*
- **Nothing blocking.** The rehearsal is finished and demo-hp is back in service: agent 0.128.0,
controller 0.210.0, claimed, off-site backups unlocked and intact.
- **One decision worth taking before the tester comes:** whether to fix the drive wall *(R-280)* now.
It is the only finding that would stop his visit outright, and it is the difference between "his
data comes back" and "his data comes back if someone types a path for him."
- **Two guards are still owed** so the install cannot break the same way twice: refuse to vouch a
version whose label does not resolve, and check that a published version and its label ship
together. *(R-273's tail.)*
## DooPlex infrastructure — separate from the product
@@ -97,11 +176,8 @@ being readable.*
connections are kept, the answer is held for a minute, and the old artifacts are gone. Worst case
is 5 s, once a minute at most. **Your instinct to prune was right and my measurement said
otherwise** — trimming to ten of each halved the slow path. *(R-267 — closed.)*
- **I printed a live access token into a session log** while setting up today's drill, and I am
telling you rather than quietly rotating it. It only opens the agent's private channel to one
demo guest, on a wire that exists solely between that host and that guest — not reachable from
your network or the internet, on a disposable machine with no customer data. Rotating it also
means updating the guest, so it is a deliberate act, not a background one. *(R-268)*
- **The access token I printed into a log yesterday is rotated**, and I checked it both ways: the old
one is refused, the new one works, and the machine's own channel is back up. *(R-268 — closed.)*
- **Our build-check alarm has one gap left.** A run that hangs is now cut off after five minutes and
the mail says how long it took — but **whether the alarm fires at all when the machinery kills a
run outright is still unverified**, and we have not claimed otherwise. *(R-265)*
@@ -0,0 +1,712 @@
# REHEARSAL — the BYO reinstall walk (2026-08-09)
> **Status: COMPLETE.** All phases walked; the integrity verdict is **BYTE-IDENTICAL**. §1–§8 were
> written before the destructive phase deliberately — a finding that exists only in a session that
> later crashes is a finding nobody has — and are left as written, including one claim later refuted
> by test (§8.3 → §9a) and one by measurement (§7.2 F-9).
**Venue: `demo-hp` (HP t740, `felhom-host`, guest 9201, customer `demo-hp`).** Operator-approved at
STOP 1. **Driven from DooPlex.** All times UTC unless marked; the host runs CEST (UTC+2).
---
## 1. Baselines — re-confirmed live on arrival, not taken from the spec
| | spec said | live reading | source |
|---|---|---|---|
| `felhom-agent` | v0.128.0 @ `28ba8593b8` | **0.128.0** on demo-felhom, **0.127.0** on demo-hp; HEAD == `origin/main` == `28ba8593b8` | `felhom-agent --version` on both nodes; `git rev-parse` |
| `felhom-controller` | v0.210.0 @ `c732fe1283` | **0.210.0** demo-felhom, **0.208.0** demo-hp; HEAD == `origin/main` == `c732fe1283` | hub `/configs`; `git rev-parse` |
| hub | v0.101.0 @ `56f8aa611c` | **0.101.0** (deployed image tag matches) | `kubectl get deploy hub`; page footer |
| Day-0 manifest | golden 0.210.0 / agent 0.128.0 / min 0.127.0 | **all three already saved** — golden `0.210.0` (`b9f701fa…`), agent `0.128.0` (`c6eba73b…`), min agent `0.127.0`, wrapper `104db0a4…` | hub `/configuration`, selected `<option>` values |
All three repos were clean and at `origin/main` before anything ran.
**The fleet is not uniform, and the venue is the box that is behind** — demo-hp runs agent 0.127.0 and
controller 0.208.0 against a manifest vouching 0.128.0/0.210.0. That is an argument *for* the venue: the
reinstall genuinely exercises the manifest rather than re-installing what is already there.
---
## 2. Venue comparison — the fidelity table that decided it
The runbook's criterion is *"the box whose current state most resembles what the tester will have"*.
| criterion | `demo-felhom` (N100) | `demo-hp` (t740) |
|---|---|---|
| customer apps deployed | 1 — opengist | **3 — privatebin, opengist, calibre-web** |
| enrolled user-data drive | **none registered.** `/mnt/felhom-drives` empty; controller logs `[WARN] Storage paths: no storage paths registered` | `/mnt/felhom-drives/Felhom-Share` + NVMe enrolled 2026-07-22; one registered storage path `/mnt/sys_drive` |
| off-site tier has actually run | **NO**`offsite.state=needs_credential`, no run has ever succeeded | **YES** — 18 restic snapshots, daily, unbroken |
| escrow / recovery ceremony | present, generation 2 | present, generation 2 |
| install shape | mkimage loader, **Secure Boot OFF** (firmware workaround) | shim loader, **Secure Boot ENABLED** — the customer shape |
| collateral on the box | none | `drill-r50` (VM 300); leftover `c11-scratch` storage |
**Chosen: demo-hp.** It wins every fidelity criterion that distinguishes them.
**`drill-r50` is not at risk, and this was checked rather than assumed:** VM 300 is **not** in the
`felhom` pool (`pvesh get /pools/felhom` → members `[9201]`), its disks are on `local-lvm`, and
`--uninstall` removes no storage definitions and no non-pool guest — its own end-of-teardown statement
says the enrolled drives are *"unmounted only, NEVER wiped"* and that it never deletes PBS backups, hub
records or escrow blobs.
**Correction recorded, because it was part of the comparison put in front of the operator:** the
enrolled-drive row was first reported as *"real files on an enrolled drive"*. `Felhom-Share` is
**empty** — the share exists and is enrolled, but holds no files. demo-hp still wins the row (demo-hp
has a registered storage path; demo-felhom has none), on a narrower margin than first stated.
---
## 3. Pre-phase — R-268, the leaked per-guest local-API token
**Done first and independently, on `demo-felhom` guest 9201, as the runbook required.** No token value
reached stdout at any point: old value captured file→file from `bootstrap.json`, new value minted into
a file, both shredded at the end.
### 3.1 The rotation, and its proof
| step | old token | new token |
|---|---|---|
| before rotation | **HTTP 200** | — |
| bogus token (negative control) | **HTTP 401** | — |
| after the on-disk rotation, before any reload | **HTTP 200***see 3.2* | 200 |
| after that lookup forced a reload | **HTTP 401** | — |
| after `systemctl restart felhom-agent` | **HTTP 401** | **HTTP 200** |
**Positive observable for the controller half:** `local-api: channel up (agent 169.254.253.1:8443) —
guest 9201, 3 mount(s) visible`, enumerating mp8/mp9/mp0 — data only the agent can supply. An absent
error would not have been evidence.
**R-268 is CLOSED.** The old token is refused; the new one works; the controller's channel is up.
### 3.2 FINDING — an out-of-process token rotation does not revoke the old token
`localapi.TokenStore.Mint`'s doc comment states *"last-write wins — any previous token for this guest is
revoked"*. **Across processes that is false until something else forces a reload.** The long-lived agent
serves `Lookup` from an in-memory index and re-reads the store **only on a miss** (the B3
reload-on-miss optimisation). A superseded token is still a direct map **hit**, so it authorises.
**Red-proved twice.** A temporary unit probe (Scenario C with the lookup order reversed — present the
rotated-out token *first*) fails on the claim; the shipped
`TestTokenStore_ReloadOnMiss_RemintCoherence` passes **only because it looks up the NEW token first**,
which is what triggers the reload that then evicts the old hash. Then proved on real hardware: the row
marked ← above is a live 200 from a token that the store on disk had already superseded.
This is the `CLAUDE.md` case exactly — a comment asserting an invariant with a test that appears to pin
it and does not. The honest fix is either a test that pins the reversed order, or an eviction that does
not depend on an unrelated lookup missing first.
### 3.3 FINDING — R-268's own stated rotation recipe is incomplete
The register row says the new plaintext *"must also be written into the guest's
`/etc/felhom-bootstrap/bootstrap.json` or the in-guest controller loses its agent access"*. **Writing
bootstrap.json is not sufficient.** `bootstrap.ensureLocalAPI` returns early when
`cfg.LocalAPI.Endpoint != ""` — it fills an *absent* block and never refreshes a present one — so after
a rotation the controller keeps serving the old token from its own `controller.yaml` **across a
restart**. Proved live: two controller restarts, still HTTP 401. The rotation only completed once
`local_api.token` was written into `controller.yaml` itself.
The neighbouring `DetectEndpointDrift` compares the **endpoint** and deliberately does not compare the
token (*"a token mismatch is a different failure"*). So this shape is known to be unmodelled, and R-78
("which file is authoritative") is its parent.
### 3.4 FINDING — the agent-channel alarm never closes, and its own prescribed remedy is why
`channelhealth` alerted correctly and promptly: `agent_channel_unauthorized`, no debounce, pushed to
the hub — *"agent rejected the controller token (HTTP 401) — token stale/rotated (**re-bootstrap**)"*.
The channel then recovered, and **no recovery event was ever sent**. In `Checker.Check`'s UP branch the
notify is guarded by `prev != "" && prev != "up"`, and a controller restart resets `state` to `""`. The
alert's own instruction is to re-bootstrap — i.e. restart the controller — so **following the alarm's
advice structurally guarantees the recovery is silent.** The down side is deliberately asymmetric
(F2: a born-down channel alerts on cycle 1); the up side is not.
Operator-visible result on the hub: two `agent_channel_unauthorized` errors, one `sent` and one
`suppressed` by cooldown, and nothing afterwards. The customer dashboard is fine — `SetDashboard`
reflects current state every cycle. It is the operator's trail that ends on "down".
---
## 4. Gate 0
### 4.1 The recovery code — CONFIRMED IN THE OPERATOR'S HANDS (STOP 1)
Not printed, not logged, not requested. It will be supplied at the screen at STOP 3.
### 4.2 The dataset — planted and fingerprinted
Planted at `/var/lib/felhom/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/` — inside
the **Calibre library**, which is real app-owned customer data and is **demonstrably** carried off-site
(the path appears in every `felhom-offbox,calibre-web` snapshot). Chosen after checking coverage rather
than assuming it: the per-app tier-1 unit carries only the named docker volume, so a file dropped in an
arbitrary place would not have travelled.
| sha256 (16) | bytes | name | name bytes (hex) |
|---|---|---|---|
| `54b773c46bbfd994…` | 3 145 728 | `binary-3mb.bin` | `62696e6172792d336d622e62696e` |
| `52a5c5ebfcac247f…` | 59 | **`árvíztűrő-tükörfúrógép.txt`** | `c3a17276c3ad7a74c5b172c5912d74c3bc6bc3b67266c3ba72c3b367c3a9702e747874` |
| `15d2024dfc224162…` | 25 | **`nested/őszibarack.md`** | `6e65737465642fc591737a6962617261636b2e6d64` |
| `924497918e55fe6d…` | 21 | `plain.txt` | `706c61696e2e747874` |
Manifest: `documentation/audits/evidence/` → recorded in the session scratchpad as
`GATE0-before-manifest.json`. The manifest keys on **raw name bytes**, not rendered text.
### 4.3 The positive control — the comparator was watched failing, three ways
| control | expected | result |
|---|---|---|
| identical copy | PASS | **PASS**, exit 0 |
| one bit flipped at byte 1 500 000 of the 3 MB binary | FAIL | **FAIL** — named the file and both sha256s |
| **NFC→NFD rename of the accented name** (renders *identically*) | FAIL | **FAIL** — caught as MISSING+EXTRA on the name bytes |
| one file deleted | FAIL | **FAIL** — named the missing file |
The NFD case is the one that matters: a comparison of *rendered* filenames would have passed it. Every
altered copy was discarded after the control ran.
### 4.4 The off-site tier — has run since the dataset was planted
Triggered through the product's own dashboard button (`POST /backup/offbox/run` → flash *„A távoli
mentés elindult"*), not by invoking restic by hand.
```
[offbox] backup run started (3 app(s) toggled)
[offbox] pre-push dump leg completed in 7.329s — snapshot pair is coherent
[offbox] backed up calibre-web (…/backups/primary/calibre-web, 1 mandatory path(s))
[offbox] backup OK: 3 app(s) backed up, 18 snapshot(s), 1m22s
```
**The restore point, by identity — never by size:**
| snapshot | time (UTC) | tag | paths |
|---|---|---|---|
| **`41c830db`** | 2026-08-09 08:30:38 | `felhom-offbox,calibre-web` | `…/backups/primary/calibre-web` + `…/userdata/media/books` |
| `9e38b84c` | 2026-08-09 08:30:49 | `felhom-offbox,opengist` | `…/backups/primary/opengist` |
| `78b93f04` | 2026-08-09 08:30:53 | `felhom-offbox,privatebin` | `…/backups/primary/privatebin` |
`restic ls 41c830db` confirms all four planted files are present, **both accented filenames included**.
### 4.5 The box, captured
`GATE0-demo-hp-before.txt` (282 lines): agent version, `pveversion`, `pct`/`qm` lists, pool membership,
`pvesm status`, `storage.cfg`, `df`, `lsblk` with serials, the agent unit + service user, sudoers, the
pveum users/tokens/roles/ACL, the guest hook, the WireGuard tunnel, the OOB belt, the agent state dir,
the install state file, PBS snapshots, local vzdump archives, and the enrolled-drive tree.
---
## 5. Findings so far, ranked by what they cost the person in front of you
Nothing yet is rank 1 or 2 — the walk has not started.
### Rank 3 — misleads
- **F-1 · A rotated-out local-API token still authorises** (§3.2). *A customer would do nothing — they
cannot see it. An operator rotating a leaked token would believe it revoked when it is not.* The
shipped test passes for the wrong reason.
- **F-2 · R-268's rotation recipe is incomplete** (§3.3). *Whoever follows it gets a box whose agent
channel is dead across restarts, with a correct-sounding alarm and no hint that `controller.yaml` is
the file that matters.*
- **F-3 · The agent-channel alarm never closes** (§3.4). *The operator fixes it as instructed and is
never told it worked; the hub's last word stays "down".*
- **F-4 · Three hub surfaces jointly misrepresent a healthy off-site tier as absent.** The customer page
reads `Snapshots 0 / Repo Size 0 MB / Integrity Unknown` (that panel renders the **local** tier, while
the healthy `offsite` object — `snapshot_count: 18`, `last_status: ok` — sits unrendered in the same
report); the Offsite page shows `0.0 GB` (true, but a 162 KB repo rounds to nothing); and a stale
`offsite_delivery_stuck` event from 2026-08-07 reads as current. *Three independent surfaces agreeing
on a wrong picture is how a working backup gets "fixed".* **This one caught me** — it produced a wrong
statement to the operator at STOP 1, corrected before it changed anything.
- **F-5 · There is no operator-triggerable off-site backup.** The only route is the customer's dashboard
button; `signed_jobs` needs a signing key the hub does not hold. Sibling of R-177.
### Rank 4 — cosmetic / hygiene
- **F-6 · Leave-behinds from earlier sessions**, found while capturing state:
- demo-hp host: storage `c11-scratch` (Campaign 11), and `felhom-backup:…vzdump-lxc-9100-2026_08_03…`
for a guest that no longer exists;
- demo-hp guest 9201 `/root`: `.dpw` (13 bytes), `.h`, `.sec.html`, all 2026-08-07 16:08;
- demo-felhom guest 9201: `/root/r204-backup/offbox.orig`.
- **F-7 · The managed-update floor is `0.200.0`, not `0.156.0`** — the runbook's §8.3 premise is stale.
Confirmed twice: hub `/configuration` reads *"Effective floor v0.200.0 — source: DB (hub_settings);
env fallback would be v0.120.0"*, and both boxes log `settle-gate: GO — at/above floor 0.200.0`. The
ruling's **substance** still needs recording; its **number** must not be written down as 0.156.0.
### Not a finding, checked and cleared
- The two `userdata` trees under `sys_drive` are the **same directory** surfaced twice, not a
backed-up copy beside an unprotected one. Checked against calibre-web's actual bind mounts.
---
## 6. Steps taken off-path, and what they cost
Rule 2 (no shell after the install command returns) does not bind yet — the walk has not begun. These
are Gate 0 preparation acts by the operator, recorded because they are mutations:
1. **R-268 rotation on demo-felhom** — token store appended, `bootstrap.json` and `controller.yaml`
rewritten, agent and controller restarted. Required by the runbook's pre-phase. *Not the venue.*
2. **demo-hp's dashboard password was re-set to the value in the operator's credentials file.** The
customer-owned password was not known to this session, and no operator-side route to the off-site
button exists (F-5). The prior `password_hash` is preserved at `/root/.orighash` and
`/root/settings.json.rehearsal-orig` in guest 9201; `claimed` was left `true`. **Done on operator
instruction.** Cost to fidelity: none for the walk — P1 destroys this guest, and P4 re-claims with a
password the operator chooses.
3. **The off-site run was started by a script pressing the dashboard's own endpoint**, with a real
session and a real CSRF token, rather than by a person clicking. The server path is identical; only
the click is synthetic.
---
## 7. P1 — removing Felhom from the host
`--uninstall`, the product's own path, dry-run first. **60 seconds** (08:37:23 → 08:38:23 UTC), exit 0.
The typed-vmid confirmation was satisfied, not bypassed.
### 7.1 The reversal, proved rather than trusted
| component | before | after |
|---|---|---|
| agent binary | `/usr/local/bin/felhom-agent` 0.127.0 | **ABSENT** |
| `felhom-agent` service user | present | **ABSENT** |
| `felhom-op` (OOB) user | present | **ABSENT** |
| `/etc/felhom-sshd` (OOB belt) | present | **ABSENT** |
| systemd units | agent, mgmt-watchdog, felhom-sshd, oob-nft, shared-parent | **all removed** |
| pveum roles | `FelhomAgentBase/Guest/Store` | **none** |
| pveum user + token | `felhom-agent@pve!agent` | **gone**; user list is `root@pam` only |
| pveum ACL | pool + 9 scratch-vmid + 3 storage grants | **empty** |
| `felhom` pool | members `[9201]` | **`[]` — pool removed** |
| guest 9201 | running | **destroyed** |
| guest-hook snippet | present | **removed** |
| dnsmasq snippets | present | **removed** |
| `/var/lib/felhom-agent`, `/var/lib/felhom-install` | present | **removed** |
| **KEPT as declared** | | PBS snapshots (2), local vzdump (4), `/mnt/felhom-drives/Felhom-Share`, storage defs |
Space returned: `local-lvm` 20 904 790 → 12 355 143 KiB (**≈8.5 GiB**); `local` ≈64 MiB; the NVMe
unchanged (backups deliberately kept). `drill-r50` (VM 300) untouched, as predicted.
### 7.2 What it left behind that it did not declare
**F-8 · Five orphaned `agent.json.*` backups survive, each carrying a hub API key and a Proxmox
token.** `/etc/felhom-agent/` remains with `agent.json.campaign8-before`, `.campaign9-before`,
`.campaign9-prev`, `.pre-e-target-move`, `.pre-prunegate.bak` — all 0600, three owned by the **now-deleted
service user's uid 999**, each holding a 64-char `hub.api_key` and a 59-char `proxmox.token`.
The uninstall's own WIPED statement claims it removes *"config (+ its .bak backups)"*, and
`scripts/CHANGELOG` records F1 as *"uninstall now purges the agent config's `.bak*` siblings (one held a
live hub api_key)"*. **That fix does not match the filenames in use** — and note it misses
`agent.json.pre-prunegate.bak`, a file that literally ends in `.bak`.
**Assessed rather than assumed: these particular secrets are SUPERSEDED, not live.** The orphaned
`hub.api_key` hashes to `a5d2222a…`; the hub's current key for demo-hp is `8c59d1b6…`. The Proxmox token
was deleted by this same uninstall. **The exposure is timing-dependent, not absent** — a backup taken
shortly before an uninstall would hold a current key, which is exactly what R-179's precedent records.
The broken thing is the stated guarantee.
**And the reinstall hands them to the new service account.** The rebuilt box recreates `felhom-agent` at
**uid 999 — the same uid the deleted one had** — so the three backups owned by uid 999 become the new
account's files. Verified: `su -s /bin/sh felhom-agent -c 'test -r …'` succeeds on
`agent.json.campaign8-before` and `agent.json.pre-prunegate.bak`. A fresh install's service account
inherits read access to the previous install's credentials. Superseded here; live if the backups are recent.
**F-9 · A stale sudoers file survives — and it is INERT. Hypothesis raised, then refuted by test.**
`/etc/sudoers.d/felhom-agent.bak-pre-e2a` (21 833 bytes, 2026-07-28, 0440 root:root) survives the
uninstall and grants `felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, …`,
and `visudo -c -f` on it says **`parsed OK`**.
The obvious worry — that the reinstall recreates the `felhom-agent` user and the box silently regains a
pre-E-2a privilege set — **was tested against the rebuilt box and is FALSE.** With the user recreated
(uid 999) and this as the *only* felhom file in `/etc/sudoers.d`, `sudo -l -U felhom-agent` returns
**"User felhom-agent is not allowed to run sudo on felhom-host."** sudo ignores files in `sudoers.d`
whose names contain a dot, so `…​.bak-pre-e2a` is never loaded. That is a robust reason, not luck —
`visudo -c -f <file>` checks a file explicitly and therefore does **not** tell you whether sudo loads it.
**What is left is rank 4:** 21 KB of the privilege grammar lingering on a host Felhom was told to leave,
inside a teardown that claims to have removed the sudoers file.
**F-10 · The WireGuard tunnel to Felhom's off-site endpoint stays up.** After an uninstall,
`wg-quick@wg-felhom` is **enabled and active**, `/etc/wireguard/wg-felhom.conf` present, with a live
handshake to `167.233.158.164:443` 52 seconds old and 5.86 GiB received / 2.48 GiB sent on the counter.
It appears in **neither** the WIPED nor the KEPT list. A box told to leave Felhom keeps a live inbound
route into Felhom's infrastructure, and the operator is not told.
**F-11 · Smaller undeclared residue:** `/etc/felhom/` (`.bootstrap-done`, `appliance-pairing-code` —
the console pairing code); `/etc/systemd/system/felhom-bootstrap.service` + `/usr/local/sbin/felhom-bootstrap.sh`;
the `vmbr9` island-bridge stanza (169.254.253.1/30) still in `/etc/network/interfaces`.
**F-12 · A teardown-ordering warning:** `user config - ignore invalid acl role 'FelhomAgentStore'` —
the role is deleted before an ACL that still references it. Harmless here; noise that would mask a real one.
---
## 8. P2 — the preflight, as the owner runs it
### 8.1 The refusals, judged on their messages
**First run, no caps:** `byo mode requires explicit --cores and --memory (noisy-neighbor caps on a host
you do not own)`. **Good** — names the missing thing and the reason.
**Second run:** it refused again —
```
[ERROR] a resolver is already bound to :53 on this host — Felhom needs the guest reachable by name on your LAN.
Stop or reconfigure that resolver, OR point your LAN DNS at the guest's address, then re-run.
(Felhom does NOT touch DNS services on a host it does not own — this is a refusal, not a change.)
```
As a *message* this is excellent: the finding, the evidence, two routes out, and an explicit promise
about what it will not do.
### 8.2 F-13 (RANK 1 — STOPS THE VISIT) · Felhom's uninstall leaves the exact condition that makes Felhom's reinstall refuse
The resolver it is refusing over **is Felhom's own**, and the chain is fully evidenced:
| fact | evidence |
|---|---|
| Felhom installed `dnsmasq` | `/var/lib/dpkg/info/dnsmasq.list` dated **2026-07-21 18:24 CEST** — demo-hp's Felhom day-0 |
| it was constrained by a Felhom snippet | `/etc/dnsmasq.d/` held Felhom snippets before; only `README` after |
| **the uninstall restarted it unconstrained** | the running process started **2026-08-09 10:37:39 CEST — inside the uninstall window (10:37:2310:38:23)** |
| unconstrained, it grabs the wildcard | `0.0.0.0:53` and `[::]:53`, `--local-service`, empty `dnsmasq.conf` |
| it is not PVE's | `/etc/pve/sdn/` empty; unit is stock `/usr/lib/systemd/system/dnsmasq.service`, **enabled** |
The uninstall mentions it only as *"the 'sudo' and 'dnsmasq' packages were left installed (**system
packages**)"* — but dnsmasq is not a system package here; Felhom installed it, and it left it **enabled
and running as a wildcard resolver**.
**What a customer would do next:** read a message that says a resolver is in the way, conclude their own
network setup is at fault, and start debugging DNS they never configured. On a BYO box with no shell
guidance, this is where the visit stops.
**Confirmed by counterfactual:** `systemctl stop dnsmasq && systemctl disable dnsmasq` → `host DNS (:53):
free` → **PRE-FLIGHT PASS**. Nothing else changed. *(Recorded as an intervention — it is the action the
message tells the owner to take.)*
### 8.3 F-14 (RANK 2) · A local golden is adopted with NO version and NO checksum check
Preflight reported `golden (local): local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst`. Step 7 is:
```sh
if [[ -n "$GOLDEN_VOLID" ]] && ! $FORCE_GITEA_GOLDEN; then
log_skip " using local golden: $GOLDEN_VOLID"
_state_mark golden; return 0
fi
```
**The hub manifest's `golden.sha256` — the whole point of the hub vouching a different trust root than
Gitea — is consulted only on the fetch path.** A locally-present archive bypasses the vouch entirely: no
version comparison, no digest, no warning.
**What that archive actually contains, read out of the tar:**
`gitea.dooplex.hu/admin/felhom-controller:`**`0.192.0`**, against a vouched golden of **0.210.0**.
Two consequences, and the second is the serious one:
1. The rebuilt box comes up **18 controller releases stale**, reporting success.
2. **0.192.0 is below 0.200.0 — the release where R-193's off-site recovery SCREEN shipped.** So a
customer reinstalled today returns on a controller that cannot run the recovery ceremony their data
depends on. It is also born below the managed-update floor (0.200.0), and the updater's auto-target
is the floor, never the newest — so it converges to 0.200.0, still not the vouched build.
**This is the R-111/R-115/R-120 drift family, one layer down:** the R-120 gate guards what the operator
may *vouch*; nothing guards what an install actually *takes*. It also compounds with P1 — the uninstall
deliberately keeps the old golden (*"golden vzdump left in place"*), and the reinstall then adopts it.
**The walk continues unflagged** (`--force-gitea-golden` NOT passed), because measuring the path a real
reinstall takes is the point. §9 records which controller version actually came up.
### 8.4 The BYO mutation disclosure — read as a customer
Comprehensive and honest: every pveum artifact, the sudoers file, the state and config paths, the
packages it will install, the guest's capped size, the self-update authority, and — creditably — the
**outbound WireGuard tunnel**, in plain words, with the note that the operator can disable it per
customer. It closes with `NOT touched in byo mode: root@pam (no break-glass), host DNS (:53), OOB sshd.`
**One material omission, given §8.3:** it does not say that a pre-existing local golden may be used
instead of the vouched one. The disclosure lists everything the install will *create* and nothing about
what it may *reuse*.
---
## 9. P3 — the install. IT FAILED, AND THE WALK STOPS HERE
Started 08:51:33 UTC, **failed 08:52:17 UTC (44 s), exit 1, at step 5/8.** Run unflagged, as a real
owner would. The typed hostname acknowledgement was given, not bypassed.
```
[INFO] manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.210.0
[OK] verified sha256 c6eba73bf9b9ad69… matches the hub manifest
[OK] installed /usr/local/bin/felhom-agent (felhom-agent 0.128.0)
[OK] created service user felhom-agent
curl: (22) The requested URL returned error: 404
[ERROR] raw fetch failed: configs/felhom-mkfs-guarded.sh (agent tag v0.128.0 — is that version tagged in felhom-agent?)
```
### 9.1 F-15 (RANK 1 — STOPS EVERY INSTALL, FLEET-WIDE) · The vouched agent version was never tagged
**`v0.128.0` does not exist as a git tag** — not locally, not on Gitea. The newest tag is `v0.127.0`.
The *binary* was published to the Gitea package registry (which is why it is selectable in the hub's
artifact dropdown and why its sha256 verified), but the release's tag step never happened.
| ref | HTTP |
|---|---|
| `raw/branch/main/configs/felhom-mkfs-guarded.sh` | **200** |
| `raw/tag/v0.127.0/configs/felhom-mkfs-guarded.sh` | **200** |
| **`raw/tag/v0.128.0/configs/felhom-mkfs-guarded.sh`** | **404** |
The commit that *is* 0.128.0 exists and is on `main`: `28ba8593b8 — v0.128.0 — R-221: the escrow seed is
asserted every tick, not remembered once`. Only the tag is missing.
**This is R-184 happening.** R-184 reads: *"Nothing prevents the hub from vouching an agent version that
was never released. The R-115 gate proves every RELEASED version is installable, but it works from git
tags."* It was filed as a gap. It is now a live outage.
**It is the collision of a good fix with an open gap.** R-183 correctly hardened the installer to pin
config fetches to the vouched tag (`hostinstall_gates.py` even asserts the pin exists —
`ok: fetch_raw pins the agent configs to the vouched agent version`). Nothing asserts that the pinned ref
**resolves**. So the first vouch naming an untagged version fails every install closed.
**The release script already knows.** `felhom-agent/scripts/release-agent.sh:23` says in as many words:
*"`raw/tag/v<version>`, a released version without a git tag 404s a box mid-install, as root."* The
hazard was understood, documented, and the tag step was still missed.
**Blast radius — stated precisely, not inflated:**
- **Every fresh install and every reinstall, on every host, fails at step 5/8** for as long as the Day-0
manifest vouches agent 0.128.0. That manifest was approved on 2026-08-09; the approval is what armed it.
- **Existing boxes are unaffected.** demo-felhom runs 0.128.0 happily — the self-update path takes the
binary from the package registry, not from a tag.
- So the damage is confined to the install path. That is also the only path a new customer, or a
reinstalling tester, ever walks.
**What a customer would do next:** nothing. `curl: (22) … 404` and a question about a git tag is not a
message a household can act on. The visit ends here and a developer is required.
### 9.2 The box's state, left as found for inspection
`--uninstall` is not re-run and no manual repair is attempted (rule 1). demo-hp is mid-install:
```
state.json completed: [preflight, token, grows, enroll] mode: byo
agent 0.128.0 installed, unit NOT created (not-found), inactive
user felhom-agent uid=999 created
pveum 2 users, 3 roles, 63 ACL lines, pool 'felhom' recreated (empty)
guest none
```
The documented `--resume` path exists and **was deliberately not used: it cannot help.** Resume replays
from `enroll`, hits the same 404, and fails identically — the missing tag is not a transient.
### 9.3 The one-line unblock, for the operator to decide
```sh
cd /mnt/5_hdd/felhom.eu/git/felhom-agent
git tag v0.128.0 28ba8593b8 && git push origin v0.128.0
```
This publishes the tag for a commit that is already on `main` and already released as a package — it
completes a half-finished release rather than changing any code. The alternative, rolling the manifest
back to agent 0.127.0, costs R-221 — the fix `STATUS.md` calls *"the last item blocking a customer from
something we promise them"* — so it is the worse option.
**Not done in this session.** Publishing a release tag is an outward-facing act and the runbook is
explicit: *"If a defect is so severe that the run cannot continue, stop and report; do not fix it here."*
### 9.4 What F-14 still owes
Step 7 never ran, so **whether the install would actually have adopted the stale 0.192.0 golden is not
yet observed** — F-14 rests on the code path plus the preflight's `golden (local): …9100…2026_08_03…`
line and the archive's own `felhom-controller:0.192.0` marker. That is strong, but it is not the
observation. **When the tag is published and the walk resumes, the first thing to record at step 7/8 is
which golden it takes.**
---
## 9a. P3 resumed — and the install succeeded
**Unblocked on operator instruction ("proceed").** `v0.128.0` was annotated at `28ba8593b8` and pushed
after an **independent download** confirmed the published package's sha256 equals the hub's vouched
`c6eba73b…`. Both config URLs then served 200. `--resume` completed in **3 m 49 s**
(09:30:30 → 09:34:19 UTC): `Day-0 provision SUCCESS`, controller **0.210.0**, agent **0.128.0**, guest
9201 onboot, pool member, all 16 ACL assertions green, controller healthy in ~18 s.
**R-274 partly refuted, and recorded as such.** Step 7 fetched the **vouched 0.210.0** — because
`--resume` skips preflight, and preflight is where local auto-discovery sets `GOLDEN_VOLID`. So the
fresh and resume paths disagree, and the resume path is the safe one. What survives: discovery is
`sort | tail -1` (newest local) with **no manifest comparison**, so a box whose newest local golden
predates the vouched one still installs stale. Full text in the register.
## 10. P4 — first contact, as the customer
**A clean pass, and worth saying so plainly.** Within four minutes of the install finishing, the
customer's own URL — fetched from outside the box, over the public internet — served:
> **A szerver beállítása** · Demo HP
> *Add meg az e-mailben kapott beállító kódot, majd válassz saját jelszót a vezérlőpult védelméhez.*
> Beállító kód · Új jelszó (min. 12 karakter) · Új jelszó megerősítése
> *Nem kaptad meg a kódot? Új kód kérése*
Unprompted, in Hungarian, naming the customer, with a self-service route if the code never arrived,
and nothing anywhere asking for a command line. The hub showed the host **ONLINE** at agent 0.128.0.
**Two findings here, neither fatal:** the hub still read *"Claimed 18d ago"* while the box was serving
its first-run page (**R-283**), and the code that arrives is named three different things across the
three surfaces, with the mail pointing at an „Elfelejtett jelszó" page the box does not show
(**R-282**). The reset code was nonetheless **accepted on the setup page** — 302 and a session — so
this is naming, not function.
## 11. P5 — getting the machine back to work
### 11.1 The recovery screen, unsought — the headline pass
The **first thing** on the dashboard after claiming, with nothing sought:
> **Adatok visszaszerzése** — *Ezt a gépet újratelepítették. A korábbi, házon kívüli mentéseid
> megvannak* — a sealed package held centrally, **sealed at 2026-08-04T11:11:37Z**, openable only with
> the customer's code; *nobody can replace it — not Felhom, not support, not the operator*; and
> **„Ebben a lépésben semmit nem állítunk vissza és semmi nem változik."**
> Field: **Helyreállítási kód (tíz szó)**.
R-193's screen meeting reality on a genuinely rebuilt box. It answered all three of its questions and
its seal date matches `host_escrow.created_at` exactly.
*(Nit: the seal date is rendered raw as `2026-08-04T11:11:37Z` to a Hungarian household — ISO-8601 in
UTC where a localised date belongs.)*
### 11.2 STOP 3 — the unlock
Entered at the box's own screen, from a file, never on a command line. **21 seconds**, and it listed
what it found without restoring anything:
| app | legutóbbi mentés | méret |
|---|---|---|
| **calibre-web** | 2026-08-09 10:30 (CEST) | **3.8 MB** |
| felhom-offbox | ″ | 6.8 KB |
| opengist | ″ | 182.3 KB |
| privatebin | ″ | 6.8 KB |
That is exactly the Gate 0 snapshot set, seen from the customer's side.
### 11.3 The wall — R-280, and it is rank 1
The restore page diagnoses the situation perfectly and then sends the customer to an empty page:
> *„Előbb csatold vissza az adatmeghajtót. … **Ez két kattintás:** Tárhely → Meghajtók, »Meglévő
> meghajtó csatolása«. Utána gyere vissza ide."*
**It is not two clicks; it is zero possible clicks.** `GET /api/disks/candidates` →
`{"initialize":[],"attach":[]}`. The agent is fine — `GET /api/disks` returns the NVMe in full — but
`handleDiskCandidates` builds *both* lists from the **unclaimed-disk** scan, and demo-hp's NVMe is
deliberately both the user-data drive and the `felhom-backup` target, so it is claimed and never
offered. Correct for `initialize`; over-broad for `attach`, which is non-destructive by definition.
It cascades: no store → Calibre-Web's install page degrades to *„Nincs regisztrált adattároló — adja
meg kézzel az útvonalat"*; no app → every restore row reads „Nincs telepítve".
**Escape hatch used, and recorded as off-path:** `POST /settings/storage/add` with
`storage_path=/mnt/sys_drive` succeeded first try — an internal path, the very one registered before
the wipe, that no household customer could produce. Everything unblocked immediately afterwards and
the deploy form became a proper picker („Tárhely (sys_drive) — 64.2 GB szabad").
### 11.4 App redeploy
`calibre-web` deployed from the catalogue through the dashboard's own API in **1 m 36 s**, running and
healthy. The catalogue itself was complete and in Hungarian, every app „Nincs telepítve".
*(Also observed: the deploy page warned „A kiválasztott tárhely majdnem megtelt." over an option
carrying `data-free-percent="93"` and „64.2 GB szabad" — an apparent inverted threshold, **R-284**.)*
## 12. P6 — getting the data back
Three options, each explained, none starting without a question. **R-204 item 3 is visibly fixed:**
option 1 states in as many words that it does *not* bring the customer's own files back.
| step | wall clock |
|---|---|
| full-restore **prepare** (3.8 MB) | **8 s** |
| full-restore **execute** | **13.2 s** (10:27:32.08 → 10:27:45.29 UTC) |
> *„A(z) calibre-web teljes mentése visszaállítva ellenőrző mappába:
> `/mnt/sys_drive/felhom-data/backups/offsite-restore/calibre-web` — a saját fájljaiddal együtt. A
> meglévő adatok változatlanok."*
Restored to a **verification folder**, not into place — honest about it, and R-213's territory.
### THE INTEGRITY VERDICT — **BYTE-IDENTICAL**
```
expected 4 file(s); found 4
VERDICT: BYTE-IDENTICAL
```
Compared against `GATE0-before-manifest.json`, keyed on **raw name bytes**:
| sha256 | bytes | name |
|---|---|---|
| `54b773c46bbfd994…` | 3 145 728 | `binary-3mb.bin` |
| `52a5c5ebfcac247f…` | 59 | **`árvíztűrő-tükörfúrógép.txt`** — name bytes `c3a1…` identical, NFC preserved |
| `15d2024dfc224162…` | 25 | **`nested/őszibarack.md`** — name bytes identical |
| `924497918e55fe6d…` | 21 | `plain.txt` |
Four expected, four restored, zero differences. The accented filenames survived as **bytes**, not
merely as rendered text — the discriminator the Gate 0 positive control was built to enforce.
## 13. P7 — what the hub thought was happening
**It thought nothing.** Across the whole reinstall the hub recorded **zero events and zero
notifications** for demo-hp. Positive control (standing rule 3): the hub logged **2 events all day
across all customers**, newest `db_dump_completed` at 00:30:07 — the store is reachable and the
silence is real.
- **The good half:** no false alarm fired during a legitimate reinstall, which is what P7 watches for.
- **The owed half (R-281):** `escrow_blob_served` exists as the tripwire for exactly this moment —
*"If no recovery is in progress on that box, investigate"* — and **has fired for demo-hp before**
(last 2026-08-04 20:12:54). Today's real unseal fired it not at all. A reinstall and a stolen
machine look identical to the operator.
## 14. Teardown — all four layers
1. **The machine** — my instruments (`fp.py`, `restored.json`, the installer, the passphrase file)
removed. Guest 9201 running and healthy. **`drill-r50` (VM 300) untouched throughout**, as predicted.
2. **The host** — `local-lvm` 36.97 % → 21.85 % (bare) → **32.35 %** (rebuilt). Pre-existing leftovers
found and deliberately **not** removed, recorded instead: `c11-scratch`, the orphaned
`vzdump-lxc-9100` archives (now three).
3. **The hub** — **no customer or appliance record created**; `demo-hp` retained deliberately per the
runbook. Nothing to delete.
4. **The off-site side** — **18 snapshots, newest still `9e38b84c` / `78b93f04` at 08:30 UTC**, i.e.
unchanged since Gate 0. The restore was a pure read. **No prune, no forget, no delete by me**; the
only retention that ran was inside the product's own backup call at Gate 0, which reported
`18 snapshot(s)` itself.
## 15. The answer to §2's question
**Yes — the data comes back, byte for byte. No — not in one sitting, and not without a shell.**
The walk completed: **P1 ✓ P2 ✓ P3 ✓ P4 ✓ P5 ✓ P6 ✓ P7 ✓**, and the verdict is BYTE-IDENTICAL.
But it completed only because two hard stops were cleared by someone who could open a terminal and
read source code:
1. **R-273** — the install died at 5/8 on a git tag that was never pushed. Cleared by a release action.
2. **R-280** — the data drive could not be re-attached through any dashboard route, while the page
promised „két kattintás". Cleared by POSTing an internal path a customer could not know.
Neither is a data-integrity problem. Both are **journey** problems, and both stop a household dead.
This is the same shape the R-201 walks kept finding: **the data half passes, the journey half fails.**
**What the product did beautifully, and should not be lost in the finding count:** the box came up on
its own at the vouched version; the setup page appeared unprompted, in Hungarian, naming the customer;
the recovery screen appeared **without being sought** and stated plainly that nothing would be changed
by unlocking; the unlock took **21 s**; the restore took **13.2 s** and said honestly that it had put
the files in a verification folder rather than back in place. Sixteen ACL assertions verified
themselves. No false alarm fired.
### Wall clocks
| segment | |
|---|---|
| P1 uninstall | **60 s** |
| P2 preflight (2 refusals, then pass) | ~6 min |
| P3 install — first attempt, FAILED | 44 s |
| P3 install — resumed, SUCCESS | **3 m 49 s** |
| P4 first contact (box live on its own URL) | within ~4 min of install |
| STOP 3 unlock | **21 s** |
| app redeploy (calibre-web) | **1 m 36 s** |
| restore prepare + execute | **8 s + 13.2 s** |
| **bare machine → verified files** | **1 h 49 m 22 s** (08:38:23 → 10:27:45 UTC) |
| — of which the product's own work | **≈ 7 m 47 s** |
**The 1 h 49 m must not be quoted as the customer number** — it is dominated by the R-273 diagnosis and
release fix (~38 min) and by two waits on a human. **The ≈ 7 m 47 s must not be quoted either**: it is
what the product costs when someone already knows every answer. The honest figure for an unaided
household is **undefined, because an unaided household does not finish.**
@@ -0,0 +1,26 @@
{
"count": 4,
"files": {
"62696e6172792d336d622e62696e": {
"name_utf8": "binary-3mb.bin",
"sha256": "54b773c46bbfd99465b68247bcb4d3cf5bf8f57a4024c6ba169dc0c326979dd2",
"size": 3145728
},
"6e65737465642fc591737a6962617261636b2e6d64": {
"name_utf8": "nested/őszibarack.md",
"sha256": "15d2024dfc22416261c56ed51741f8d7ae734350b3ac79ef1cb982d667ddef64",
"size": 25
},
"706c61696e2e747874": {
"name_utf8": "plain.txt",
"sha256": "924497918e55fe6dd75ff5ed8a9089aa64752e05540bdf804d71a03afbc48acd",
"size": 21
},
"c3a17276c3ad7a74c5b172c5912d74c3bc6bc3b67266c3ba72c3b367c3a9702e747874": {
"name_utf8": "árvíztűrő-tükörfúrógép.txt",
"sha256": "52a5c5ebfcac247f1753e86f69db46815adf4edaf28375396324e4ea6163611c",
"size": 59
}
},
"root": "/var/lib/felhom/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09"
}
@@ -0,0 +1,282 @@
=== captured 2026-08-09T08:17:53Z UTC / 2026-08-09T10:17:53+0200 host-local ===
### felhom-agent
felhom-agent 0.127.0
### pveversion
pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
### pct list
VMID Status Lock Name
9201 running demo-hp
### qm list
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
300 drill-r50 stopped 8192 32.00 0
### pool members
{"comment":"Felhom-managed guests","members":[{"cpu":0.00448653868698896,"disk":990715904,"diskread":4608909312,"diskwrite":127439912960,"id":"lxc/9201","maxcpu":7,"maxdisk":33501757440,"maxmem":28212985856,"mem":641748992,"memhost":0,"name":"demo-hp","netin":2213810807,"netout":376936192,"node":"felhom-host","status":"running","template":0,"type":"lxc","uptime":525808,"vmid":9201}],"poolid":"felhom"}
### pvesm status
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
c11-scratch dir active 983379700 6940296 926412792 0.71%
felhom-backup dir active 983379700 6940296 926412792 0.71%
felhom-pbs pbs active 0 0 0 0.00%
local dir active 40516856 19132716 19293748 47.22%
local-lvm lvmthin active 56545280 20904790 35640489 36.97%
### storage.cfg
dir: local
path /var/lib/vz
content iso,import,vztmpl,backup
lvmthin: local-lvm
thinpool data
vgname pve
content rootdir,images
pbs: felhom-pbs
datastore felhom-offsite
server 10.77.0.1
content backup
encryption-key 3f:4f:65:c0:d8:f3:9f:3c:43:29:8c:c8:61:3c:89:40:e6:7a:c8:ae:b1:44:70:09:0e:f1:f7:1b:60:a0:c4:99
fingerprint c6:07:28:3f:5b:7b:5a:41:90:28:d7:ca:4f:37:14:70:56:39:2e:2f:0b:71:e8:06:ca:60:4a:d5:56:5f:3c:fd
namespace demo-hp
username felhom@pbs!demo-hp
dir: felhom-backup
path /mnt/nvme-1tb
content backup
is_mountpoint 1
dir: c11-scratch
path /mnt/nvme-1tb
content images
### df
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-root 39G 19G 19G 50% /
/dev/nvme0n1 938G 6.7G 884G 1% /mnt/nvme-1tb
### lsblk
NAME SIZE MODEL SERIAL FSTYPE MOUNTPOINT
sda 119.2G SanDisk X600 M.2 2280 SATA 128GB 182195804614
|-sda1 1007K
|-sda2 1G vfat /boot/efi
`-sda3 118.2G LVM2_member
|-pve-swap 8G swap [SWAP]
|-pve-root 39.6G ext4 /
|-pve-data_tmeta 1G
| `-pve-data-tpool 53.9G
| |-pve-data 53.9G
| |-pve-vm--9201--disk--0 32G ext4
| |-pve-vm--9201--disk--1 70G ext4
| |-pve-vm--300--disk--0 4M
| `-pve-vm--300--disk--1 32G
`-pve-data_tdata 53.9G
`-pve-data-tpool 53.9G
|-pve-data 53.9G
|-pve-vm--9201--disk--0 32G ext4
|-pve-vm--9201--disk--1 70G ext4
|-pve-vm--300--disk--0 4M
`-pve-vm--300--disk--1 32G
nvme0n1 953.9G KXG50PNV1T02 NVMe TOSHIBA 1024GB 58BS11AFT8MQ ext4 /mnt/nvme-1tb
### felhom-agent unit + user
active
uid=999(felhom-agent) gid=990(felhom-agent) groups=990(felhom-agent),999(systemd-journal)
### sudoers entries
total 68
drwxr-xr-x 2 root root 4096 Aug 3 08:09 .
drwxr-xr-x 102 root root 4096 Aug 3 08:09 ..
-r--r----- 1 root root 1068 Apr 11 14:21 README
-r--r----- 1 root root 22483 Aug 3 08:08 felhom-agent
-r--r----- 1 root root 21833 Jul 28 11:29 felhom-agent.bak-pre-e2a
-r--r----- 1 root root 1241 Aug 3 08:09 felhom-op
-r--r----- 1 root root 666 May 20 2023 zfs
--
287
### pveum users/tokens/roles/acl
+------------------+---------+-------------------+--------+--------+-----------+--------+------+----------+------------+------------------+--------+-------------+
| userid | comment | email | enable | expire | firstname | groups | keys | lastname | realm-type | tfa-locked-until | tokens | totp-locked |
+==================+=========+===================+========+========+===========+========+======+==========+============+==================+========+=============+
| felhom-agent@pve | | | 1 | 0 | | | | | pve | | | |
+------------------+---------+-------------------+--------+--------+-----------+--------+------+----------+------------+------------------+--------+-------------+
| root@pam | | noreply@felhom.eu | 1 | 0 | | | | | pam | | | |
+------------------+---------+-------------------+--------+--------+-----------+--------+------+----------+------------+------------------+--------+-------------+
--tokens--
+---------+---------+--------+---------+
| tokenid | comment | expire | privsep |
+=========+=========+========+=========+
| agent | | 0 | 1 |
+---------+---------+--------+---------+
--roles--
| FelhomAgentBase | Datastore.Audit,SDN.Use,Sys.Audit | 0 |
| FelhomAgentGuest | Pool.Allocate,Pool.Audit,VM.Allocate,VM.Audit,VM.Backup,VM.Config.CPU,VM.Config.Disk,VM.Config.Memory,VM.Config.Network,VM.Config.Options,VM.PowerMgmt,VM.Snapshot,VM.Snapshot.Rollback | 0 |
| FelhomAgentStore | Datastore.Allocate,Datastore.AllocateSpace | 0 |
--acl--
+------------------------+------------------+-------+------------------------+-----------+
| path | roleid | type | ugid | propagate |
+========================+==================+=======+========================+===========+
| / | FelhomAgentBase | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| / | FelhomAgentBase | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /pool/felhom | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /pool/felhom | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/felhom-backup | FelhomAgentStore | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/felhom-backup | FelhomAgentStore | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/felhom-pbs | FelhomAgentStore | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/felhom-pbs | FelhomAgentStore | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/local | FelhomAgentStore | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/local | FelhomAgentStore | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/local-lvm | FelhomAgentStore | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/local-lvm | FelhomAgentStore | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990001 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990001 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990002 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990002 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990003 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990003 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990004 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990004 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990005 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990005 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990006 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990006 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990007 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990007 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990008 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990008 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990009 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990009 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
### guest hook
total 12
drwxr-xr-x 2 root root 4096 Aug 3 08:11 .
drwxr-xr-x 7 root root 4096 Jul 30 12:36 ..
-rwxr-xr-x 1 root root 166 Aug 3 08:11 felhom-guest-hook.sh
/var/lib/vz/snippets/felhom-guest-hook.sh
### wg tunnel
interface: wg-felhom
public key: KEYSzMyTKEZVDqDbikDXnadyeX9yJUjy8T58q0f4/1U=
private key: (hidden)
listening port: 59688
peer: f3d1ZI7/2/r+FWz8eo8Wl2HVygTZK9wBIN2M3dNHsgA=
endpoint: 167.233.158.164:443
allowed ips: 10.77.0.1/32, 10.77.0.250/32
latest handshake: 8 seconds ago
transfer: 5.86 GiB received, 2.48 GiB sent
persistent keepalive: every 25 seconds
### OOB belt
active
LISTEN 0 128 0.0.0.0:8822 0.0.0.0:* users:(("sshd",pid=1027109,fd=7))
LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=1052,fd=6))
LISTEN 0 128 [::]:8822 [::]:* users:(("sshd",pid=1027109,fd=6))
LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=1052,fd=7))
### agent state dir
total 56
drwxr-x--- 7 felhom-agent felhom-agent 4096 Aug 9 06:59 .
drwxr-xr-x 45 root root 4096 Aug 3 08:08 ..
-rw------- 1 felhom-agent felhom-agent 61 Aug 3 08:09 drive-intents.json
drwx------ 2 felhom-agent felhom-agent 4096 Aug 4 23:11 escrow-stage
drwx------ 2 felhom-agent felhom-agent 4096 Aug 3 08:09 felhom-sshd
drwx------ 3 felhom-agent felhom-agent 4096 Aug 3 08:11 guests
-rw------- 1 felhom-agent felhom-agent 5805 Aug 9 06:59 journal.log
-rw-r--r-- 1 felhom-agent felhom-agent 591 Aug 3 08:09 local-api.crt
-rw------- 1 felhom-agent felhom-agent 227 Aug 3 08:09 local-api.key
-rw------- 1 felhom-agent felhom-agent 82 Aug 3 08:11 local-tokens.log
-rw------- 1 felhom-agent felhom-agent 0 Aug 3 08:09 nonces.log
drwx------ 2 felhom-agent felhom-agent 4096 Aug 3 08:09 pbsdr
-rw------- 1 felhom-agent felhom-agent 383 Aug 9 06:59 restore-test-state.json
drwx------ 2 felhom-agent felhom-agent 4096 Aug 3 08:09 wg
### install state file
-rw-r--r-- 1 root root 1722 Aug 3 08:09 /etc/felhom-oob.nft
/etc/felhom:
total 12
drwxr-xr-x 2 root root 4096 Jul 21 18:28 .
drwxr-xr-x 102 root root 4096 Aug 3 08:09 ..
-rw-r--r-- 1 root root 0 Jul 21 18:28 .bootstrap-done
-rw-r--r-- 1 root root 7 Jul 21 18:20 appliance-pairing-code
/etc/felhom-agent:
total 36
drwxr-xr-x 2 root root 4096 Aug 4 09:32 .
drwxr-xr-x 102 root root 4096 Aug 3 08:09 ..
-rw------- 1 felhom-agent felhom-agent 2463 Aug 4 09:32 agent.json
-rw------- 1 root root 2462 Aug 4 09:32 agent.json.bak-r191
-rw------- 1 felhom-agent felhom-agent 2460 Jul 27 11:31 agent.json.campaign8-before
-rw------- 1 root root 2459 Jul 28 17:36 agent.json.campaign9-before
-rw------- 1 root root 2455 Jul 28 17:37 agent.json.campaign9-prev
-rw------- 1 felhom-agent felhom-agent 2459 Jul 28 18:10 agent.json.pre-e-target-move
-rw------- 1 felhom-agent felhom-agent 2459 Jul 27 08:08 agent.json.pre-prunegate.bak
/etc/felhom-sshd:
total 24
drwxr-xr-x 3 root root 4096 Aug 6 12:58 .
drwxr-xr-x 102 root root 4096 Aug 3 08:09 ..
drwxr-xr-x 2 root root 4096 Aug 6 12:58 authorized_keys
-rw------- 1 root root 411 Aug 3 08:09 ssh_host_ed25519_key
-rw-r--r-- 1 root root 101 Aug 3 08:09 ssh_host_ed25519_key.pub
-rw-r--r-- 1 root root 433 Aug 6 12:58 sshd_config
### PBS snapshots
Volid Format Type Size VMID
felhom-pbs:backup/ct/9201/2026-07-28T19:19:45Z pbs-ct backup 6264034049 9201
felhom-pbs:backup/ct/9201/2026-08-04T19:24:16Z pbs-ct backup 4637840513 9201
### local vzdump archives
Volid Format Type Size VMID
felhom-backup:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst tar.zst backup 649547835 9100
felhom-backup:backup/vzdump-lxc-9201-2026_08_07-04_34_58.tar.zst tar.zst backup 2073915792 9201
felhom-backup:backup/vzdump-lxc-9201-2026_08_08-04_37_15.tar.zst tar.zst backup 2142232544 9201
felhom-backup:backup/vzdump-lxc-9201-2026_08_09-04_42_15.tar.zst tar.zst backup 2143026831 9201
### enrolled drives
/mnt/felhom-drives/:
total 12
drwxr-xr-x 3 root root 4096 Aug 3 08:05 .
drwxr-xr-x 6 root root 4096 Aug 6 14:39 ..
drwxr-xr-x 2 root root 4096 Jul 22 13:53 Felhom-Share
/mnt/felhom-drives/Felhom-Share:
total 8
drwxr-xr-x 2 root root 4096 Jul 22 13:53 .
drwxr-xr-x 3 root root 4096 Aug 3 08:05 ..
@@ -0,0 +1,167 @@
=== captured 2026-08-09T08:38:48Z UTC ===
### agent binary
ABSENT
### agent unit
felhom-bootstrap.service disabled enabled
### service user
id: 'felhom-agent': no such user
ABSENT
### felhom-op user (OOB)
id: 'felhom-op': no such user
ABSENT
### sudoers
total 40
drwxr-xr-x 2 root root 4096 Aug 9 10:37 .
drwxr-xr-x 101 root root 4096 Aug 9 10:37 ..
-r--r----- 1 root root 1068 Apr 11 14:21 README
-r--r----- 1 root root 21833 Jul 28 11:29 felhom-agent.bak-pre-e2a
-r--r----- 1 root root 666 May 20 2023 zfs
/etc/sudoers.d/felhom-agent.bak-pre-e2a
### pct list
### qm list
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
300 drill-r50 stopped 8192 32.00 0
### pools
[]
### pveum users/roles/acl
+----------+---------+-------------------+--------+--------+-----------+--------+------+----------+------------+------------------+--------+-------------+
| userid | comment | email | enable | expire | firstname | groups | keys | lastname | realm-type | tfa-locked-until | tokens | totp-locked |
+==========+=========+===================+========+========+===========+========+======+==========+============+==================+========+=============+
| root@pam | | noreply@felhom.eu | 1 | 0 | | | | | pam | | | |
+----------+---------+-------------------+--------+--------+-----------+--------+------+----------+------------+------------------+--------+-------------+
--roles--
no felhom roles
--acl--
### pvesm status
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
c11-scratch dir active 983379700 6940296 926412792 0.71%
felhom-backup dir active 983379700 6940296 926412792 0.71%
felhom-pbs pbs active 0 0 0 0.00%
local dir active 40516856 19063736 19362728 47.05%
local-lvm lvmthin active 56545280 12355143 44190136 21.85%
### storage.cfg
dir: local
path /var/lib/vz
content iso,import,vztmpl,backup
lvmthin: local-lvm
thinpool data
vgname pve
content rootdir,images
pbs: felhom-pbs
datastore felhom-offsite
server 10.77.0.1
content backup
encryption-key 3f:4f:65:c0:d8:f3:9f:3c:43:29:8c:c8:61:3c:89:40:e6:7a:c8:ae:b1:44:70:09:0e:f1:f7:1b:60:a0:c4:99
fingerprint c6:07:28:3f:5b:7b:5a:41:90:28:d7:ca:4f:37:14:70:56:39:2e:2f:0b:71:e8:06:ca:60:4a:d5:56:5f:3c:fd
namespace demo-hp
username felhom@pbs!demo-hp
dir: felhom-backup
path /mnt/nvme-1tb
content backup
is_mountpoint 1
dir: c11-scratch
path /mnt/nvme-1tb
content images
### df
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-root 39G 19G 19G 50% /
/dev/nvme0n1 938G 6.7G 884G 1% /mnt/nvme-1tb
### mounts under /mnt
/mnt/nvme-1tb /dev/nvme0n1
### felhom-drives tree
/mnt/felhom-drives/:
total 12
drwxr-xr-x 3 root root 4096 Aug 3 08:05 .
drwxr-xr-x 6 root root 4096 Aug 6 14:39 ..
drwxr-xr-x 2 root root 4096 Jul 22 13:53 Felhom-Share
/mnt/felhom-drives/Felhom-Share:
total 8
drwxr-xr-x 2 root root 4096 Jul 22 13:53 .
drwxr-xr-x 3 root root 4096 Aug 3 08:05 ..
### leftover felhom paths
/etc/felhom
/etc/felhom-agent
/usr/local/sbin/felhom-bootstrap.sh
none of the usual paths remain
### systemd leftovers on disk
-rw-r--r-- 1 root root 1662 Jul 21 18:20 felhom-bootstrap.service
### guest hook snippets
total 8
drwxr-xr-x 2 root root 4096 Aug 9 10:37 .
drwxr-xr-x 7 root root 4096 Jul 30 12:36 ..
### dnsmasq snippets
no felhom dnsmasq snippets
### wg tunnel
interface: wg-felhom
public key: KEYSzMyTKEZVDqDbikDXnadyeX9yJUjy8T58q0f4/1U=
private key: (hidden)
listening port: 59688
peer: f3d1ZI7/2/r+FWz8eo8Wl2HVygTZK9wBIN2M3dNHsgA=
endpoint: 167.233.158.164:443
allowed ips: 10.77.0.1/32, 10.77.0.250/32
latest handshake: 52 seconds ago
transfer: 5.86 GiB received, 2.48 GiB sent
vmbr9 DOWN 00:00:00:00:00:00 <NO-CARRIER,BROADCAST,MULTICAST,UP>
wg-felhom UNKNOWN <POINTOPOINT,NOARP,UP,LOWER_UP>
### network bridges
auto vmbr9
iface vmbr9 inet static
address 169.254.253.1/30
bridge-ports none
bridge-stp off
bridge-fd 0
### listening ports
State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0 4096 100.76.96.79:39440 0.0.0.0:* users:(("tailscaled",pid=1152166,fd=24))
LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=1052,fd=6))
LISTEN 0 32 0.0.0.0:53 0.0.0.0:* users:(("dnsmasq",pid=1291306,fd=5))
LISTEN 0 4096 0.0.0.0:111 0.0.0.0:* users:(("rpcbind",pid=701,fd=4),("systemd",pid=1,fd=97))
LISTEN 0 100 127.0.0.1:25 0.0.0.0:* users:(("master",pid=1230,fd=13))
LISTEN 0 4096 127.0.0.1:85 0.0.0.0:* users:(("pvedaemon worke",pid=1179410,fd=6),("pvedaemon worke",pid=1167090,fd=6),("pvedaemon worke",pid=1015859,fd=6),("pvedaemon",pid=1283,fd=6))
LISTEN 0 100 [::1]:25 [::]:* users:(("master",pid=1230,fd=14))
LISTEN 0 4096 *:3128 *:* users:(("spiceproxy work",pid=681975,fd=6),("spiceproxy",pid=1370,fd=6))
LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=1052,fd=7))
LISTEN 0 32 [::]:53 [::]:* users:(("dnsmasq",pid=1291306,fd=7))
LISTEN 0 4096 [::]:111 [::]:* users:(("rpcbind",pid=701,fd=6),("systemd",pid=1,fd=99))
LISTEN 0 4096 [fd7a:115c:a1e0::ce36:6051]:51957 [::]:* users:(("tailscaled",pid=1152166,fd=25))
LISTEN 0 4096 *:8006 *:* users:(("pveproxy worker",pid=1245509,fd=6),("pveproxy worker",pid=1244391,fd=6),("pveproxy worker",pid=1241021,fd=6),("pveproxy",pid=1295,fd=6))
### PBS snapshots (must be KEPT)
Volid Format Type Size VMID
felhom-pbs:backup/ct/9201/2026-07-28T19:19:45Z pbs-ct backup 6264034049 9201
felhom-pbs:backup/ct/9201/2026-08-04T19:24:16Z pbs-ct backup 4637840513 9201
### local vzdump (must be KEPT)
Volid Format Type Size VMID
felhom-backup:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst tar.zst backup 649547835 9100
felhom-backup:backup/vzdump-lxc-9201-2026_08_07-04_34_58.tar.zst tar.zst backup 2073915792 9201
felhom-backup:backup/vzdump-lxc-9201-2026_08_08-04_37_15.tar.zst tar.zst backup 2142232544 9201
felhom-backup:backup/vzdump-lxc-9201-2026_08_09-04_42_15.tar.zst tar.zst backup 2143026831 9201
@@ -0,0 +1,755 @@
[INFO] felhom-host-install v1.25.0 — mode=uninstall
[STEP] UNINSTALL — local host teardown
9201
[INFO] no --vmid given; using recorded provisioned_vmid=9201 from /var/lib/felhom-install/state.json
[INFO] vmid 9201 looks like a Felhom guest (has the /etc/felhom-bootstrap mount)
[INFO] pct config 9201:
arch: amd64
cores: 7
features: nesting=1,keyctl=1
hookscript: local:snippets/felhom-guest-hook.sh
hostname: demo-hp
memory: 26906
mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:F6:C5:F5,ip=dhcp,type=veth
net1: name=eth1,bridge=vmbr9,hwaddr=BC:24:11:CC:43:3E,ip=169.254.253.2/30,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-9201-disk-0,size=32G
swap: 512
unprivileged: 1
Type the vmid (9201) to confirm PERMANENT destruction: perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
Logical volume "vm-9201-disk-0" successfully removed.
Logical volume "vm-9201-disk-1" successfully removed.
[OK] guest 9201 destroyed
[STEP] host-level removal
Removed '/etc/systemd/system/multi-user.target.wants/felhom-agent.service'.
Removed '/etc/systemd/system/timers.target.wants/felhom-mgmt-watchdog.timer'.
Removed '/etc/systemd/system/multi-user.target.wants/felhom-sshd.service'.
Removed '/etc/systemd/system/multi-user.target.wants/felhom-oob-nft.service'.
The unit files have no installation config (WantedBy=, RequiredBy=, UpheldBy=,
Also=, or Alias= settings in the [Install] section, and DefaultInstance= for
template units). This means they are not meant to be enabled or disabled using systemctl.
 
Possible reasons for having these kinds of units are:
• A unit may be statically enabled by being symlinked from another unit's
 .wants/, .requires/, or .upholds/ directory.
• A unit's purpose may be to act as a helper for some other unit which has
 a requirement dependency on it.
• A unit may be started when needed via activation (socket, path, timer,
 D-Bus, udev, scripted systemctl call, ...).
• In case of template units, the unit is meant to be enabled with some
 instance name specified.
Removed '/etc/systemd/system/pve-guests.service.wants/felhom-shared-parent.service'.
Removed '/etc/systemd/system/multi-user.target.wants/felhom-shared-parent.service'.
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
user config - ignore invalid acl role 'FelhomAgentStore'
[SKIP] old broad role FelhomAgent already absent
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[SKIP] golden vzdump left in place (pass --remove-golden to remove)
[STEP] kept vs wiped — read before pulling drives or closing the customer out
@@ -0,0 +1,63 @@
[INFO] felhom-host-install v1.25.0 — mode=byo customer=demo-hp vmid=9201
[STEP] 1/8 pre-flight
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
[INFO] node: felhom-host (auto)
[INFO] agent config: /etc/felhom-agent/agent.json
[INFO] agent: not installed yet — will be fetched + installed in step 5/8
[INFO] local-lvm free: ~42 GiB
[WARN] local-lvm free ~42 GiB < hard min 120 GiB
[INFO] free RAM: ~26457 MiB
[INFO] existing guests on this host: 1 (pct+qm)
[INFO] archive storage 'local' present
@@ -0,0 +1,40 @@
[INFO] felhom-host-install v1.25.0 — mode=byo customer=demo-hp vmid=9201
[STEP] 1/8 pre-flight
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
[INFO] node: felhom-host (auto)
[INFO] agent config: /etc/felhom-agent/agent.json
[INFO] agent: not installed yet — will be fetched + installed in step 5/8
[INFO] local-lvm free: ~42 GiB
[WARN] local-lvm free ~42 GiB < hard min 120 GiB
[INFO] free RAM: ~26535 MiB
[INFO] existing guests on this host: 1 (pct+qm)
[INFO] archive storage 'local' present
[INFO] acl storages all present or pre-positioned: local local-lvm felhom-pbs
[ERROR] a resolver is already bound to :53 on this host:
udp UNCONN 0 0 0.0.0.0:53 0.0.0.0:* users:(("dnsmasq",pid=1291306,fd=4)) udp UNCONN 0 0 [::]:53 [::]:* users:(("dnsmasq",pid=1291306,fd=6)) tcp LISTEN 0 32 0.0.0.0:53 0.0.0.0:* users:(("dnsmasq",pid=1
[ERROR] a resolver is already bound to :53 on this host — Felhom needs the guest reachable by name on your LAN.
Stop or reconfigure that resolver, OR point your LAN DNS at the guest's address, then re-run.
(Felhom does NOT touch DNS services on a host it does not own — this is a refusal, not a change.)
[ERROR] PRE-FLIGHT FAIL (exit 1) — fix the finding above and re-run
Connection to 100.76.96.79 closed.
@@ -0,0 +1,28 @@
=== half-installed state 2026-08-09T08:54:00Z ===
### agent
felhom-agent 0.128.0
inactive
not-found
### install state file
{
"completed": [
"preflight",
"token",
"grows",
"enroll"
],
"customer_id": "demo-hp",
"mode": "byo"
}### service user
uid=999(felhom-agent) gid=990(felhom-agent) groups=990(felhom-agent),999(systemd-journal)
### sudoers now present
-r--r----- 1 root root 21833 Jul 28 11:29 felhom-agent.bak-pre-e2a
### pveum
2
3
63
### pool
[{"comment":"Felhom-managed guests","poolid":"felhom"}]
### guest
### wrappers
-rwxr-xr-x 1 root root 15730 Jul 21 18:20 /usr/local/sbin/felhom-bootstrap.sh
@@ -0,0 +1,835 @@
felhom-host
[INFO] felhom-host-install v1.25.0 — mode=byo customer=demo-hp vmid=9201
[SKIP] pre-flight (resumed)
[STEP] 2/8 Proxmox API token
[SKIP] pool felhom already exists
[INFO] user felhom-agent@pve exists
[INFO] removing stale token agent (secret unrecoverable — rotating)
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
user config - ignore invalid acl token 'felhom-agent@pve!agent'
[OK] token minted (secret captured, not logged)
[INFO] role FelhomAgentBase exists — ensuring exact privileges
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[INFO] role FelhomAgentGuest exists — ensuring exact privileges
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[INFO] role FelhomAgentStore exists — ensuring exact privileges
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[OK] scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm felhom-pbs])
[SKIP] old broad role FelhomAgent already absent
[STEP] 3/8 compute volume grows
[INFO] auto-computed from ~42 GiB free (ONE volume since R-165)
[INFO] grows: rootfs +0G (->32G), data +46G (->70G, ONE volume)
[STEP] 4/8 host enrollment (POST /host-enroll)
[OK] host REUSED (idempotent — existing credential)
[INFO] host_id: demo-hp-bb76ea (api_key captured, not logged)
[SKIP] break-glass (step 4b) SKIPPED (byo: the host owner manages break-glass; root@pam untouched, nothing vaulted)
[STEP] 5/8 agent install (fetch + verify + install)
[SKIP] apt repo alignment skipped (byo — the owner manages repos)
[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
[INFO] manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.210.0
[INFO] fetching agent binary v0.128.0 from Gitea …
[OK] verified sha256 c6eba73bf9b9ad69… matches the hub manifest
[OK] installed /usr/local/bin/felhom-agent (felhom-agent 0.128.0)
[INFO] service user felhom-agent exists
[INFO] felhom-agent already in systemd-journal
[OK] installed /usr/local/sbin/felhom-mkfs-guarded (0755, the guarded mkfs path)
[OK] installed /usr/local/sbin/felhom-selfupdate-guarded (0755, the guarded A/B binary-swap path)
[OK] installed /usr/local/sbin/felhom-pbs-apply (0755, the guarded PBS-DR apply path)
[OK] installed /usr/local/sbin/felhom-backup-target-apply (0755, the guarded backup-target path)
[OK] installed /etc/sudoers.d/felhom-agent (0440, visudo-validated)
[OK] installed /etc/systemd/system/felhom-agent.service + enabled (started in step 6 after config)
[OK] installed self-update rollback unit + start-limit drop-in (auto-rollback armed)
[OK] installed break-glass layers 1+2 (tmpfiles /run/sshd + agent-independent watchdog timer)
[SKIP] OOB (felhom-sshd) off (byo, or appliance --no-oob) — skipping
[STEP] 6/8 agent config + service
[SKIP] backup target 'felhom-backup' already exists — leaving its DEFINITION exactly as it is (Scenario F)
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
felhom-backup-target-apply: granted FelhomAgentStore on /storage/felhom-backup (user + token)
[INFO] node=felhom-host local_api=192.168.0.87:8443 tls_fp=E5:6D:79:3B:67:A1…
[OK] wrote /etc/felhom-agent/agent.json (0600 felhom-agent)
[OK] byo config asserts passed (lan_resolver.enable / oob.enabled off; deployment_mode=byo; wg_tunnel intentionally ON — base infrastructure)
[OK] agent --selftest (read-only) passed
[OK] felhom-agent service active (non-root felhom-agent reads the config OK)
[STEP] 7/8 golden archive
[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
[INFO] fetching golden v0.210.0 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_08_09-11_31_16.tar.zst
[OK] verified sha256 b9f701fab813c051… matches the hub manifest
[OK] golden imported + verified: local:backup/vzdump-lxc-9100-2026_08_09-11_31_16.tar.zst
[STEP] 8/8 provision guest 9201
[SKIP] pool felhom already exists
=== felhom-agent 0.128.0 selftest=provision (vmid=9201 customer=demo-hp hostname=demo-hp) ===
--- front half: bring-up (provision) local:backup/vzdump-lxc-9100-2026_08_09-11_31_16.tar.zst → vmid 9201 ---
time=2026-08-09T11:33:34.336+02:00 level=INFO msg="bring-up: pool membership re-asserted" vmid=9201 pool=felhom
[OK] front half: vmid 9201 up (boot+running) in 1m9s; MAC=BC:24:11:65:7F:69
--- back half: mint per-guest token + populate bootstrap config mount ---
time=2026-08-09T11:33:38.654+02:00 level=INFO msg="provision: back-half complete" vmid=9201 mount=mp9 guest_path=/etc/felhom-bootstrap endpoint=192.168.0.87:8443
[OK] back half: bootstrap mount mp9 → /etc/felhom-bootstrap on vmid 9201 (host dir /var/lib/felhom-agent/guests/9201/bootstrap)
local-api endpoint 192.168.0.87:8443 · leaf fp e3c7aaafbe7643d35eb5a09d2c3fe5128f1ef626010e7c17060dd55a5cda37d5 · token: minted (not printed)
=== selftest=provision OK — guest 9201 provisioned + bootstrap-mounted (KEPT) ===
next: reboot the guest → the golden's baked controller-bootstrap unit deploys the controller,
which PULLS its controller.yaml from the hub (retrieval passphrase) and merges in this local_api.
[OK] provision completed
[INFO] rebooting guest 9201 so the baked controller-bootstrap unit picks up the mount
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[STEP] verify
[OK] pct status: running
[OK] onboot: 1
mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
rootfs: local-lvm:vm-9201-disk-0,size=32G
[OK] pool: guest 9201 is a member of felhom
[OK] acl: FelhomAgentBase@/ present (user+token)
[OK] acl: FelhomAgentGuest@/pool/felhom present (user+token)
[OK] acl: FelhomAgentStore@/storage/local present (user+token)
[OK] acl: FelhomAgentStore@/storage/local-lvm present (user+token)
[OK] acl: FelhomAgentStore@/storage/felhom-pbs present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990000 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990001 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990002 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990003 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990004 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990005 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990006 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990007 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990008 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990009 present (user+token)
[OK] authz signers: 2 (operator-signed self-update armed)
[OK] controller: Up 18 seconds (healthy) (after ~0s)
[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.210.0
[INFO] cloudflared: Up 18 seconds
[INFO] (confirm in the hub UI that host demo-hp-bb76ea reports guest 9201)
[OK] Day-0 provision SUCCESS — vmid=9201 host_id=demo-hp-bb76ea customer=demo-hp golden=local:backup/vzdump-lxc-9100-2026_08_09-11_31_16.tar.zst
Connection to 100.76.96.79 closed.
@@ -0,0 +1,768 @@
[INFO] felhom-host-install v1.25.0 — mode=byo customer=demo-hp vmid=9201
[STEP] 1/8 pre-flight
felhom-host
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
[INFO] node: felhom-host (auto)
[INFO] agent config: /etc/felhom-agent/agent.json
[INFO] agent: not installed yet — will be fetched + installed in step 5/8
[INFO] local-lvm free: ~42 GiB
[WARN] local-lvm free ~42 GiB < hard min 120 GiB
[INFO] free RAM: ~26440 MiB
[INFO] existing guests on this host: 1 (pct+qm)
[INFO] archive storage 'local' present
[INFO] acl storages all present or pre-positioned: local local-lvm felhom-pbs
[INFO] host DNS (:53): free
[INFO] hub reachable (https://hub.felhom.eu)
[OK] customer 'demo-hp' exists + passphrase valid
[INFO] golden (local): local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst
[STEP] BYO disclosure — installing Felhom on a host the operator does NOT own will create:
pveum: roles FelhomAgentBase/FelhomAgentGuest/FelhomAgentStore + user felhom-agent@pve + token felhom-agent@pve!agent
+ scoped ACL grants (Base@/, Guest@/pool/felhom, Store@[local local-lvm felhom-pbs]) + the 'felhom' pool
agent: service user felhom-agent + /usr/local/bin/felhom-agent (+ .bak-* backups; A/B slots via the guarded self-update)
+ /etc/systemd/system/felhom-agent.service (+ felhom-agent-rollback.service + start-limit drop-in)
+ /etc/sudoers.d/felhom-agent + state dir /var/lib/felhom-agent + config /etc/felhom-agent/agent.json (0600)
host: /usr/local/sbin/felhom-mkfs-guarded + /usr/local/sbin/felhom-selfupdate-guarded
+ /usr/local/sbin/felhom-pbs-apply (PBS-DR apply wrapper — DR capability is baked
on every install; ACTIVATION stays a hub flag, off = zero effect on this host)
+ felhom-mgmt-watchdog service+timer+script + /etc/tmpfiles.d/felhom-privsep.conf
+ guest-hook snippet under /var/lib/vz/snippets/ (agent-installed at runtime)
+ the 'sudo' and 'age' packages if absent + install state dir /var/lib/felhom-install
wg: an OUTBOUND WireGuard tunnel to the Felhom hub (wg_tunnel.enabled=true — base
infrastructure like the cloudflared tunnel; hands-free peer registration; the
operator can disable it per-customer hub-side on request)
guest: the provisioned Felhom LXC (vmid 9201, capped 4 cores / 8192 MiB) + its volumes
+ the golden vzdump imported onto storage 'local'
update: operator-signed self-update authority: felhom-op-1
NOT touched in byo mode: root@pam (no break-glass), host DNS (:53), OOB sshd.
Type this host's short hostname (felhom-host) to acknowledge the byo install: [OK] byo install acknowledged for host felhom-host
[OK] pre-flight passed
[STEP] 2/8 Proxmox API token
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[OK] token minted (secret captured, not logged)
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
[OK] scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm felhom-pbs])
[SKIP] old broad role FelhomAgent already absent
[STEP] 3/8 compute volume grows
[INFO] auto-computed from ~42 GiB free (ONE volume since R-165)
[INFO] grows: rootfs +0G (->32G), data +46G (->70G, ONE volume)
[STEP] 4/8 host enrollment (POST /host-enroll)
[OK] host REUSED (idempotent — existing credential)
[INFO] host_id: demo-hp-bb76ea (api_key captured, not logged)
[SKIP] break-glass (step 4b) SKIPPED (byo: the host owner manages break-glass; root@pam untouched, nothing vaulted)
[STEP] 5/8 agent install (fetch + verify + install)
[SKIP] apt repo alignment skipped (byo — the owner manages repos)
[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
[INFO] manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.210.0
[INFO] fetching agent binary v0.128.0 from Gitea …
[OK] verified sha256 c6eba73bf9b9ad69… matches the hub manifest
[OK] installed /usr/local/bin/felhom-agent (felhom-agent 0.128.0)
[OK] created service user felhom-agent
[OK] added felhom-agent to systemd-journal (unprivileged journal read for NAS verify)
curl: (22) The requested URL returned error: 404
[ERROR] raw fetch failed: configs/felhom-mkfs-guarded.sh (agent tag v0.128.0 — is that version tagged in felhom-agent?)
Connection to 100.76.96.79 closed.
@@ -0,0 +1,26 @@
{
"count": 4,
"files": {
"62696e6172792d336d622e62696e": {
"name_utf8": "binary-3mb.bin",
"sha256": "54b773c46bbfd99465b68247bcb4d3cf5bf8f57a4024c6ba169dc0c326979dd2",
"size": 3145728
},
"6e65737465642fc591737a6962617261636b2e6d64": {
"name_utf8": "nested/őszibarack.md",
"sha256": "15d2024dfc22416261c56ed51741f8d7ae734350b3ac79ef1cb982d667ddef64",
"size": 25
},
"706c61696e2e747874": {
"name_utf8": "plain.txt",
"sha256": "924497918e55fe6dd75ff5ed8a9089aa64752e05540bdf804d71a03afbc48acd",
"size": 21
},
"c3a17276c3ad7a74c5b172c5912d74c3bc6bc3b67266c3ba72c3b367c3a9702e747874": {
"name_utf8": "árvíztűrő-tükörfúrógép.txt",
"sha256": "52a5c5ebfcac247f1753e86f69db46815adf4edaf28375396324e4ea6163611c",
"size": 59
}
},
"root": "/var/lib/felhom/sys_drive/felhom-data/backups/offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09"
}
+17 -1
View File
@@ -496,7 +496,23 @@ applied.** The one that matters: Scenario A **fails against today's tree** with
| **R-267** | **The Configuration page is 2.6× faster and is still ~10 s, and the remaining cost is ONE Gitea call whose latency swings 20× with load.** Reported by the operator as *"almost minutes to load"*, with the reasonable guess that it hashes artifacts on page load. **THAT GUESS DOES NOT HOLD and the code already said so** — Gitea stores each file's sha256 and `gitea.FileSHA256` reads it as metadata (*"the artifact bytes are never downloaded"*). The cost was latency × count. **Fixed in hub v0.100.00.100.2, three legs, each found by refusing to accept a number that did not match the arithmetic:** (1) the per-version sha lookups were **serial** — 2 packages × (1 search + 20 lookups) = 42 sequential calls; now concurrent, bounded 8. (2) The two dropdowns resolved **one after the other**; now side by side. (3) The client used `http.DefaultTransport`, whose **`MaxIdleConnsPerHost` is 2**, so under a 16-way fan-out nearly every call paid a fresh TCP setup *and* a fresh authentication — and authentication is the expensive half (`/api/v1/version` unauthenticated **0.03 s** vs an authenticated package call **0.24 s**). **Measured: 26.2 s → mean 9.85 s over 8 samples (min 5.13, max 18.13).** **THE REMAINING COST IS THE PACKAGE SEARCH** `/api/v1/packages/admin?type=generic&q=…&limit=100`, one per dropdown: **0.203.8 s each depending on load**, and running the two concurrently does not help (3.68 s for both together vs 3.82 s for one alone — Gitea appears to serialise them). By contrast **16 concurrent file-metadata calls take 0.58 s**. **⚠ EVERY NUMBER HERE IS CONTAMINATED and that is stated rather than hidden:** they were taken on DooPlex at load average 711 while this same session was building images, running two Go suites and baking a golden. The same search measured **3.8 s** in-cluster and **0.44 s** from the host ninety seconds later. **Re-measure on an idle box before deciding anything.** **The operator's two proposals, answered on the measurement:** *"reduce the number of artifacts"* — only **50 generic versions exist in total** (33 agent + 17 golden) and `limit=25` costs 0.20 s against `limit=100`'s 0.44 s, so pruning helps **somewhat and sub-linearly**; it is worth doing for its own sake (it pairs with R-210) but it is not the lever. *"hash on creation, store in the DB"***NOT RECOMMENDED, and the reason is a rule this project already holds**: Gitea IS the store, and a copy in `hub_settings` would be a second source of truth that can drift from the registry it describes, while the operator reads exactly that value to confirm what they are about to vouch. `golden_currency_gate.py` records the same reasoning for the vouched version. **The lever that would actually work, and the trade-off that makes it a DECISION rather than an implementation:** cache the version list + shas **in memory** with a short TTL and refresh in the background — the page becomes instant and bounded-stale, at the cost that a just-published artifact does not appear for up to the TTL. That is an operator call about how fresh the dropdown must be, and it is why this row is open rather than closed. **Cheaper interim, also a decision:** drop the dropdown cap from 20 to ~8, which cuts the fan-out but hides older versions from a rollback | **CLOSED 2026-08-08 — hub v0.101.0 + a registry prune.** **Final: cold 5.4 s, warm 0.14 s** (was 26.2 s). Three serialisation legs took it to 9.85 s mean, the 60 s in-memory memo took the warm path to a quarter-second, and the prune halved what remains of the cold path. **⚠ TWO CORRECTIONS TO THIS ROW'S OWN EARLIER TEXT, because both were wrong and both mattered.** **(1) "Only 50 generic versions exist" WAS NOT A COUNT, IT WAS A PAGE LIMIT.** `?type=generic&limit=1000` returns at most 50; the 50 I measured was exactly the cap, and three older agent versions (0.81.0, 0.80.0, 0.79.0) only became visible after the first 30 deletions moved them onto page one. **An unpaginated listing is not evidence of a total** — this repo's own "an empty listing is not evidence of emptiness" rule, walked into while measuring. **(2) THE OPERATOR'S "REDUCE THE NUMBER OF ARTIFACTS" WAS THE BETTER CALL AND MY MEASUREMENT SAID OTHERWISE.** I reported it helps "sub-linearly" and "is not the lever". Measured after: trimming to 10+10 took the COLD load from 13.4 s to 5.4 s — a 2.5× improvement on the path the memo cannot help, because the fan-out is per-version. Recorded rather than quietly dropped (the R-96 standing rule). **Pruned to the newest 10 per package on the operator's rule**, with the live-vouched golden/agent/floor asserted into the KEEP set before a single DELETE was issued; 33 deletions, all HTTP 204, and golden 0.210.0 / agent 0.128.0 / agent 0.127.0 verified still fetchable afterwards. `drill-r50` runs agent 0.113.0, now deleted — flagged to the operator first; it is a disposable nested drill VM and only its re-download path is gone |
| **R-268** | **A live per-guest local-API token was printed into a session transcript.** Done by me on 2026-08-08 while setting up R-221's live drill: a `python3` one-liner meant to list `bootstrap.json`'s keys printed the `local_api` object whole, including its `token`, for guest **9201 on demo-felhom**. **Reported immediately rather than quietly rotated**, because this project's rule is that a secret reaching a transcript is a finding whatever its blast radius. **THE EXPOSURE, ASSESSED RATHER THAN ASSUMED — it is small, and saying so is not the same as excusing it.** The token authorises only the agent's **per-guest** local API on `169.254.253.1:8443`, which listens on the island bridge `vmbr9` — an L2 segment between the PVE host and that one guest. It is not routable from the LAN or the internet, it is self-scoped to guest 9201 (the agent authorises against the token's own guest, never a caller-supplied id), and using it already requires code execution on that host or guest, at which point an attacker has strictly more than the token. The box is **Tier 0, disposable, no customer data**. **ROTATION IS AVAILABLE BUT IS NOT A ONE-LINER, which is why it was not done unannounced:** `localapi.TokenStore.Mint(vmid)` is last-write-wins per VMID (`tokenstore.go:127`, the store keeps only hashes), but the new plaintext must also be written into the guest's `/etc/felhom-bootstrap/bootstrap.json` or the in-guest controller loses its agent access — so it is an operator-timed act, not a background one. **The honest general fix is upstream of the incident:** reading a secret-bearing JSON should go through a helper that prints keys and never values, the same discipline the bake uses for the Gitea token (file → file, `grep -c` on the shape, never the value). Two of this fortnight's findings (R-249, R-132) are the same class from the other direction | **READY** — owner Viktor |
| **R-268** | **A live per-guest local-API token was printed into a session transcript.** Done by me on 2026-08-08 while setting up R-221's live drill: a `python3` one-liner meant to list `bootstrap.json`'s keys printed the `local_api` object whole, including its `token`, for guest **9201 on demo-felhom**. **Reported immediately rather than quietly rotated**, because this project's rule is that a secret reaching a transcript is a finding whatever its blast radius. **THE EXPOSURE, ASSESSED RATHER THAN ASSUMED — it is small, and saying so is not the same as excusing it.** The token authorises only the agent's **per-guest** local API on `169.254.253.1:8443`, which listens on the island bridge `vmbr9` — an L2 segment between the PVE host and that one guest. It is not routable from the LAN or the internet, it is self-scoped to guest 9201 (the agent authorises against the token's own guest, never a caller-supplied id), and using it already requires code execution on that host or guest, at which point an attacker has strictly more than the token. The box is **Tier 0, disposable, no customer data**. **ROTATION IS AVAILABLE BUT IS NOT A ONE-LINER, which is why it was not done unannounced:** `localapi.TokenStore.Mint(vmid)` is last-write-wins per VMID (`tokenstore.go:127`, the store keeps only hashes), but the new plaintext must also be written into the guest's `/etc/felhom-bootstrap/bootstrap.json` or the in-guest controller loses its agent access — so it is an operator-timed act, not a background one. **The honest general fix is upstream of the incident:** reading a secret-bearing JSON should go through a helper that prints keys and never values, the same discipline the bake uses for the Gitea token (file → file, `grep -c` on the shape, never the value). Two of this fortnight's findings (R-249, R-132) are the same class from the other direction | **CLOSED — ROTATED + PROVEN LIVE 2026-08-09** (rehearsal pre-phase, `audits/REHEARSAL-byo-reinstall-2026-08-09.md` §3) | — | Old token **HTTP 401**, new token **HTTP 200**, controller channel back up with a positive observable (`local-api: channel up … 3 mount(s) visible`). No value reached stdout; both shredded. **The recipe in this row was INCOMPLETE and rotating it proved so** — writing `bootstrap.json` is not enough (→ **R-270**), and the store's own revocation claim is false across processes (→ **R-269**) | CC |
| **R-269** | **A rotated-out per-guest local-API token still authorises, and the test that appears to pin the opposite passes only because of its lookup ORDER.** `localapi.TokenStore.Mint` documents *"last-write wins — any previous token for this guest is revoked"*. Across processes that is FALSE until something unrelated forces a reload: the long-lived agent serves `Lookup` from an in-memory index and re-reads the store **only on a miss** (the B3 reload-on-miss optimisation, `tokenstore.go`), so a superseded token is a direct map **hit** and returns `(vmid, true)`. **Red-proved twice.** (a) A unit probe — `TestTokenStore_ReloadOnMiss_RemintCoherence` with the two lookups swapped, i.e. present the rotated-out token FIRST — **fails**; the shipped test passes only because it looks up the NEW token first, and that miss is what evicts the old hash. (b) On hardware, 2026-08-09: after the on-disk rotation the old token returned **HTTP 200**, then 401 only once a new-token lookup had forced the reload, and reliably 401 after `systemctl restart felhom-agent`. **This is the `CLAUDE.md` invariant-comment case exactly** — the comment reads as settled and the test that looks like its pin is order-dependent. **Fix options:** pin the reversed order with a test, or make eviction not depend on an unrelated miss. Until then, **an operator rotating a leaked token MUST restart the agent** — the runbook step is not optional | **READY (S) — NEW 2026-08-09** | — | Found by doing R-268's rotation rather than reading about it | CC |
| **R-270** | **R-268's stated rotation recipe is incomplete: the controller never re-reads `bootstrap.json`'s `local_api`, so a rotation leaves the agent channel dead across restarts.** `bootstrap.ensureLocalAPI` returns early when `cfg.LocalAPI.Endpoint != ""` — by design it FILLS an absent block and never refreshes a present one — so the token the controller uses lives in its own `controller.yaml`, not in the mount. Proved live 2026-08-09: two controller restarts after a correct `bootstrap.json` rotation, still `HTTP 401`; the channel came up only once `local_api.token` was written into `controller.yaml`. The neighbouring `DetectEndpointDrift` compares the ENDPOINT and deliberately does not compare the token (*"a token mismatch is a different failure"*), so this shape is knowingly unmodelled. Parent question — which file is authoritative — is **R-78** | **READY (S) — NEW 2026-08-09** | — | Either teach the drift detector the token, or make the rotation path write both files. Correct the R-268 row's recipe either way | CC |
| **R-271** | **The `agent_channel_unauthorized` alarm can never be closed, because its own prescribed remedy is what silences the recovery.** `channelhealth.Checker.Check`'s UP branch notifies only when `prev != "" && prev != "up"`; a controller restart resets `state` to `""`, so an unseeded→up transition is silent by construction. The alert text says *"token stale/rotated (**re-bootstrap**)"* — i.e. restart the controller — so **following the instruction guarantees no recovery event.** Observed live 2026-08-09: two `agent_channel_unauthorized` errors on the hub (one `sent`, one `suppressed` by the 1 h operator cooldown) and **nothing afterwards**, though the channel came up 3 minutes later and stayed up. The down side is deliberately asymmetric (F2: a born-down channel alerts on cycle 1); the up side never got the matching treatment. Customer dashboard is fine — `SetDashboard` reflects current state every cycle. It is the OPERATOR's trail that ends on "down" | **READY (S) — NEW 2026-08-09** | — | Notify on unseeded→up when the previous *persisted* state was down, or seed from the hub's last event | CC |
| **R-272** | **RANK 1 — Felhom's `--uninstall` leaves the exact condition that makes Felhom's own reinstall REFUSE.** Chain, fully evidenced on demo-hp 2026-08-09: Felhom installs `dnsmasq` at day-0 (`/var/lib/dpkg/info/dnsmasq.list` dated **2026-07-21 18:24 CEST**, demo-hp's day-0) and constrains it with a snippet in `/etc/dnsmasq.d/`; `--uninstall` removes the snippet and **restarts the daemon** (running process start time **2026-08-09 10:37:39 CEST — inside the 10:37:2310:38:23 uninstall window**) but leaves the package installed and the unit **enabled**; unconstrained, dnsmasq binds `0.0.0.0:53`; the next install's preflight then hard-refuses with *"a resolver is already bound to :53"*. It is **not** PVE SDN's (`/etc/pve/sdn/` empty; stock unit). The teardown mentions it only as *"the 'sudo' and 'dnsmasq' packages were left installed (**system packages**)"* — **dnsmasq is not a system package here, Felhom installed it.** **What a customer does next:** reads a message blaming a resolver, concludes their own LAN DNS is at fault, and debugs something they never configured. **Counterfactual confirmed:** `systemctl stop dnsmasq && systemctl disable dnsmasq``host DNS (:53): free` → PRE-FLIGHT PASS, nothing else changed. **The refusal MESSAGE is good** (finding, evidence, two routes, and an explicit promise not to touch DNS on a host it does not own) — the defect is that Felhom caused the condition and does not say so | **READY (M) — NEW 2026-08-09** | — | Either stop+disable dnsmasq on uninstall when Felhom installed it, or have the preflight recognise its own leftover and say so | CC |
| **R-273** | **RANK 1 — the hub vouched an agent version that was never git-tagged, and every install fleet-wide now fails at step 5/8.** `v0.128.0` exists as a Gitea **package** (so it is selectable in the artifact manifest and its sha256 verifies) but **not as a git tag** — newest tag is `v0.127.0`. Since R-183, `felhom-host-install.sh` correctly pins config fetches to `raw/tag/v$ART_AGENT_VER`, so the install dies: `curl: (22) … 404``raw fetch failed: configs/felhom-mkfs-guarded.sh (agent tag v0.128.0 — is that version tagged in felhom-agent?)`. Measured: `raw/branch/main/…`**200**, `raw/tag/v0.127.0/…`**200**, `raw/tag/v0.128.0/…`**404**. The commit that IS 0.128.0 is on `main` (`28ba8593b8`); only the tag is missing. **THIS IS R-184 ARRIVING** — that row says nothing prevents the hub vouching a never-released version because the R-115 gate works from tags. **A good fix met an open gap:** `hostinstall_gates.py` asserts the pin EXISTS (`ok: fetch_raw pins the agent configs to the vouched agent version`); nothing asserts the pinned ref RESOLVES. And `felhom-agent/scripts/release-agent.sh:23` already documents the hazard verbatim — *"a released version without a git tag 404s a box mid-install, as root"* — so it was understood and the step was still missed. **Blast radius, precise:** every fresh install and every reinstall is dead while the manifest vouches 0.128.0 (approved 2026-08-09 — the approval armed it); **existing boxes are unaffected** (self-update takes the binary from the package registry, and demo-felhom runs 0.128.0 healthily). **Unblock:** `git tag v0.128.0 28ba8593b8 && git push origin v0.128.0` — publishes a tag for a commit already on `main` and already released as a package. Rolling the manifest back to 0.127.0 instead would cost R-221. **Deliberately NOT done in the rehearsal session** (diagnose-not-fix; publishing a tag is outward-facing). **Owed gates:** (a) refuse a vouch whose tag does not resolve, (b) a release-completeness check that the package and the tag ship together | **CLOSED 2026-08-09 — tag pushed, install PROVEN** | — | On operator instruction ("proceed"), `v0.128.0` was annotated at `28ba8593b8` and pushed, after an INDEPENDENT download confirmed the published package's sha256 equals the hub's vouched `c6eba73b…`. `raw/tag/v0.128.0/configs/{felhom-agent.service,felhom-mkfs-guarded.sh}` then both served **200**, and `--resume` carried the install to `Day-0 provision SUCCESS` in **3m49s**. **THE TWO GATES ARE STILL OWED AND ARE THE REAL FIX** — this close records that one box was unblocked, not that the class is. | CC |
| **R-274** | **A local golden is adopted with NO version and NO checksum check, so a reinstall can silently come up releases behind.** `felhom-host-install.sh` step 7: `if [[ -n "$GOLDEN_VOLID" ]] && ! $FORCE_GITEA_GOLDEN; then log_skip "using local golden"; return 0; fi`**the hub manifest's `golden.sha256`, whose entire purpose is to vouch from a different trust root than Gitea, is consulted only on the FETCH path.** A locally-present archive bypasses the vouch: no version compare, no digest, no warning. On demo-hp 2026-08-09 the preflight selected `local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst`, whose baked marker reads `felhom-controller:`**`0.192.0`**, against a vouched golden of **0.210.0** — 18 releases stale. **The sharp consequence:** 0.192.0 is **below 0.200.0, where R-193's off-site recovery SCREEN shipped**, so a customer reinstalled today returns on a controller that cannot run the recovery ceremony their data depends on; it is also born below the managed-update floor (0.200.0), and the updater's auto-target is the floor, never the newest. **It compounds with the teardown**, which deliberately keeps the old golden (*"golden vzdump left in place"*). This is the R-111/R-115/R-120 drift family one layer down: the R-120 gate guards what may be VOUCHED, nothing guards what an install TAKES. **OBSERVED 2026-08-09, AND THE RESULT NARROWS THE ROW — recorded because it partly refutes what was written above.** On the RESUME path step 7 **fetched the vouched 0.210.0 correctly** (`fetching golden v0.210.0 from Gitea`), because `--resume` skips preflight and preflight is where local auto-discovery sets `GOLDEN_VOLID` (the GL6-F4 comment says so). **So the fresh-install and resume paths disagree on golden selection, and the resume path is the safe one.** Discovery is `… | sort | tail -1`, i.e. the NEWEST local archive by filename — a sensible heuristic, **and still no comparison against the manifest's version or sha**. The defect therefore stands as: *a box whose newest local golden predates the vouched one installs stale, silently* — which is exactly the state demo-hp was in before this run (newest local 0.192.0 vs vouched 0.210.0). It is now masked on this box because the freshly fetched 0.210.0 is the newest — **correct by recency, not by verification**. There are now **three** goldens on `local` (07-21, 08-03, 08-09), because the teardown keeps them. **Still not observed: a FRESH (non-resume) install taking a stale local golden.** | **READY (S) — NEW 2026-08-09, NARROWED same day** | — | Compare the local golden's version/sha against the manifest and refuse or re-fetch on mismatch; say so in the BYO disclosure, which today lists only what the install CREATES, never what it REUSES | CC |
| **R-275** | **`--uninstall` leaves five 0600 `agent.json.*` credential backups, and the reinstall hands them to the new service account.** `/etc/felhom-agent/` survives with `agent.json.{campaign8-before,campaign9-before,campaign9-prev,pre-e-target-move,pre-prunegate.bak}`, each carrying a 64-char `hub.api_key` and a 59-char `proxmox.token`. The teardown claims to remove *"config (+ its .bak backups)"* and `scripts/CHANGELOG` F1 records *"uninstall now purges the agent config's `.bak*` siblings (one held a live hub api_key)"* — **that fix does not match the filenames in use, and it misses `agent.json.pre-prunegate.bak`, a file that literally ends in `.bak`.** **Exposure assessed, not assumed:** these are SUPERSEDED — the orphaned key hashes to `a5d2222a…`, the hub's current demo-hp key to `8c59d1b6…`, and the Proxmox token was deleted by the same uninstall. **But the reinstall recreates `felhom-agent` at uid 999, the same uid the deleted account had**, so three of the backups become the new account's files — verified readable as `felhom-agent`. A fresh install's service account inherits read access to the prior install's credentials; superseded today, live if the backups were recent (R-179's precedent). **Also left, undeclared:** `/etc/felhom/{.bootstrap-done,appliance-pairing-code}`, `felhom-bootstrap.service` + `/usr/local/sbin/felhom-bootstrap.sh`, the `vmbr9` stanza in `/etc/network/interfaces`, and `/etc/sudoers.d/felhom-agent.bak-pre-e2a` (21 KB — **INERT: sudo skips dotted filenames, verified with `sudo -l -U felhom-agent`; `visudo -c -f` parsing it OK is NOT evidence sudo loads it**) | **READY (S) — NEW 2026-08-09** | — | Purge by directory, not by glob; and do not let a new service account reuse a uid that owns old secrets | CC |
| **R-276** | **RANK 2 — an uninstalled box keeps a live WireGuard tunnel into Felhom's off-site endpoint, and the teardown says nothing.** After `--uninstall` on demo-hp, `wg-quick@wg-felhom` is **enabled and active**, `/etc/wireguard/wg-felhom.conf` present, handshake to `167.233.158.164:443` **52 s old**, counters 5.86 GiB in / 2.48 GiB sent. It appears in **neither** the WIPED nor the KEPT list, though the BYO install disclosure names it prominently on the way in (*"an OUTBOUND WireGuard tunnel to the Felhom hub"*). A host told to leave Felhom retains a live network path into Felhom infrastructure, its hub-side peer registration intact, and nobody is told | **READY (S) — NEW 2026-08-09** | — | Tear the tunnel down and deregister the peer, or list it under KEPT with the reason and the removal command | CC |
| **R-277** | **Three hub surfaces jointly present a HEALTHY off-site tier as an absent one — and it produced a wrong operator statement during this run.** For demo-hp on 2026-08-09 the box was pushing off-site daily without a gap (18 restic snapshots, `last_status: ok`), yet: (a) the customer page's Backup panel read `Snapshots 0 / Repo Size 0 MB / Integrity Unknown` — it renders the **local disk tier**, while the healthy `offsite` object sits **in the same report** unrendered on that panel; (b) the Offsite page read `0.0 GB` — true, but a 162 KB repo rounds to nothing; (c) a stale `offsite_delivery_stuck` event from **2026-08-07 10:19** (not recurring) reads as current state. **Three independent surfaces agreeing on a wrong picture is how a working backup gets "fixed".** It did exactly that here: the rehearsal reported a fleet-wide off-site outage to the operator and had to retract it. **Note the true half:** demo-felhom IS genuinely stuck (`offsite.state=needs_credential`, no run has ever succeeded) → **R-278** | **READY (S) — NEW 2026-08-09** | — | Render the offsite object on the offsite row; show bytes not rounded GB; distinguish a live alarm from event history | CC |
| **R-278** | **demo-felhom's off-site tier has never completed a run and has been stuck for six days.** `offsite.state=needs_credential` since the 2026-08-03 guest rebuild; the hub's own alarm reads *"enabled + escrowed but no run has EVER succeeded"*; the controller's `offsite-credential-retry` job runs every 5 minutes and completes in 0 s, doing nothing. R-193's fix (the recovery SCREEN, controller 0.200.0) is present on the box, so the remedy exists — it just needs the customer-present ceremony that nobody has run, which is R-243's shape (*"a machine waiting for its recovery code can stop backing up off-site without alarming us"*) landing on a real box. **Contrast that makes it a defect and not a chore:** demo-hp, same rebuild, same day, recovered and has 18 snapshots | **READY (S) — NEW 2026-08-09** | — | Either the self-heal reconciler owns this shape end-to-end, or the box must say plainly on the dashboard that it is unprotected pending the recovery code | CC |
| **R-279** | **There is no operator-triggerable off-site backup.** The only route to `POST /backup/offbox/run` is the customer's own dashboard session; `signed_jobs` carries opaque operator-SIGNED blobs and the hub holds no signing key. This cost the rehearsal a stop: preparing the run needed one off-site push and there was no operator path to it. Sibling of **R-177** (no operator-triggerable fill check) | **READY (XS) — NEW 2026-08-09** | — | Same shape as R-177; solve both together | CC |
| **R-280** | **RANK 1 — after a reinstall the data drive cannot be re-attached through ANY dashboard route, and the restore page promises it is "two clicks".** Measured on the rebuilt demo-hp, 2026-08-09. The restore page diagnoses the situation perfectly and then sends the customer to an empty page: *„Előbb csatold vissza az adatmeghajtót. A mentéseid megvannak, és a meghajtók is megvannak — újratelepítés után viszont a gép még nem ismeri őket, ezért most nincs hová visszaállítani. **Ez két kattintás:** Tárhely → Meghajtók, »Meglévő meghajtó csatolása«."* **It is not two clicks; it is zero possible clicks.** `GET /api/disks/candidates``{"initialize":[],"attach":[]}`, so both wizards render an empty selector, and `Tárhely → Meghajtók` reads „Nincs regisztrált adattároló" with an empty unregistered list. **The agent is not at fault**`GET /api/disks` returns the NVMe in full (1.0 TB, SMART PASSED, `mount_path:/mnt/nvme-1tb`, `guest_attached:false`), so the channel and enumeration work. **ROOT CAUSE:** `handleDiskCandidates` builds both lists from `ListCandidateDisks`, the UNCLAIMED-disk scan; demo-hp's NVMe is deliberately BOTH the user-data drive and the `felhom-backup` target (`operations/nodes.md`), so it is claimed and never offered. That filter is **correct for `initialize`** (never offer to format a disk in use — `/storage/init` even says so: *„Rendszer- és biztonsági-mentés meghajtók itt nem jelennek meg — azok védettek"*) and **over-broad for `attach`**, which is non-destructive by definition and whose own page says *„A meghajtón lévő adatok nem törlődnek — a csatolás csak elérhetővé teszi azokat."* **It cascades:** no store → Calibre-Web's install page degrades to *„Nincs regisztrált adattároló — adja meg kézzel az útvonalat"* and demands a hand-typed `E-könyvtár útvonal`; no app → the restore rows read „Nincs telepítve". **THE ESCAPE HATCH WORKS AND NO CUSTOMER COULD FIND IT:** `POST /settings/storage/add` with `storage_path=/mnt/sys_drive` succeeded first try (*„Adattároló sikeresen hozzáadva"*) — and `/mnt/sys_drive` is an internal path, the very one registered before the wipe. Once registered, everything unblocked and the deploy form became a proper picker (*„Tárhely (sys_drive) — 64.2 GB szabad"*). **This is R-220's successor:** R-220 was closed as "drives unenrollable after a rebuild — fixed"; enumeration is fixed, OFFERING is not | **READY (M) — NEW 2026-08-09** | — | Populate `attach` from mounted-but-unregistered filesystems rather than from the unclaimed-DISK scan; and never print "two clicks" without asserting the destination is non-empty | CC |
| **R-281** | **The hub said NOTHING through an entire reinstall — and the tripwire for a sealed-backup unseal did not fire on a real unseal.** Between the uninstall (08:38 UTC) and the verified restore (10:27 UTC) demo-hp's guest was destroyed, the agent and its pveum identity removed, the host re-enrolled, a new guest provisioned, the box re-claimed, the sealed off-site package opened with the customer's recovery code, an app redeployed and 3.8 MB restored. **Events recorded for demo-hp in that window: ZERO. Notifications: ZERO.** **Positive control on the query** (standing rule 3): the hub recorded **2 events all day across all customers**, newest `db_dump_completed` at 00:30:07 — so the store is reachable and the silence is real, not a bad filter. **The good half, stated first:** no FALSE alarm fired during a legitimate reinstall, which is what P7 was watching for. **The owed half:** `escrow_blob_served` exists precisely as the tripwire for this moment — its text is *"the blob cannot be opened without the customer's recovery code… If no recovery is in progress on that box, investigate"* — and it **has fired for demo-hp before** (twice, last 2026-08-04 20:12:54). Today's unseal, through the R-193 recovery screen, fired it **not at all**. Either the screen's unlock path does not emit it or the rebuilt-box path bypasses it; **which of those is not established here.** A reinstall and a theft of a machine look identical to the operator | **READY (M) — NEW 2026-08-09** | — | Emit on the recovery-screen unlock path; and decide which reinstall milestones are worth one line each | CC |
| **R-282** | **One secret, three different Hungarian names, and the email sends the customer to a page their box is not showing.** Sending it from the hub is „**Visszaállító** kód küldése"; the email that arrives is subject „Jelszó-**visszaállítási** kód", body „**Visszaállító** kód: …", and it instructs *„Add meg a vezérlőpult »**Elfelejtett jelszó**« oldalán"*; the page the box actually serves is „A szerver **beállítása**" asking for a „**Beállító** kód". **A rebuilt box shows a SETUP page and the hub can only send a RESET mail** (because hub-side the customer is still `claimed_at 2026-07-21`), so the instruction names a route that does not exist on screen. **It does work if you ignore the instructions** — the reset code was accepted on the setup page (302 + session), so this is naming, not function. **It cost this session real time and one wasted code:** the operator supplied a 3-word Hungarian code believing it was the recovery code, because the hub calls the claim code „Visszaállító kód" and the ESCROW code is also „Visszaállító kód" — the only reliable discriminator is length (claim = 3 Hungarian words; recovery = **10** EFF-list words, and the recovery screen does say „(tíz szó)") | **READY (S) — NEW 2026-08-09** | — | Pick one name per secret and use it on all three surfaces; make the mail's page reference match what a rebuilt box actually shows | CC |
| **R-283** | **After a rebuild the hub says "Claimed 18d ago" while the box serves its first-run setup page.** `customer_claims` for demo-hp still read `claimed_at 2026-07-21 16:29:25`, `generation 2`, `issued_at 2026-08-03` while the freshly provisioned guest — whose `settings.json` is new — correctly showed „A szerver beállítása". The two sides never reconcile: the hub's claim state survives a guest rebuild and the box's does not. Consequences: the operator's screen says the box is claimed when it is not, a resend produces a RESET code instead of a SETUP code (→ **R-282**), and any previously issued code fails with *„Hibás vagy lejárt kód"* — a message that is technically true and tells the customer nothing about the real cause, namely their own reinstall. Mirror image of **R-214/R-235** (an already-paired box still told to pair itself) | **READY (S) — NEW 2026-08-09** | — | Let a report from a box carrying no claim state clear the hub's, or show both sides on the operator page | CC |
| **R-284** | **„A kiválasztott tárhely majdnem megtelt." on a store that is 93 % FREE — an apparent inverted threshold.** Calibre-Web's deploy page rendered `<option value="/mnt/sys_drive" data-free-percent="93">` alongside „Tárhely (sys_drive) — **64.2 GB szabad**" and the warning „A kiválasztott tárhely majdnem megtelt." 93 % free read as 93 % used is the obvious candidate, and `checkStorageSpace(this)` is the function to look at. **Not confirmed by reading the code** — reported as measured output only. A capacity warning that cries wolf on an empty disk is one a customer learns to click past | **READY (XS) — NEW 2026-08-09** | — | Check `checkStorageSpace`'s comparison against `data-free-percent`; add a render test per branch | CC |
**Explicitly still open, untouched by this session:** R-246, R-255, R-256, R-257, R-261, R-262,
R-263, **R-264** (the twenty-one undecided facts — a design session of its own), R-240, R-243,