admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rehearsal 2026-08-09: pre-phase + Gate 0 recorded before the destructive walk
gates / gates (push) Successful in 29s
Details

Browse Source 
Venue demo-hp, operator-approved at STOP 1. Records the state that P1 destroys,
plus seven pre-walk findings, while they can still be checked against a live box.

R-268 CLOSED — the leaked per-guest local-API token is rotated and the rotation
is PROVEN in both directions (old refused, new accepted, channel up with a
positive observable). Rotating it surfaced three defects:

- an out-of-process rotation does NOT revoke the old token. The daemon serves
  Lookup from a stale index and re-reads only on a MISS, so a superseded token
  is a direct hit. Red-proved in a unit probe AND live on hardware; the shipped
  RemintCoherence test passes only because it looks up the NEW token first.
- R-268's own recipe is incomplete: ensureLocalAPI returns early on a present
  local_api block, so writing bootstrap.json is not enough — the controller
  serves the old token from controller.yaml across restarts.
- the agent-channel alarm never closes: the UP branch does not notify from an
  unseeded state, and the alarm's own remedy ("re-bootstrap") resets it.

Gate 0 complete: dataset planted in the Calibre library (coverage verified, not
assumed) with two Hungarian accented filenames; the comparator watched FAILING
three ways including an NFC->NFD rename that renders identically; off-site run
driven through the product's own button; restore point recorded by identity as
snapshot 41c830db, confirmed to carry all four files.

Also corrects the record: demo-hp's off-site tier is HEALTHY. Three hub surfaces
agreed it was absent and all three mislead — the panel showing 0 snapshots renders
the LOCAL tier, 162 KB rounds to 0.0 GB, and a two-day-old stuck event reads as
current. And the managed-update floor is live at 0.200.0, not 0.156.0.
This commit is contained in:
authentik Default Admin
2026-08-09 10:34:57 +02:00
parent 56f8aa611c
commit 34646295dc
3 changed files with 564 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md
+256
View File
@@ -0,0 +1,256 @@
# REHEARSAL — the BYO reinstall walk (2026-08-09)
> **Status: IN PROGRESS.** Pre-phase and Gate 0 are complete; STOP 1 passed. The walk (P1–P7) has not
> started. This file is written before the destructive phase deliberately — a finding that exists only
> in a session that later crashes is a finding nobody has.
**Venue: `demo-hp` (HP t740, `felhom-host`, guest 9201, customer `demo-hp`).** Operator-approved at
STOP 1. **Driven from DooPlex.** All times UTC unless marked; the host runs CEST (UTC+2).
---
## 1. Baselines — re-confirmed live on arrival, not taken from the spec
| | spec said | live reading | source |
|---|---|---|---|
| `felhom-agent` | v0.128.0 @ `28ba8593b8` | **0.128.0** on demo-felhom, **0.127.0** on demo-hp; HEAD == `origin/main` == `28ba8593b8` | `felhom-agent --version` on both nodes; `git rev-parse` |
| `felhom-controller` | v0.210.0 @ `c732fe1283` | **0.210.0** demo-felhom, **0.208.0** demo-hp; HEAD == `origin/main` == `c732fe1283` | hub `/configs`; `git rev-parse` |
| hub | v0.101.0 @ `56f8aa611c` | **0.101.0** (deployed image tag matches) | `kubectl get deploy hub`; page footer |
| Day-0 manifest | golden 0.210.0 / agent 0.128.0 / min 0.127.0 | **all three already saved** — golden `0.210.0` (`b9f701fa…`), agent `0.128.0` (`c6eba73b…`), min agent `0.127.0`, wrapper `104db0a4…` | hub `/configuration`, selected `<option>` values |
All three repos were clean and at `origin/main` before anything ran.
**The fleet is not uniform, and the venue is the box that is behind** — demo-hp runs agent 0.127.0 and
controller 0.208.0 against a manifest vouching 0.128.0/0.210.0. That is an argument *for* the venue: the
reinstall genuinely exercises the manifest rather than re-installing what is already there.
---
## 2. Venue comparison — the fidelity table that decided it
The runbook's criterion is *"the box whose current state most resembles what the tester will have"*.
| criterion | `demo-felhom` (N100) | `demo-hp` (t740) |
|---|---|---|
| customer apps deployed | 1 — opengist | **3 — privatebin, opengist, calibre-web** |
| enrolled user-data drive | **none registered.** `/mnt/felhom-drives` empty; controller logs `[WARN] Storage paths: no storage paths registered` | `/mnt/felhom-drives/Felhom-Share` + NVMe enrolled 2026-07-22; one registered storage path `/mnt/sys_drive` |
| off-site tier has actually run | **NO** — `offsite.state=needs_credential`, no run has ever succeeded | **YES** — 18 restic snapshots, daily, unbroken |
| escrow / recovery ceremony | present, generation 2 | present, generation 2 |
| install shape | mkimage loader, **Secure Boot OFF** (firmware workaround) | shim loader, **Secure Boot ENABLED** — the customer shape |
| collateral on the box | none | `drill-r50` (VM 300); leftover `c11-scratch` storage |
**Chosen: demo-hp.** It wins every fidelity criterion that distinguishes them.
**`drill-r50` is not at risk, and this was checked rather than assumed:** VM 300 is **not** in the
`felhom` pool (`pvesh get /pools/felhom` → members `[9201]`), its disks are on `local-lvm`, and
`--uninstall` removes no storage definitions and no non-pool guest — its own end-of-teardown statement
says the enrolled drives are *"unmounted only, NEVER wiped"* and that it never deletes PBS backups, hub
records or escrow blobs.
**Correction recorded, because it was part of the comparison put in front of the operator:** the
enrolled-drive row was first reported as *"real files on an enrolled drive"*. `Felhom-Share` is
**empty** — the share exists and is enrolled, but holds no files. demo-hp still wins the row (demo-hp
has a registered storage path; demo-felhom has none), on a narrower margin than first stated.
---
## 3. Pre-phase — R-268, the leaked per-guest local-API token
**Done first and independently, on `demo-felhom` guest 9201, as the runbook required.** No token value
reached stdout at any point: old value captured file→file from `bootstrap.json`, new value minted into
a file, both shredded at the end.
### 3.1 The rotation, and its proof
| step | old token | new token |
|---|---|---|
| before rotation | **HTTP 200** | — |
| bogus token (negative control) | **HTTP 401** | — |
| after the on-disk rotation, before any reload | **HTTP 200** ← *see 3.2* | 200 |
| after that lookup forced a reload | **HTTP 401** | — |
| after `systemctl restart felhom-agent` | **HTTP 401** | **HTTP 200** |
**Positive observable for the controller half:** `local-api: channel up (agent 169.254.253.1:8443) —
guest 9201, 3 mount(s) visible`, enumerating mp8/mp9/mp0 — data only the agent can supply. An absent
error would not have been evidence.
**R-268 is CLOSED.** The old token is refused; the new one works; the controller's channel is up.
### 3.2 FINDING — an out-of-process token rotation does not revoke the old token
`localapi.TokenStore.Mint`'s doc comment states *"last-write wins — any previous token for this guest is
revoked"*. **Across processes that is false until something else forces a reload.** The long-lived agent
serves `Lookup` from an in-memory index and re-reads the store **only on a miss** (the B3
reload-on-miss optimisation). A superseded token is still a direct map **hit**, so it authorises.
**Red-proved twice.** A temporary unit probe (Scenario C with the lookup order reversed — present the
rotated-out token *first*) fails on the claim; the shipped
`TestTokenStore_ReloadOnMiss_RemintCoherence` passes **only because it looks up the NEW token first**,
which is what triggers the reload that then evicts the old hash. Then proved on real hardware: the row
marked ← above is a live 200 from a token that the store on disk had already superseded.
This is the `CLAUDE.md` case exactly — a comment asserting an invariant with a test that appears to pin
it and does not. The honest fix is either a test that pins the reversed order, or an eviction that does
not depend on an unrelated lookup missing first.
### 3.3 FINDING — R-268's own stated rotation recipe is incomplete
The register row says the new plaintext *"must also be written into the guest's
`/etc/felhom-bootstrap/bootstrap.json` or the in-guest controller loses its agent access"*. **Writing
bootstrap.json is not sufficient.** `bootstrap.ensureLocalAPI` returns early when
`cfg.LocalAPI.Endpoint != ""` — it fills an *absent* block and never refreshes a present one — so after
a rotation the controller keeps serving the old token from its own `controller.yaml` **across a
restart**. Proved live: two controller restarts, still HTTP 401. The rotation only completed once
`local_api.token` was written into `controller.yaml` itself.
The neighbouring `DetectEndpointDrift` compares the **endpoint** and deliberately does not compare the
token (*"a token mismatch is a different failure"*). So this shape is known to be unmodelled, and R-78
("which file is authoritative") is its parent.
### 3.4 FINDING — the agent-channel alarm never closes, and its own prescribed remedy is why
`channelhealth` alerted correctly and promptly: `agent_channel_unauthorized`, no debounce, pushed to
the hub — *"agent rejected the controller token (HTTP 401) — token stale/rotated (**re-bootstrap**)"*.
The channel then recovered, and **no recovery event was ever sent**. In `Checker.Check`'s UP branch the
notify is guarded by `prev != "" && prev != "up"`, and a controller restart resets `state` to `""`. The
alert's own instruction is to re-bootstrap — i.e. restart the controller — so **following the alarm's
advice structurally guarantees the recovery is silent.** The down side is deliberately asymmetric
(F2: a born-down channel alerts on cycle 1); the up side is not.
Operator-visible result on the hub: two `agent_channel_unauthorized` errors, one `sent` and one
`suppressed` by cooldown, and nothing afterwards. The customer dashboard is fine — `SetDashboard`
reflects current state every cycle. It is the operator's trail that ends on "down".
---
## 4. Gate 0
### 4.1 The recovery code — CONFIRMED IN THE OPERATOR'S HANDS (STOP 1)
Not printed, not logged, not requested. It will be supplied at the screen at STOP 3.
### 4.2 The dataset — planted and fingerprinted
Planted at `/var/lib/felhom/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/` — inside
the **Calibre library**, which is real app-owned customer data and is **demonstrably** carried off-site
(the path appears in every `felhom-offbox,calibre-web` snapshot). Chosen after checking coverage rather
than assuming it: the per-app tier-1 unit carries only the named docker volume, so a file dropped in an
arbitrary place would not have travelled.
| sha256 (16) | bytes | name | name bytes (hex) |
|---|---|---|---|
| `54b773c46bbfd994…` | 3 145 728 | `binary-3mb.bin` | `62696e6172792d336d622e62696e` |
| `52a5c5ebfcac247f…` | 59 | **`árvíztűrő-tükörfúrógép.txt`** | `c3a17276c3ad7a74c5b172c5912d74c3bc6bc3b67266c3ba72c3b367c3a9702e747874` |
| `15d2024dfc224162…` | 25 | **`nested/őszibarack.md`** | `6e65737465642fc591737a6962617261636b2e6d64` |
| `924497918e55fe6d…` | 21 | `plain.txt` | `706c61696e2e747874` |
Manifest: `documentation/audits/evidence/` → recorded in the session scratchpad as
`GATE0-before-manifest.json`. The manifest keys on **raw name bytes**, not rendered text.
### 4.3 The positive control — the comparator was watched failing, three ways
| control | expected | result |
|---|---|---|
| identical copy | PASS | **PASS**, exit 0 |
| one bit flipped at byte 1 500 000 of the 3 MB binary | FAIL | **FAIL** — named the file and both sha256s |
| **NFC→NFD rename of the accented name** (renders *identically*) | FAIL | **FAIL** — caught as MISSING+EXTRA on the name bytes |
| one file deleted | FAIL | **FAIL** — named the missing file |
The NFD case is the one that matters: a comparison of *rendered* filenames would have passed it. Every
altered copy was discarded after the control ran.
### 4.4 The off-site tier — has run since the dataset was planted
Triggered through the product's own dashboard button (`POST /backup/offbox/run` → flash *„A távoli
mentés elindult"*), not by invoking restic by hand.
```
[offbox] backup run started (3 app(s) toggled)
[offbox] pre-push dump leg completed in 7.329s — snapshot pair is coherent
[offbox] backed up calibre-web (…/backups/primary/calibre-web, 1 mandatory path(s))
[offbox] backup OK: 3 app(s) backed up, 18 snapshot(s), 1m22s
```
**The restore point, by identity — never by size:**
| snapshot | time (UTC) | tag | paths |
|---|---|---|---|
| **`41c830db`** | 2026-08-09 08:30:38 | `felhom-offbox,calibre-web` | `…/backups/primary/calibre-web` + `…/userdata/media/books` |
| `9e38b84c` | 2026-08-09 08:30:49 | `felhom-offbox,opengist` | `…/backups/primary/opengist` |
| `78b93f04` | 2026-08-09 08:30:53 | `felhom-offbox,privatebin` | `…/backups/primary/privatebin` |
`restic ls 41c830db` confirms all four planted files are present, **both accented filenames included**.
### 4.5 The box, captured
`GATE0-demo-hp-before.txt` (282 lines): agent version, `pveversion`, `pct`/`qm` lists, pool membership,
`pvesm status`, `storage.cfg`, `df`, `lsblk` with serials, the agent unit + service user, sudoers, the
pveum users/tokens/roles/ACL, the guest hook, the WireGuard tunnel, the OOB belt, the agent state dir,
the install state file, PBS snapshots, local vzdump archives, and the enrolled-drive tree.
---
## 5. Findings so far, ranked by what they cost the person in front of you
Nothing yet is rank 1 or 2 — the walk has not started.
### Rank 3 — misleads
- **F-1 · A rotated-out local-API token still authorises** (§3.2). *A customer would do nothing — they
cannot see it. An operator rotating a leaked token would believe it revoked when it is not.* The
shipped test passes for the wrong reason.
- **F-2 · R-268's rotation recipe is incomplete** (§3.3). *Whoever follows it gets a box whose agent
channel is dead across restarts, with a correct-sounding alarm and no hint that `controller.yaml` is
the file that matters.*
- **F-3 · The agent-channel alarm never closes** (§3.4). *The operator fixes it as instructed and is
never told it worked; the hub's last word stays "down".*
- **F-4 · Three hub surfaces jointly misrepresent a healthy off-site tier as absent.** The customer page
reads `Snapshots 0 / Repo Size 0 MB / Integrity Unknown` (that panel renders the **local** tier, while
the healthy `offsite` object — `snapshot_count: 18`, `last_status: ok` — sits unrendered in the same
report); the Offsite page shows `0.0 GB` (true, but a 162 KB repo rounds to nothing); and a stale
`offsite_delivery_stuck` event from 2026-08-07 reads as current. *Three independent surfaces agreeing
on a wrong picture is how a working backup gets "fixed".* **This one caught me** — it produced a wrong
statement to the operator at STOP 1, corrected before it changed anything.
- **F-5 · There is no operator-triggerable off-site backup.** The only route is the customer's dashboard
button; `signed_jobs` needs a signing key the hub does not hold. Sibling of R-177.
### Rank 4 — cosmetic / hygiene
- **F-6 · Leave-behinds from earlier sessions**, found while capturing state:
- demo-hp host: storage `c11-scratch` (Campaign 11), and `felhom-backup:…vzdump-lxc-9100-2026_08_03…`
for a guest that no longer exists;
- demo-hp guest 9201 `/root`: `.dpw` (13 bytes), `.h`, `.sec.html`, all 2026-08-07 16:08;
- demo-felhom guest 9201: `/root/r204-backup/offbox.orig`.
- **F-7 · The managed-update floor is `0.200.0`, not `0.156.0`** — the runbook's §8.3 premise is stale.
Confirmed twice: hub `/configuration` reads *"Effective floor v0.200.0 — source: DB (hub_settings);
env fallback would be v0.120.0"*, and both boxes log `settle-gate: GO — at/above floor 0.200.0`. The
ruling's **substance** still needs recording; its **number** must not be written down as 0.156.0.
### Not a finding, checked and cleared
- The two `userdata` trees under `sys_drive` are the **same directory** surfaced twice, not a
backed-up copy beside an unprotected one. Checked against calibre-web's actual bind mounts.
---
## 6. Steps taken off-path, and what they cost
Rule 2 (no shell after the install command returns) does not bind yet — the walk has not begun. These
are Gate 0 preparation acts by the operator, recorded because they are mutations:
1. **R-268 rotation on demo-felhom** — token store appended, `bootstrap.json` and `controller.yaml`
rewritten, agent and controller restarted. Required by the runbook's pre-phase. *Not the venue.*
2. **demo-hp's dashboard password was re-set to the value in the operator's credentials file.** The
customer-owned password was not known to this session, and no operator-side route to the off-site
button exists (F-5). The prior `password_hash` is preserved at `/root/.orighash` and
`/root/settings.json.rehearsal-orig` in guest 9201; `claimed` was left `true`. **Done on operator
instruction.** Cost to fidelity: none for the walk — P1 destroys this guest, and P4 re-claims with a
password the operator chooses.
3. **The off-site run was started by a script pressing the dashboard's own endpoint**, with a real
session and a real CSRF token, rather than by a person clicking. The server path is identical; only
the click is synthetic.
---
## 7. What happens next
P1 — remove Felhom from demo-hp with `--uninstall`, then measure what it leaves behind. Nothing in
§1–§4 is repeated after that point; the "before" is fixed as of this file.
documentation/audits/evidence-rehearsal-2026-08-09/GATE0-before-manifest.json
+26
View File
@@ -0,0 +1,26 @@
{
"count": 4,
"files": {
"62696e6172792d336d622e62696e": {
"name_utf8": "binary-3mb.bin",
"sha256": "54b773c46bbfd99465b68247bcb4d3cf5bf8f57a4024c6ba169dc0c326979dd2",
"size": 3145728
},
"6e65737465642fc591737a6962617261636b2e6d64": {
"name_utf8": "nested/őszibarack.md",
"sha256": "15d2024dfc22416261c56ed51741f8d7ae734350b3ac79ef1cb982d667ddef64",
"size": 25
},
"706c61696e2e747874": {
"name_utf8": "plain.txt",
"sha256": "924497918e55fe6dd75ff5ed8a9089aa64752e05540bdf804d71a03afbc48acd",
"size": 21
},
"c3a17276c3ad7a74c5b172c5912d74c3bc6bc3b67266c3ba72c3b367c3a9702e747874": {
"name_utf8": "árvíztűrő-tükörfúrógép.txt",
"sha256": "52a5c5ebfcac247f1753e86f69db46815adf4edaf28375396324e4ea6163611c",
"size": 59
}
},
"root": "/var/lib/felhom/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09"
}
documentation/audits/evidence-rehearsal-2026-08-09/GATE0-demo-hp-before.txt
+282
View File
@@ -0,0 +1,282 @@
=== captured 2026-08-09T08:17:53Z UTC / 2026-08-09T10:17:53+0200 host-local ===
### felhom-agent
felhom-agent 0.127.0
### pveversion
pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
### pct list
VMID Status Lock Name
9201 running demo-hp
### qm list
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
300 drill-r50 stopped 8192 32.00 0
### pool members
{"comment":"Felhom-managed guests","members":[{"cpu":0.00448653868698896,"disk":990715904,"diskread":4608909312,"diskwrite":127439912960,"id":"lxc/9201","maxcpu":7,"maxdisk":33501757440,"maxmem":28212985856,"mem":641748992,"memhost":0,"name":"demo-hp","netin":2213810807,"netout":376936192,"node":"felhom-host","status":"running","template":0,"type":"lxc","uptime":525808,"vmid":9201}],"poolid":"felhom"}
### pvesm status
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
c11-scratch dir active 983379700 6940296 926412792 0.71%
felhom-backup dir active 983379700 6940296 926412792 0.71%
felhom-pbs pbs active 0 0 0 0.00%
local dir active 40516856 19132716 19293748 47.22%
local-lvm lvmthin active 56545280 20904790 35640489 36.97%
### storage.cfg
dir: local
path /var/lib/vz
content iso,import,vztmpl,backup
lvmthin: local-lvm
thinpool data
vgname pve
content rootdir,images
pbs: felhom-pbs
datastore felhom-offsite
server 10.77.0.1
content backup
encryption-key 3f:4f:65:c0:d8:f3:9f:3c:43:29:8c:c8:61:3c:89:40:e6:7a:c8:ae:b1:44:70:09:0e:f1:f7:1b:60:a0:c4:99
fingerprint c6:07:28:3f:5b:7b:5a:41:90:28:d7:ca:4f:37:14:70:56:39:2e:2f:0b:71:e8:06:ca:60:4a:d5:56:5f:3c:fd
namespace demo-hp
username felhom@pbs!demo-hp
dir: felhom-backup
path /mnt/nvme-1tb
content backup
is_mountpoint 1
dir: c11-scratch
path /mnt/nvme-1tb
content images
### df
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-root 39G 19G 19G 50% /
/dev/nvme0n1 938G 6.7G 884G 1% /mnt/nvme-1tb
### lsblk
NAME SIZE MODEL SERIAL FSTYPE MOUNTPOINT
sda 119.2G SanDisk X600 M.2 2280 SATA 128GB 182195804614
|-sda1 1007K
|-sda2 1G vfat /boot/efi
`-sda3 118.2G LVM2_member
|-pve-swap 8G swap [SWAP]
|-pve-root 39.6G ext4 /
|-pve-data_tmeta 1G
| `-pve-data-tpool 53.9G
| |-pve-data 53.9G
| |-pve-vm--9201--disk--0 32G ext4
| |-pve-vm--9201--disk--1 70G ext4
| |-pve-vm--300--disk--0 4M
| `-pve-vm--300--disk--1 32G
`-pve-data_tdata 53.9G
`-pve-data-tpool 53.9G
|-pve-data 53.9G
|-pve-vm--9201--disk--0 32G ext4
|-pve-vm--9201--disk--1 70G ext4
|-pve-vm--300--disk--0 4M
`-pve-vm--300--disk--1 32G
nvme0n1 953.9G KXG50PNV1T02 NVMe TOSHIBA 1024GB 58BS11AFT8MQ ext4 /mnt/nvme-1tb
### felhom-agent unit + user
active
uid=999(felhom-agent) gid=990(felhom-agent) groups=990(felhom-agent),999(systemd-journal)
### sudoers entries
total 68
drwxr-xr-x 2 root root 4096 Aug 3 08:09 .
drwxr-xr-x 102 root root 4096 Aug 3 08:09 ..
-r--r----- 1 root root 1068 Apr 11 14:21 README
-r--r----- 1 root root 22483 Aug 3 08:08 felhom-agent
-r--r----- 1 root root 21833 Jul 28 11:29 felhom-agent.bak-pre-e2a
-r--r----- 1 root root 1241 Aug 3 08:09 felhom-op
-r--r----- 1 root root 666 May 20 2023 zfs
--
287
### pveum users/tokens/roles/acl
+------------------+---------+-------------------+--------+--------+-----------+--------+------+----------+------------+------------------+--------+-------------+
| userid | comment | email | enable | expire | firstname | groups | keys | lastname | realm-type | tfa-locked-until | tokens | totp-locked |
+==================+=========+===================+========+========+===========+========+======+==========+============+==================+========+=============+
| felhom-agent@pve | | | 1 | 0 | | | | | pve | | | |
+------------------+---------+-------------------+--------+--------+-----------+--------+------+----------+------------+------------------+--------+-------------+
| root@pam | | noreply@felhom.eu | 1 | 0 | | | | | pam | | | |
+------------------+---------+-------------------+--------+--------+-----------+--------+------+----------+------------+------------------+--------+-------------+
--tokens--
+---------+---------+--------+---------+
| tokenid | comment | expire | privsep |
+=========+=========+========+=========+
| agent | | 0 | 1 |
+---------+---------+--------+---------+
--roles--
| FelhomAgentBase | Datastore.Audit,SDN.Use,Sys.Audit | 0 |
| FelhomAgentGuest | Pool.Allocate,Pool.Audit,VM.Allocate,VM.Audit,VM.Backup,VM.Config.CPU,VM.Config.Disk,VM.Config.Memory,VM.Config.Network,VM.Config.Options,VM.PowerMgmt,VM.Snapshot,VM.Snapshot.Rollback | 0 |
| FelhomAgentStore | Datastore.Allocate,Datastore.AllocateSpace | 0 |
--acl--
+------------------------+------------------+-------+------------------------+-----------+
| path | roleid | type | ugid | propagate |
+========================+==================+=======+========================+===========+
| / | FelhomAgentBase | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| / | FelhomAgentBase | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /pool/felhom | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /pool/felhom | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/felhom-backup | FelhomAgentStore | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/felhom-backup | FelhomAgentStore | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/felhom-pbs | FelhomAgentStore | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/felhom-pbs | FelhomAgentStore | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/local | FelhomAgentStore | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/local | FelhomAgentStore | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/local-lvm | FelhomAgentStore | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/local-lvm | FelhomAgentStore | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990001 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990001 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990002 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990002 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990003 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990003 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990004 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990004 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990005 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990005 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990006 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990006 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990007 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990007 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990008 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990008 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990009 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990009 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
### guest hook
total 12
drwxr-xr-x 2 root root 4096 Aug 3 08:11 .
drwxr-xr-x 7 root root 4096 Jul 30 12:36 ..
-rwxr-xr-x 1 root root 166 Aug 3 08:11 felhom-guest-hook.sh
/var/lib/vz/snippets/felhom-guest-hook.sh
### wg tunnel
interface: wg-felhom
public key: KEYSzMyTKEZVDqDbikDXnadyeX9yJUjy8T58q0f4/1U=
private key: (hidden)
listening port: 59688
peer: f3d1ZI7/2/r+FWz8eo8Wl2HVygTZK9wBIN2M3dNHsgA=
endpoint: 167.233.158.164:443
allowed ips: 10.77.0.1/32, 10.77.0.250/32
latest handshake: 8 seconds ago
transfer: 5.86 GiB received, 2.48 GiB sent
persistent keepalive: every 25 seconds
### OOB belt
active
LISTEN 0 128 0.0.0.0:8822 0.0.0.0:* users:(("sshd",pid=1027109,fd=7))
LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=1052,fd=6))
LISTEN 0 128 [::]:8822 [::]:* users:(("sshd",pid=1027109,fd=6))
LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=1052,fd=7))
### agent state dir
total 56
drwxr-x--- 7 felhom-agent felhom-agent 4096 Aug 9 06:59 .
drwxr-xr-x 45 root root 4096 Aug 3 08:08 ..
-rw------- 1 felhom-agent felhom-agent 61 Aug 3 08:09 drive-intents.json
drwx------ 2 felhom-agent felhom-agent 4096 Aug 4 23:11 escrow-stage
drwx------ 2 felhom-agent felhom-agent 4096 Aug 3 08:09 felhom-sshd
drwx------ 3 felhom-agent felhom-agent 4096 Aug 3 08:11 guests
-rw------- 1 felhom-agent felhom-agent 5805 Aug 9 06:59 journal.log
-rw-r--r-- 1 felhom-agent felhom-agent 591 Aug 3 08:09 local-api.crt
-rw------- 1 felhom-agent felhom-agent 227 Aug 3 08:09 local-api.key
-rw------- 1 felhom-agent felhom-agent 82 Aug 3 08:11 local-tokens.log
-rw------- 1 felhom-agent felhom-agent 0 Aug 3 08:09 nonces.log
drwx------ 2 felhom-agent felhom-agent 4096 Aug 3 08:09 pbsdr
-rw------- 1 felhom-agent felhom-agent 383 Aug 9 06:59 restore-test-state.json
drwx------ 2 felhom-agent felhom-agent 4096 Aug 3 08:09 wg
### install state file
-rw-r--r-- 1 root root 1722 Aug 3 08:09 /etc/felhom-oob.nft
/etc/felhom:
total 12
drwxr-xr-x 2 root root 4096 Jul 21 18:28 .
drwxr-xr-x 102 root root 4096 Aug 3 08:09 ..
-rw-r--r-- 1 root root 0 Jul 21 18:28 .bootstrap-done
-rw-r--r-- 1 root root 7 Jul 21 18:20 appliance-pairing-code
/etc/felhom-agent:
total 36
drwxr-xr-x 2 root root 4096 Aug 4 09:32 .
drwxr-xr-x 102 root root 4096 Aug 3 08:09 ..
-rw------- 1 felhom-agent felhom-agent 2463 Aug 4 09:32 agent.json
-rw------- 1 root root 2462 Aug 4 09:32 agent.json.bak-r191
-rw------- 1 felhom-agent felhom-agent 2460 Jul 27 11:31 agent.json.campaign8-before
-rw------- 1 root root 2459 Jul 28 17:36 agent.json.campaign9-before
-rw------- 1 root root 2455 Jul 28 17:37 agent.json.campaign9-prev
-rw------- 1 felhom-agent felhom-agent 2459 Jul 28 18:10 agent.json.pre-e-target-move
-rw------- 1 felhom-agent felhom-agent 2459 Jul 27 08:08 agent.json.pre-prunegate.bak
/etc/felhom-sshd:
total 24
drwxr-xr-x 3 root root 4096 Aug 6 12:58 .
drwxr-xr-x 102 root root 4096 Aug 3 08:09 ..
drwxr-xr-x 2 root root 4096 Aug 6 12:58 authorized_keys
-rw------- 1 root root 411 Aug 3 08:09 ssh_host_ed25519_key
-rw-r--r-- 1 root root 101 Aug 3 08:09 ssh_host_ed25519_key.pub
-rw-r--r-- 1 root root 433 Aug 6 12:58 sshd_config
### PBS snapshots
Volid Format Type Size VMID
felhom-pbs:backup/ct/9201/2026-07-28T19:19:45Z pbs-ct backup 6264034049 9201
felhom-pbs:backup/ct/9201/2026-08-04T19:24:16Z pbs-ct backup 4637840513 9201
### local vzdump archives
Volid Format Type Size VMID
felhom-backup:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst tar.zst backup 649547835 9100
felhom-backup:backup/vzdump-lxc-9201-2026_08_07-04_34_58.tar.zst tar.zst backup 2073915792 9201
felhom-backup:backup/vzdump-lxc-9201-2026_08_08-04_37_15.tar.zst tar.zst backup 2142232544 9201
felhom-backup:backup/vzdump-lxc-9201-2026_08_09-04_42_15.tar.zst tar.zst backup 2143026831 9201
### enrolled drives
/mnt/felhom-drives/:
total 12
drwxr-xr-x 3 root root 4096 Aug 3 08:05 .
drwxr-xr-x 6 root root 4096 Aug 6 14:39 ..
drwxr-xr-x 2 root root 4096 Jul 22 13:53 Felhom-Share
/mnt/felhom-drives/Felhom-Share:
total 8
drwxr-xr-x 2 root root 4096 Jul 22 13:53 .
drwxr-xr-x 3 root root 4096 Aug 3 08:05 ..