admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b1afbb8a4d6c06e782da726785502323e4c163bb
felhom.eu/documentation/audits/evidence-rehearsal-2026-08-09/GATE0-demo-hp-before.txt
T
admin 34646295dc
gates / gates (push) Successful in 29s
Details
Rehearsal 2026-08-09: pre-phase + Gate 0 recorded before the destructive walk 
Venue demo-hp, operator-approved at STOP 1. Records the state that P1 destroys,
plus seven pre-walk findings, while they can still be checked against a live box.

R-268 CLOSED — the leaked per-guest local-API token is rotated and the rotation
is PROVEN in both directions (old refused, new accepted, channel up with a
positive observable). Rotating it surfaced three defects:

- an out-of-process rotation does NOT revoke the old token. The daemon serves
  Lookup from a stale index and re-reads only on a MISS, so a superseded token
  is a direct hit. Red-proved in a unit probe AND live on hardware; the shipped
  RemintCoherence test passes only because it looks up the NEW token first.
- R-268's own recipe is incomplete: ensureLocalAPI returns early on a present
  local_api block, so writing bootstrap.json is not enough — the controller
  serves the old token from controller.yaml across restarts.
- the agent-channel alarm never closes: the UP branch does not notify from an
  unseeded state, and the alarm's own remedy ("re-bootstrap") resets it.

Gate 0 complete: dataset planted in the Calibre library (coverage verified, not
assumed) with two Hungarian accented filenames; the comparator watched FAILING
three ways including an NFC->NFD rename that renders identically; off-site run
driven through the product's own button; restore point recorded by identity as
snapshot 41c830db, confirmed to carry all four files.

Also corrects the record: demo-hp's off-site tier is HEALTHY. Three hub surfaces
agreed it was absent and all three mislead — the panel showing 0 snapshots renders
the LOCAL tier, 162 KB rounds to 0.0 GB, and a two-day-old stuck event reads as
current. And the managed-update floor is live at 0.200.0, not 0.156.0.
2026-08-09 10:34:57 +02:00

283 lines
19 KiB
Plaintext
Raw Blame History

=== captured 2026-08-09T08:17:53Z UTC / 2026-08-09T10:17:53+0200 host-local ===
### felhom-agent
felhom-agent 0.127.0
### pveversion
pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
### pct list
VMID Status Lock Name
9201 running demo-hp
### qm list
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
300 drill-r50 stopped 8192 32.00 0
### pool members
{"comment":"Felhom-managed guests","members":[{"cpu":0.00448653868698896,"disk":990715904,"diskread":4608909312,"diskwrite":127439912960,"id":"lxc/9201","maxcpu":7,"maxdisk":33501757440,"maxmem":28212985856,"mem":641748992,"memhost":0,"name":"demo-hp","netin":2213810807,"netout":376936192,"node":"felhom-host","status":"running","template":0,"type":"lxc","uptime":525808,"vmid":9201}],"poolid":"felhom"}
### pvesm status
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
c11-scratch dir active 983379700 6940296 926412792 0.71%
felhom-backup dir active 983379700 6940296 926412792 0.71%
felhom-pbs pbs active 0 0 0 0.00%
local dir active 40516856 19132716 19293748 47.22%
local-lvm lvmthin active 56545280 20904790 35640489 36.97%
### storage.cfg
dir: local
path /var/lib/vz
content iso,import,vztmpl,backup
lvmthin: local-lvm
thinpool data
vgname pve
content rootdir,images
pbs: felhom-pbs
datastore felhom-offsite
server 10.77.0.1
content backup
encryption-key 3f:4f:65:c0:d8:f3:9f:3c:43:29:8c:c8:61:3c:89:40:e6:7a:c8:ae:b1:44:70:09:0e:f1:f7:1b:60:a0:c4:99
fingerprint c6:07:28:3f:5b:7b:5a:41:90:28:d7:ca:4f:37:14:70:56:39:2e:2f:0b:71:e8:06:ca:60:4a:d5:56:5f:3c:fd
namespace demo-hp
username felhom@pbs!demo-hp
dir: felhom-backup
path /mnt/nvme-1tb
content backup
is_mountpoint 1
dir: c11-scratch
path /mnt/nvme-1tb
content images
### df
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-root 39G 19G 19G 50% /
/dev/nvme0n1 938G 6.7G 884G 1% /mnt/nvme-1tb
### lsblk
NAME SIZE MODEL SERIAL FSTYPE MOUNTPOINT
sda 119.2G SanDisk X600 M.2 2280 SATA 128GB 182195804614
|-sda1 1007K
|-sda2 1G vfat /boot/efi
`-sda3 118.2G LVM2_member
|-pve-swap 8G swap [SWAP]
|-pve-root 39.6G ext4 /
|-pve-data_tmeta 1G
| `-pve-data-tpool 53.9G
| |-pve-data 53.9G
| |-pve-vm--9201--disk--0 32G ext4
| |-pve-vm--9201--disk--1 70G ext4
| |-pve-vm--300--disk--0 4M
| `-pve-vm--300--disk--1 32G
`-pve-data_tdata 53.9G
`-pve-data-tpool 53.9G
|-pve-data 53.9G
|-pve-vm--9201--disk--0 32G ext4
|-pve-vm--9201--disk--1 70G ext4
|-pve-vm--300--disk--0 4M
`-pve-vm--300--disk--1 32G
nvme0n1 953.9G KXG50PNV1T02 NVMe TOSHIBA 1024GB 58BS11AFT8MQ ext4 /mnt/nvme-1tb
### felhom-agent unit + user
active
uid=999(felhom-agent) gid=990(felhom-agent) groups=990(felhom-agent),999(systemd-journal)
### sudoers entries
total 68
drwxr-xr-x 2 root root 4096 Aug 3 08:09 .
drwxr-xr-x 102 root root 4096 Aug 3 08:09 ..
-r--r----- 1 root root 1068 Apr 11 14:21 README
-r--r----- 1 root root 22483 Aug 3 08:08 felhom-agent
-r--r----- 1 root root 21833 Jul 28 11:29 felhom-agent.bak-pre-e2a
-r--r----- 1 root root 1241 Aug 3 08:09 felhom-op
-r--r----- 1 root root 666 May 20 2023 zfs
--
287
### pveum users/tokens/roles/acl
+------------------+---------+-------------------+--------+--------+-----------+--------+------+----------+------------+------------------+--------+-------------+
| userid | comment | email | enable | expire | firstname | groups | keys | lastname | realm-type | tfa-locked-until | tokens | totp-locked |
+==================+=========+===================+========+========+===========+========+======+==========+============+==================+========+=============+
| felhom-agent@pve | | | 1 | 0 | | | | | pve | | | |
+------------------+---------+-------------------+--------+--------+-----------+--------+------+----------+------------+------------------+--------+-------------+
| root@pam | | noreply@felhom.eu | 1 | 0 | | | | | pam | | | |
+------------------+---------+-------------------+--------+--------+-----------+--------+------+----------+------------+------------------+--------+-------------+
--tokens--
+---------+---------+--------+---------+
| tokenid | comment | expire | privsep |
+=========+=========+========+=========+
| agent | | 0 | 1 |
+---------+---------+--------+---------+
--roles--
| FelhomAgentBase | Datastore.Audit,SDN.Use,Sys.Audit | 0 |
| FelhomAgentGuest | Pool.Allocate,Pool.Audit,VM.Allocate,VM.Audit,VM.Backup,VM.Config.CPU,VM.Config.Disk,VM.Config.Memory,VM.Config.Network,VM.Config.Options,VM.PowerMgmt,VM.Snapshot,VM.Snapshot.Rollback | 0 |
| FelhomAgentStore | Datastore.Allocate,Datastore.AllocateSpace | 0 |
--acl--
+------------------------+------------------+-------+------------------------+-----------+
| path | roleid | type | ugid | propagate |
+========================+==================+=======+========================+===========+
| / | FelhomAgentBase | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| / | FelhomAgentBase | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /pool/felhom | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /pool/felhom | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/felhom-backup | FelhomAgentStore | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/felhom-backup | FelhomAgentStore | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/felhom-pbs | FelhomAgentStore | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/felhom-pbs | FelhomAgentStore | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/local | FelhomAgentStore | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/local | FelhomAgentStore | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/local-lvm | FelhomAgentStore | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /storage/local-lvm | FelhomAgentStore | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990001 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990001 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990002 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990002 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990003 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990003 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990004 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990004 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990005 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990005 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990006 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990006 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990007 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990007 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990008 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990008 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990009 | FelhomAgentGuest | user | felhom-agent@pve | 1 |
+------------------------+------------------+-------+------------------------+-----------+
| /vms/990009 | FelhomAgentGuest | token | felhom-agent@pve!agent | 1 |
+------------------------+------------------+-------+------------------------+-----------+
### guest hook
total 12
drwxr-xr-x 2 root root 4096 Aug 3 08:11 .
drwxr-xr-x 7 root root 4096 Jul 30 12:36 ..
-rwxr-xr-x 1 root root 166 Aug 3 08:11 felhom-guest-hook.sh
/var/lib/vz/snippets/felhom-guest-hook.sh
### wg tunnel
interface: wg-felhom
public key: KEYSzMyTKEZVDqDbikDXnadyeX9yJUjy8T58q0f4/1U=
private key: (hidden)
listening port: 59688
peer: f3d1ZI7/2/r+FWz8eo8Wl2HVygTZK9wBIN2M3dNHsgA=
endpoint: 167.233.158.164:443
allowed ips: 10.77.0.1/32, 10.77.0.250/32
latest handshake: 8 seconds ago
transfer: 5.86 GiB received, 2.48 GiB sent
persistent keepalive: every 25 seconds
### OOB belt
active
LISTEN 0 128 0.0.0.0:8822 0.0.0.0:* users:(("sshd",pid=1027109,fd=7))
LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=1052,fd=6))
LISTEN 0 128 [::]:8822 [::]:* users:(("sshd",pid=1027109,fd=6))
LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=1052,fd=7))
### agent state dir
total 56
drwxr-x--- 7 felhom-agent felhom-agent 4096 Aug 9 06:59 .
drwxr-xr-x 45 root root 4096 Aug 3 08:08 ..
-rw------- 1 felhom-agent felhom-agent 61 Aug 3 08:09 drive-intents.json
drwx------ 2 felhom-agent felhom-agent 4096 Aug 4 23:11 escrow-stage
drwx------ 2 felhom-agent felhom-agent 4096 Aug 3 08:09 felhom-sshd
drwx------ 3 felhom-agent felhom-agent 4096 Aug 3 08:11 guests
-rw------- 1 felhom-agent felhom-agent 5805 Aug 9 06:59 journal.log
-rw-r--r-- 1 felhom-agent felhom-agent 591 Aug 3 08:09 local-api.crt
-rw------- 1 felhom-agent felhom-agent 227 Aug 3 08:09 local-api.key
-rw------- 1 felhom-agent felhom-agent 82 Aug 3 08:11 local-tokens.log
-rw------- 1 felhom-agent felhom-agent 0 Aug 3 08:09 nonces.log
drwx------ 2 felhom-agent felhom-agent 4096 Aug 3 08:09 pbsdr
-rw------- 1 felhom-agent felhom-agent 383 Aug 9 06:59 restore-test-state.json
drwx------ 2 felhom-agent felhom-agent 4096 Aug 3 08:09 wg
### install state file
-rw-r--r-- 1 root root 1722 Aug 3 08:09 /etc/felhom-oob.nft
/etc/felhom:
total 12
drwxr-xr-x 2 root root 4096 Jul 21 18:28 .
drwxr-xr-x 102 root root 4096 Aug 3 08:09 ..
-rw-r--r-- 1 root root 0 Jul 21 18:28 .bootstrap-done
-rw-r--r-- 1 root root 7 Jul 21 18:20 appliance-pairing-code
/etc/felhom-agent:
total 36
drwxr-xr-x 2 root root 4096 Aug 4 09:32 .
drwxr-xr-x 102 root root 4096 Aug 3 08:09 ..
-rw------- 1 felhom-agent felhom-agent 2463 Aug 4 09:32 agent.json
-rw------- 1 root root 2462 Aug 4 09:32 agent.json.bak-r191
-rw------- 1 felhom-agent felhom-agent 2460 Jul 27 11:31 agent.json.campaign8-before
-rw------- 1 root root 2459 Jul 28 17:36 agent.json.campaign9-before
-rw------- 1 root root 2455 Jul 28 17:37 agent.json.campaign9-prev
-rw------- 1 felhom-agent felhom-agent 2459 Jul 28 18:10 agent.json.pre-e-target-move
-rw------- 1 felhom-agent felhom-agent 2459 Jul 27 08:08 agent.json.pre-prunegate.bak
/etc/felhom-sshd:
total 24
drwxr-xr-x 3 root root 4096 Aug 6 12:58 .
drwxr-xr-x 102 root root 4096 Aug 3 08:09 ..
drwxr-xr-x 2 root root 4096 Aug 6 12:58 authorized_keys
-rw------- 1 root root 411 Aug 3 08:09 ssh_host_ed25519_key
-rw-r--r-- 1 root root 101 Aug 3 08:09 ssh_host_ed25519_key.pub
-rw-r--r-- 1 root root 433 Aug 6 12:58 sshd_config
### PBS snapshots
Volid Format Type Size VMID
felhom-pbs:backup/ct/9201/2026-07-28T19:19:45Z pbs-ct backup 6264034049 9201
felhom-pbs:backup/ct/9201/2026-08-04T19:24:16Z pbs-ct backup 4637840513 9201
### local vzdump archives
Volid Format Type Size VMID
felhom-backup:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst tar.zst backup 649547835 9100
felhom-backup:backup/vzdump-lxc-9201-2026_08_07-04_34_58.tar.zst tar.zst backup 2073915792 9201
felhom-backup:backup/vzdump-lxc-9201-2026_08_08-04_37_15.tar.zst tar.zst backup 2142232544 9201
felhom-backup:backup/vzdump-lxc-9201-2026_08_09-04_42_15.tar.zst tar.zst backup 2143026831 9201
### enrolled drives
/mnt/felhom-drives/:
total 12
drwxr-xr-x 3 root root 4096 Aug 3 08:05 .
drwxr-xr-x 6 root root 4096 Aug 6 14:39 ..
drwxr-xr-x 2 root root 4096 Jul 22 13:53 Felhom-Share
/mnt/felhom-drives/Felhom-Share:
total 8
drwxr-xr-x 2 root root 4096 Jul 22 13:53 .
drwxr-xr-x 3 root root 4096 Aug 3 08:05 ..
Reference in New Issue View Git Blame Copy Permalink