felhom.eu/documentation/audits/plex2shell-felhom-impact-2026-10-23

Plex2Shell on DooPlex — did the attack reach Felhom's secrets? (2026-10-23)

Read-only investigation. Source of the incident facts: ~/incident-plex-2026-10-23/INCIDENT.md on DooPlex (HM-040, homelab session). Architecture read first: architecture/01-topology-and-trust.md, 04-control-plane-authorization.md §3.1, 05-hub-architecture.md §16.2, 13-threat-model.md, runbooks/secrets.md, runbooks/total-loss-of-dooplex.md, runbooks/target-selection.md (DooPlex = Tier 2). Window: 2026-10-17 04:00Z → 2026-10-23 08:04:34Z (the isolation NetworkPolicy incident-2026-10-23-plex-isolate, creationTimestamp in INCIDENT.md Phase 2a).

Verdict: NOT REACHED

Two open paths led from the hacked Plex pod to Felhom secrets. Each would have left a trace in a record that survived, and neither trace exists:

1. NFS export (/mnt/5_hdd/backup/longhorn-pvc, *(rw,no_root_squash,no_subtree_check)): the NFS server accepted zero new TCP connections from 2026-10-10 00:00Z to 2026-10-23 13:05Z — node_nfsd_connections_total flat at 288, 3902 five-minute samples, no gap. Every NFS request needs a TCP connection (nfsd listens on TCP only: portlist, rpcinfo). Positive control: after this session's probe pod made three TCP connections to :2049 today the counter read 4 (one Longhorn client + three probes), in /proc/net/rpc/nfsd and in Prometheus alike. The only NFS client in the window was Longhorn's own manager (one long-lived NFSv4.2 connection). Read operations happened only in Longhorn's backup hours; bytes read per day stayed at 18–39 MiB.
2. Longhorn manager API (no login, port 9500 — can create backups, backup targets, volumes): no backup was created outside the two recurring jobs' times (longhorn_backup_state series, 1-minute scrape); every scheduled night of the window wrote its backups to DooPlex's own NFS store (so the target was local each night); no volume existed in the window that is not explained; the backup target object is the one created 2025-11-29; DooPlex's network card sent no more than the Plex pod itself sent.

What would change this verdict: evidence of a container escape to the host (the incident session found none, INCIDENT.md „Anything OUTSIDE the Plex pod"), or a Longhorn target swap + backup + delete inside one minute (no record either way; Longhorn manager logs were lost, below).

Part 1 — the shared backup folder

what	value	evidence
export (before)	/mnt/5_hdd/backup/longhorn-pvc *(rw,sync,no_subtree_check,no_root_squash)	p1-nfs-exports-clients.txt
who can reach :2049	the host firewall accepts 192.168.0.0/24, 10.42.0.0/16 (pods), 10.43.0.0/16, 100.64.0.0/10 (tailnet)	nft list ruleset, input chain
secure option	no protection inside a pod: containerd 2.x sets net.ipv4.ip_unprivileged_port_start=0 in every pod; a uid-1000 probe with CapEff 0 bound port 700 and connected to :2049	p2-targeted-probes.txt (last line), sysctl read in the Plex pod netns = 0, host = 1024
filesystem	the export is a sub-directory of /dev/sda1 (ext4, mounted at /mnt/5_hdd); root inode 2 has generation 0	findmnt, lsattr -vd /mnt/5_hdd
contents	Longhorn backup store, 119 GB: nightly backups of every Longhorn volume, including felhom-system/hub-data (the hub database), NOT encrypted (spec.encrypted=false on all four felhom-system volumes)	p1-backup-folder-listing.txt
v4 pseudo-root	/, /mnt, /mnt/5_hdd, /mnt/5_hdd/backup appear read-only as path components only; no other directory under /mnt/5_hdd/backup is exported	/proc/fs/nfsd/exports
clients now	one: longhorn-manager-w4tf6 10.42.0.214, NFSv4.2	/proc/fs/nfsd/clients/*/info
NFS logging	NFSv3 mounts are logged only at debug level; NFSv4 attaches are not logged at all (positive control: client 3 is attached and the journal has no attach line); detaches are logged (2 lines in boot -1, none in the window); illegal-port refusals are logged (none). An absent line proves nothing for v4.	journalctl -t rpc.mountd, strings rpc.mountd

No subtree check matters beyond the folder. With no_subtree_check on a sub-directory export, the server checks only that a file handle belongs to the exported filesystem. A published technique (HvS-Consulting's nfs-security-tooling, 2024) builds the handle of the filesystem's root directory and walks the whole disk; with no_root_squash it does so as root. On /dev/sda1 that reaches the operator signing keys (/mnt/5_hdd/felhom.eu/ felhom-op-operational, felhom-rec-recovery, felhom_op_ed25519, no passphrase, R-924), the private security-review evidence, the workspace, the encrypted sec store (its age key is on /dev/sdb1, not reachable) and the other backup trees. Grade: documented technique, every precondition measured here, NOT exercised (doing it would be an attack on a Tier-2 machine).

The change (operator yes, 2026-10-23 ~15:18Z): the line became /mnt/5_hdd/backup/longhorn-pvc 10.42.0.0/16(rw,sync,no_subtree_check,no_root_squash); exportfs -ra rc 0. Read back: exportfs -v and /proc/fs/nfsd/exports show 10.42.0.0/16; Longhorn's backup target synced at 15:20:10Z with available: true; the Longhorn client is still attached. Backup of the old file: /etc/exports.bak-2026-10-23-plex2shell. Undo: sudo cp -p /etc/exports.bak-2026-10-23-plex2shell /etc/exports && sudo exportfs -ra. This closes the folder to the home network and the tailnet. It does NOT close it to a hacked pod — Longhorn connects from its pod address, so the pod network must stay allowed; limiting it to DooPlex itself would break Longhorn. Root squash is not possible either (the store is root-owned 0700 and Longhorn writes as root). p1-export-change.txt.

Part 2 — what the hacked Plex could reach without a login

Measured from probe pod default/cc-p2s-probe (uid 1000, CapEff 0, no service-account token, no NetworkPolicy in default — the Plex pod's position before the isolation). Created 15:19:46Z, removed 15:33:44Z (p2-probe-lifecycle.txt, before/after listings). 186 of 196 TCP Service ports answered (p2-service-sweep.txt).

target	no-login result	what a no-login caller gets
Longhorn manager API longhorn-backend:9500, UI longhorn-frontend:80	200	every volume (234 names), backup targets, backup volumes; actions snapshotCreate, snapshotBackup, attach, detach, pvcCreate, activate offered on hub-data
NFS on the node :2049, :111	open	see Part 1
Felhom hub hub:8080 (bypasses the ingress geo filter)	/healthz 200, /login 200, every data path 401/403/302	nothing (matches 13 §4: health, login, register, armed re-enroll)
kube-apiserver 10.43.0.1:443 / 192.168.0.180:6443	401	nothing (k3s config.yaml unchanged since 2026-08-05)
kubelet :10250	401; :10255 closed	nothing
Prometheus, Grafana (anonymous Viewer), Alertmanager, node-exporter, kube-state-metrics	200	metrics and dashboards; Alertmanager also lets anyone create a silence. No secret values
Gitea	200	the public repos (R-925 ruling); registry /v2/ 401
Vaultwarden	/alive, /api/config, /admin login page 200	nothing without the master password
ArgoCD, Headlamp, the filebrowsers	401 on the API	nothing (Headlamp's own service account can read every Secret — behind OIDC)
felhom-system/umami-db 5432, other databases	open	password needed
LAN	router admin (.1), a TP-Link (.254), a printer (.20), demo-felhom's Proxmox UI and SSH (.162), PBS UI on DooPlex :8007, Samba :445	logins needed; p2-lan-reach.txt

Part 3 — evidence of use in the window, per path

path	best record that exists	result
NFS	nfsd counters in Prometheus (connections, ops by method, bytes, RPC errors) — p3-nfsd-*.json	unused (verdict 1). RPC errors (auth/fmt/cInt) 0
Longhorn API	longhorn_backup_state, longhorn_volume_state series; nfsd bytes written per night; Backup/Snapshot/BackupTarget/Volume creationTimestamp — p3-longhorn-*	no sign of use: 106 backups in the window = 53 daily (10-23 02:00Z) + 53 weekly (10-18 03:00Z), all to the local URL; 0 backups first seen off-schedule; no unexplained volume. Longhorn CRs carry no managedFields, so who changed what is not recorded there (p3-longhorn-objects.txt, CORRECTION)
Longhorn manager/UI logs	—	no record: lost. DooPlex was rebooted at 13:20Z by the maintenance session (~/maintenance/2026-10-window); the pre-reboot container logs of longhorn-manager, longhorn-ui, the hub, the ingress, Gitea, Vaultwarden, Grafana and Prometheus are gone. The incident session's earlier greps (before the reboot) found 10.42.0.52 in none of longhorn-manager, CNPG, Gitea (from 10-21 20:59), authentik; that is their record, not re-verifiable
hub	/data/access-log (R-1140), 10-18 10:25Z → now, 12 764 lines — p3-hub-access-log.txt	no call from the Plex pod (10.42.0.52: 0; positive control: the probe's 10.42.0.107 is logged). Credentialed requests come only from the boxes' LAN addresses, the home WAN and the operator's sessions. Gap: 10-17 04:00Z → 10-18 10:25Z no record (the log started then) — but no hub route gives a secret without a credential
SSH to DooPlex	journal (boot -1 survives)	0 sshd lines from 10.42/10.43 in the window (control: 61 lines from 192.168.x)
Samba	per-client log files log.%m	no file for 10.42.0.52 (positive control: log.10.42.0.107 appeared when the probe touched :445) — p3-samba-clients.txt
Alertmanager	silences API (keeps expired ones)	only two silences, both CC's, both outside the attack
DNS	CoreDNS keeps no query log; only upstream-timeout lines name a domain, without the client	names in p3-coredns-error-names.txt; hand.genddos.st looked up 10 times on 10-19 08:31–09:50Z (client unknown)
network bytes	cAdvisor per pod (no peer label) — p3-network-bytes.txt	the Plex pod sent ~1.5 TB (143 / 202 / 528 / 441 / 206 GiB per day, 10-18 → 10-22; before: 0.04 GiB/day) and received ≤ 7.9 GiB/day. DooPlex's NIC eno1 sent the same extra out. Outbound-heavy: the shape of a flood tool or of streaming, not of a download. It did not come from the NFS store (nfsd reads flat) or from other pods (bridge-to-pod bytes flat)

Part 4 — Felhom's secrets on DooPlex

Exposure = what a path could read, judged with Part 3. „Possible path" means the door existed; it does not mean it was used.

secret	where it lives	path that could read it (Part 2)	evidence of use	exposure
hub DB: Cloudflare API tokens, tunnel tokens, git tokens (unsealed, R-975)	hub.db on Longhorn hub-data; unencrypted nightly copies in the NFS store	NFS export (direct, no escape needed); Longhorn API backup to an outside target	none (Part 3)	none
hub DB: box API keys, controller keys, owner passphrases, PBS-DR tokens, break-glass passwords, Storage Box sub-account passwords (sealed, AES-GCM, key OFFSITE_SECRET_KEY)	same	same — but the key is a k8s Secret, not in the DB or its backups	none	none (sealed values useless without the key)
escrow blobs host_escrow.blob / identity_blob	same	same	none	none — sealed means: wrapped by each household's recovery code, which the hub never holds (03-host-agent.md, escrow retention: „the hub never decrypts either table")
hub tokens (api_tokens.token_hash), self-bind / claim codes	same, hashed	same	none	none (hashes)
k8s Secrets in felhom-system (report-api global key, offsite-secret-key, storagebox = main Storage Box/Hetzner login, resend-api, tenantsync, agent-poke, wg-endpoint-ssh, gitea-creds, umami, TLS keys)	etcd/k3s datastore (/var/lib/rancher, sdb1); also the nightly GPG secrets export on sda1 (/mnt/5_hdd/backup/secrets/exports, symmetric, passphrase file on sdb1)	k8s API anonymous: 401. NFS escape → the .gpg files only (key not on sda1)	none	none
hub session / signing material	k8s Secret / hub DB	as above	none	none
operator signing keys (operational + recovery + felhom_op_ed25519, no passphrase)	/mnt/5_hdd/felhom.eu/ on sda1	NFS export → file-handle escape (documented, not exercised)	none (no NFS connection at all)	none — but the door is still open to any pod (row filed)
sec store	cc-secrets/store/cc.sops.yaml on sda1, age key ~/.config/sops/age/keys.txt on sdb1	escape → ciphertext only	none	none
registry login, Gitea tokens	~/.docker/config.json, ~/.git-credentials (sdb1, 0600)	none from the pod network (registry 401, Gitea API anonymous = public repos only)	—	none
Vaultwarden (operator's password manager, R-923)	Longhorn volume; nightly unencrypted Longhorn copy in the NFS store (vault items stay encrypted by the master password)	login page; NFS export → encrypted vault DB	none for NFS; login logs lost	none (no record of login attempts — logs lost, and the vault is encrypted at rest)
off-site ciphertext (ep0 copy, demo-felhom backups)	sda1	escape → ciphertext only	none	none

Part 5 — plan

Verdict NOT REACHED → no rotation plan. The open doors are filed instead (rows below). Nothing was rotated.

Observations

• A hacked pod on DooPlex can still read the hub's unencrypted nightly backups and, by the file-handle escape, the operator signing keys through the NFS export. → FILED: R-1241 (P1).
• The Felhom demo NAS export /mnt/5_hdd/felhom-data → 192.168.0.162 is a sub-directory of the same disk with no_subtree_check: a compromised demo-felhom host could walk all of sda1 as uid 101000 (world-readable files only, e.g. the .gpg secrets exports and every repo). → FILED: R-1242 (P3).
• The hub answers pods directly, without the ingress geo filter. → NOT-A-FINDING: the geo filter is not an authentication control; every no-token path refused (measured).
• demo-felhom's Proxmox UI and SSH are reachable from any DooPlex pod (masqueraded as 192.168.0.180). → NOT-A-FINDING: same door as R-982 (root password SSH from the LAN), now with „a hacked DooPlex pod" as one more LAN attacker; added to 13 §3.
• Pre-reboot pod logs were erased by the 13:20Z reboot. → NOT-A-FINDING for Felhom: the hub's own access log persisted (R-1140 did its job). Homelab item, below.

For the homelab session (no HM numbers minted; evidence paths in this folder)

1. The NFS export is still open to every pod, no_root_squash, no_subtree_check, on the disk that holds the Felhom signing keys. secure does nothing in pods (containerd sets ip_unprivileged_port_start=0). Options: an egress NetworkPolicy so only longhorn-system reaches :2049/:111, and/or move the store to its own filesystem (LV or loop) so the subtree escape has nothing else to reach. Not managed by Ansible — the narrowed line persists.
2. Longhorn manager API and UI need no login from any pod (create backups/targets/volumes, delete volumes).
3. The old Plex volume was deleted through Longhorn's API at 11:24:15Z by nobody identified (INCIDENT.md EVIDENCE NOTE). Ask the operator whether it was them; if not, it is an unattributed write on an open API.
4. The Plex pod sent ~1.5 TB out on 10-18 → 10-22 (not in INCIDENT.md); and CoreDNS saw hand.genddos.st lookups on 10-19. Possibly a flood tool riding the same compromise.
5. The maintenance reboot at 13:20Z erased the window's pod logs (longhorn-manager, ingress, hub, Gitea, …): no log shipping, kubelet keeps one previous container log. Evidence rule R-320 applies to the homelab too.
6. Prometheus, Alertmanager (silences!), Grafana (anonymous Viewer), kube-state-metrics, node-exporter, MinIO console, Headlamp (its service account can read every Secret and write RBAC — behind OIDC) all answer every pod.
7. A user process python3 main.py -d -i (kisfenyo) listens on 0.0.0.0:8443 on DooPlex.