# Plex2Shell on DooPlex — did the attack reach Felhom's secrets? (2026-10-23)

Read-only investigation. Source of the incident facts: `~/incident-plex-2026-10-23/INCIDENT.md` on DooPlex (HM-040,
homelab session). Architecture read first: `architecture/01-topology-and-trust.md`, `04-control-plane-authorization.md`
§3.1, `05-hub-architecture.md` §16.2, `13-threat-model.md`, `runbooks/secrets.md`, `runbooks/total-loss-of-dooplex.md`,
`runbooks/target-selection.md` (DooPlex = Tier 2). Window: **2026-10-17 04:00Z → 2026-10-23 08:04:34Z** (the isolation
NetworkPolicy `incident-2026-10-23-plex-isolate`, `creationTimestamp` in INCIDENT.md Phase 2a).

## Verdict: **NOT REACHED**

Two open paths led from the hacked Plex pod to Felhom secrets. Each would have left a trace in a record that survived,
and neither trace exists:

1. **NFS export** (`/mnt/5_hdd/backup/longhorn-pvc`, `*(rw,no_root_squash,no_subtree_check)`): the NFS server accepted
   **zero new TCP connections from 2026-10-10 00:00Z to 2026-10-23 13:05Z** — `node_nfsd_connections_total` flat at 288,
   3902 five-minute samples, no gap. Every NFS request needs a TCP connection (nfsd listens on TCP only: `portlist`,
   `rpcinfo`). **Positive control:** after this session's probe pod made three TCP connections to :2049 today the counter
   read 4 (one Longhorn client + three probes), in `/proc/net/rpc/nfsd` and in Prometheus alike. The only NFS client in
   the window was Longhorn's own manager (one long-lived NFSv4.2 connection). Read operations happened only in Longhorn's
   backup hours; bytes read per day stayed at 18–39 MiB.
2. **Longhorn manager API** (no login, port 9500 — can create backups, backup targets, volumes): no backup was created
   outside the two recurring jobs' times (`longhorn_backup_state` series, 1-minute scrape); every scheduled night of the
   window wrote its backups to DooPlex's own NFS store (so the target was local each night); no volume existed in the
   window that is not explained; the backup target object is the one created 2025-11-29; DooPlex's network card sent
   no more than the Plex pod itself sent.

**What would change this verdict:** evidence of a container escape to the host (the incident session found none,
`INCIDENT.md` „Anything OUTSIDE the Plex pod"), or a Longhorn target swap + backup + delete inside one minute (no
record either way; Longhorn manager logs were lost, below).

## Part 1 — the shared backup folder

| what | value | evidence |
|---|---|---|
| export (before) | `/mnt/5_hdd/backup/longhorn-pvc *(rw,sync,no_subtree_check,no_root_squash)` | `p1-nfs-exports-clients.txt` |
| who can reach :2049 | the host firewall accepts 192.168.0.0/24, 10.42.0.0/16 (pods), 10.43.0.0/16, 100.64.0.0/10 (tailnet) | `nft list ruleset`, input chain |
| `secure` option | **no protection inside a pod**: containerd 2.x sets `net.ipv4.ip_unprivileged_port_start=0` in every pod; a uid-1000 probe with CapEff 0 bound port 700 and connected to :2049 | `p2-targeted-probes.txt` (last line), sysctl read in the Plex pod netns = 0, host = 1024 |
| filesystem | the export is a sub-directory of `/dev/sda1` (ext4, mounted at `/mnt/5_hdd`); root inode 2 has generation 0 | `findmnt`, `lsattr -vd /mnt/5_hdd` |
| contents | Longhorn backup store, 119 GB: nightly backups of every Longhorn volume, **including `felhom-system/hub-data` (the hub database), NOT encrypted** (`spec.encrypted=false` on all four felhom-system volumes) | `p1-backup-folder-listing.txt` |
| v4 pseudo-root | `/`, `/mnt`, `/mnt/5_hdd`, `/mnt/5_hdd/backup` appear read-only as path components only; no other directory under `/mnt/5_hdd/backup` is exported | `/proc/fs/nfsd/exports` |
| clients now | one: `longhorn-manager-w4tf6` 10.42.0.214, NFSv4.2 | `/proc/fs/nfsd/clients/*/info` |
| NFS logging | NFSv3 mounts are logged only at debug level; NFSv4 attaches are not logged at all (**positive control:** client 3 is attached and the journal has no attach line); detaches are logged (2 lines in boot -1, none in the window); illegal-port refusals are logged (none). An absent line proves nothing for v4. | `journalctl -t rpc.mountd`, `strings rpc.mountd` |

**No subtree check matters beyond the folder.** With `no_subtree_check` on a sub-directory export, the server checks
only that a file handle belongs to the exported *filesystem*. A published technique (HvS-Consulting's
`nfs-security-tooling`, 2024) builds the handle of the filesystem's root directory and walks the whole disk; with
`no_root_squash` it does so as root. On `/dev/sda1` that reaches the **operator signing keys** (`/mnt/5_hdd/felhom.eu/
felhom-op-operational`, `felhom-rec-recovery`, `felhom_op_ed25519`, no passphrase, R-924), the private security-review
evidence, the workspace, the encrypted `sec` store (its age key is on `/dev/sdb1`, not reachable) and the other backup
trees. **Grade: documented technique, every precondition measured here, NOT exercised** (doing it would be an attack on
a Tier-2 machine).

**The change (operator yes, 2026-10-23 ~15:18Z):** the line became
`/mnt/5_hdd/backup/longhorn-pvc 10.42.0.0/16(rw,sync,no_subtree_check,no_root_squash)`; `exportfs -ra` rc 0. Read back:
`exportfs -v` and `/proc/fs/nfsd/exports` show `10.42.0.0/16`; Longhorn's backup target synced at 15:20:10Z with
`available: true`; the Longhorn client is still attached. Backup of the old file: `/etc/exports.bak-2026-10-23-plex2shell`.
Undo: `sudo cp -p /etc/exports.bak-2026-10-23-plex2shell /etc/exports && sudo exportfs -ra`. **This closes the folder to
the home network and the tailnet. It does NOT close it to a hacked pod** — Longhorn connects from its pod address, so
the pod network must stay allowed; limiting it to DooPlex itself would break Longhorn. Root squash is not possible either
(the store is root-owned 0700 and Longhorn writes as root). `p1-export-change.txt`.

## Part 2 — what the hacked Plex could reach without a login

Measured from probe pod `default/cc-p2s-probe` (uid 1000, CapEff 0, no service-account token, no NetworkPolicy in
`default` — the Plex pod's position before the isolation). Created 15:19:46Z, removed 15:33:44Z
(`p2-probe-lifecycle.txt`, before/after listings). 186 of 196 TCP Service ports answered (`p2-service-sweep.txt`).

| target | no-login result | what a no-login caller gets |
|---|---|---|
| Longhorn manager API `longhorn-backend:9500`, UI `longhorn-frontend:80` | 200 | every volume (234 names), backup targets, backup volumes; actions `snapshotCreate`, `snapshotBackup`, `attach`, `detach`, `pvcCreate`, `activate` offered on `hub-data` |
| NFS on the node :2049, :111 | open | see Part 1 |
| Felhom hub `hub:8080` (bypasses the ingress geo filter) | `/healthz` 200, `/login` 200, every data path 401/403/302 | nothing (matches `13` §4: health, login, register, armed re-enroll) |
| kube-apiserver 10.43.0.1:443 / 192.168.0.180:6443 | 401 | nothing (k3s `config.yaml` unchanged since 2026-08-05) |
| kubelet :10250 | 401; :10255 closed | nothing |
| Prometheus, Grafana (anonymous Viewer), Alertmanager, node-exporter, kube-state-metrics | 200 | metrics and dashboards; Alertmanager also lets anyone create a silence. No secret values |
| Gitea | 200 | the public repos (R-925 ruling); registry `/v2/` 401 |
| Vaultwarden | `/alive`, `/api/config`, `/admin` login page 200 | nothing without the master password |
| ArgoCD, Headlamp, the filebrowsers | 401 on the API | nothing (Headlamp's own service account can read every Secret — behind OIDC) |
| `felhom-system/umami-db` 5432, other databases | open | password needed |
| LAN | router admin (.1), a TP-Link (.254), a printer (.20), **demo-felhom's Proxmox UI and SSH (.162)**, PBS UI on DooPlex :8007, Samba :445 | logins needed; `p2-lan-reach.txt` |

## Part 3 — evidence of use in the window, per path

| path | best record that exists | result |
|---|---|---|
| NFS | nfsd counters in Prometheus (connections, ops by method, bytes, RPC errors) — `p3-nfsd-*.json` | **unused** (verdict 1). RPC errors (auth/fmt/cInt) 0 |
| Longhorn API | `longhorn_backup_state`, `longhorn_volume_state` series; nfsd bytes written per night; Backup/Snapshot/BackupTarget/Volume `creationTimestamp` — `p3-longhorn-*` | **no sign of use**: 106 backups in the window = 53 daily (10-23 02:00Z) + 53 weekly (10-18 03:00Z), all to the local URL; 0 backups first seen off-schedule; no unexplained volume. Longhorn CRs carry **no managedFields**, so who changed what is not recorded there (`p3-longhorn-objects.txt`, CORRECTION) |
| Longhorn manager/UI logs | — | **no record: lost.** DooPlex was rebooted at 13:20Z by the maintenance session (`~/maintenance/2026-10-window`); the pre-reboot container logs of longhorn-manager, longhorn-ui, the hub, the ingress, Gitea, Vaultwarden, Grafana and Prometheus are gone. The incident session's earlier greps (before the reboot) found 10.42.0.52 in none of longhorn-manager, CNPG, Gitea (from 10-21 20:59), authentik; that is their record, not re-verifiable |
| hub | `/data/access-log` (R-1140), 10-18 10:25Z → now, 12 764 lines — `p3-hub-access-log.txt` | **no call from the Plex pod** (10.42.0.52: 0; positive control: the probe's 10.42.0.107 is logged). Credentialed requests come only from the boxes' LAN addresses, the home WAN and the operator's sessions. **Gap:** 10-17 04:00Z → 10-18 10:25Z no record (the log started then) — but no hub route gives a secret without a credential |
| SSH to DooPlex | journal (boot -1 survives) | 0 sshd lines from 10.42/10.43 in the window (control: 61 lines from 192.168.x) |
| Samba | per-client log files `log.%m` | no file for 10.42.0.52 (positive control: `log.10.42.0.107` appeared when the probe touched :445) — `p3-samba-clients.txt` |
| Alertmanager | silences API (keeps expired ones) | only two silences, both CC's, both outside the attack |
| DNS | CoreDNS keeps **no query log**; only upstream-timeout lines name a domain, without the client | names in `p3-coredns-error-names.txt`; `hand.genddos.st` looked up 10 times on 10-19 08:31–09:50Z (client unknown) |
| network bytes | cAdvisor per pod (no peer label) — `p3-network-bytes.txt` | **the Plex pod sent ~1.5 TB** (143 / 202 / 528 / 441 / 206 GiB per day, 10-18 → 10-22; before: 0.04 GiB/day) and received ≤ 7.9 GiB/day. DooPlex's NIC `eno1` sent the same extra out. Outbound-heavy: the shape of a flood tool or of streaming, not of a download. It did **not** come from the NFS store (nfsd reads flat) or from other pods (bridge-to-pod bytes flat) |

## Part 4 — Felhom's secrets on DooPlex

Exposure = what a path could read, judged with Part 3. „Possible path" means the door existed; it does not mean it was used.

| secret | where it lives | path that could read it (Part 2) | evidence of use | exposure |
|---|---|---|---|---|
| hub DB: Cloudflare API tokens, tunnel tokens, git tokens (**unsealed**, R-975) | `hub.db` on Longhorn `hub-data`; **unencrypted nightly copies in the NFS store** | NFS export (direct, no escape needed); Longhorn API backup to an outside target | none (Part 3) | **none** |
| hub DB: box API keys, controller keys, owner passphrases, PBS-DR tokens, break-glass passwords, Storage Box sub-account passwords (**sealed**, AES-GCM, key `OFFSITE_SECRET_KEY`) | same | same — but the key is a k8s Secret, not in the DB or its backups | none | **none** (sealed values useless without the key) |
| escrow blobs `host_escrow.blob` / `identity_blob` | same | same | none | **none** — sealed means: wrapped by each household's recovery code, which the hub never holds (`03-host-agent.md`, escrow retention: „the hub never decrypts either table") |
| hub tokens (`api_tokens.token_hash`), self-bind / claim codes | same, hashed | same | none | **none** (hashes) |
| k8s Secrets in `felhom-system` (report-api global key, offsite-secret-key, storagebox = main Storage Box/Hetzner login, resend-api, tenantsync, agent-poke, wg-endpoint-ssh, gitea-creds, umami, TLS keys) | etcd/k3s datastore (`/var/lib/rancher`, sdb1); also the nightly GPG secrets export on sda1 (`/mnt/5_hdd/backup/secrets/exports`, symmetric, passphrase file on sdb1) | k8s API anonymous: 401. NFS escape → the `.gpg` files only (key not on sda1) | none | **none** |
| hub session / signing material | k8s Secret / hub DB | as above | none | **none** |
| **operator signing keys** (operational + recovery + `felhom_op_ed25519`, no passphrase) | `/mnt/5_hdd/felhom.eu/` on **sda1** | NFS export → file-handle escape (documented, not exercised) | none (no NFS connection at all) | **none** — but the door is still open to any pod (row filed) |
| `sec` store | `cc-secrets/store/cc.sops.yaml` on sda1, age key `~/.config/sops/age/keys.txt` on sdb1 | escape → ciphertext only | none | **none** |
| registry login, Gitea tokens | `~/.docker/config.json`, `~/.git-credentials` (sdb1, 0600) | none from the pod network (registry 401, Gitea API anonymous = public repos only) | — | **none** |
| Vaultwarden (operator's password manager, R-923) | Longhorn volume; nightly unencrypted Longhorn copy in the NFS store (vault items stay encrypted by the master password) | login page; NFS export → encrypted vault DB | none for NFS; login logs lost | **none** (no record of login attempts — logs lost, and the vault is encrypted at rest) |
| off-site ciphertext (ep0 copy, demo-felhom backups) | sda1 | escape → ciphertext only | none | **none** |

## Part 5 — plan

Verdict NOT REACHED → **no rotation plan.** The open doors are filed instead (rows below). Nothing was rotated.

## Observations

- **A hacked pod on DooPlex can still read the hub's unencrypted nightly backups and, by the file-handle escape, the
  operator signing keys through the NFS export.** → FILED: R-1241 (P1).
- **The Felhom demo NAS export `/mnt/5_hdd/felhom-data → 192.168.0.162` is a sub-directory of the same disk with
  `no_subtree_check`**: a compromised demo-felhom host could walk all of sda1 as uid 101000 (world-readable files only,
  e.g. the `.gpg` secrets exports and every repo). → FILED: R-1242 (P3).
- The hub answers pods directly, without the ingress geo filter. → NOT-A-FINDING: the geo filter is not an
  authentication control; every no-token path refused (measured).
- demo-felhom's Proxmox UI and SSH are reachable from any DooPlex pod (masqueraded as 192.168.0.180). → NOT-A-FINDING:
  same door as R-982 (root password SSH from the LAN), now with „a hacked DooPlex pod" as one more LAN attacker; added
  to `13` §3.
- Pre-reboot pod logs were erased by the 13:20Z reboot. → NOT-A-FINDING for Felhom: the hub's own access log
  persisted (R-1140 did its job). Homelab item, below.

## For the homelab session (no HM numbers minted; evidence paths in this folder)

1. **The NFS export is still open to every pod**, `no_root_squash`, `no_subtree_check`, on the disk that holds the
   Felhom signing keys. `secure` does nothing in pods (containerd sets `ip_unprivileged_port_start=0`). Options: an
   egress NetworkPolicy so only `longhorn-system` reaches :2049/:111, and/or move the store to its own filesystem
   (LV or loop) so the subtree escape has nothing else to reach. Not managed by Ansible — the narrowed line persists.
2. **Longhorn manager API and UI need no login** from any pod (create backups/targets/volumes, delete volumes).
3. **The old Plex volume was deleted through Longhorn's API at 11:24:15Z** by nobody identified (`INCIDENT.md`
   EVIDENCE NOTE). Ask the operator whether it was them; if not, it is an unattributed write on an open API.
4. **The Plex pod sent ~1.5 TB out on 10-18 → 10-22** (not in INCIDENT.md); and CoreDNS saw `hand.genddos.st`
   lookups on 10-19. Possibly a flood tool riding the same compromise.
5. **The maintenance reboot at 13:20Z erased the window's pod logs** (longhorn-manager, ingress, hub, Gitea, …): no log
   shipping, kubelet keeps one previous container log. Evidence rule R-320 applies to the homelab too.
6. Prometheus, Alertmanager (silences!), Grafana (anonymous Viewer), kube-state-metrics, node-exporter, MinIO console,
   Headlamp (its service account can read every Secret and write RBAC — behind OIDC) all answer every pod.
7. A user process `python3 main.py -d -i` (kisfenyo) listens on 0.0.0.0:8443 on DooPlex.