admin
8257ff1cbe
gates / gates (push) Successful in 12m37s
Read-only audit of whether the hacked Plex pod on DooPlex (HM-040, 2026-10-17 -> 10-23) reached Felhom's secrets. Two no-login doors existed (the Longhorn NFS export and the Longhorn API); neither was used: nfsd accepted 0 new connections in the window (positive control), no off-schedule Longhorn backup or volume. Evidence in documentation/audits/plex2shell-felhom-impact-2026-10-23/. The NFS export was narrowed from * to 10.42.0.0/16 on the operator's yes (host change, not in git). 13-threat-model.md: attacker j and one boundary row. STATUS: verdict, the P1, one do item.
149 lines
16 KiB
Markdown
149 lines
16 KiB
Markdown
# Plex2Shell on DooPlex — did the attack reach Felhom's secrets? (2026-10-23)
|
||
|
||
Read-only investigation. Source of the incident facts: `~/incident-plex-2026-10-23/INCIDENT.md` on DooPlex (HM-040,
|
||
homelab session). Architecture read first: `architecture/01-topology-and-trust.md`, `04-control-plane-authorization.md`
|
||
§3.1, `05-hub-architecture.md` §16.2, `13-threat-model.md`, `runbooks/secrets.md`, `runbooks/total-loss-of-dooplex.md`,
|
||
`runbooks/target-selection.md` (DooPlex = Tier 2). Window: **2026-10-17 04:00Z → 2026-10-23 08:04:34Z** (the isolation
|
||
NetworkPolicy `incident-2026-10-23-plex-isolate`, `creationTimestamp` in INCIDENT.md Phase 2a).
|
||
|
||
## Verdict: **NOT REACHED**
|
||
|
||
Two open paths led from the hacked Plex pod to Felhom secrets. Each would have left a trace in a record that survived,
|
||
and neither trace exists:
|
||
|
||
1. **NFS export** (`/mnt/5_hdd/backup/longhorn-pvc`, `*(rw,no_root_squash,no_subtree_check)`): the NFS server accepted
|
||
**zero new TCP connections from 2026-10-10 00:00Z to 2026-10-23 13:05Z** — `node_nfsd_connections_total` flat at 288,
|
||
3902 five-minute samples, no gap. Every NFS request needs a TCP connection (nfsd listens on TCP only: `portlist`,
|
||
`rpcinfo`). **Positive control:** after this session's probe pod made three TCP connections to :2049 today the counter
|
||
read 4 (one Longhorn client + three probes), in `/proc/net/rpc/nfsd` and in Prometheus alike. The only NFS client in
|
||
the window was Longhorn's own manager (one long-lived NFSv4.2 connection). Read operations happened only in Longhorn's
|
||
backup hours; bytes read per day stayed at 18–39 MiB.
|
||
2. **Longhorn manager API** (no login, port 9500 — can create backups, backup targets, volumes): no backup was created
|
||
outside the two recurring jobs' times (`longhorn_backup_state` series, 1-minute scrape); every scheduled night of the
|
||
window wrote its backups to DooPlex's own NFS store (so the target was local each night); no volume existed in the
|
||
window that is not explained; the backup target object is the one created 2025-11-29; DooPlex's network card sent
|
||
no more than the Plex pod itself sent.
|
||
|
||
**What would change this verdict:** evidence of a container escape to the host (the incident session found none,
|
||
`INCIDENT.md` „Anything OUTSIDE the Plex pod"), or a Longhorn target swap + backup + delete inside one minute (no
|
||
record either way; Longhorn manager logs were lost, below).
|
||
|
||
## Part 1 — the shared backup folder
|
||
|
||
| what | value | evidence |
|
||
|---|---|---|
|
||
| export (before) | `/mnt/5_hdd/backup/longhorn-pvc *(rw,sync,no_subtree_check,no_root_squash)` | `p1-nfs-exports-clients.txt` |
|
||
| who can reach :2049 | the host firewall accepts 192.168.0.0/24, 10.42.0.0/16 (pods), 10.43.0.0/16, 100.64.0.0/10 (tailnet) | `nft list ruleset`, input chain |
|
||
| `secure` option | **no protection inside a pod**: containerd 2.x sets `net.ipv4.ip_unprivileged_port_start=0` in every pod; a uid-1000 probe with CapEff 0 bound port 700 and connected to :2049 | `p2-targeted-probes.txt` (last line), sysctl read in the Plex pod netns = 0, host = 1024 |
|
||
| filesystem | the export is a sub-directory of `/dev/sda1` (ext4, mounted at `/mnt/5_hdd`); root inode 2 has generation 0 | `findmnt`, `lsattr -vd /mnt/5_hdd` |
|
||
| contents | Longhorn backup store, 119 GB: nightly backups of every Longhorn volume, **including `felhom-system/hub-data` (the hub database), NOT encrypted** (`spec.encrypted=false` on all four felhom-system volumes) | `p1-backup-folder-listing.txt` |
|
||
| v4 pseudo-root | `/`, `/mnt`, `/mnt/5_hdd`, `/mnt/5_hdd/backup` appear read-only as path components only; no other directory under `/mnt/5_hdd/backup` is exported | `/proc/fs/nfsd/exports` |
|
||
| clients now | one: `longhorn-manager-w4tf6` 10.42.0.214, NFSv4.2 | `/proc/fs/nfsd/clients/*/info` |
|
||
| NFS logging | NFSv3 mounts are logged only at debug level; NFSv4 attaches are not logged at all (**positive control:** client 3 is attached and the journal has no attach line); detaches are logged (2 lines in boot -1, none in the window); illegal-port refusals are logged (none). An absent line proves nothing for v4. | `journalctl -t rpc.mountd`, `strings rpc.mountd` |
|
||
|
||
**No subtree check matters beyond the folder.** With `no_subtree_check` on a sub-directory export, the server checks
|
||
only that a file handle belongs to the exported *filesystem*. A published technique (HvS-Consulting's
|
||
`nfs-security-tooling`, 2024) builds the handle of the filesystem's root directory and walks the whole disk; with
|
||
`no_root_squash` it does so as root. On `/dev/sda1` that reaches the **operator signing keys** (`/mnt/5_hdd/felhom.eu/
|
||
felhom-op-operational`, `felhom-rec-recovery`, `felhom_op_ed25519`, no passphrase, R-924), the private security-review
|
||
evidence, the workspace, the encrypted `sec` store (its age key is on `/dev/sdb1`, not reachable) and the other backup
|
||
trees. **Grade: documented technique, every precondition measured here, NOT exercised** (doing it would be an attack on
|
||
a Tier-2 machine).
|
||
|
||
**The change (operator yes, 2026-10-23 ~15:18Z):** the line became
|
||
`/mnt/5_hdd/backup/longhorn-pvc 10.42.0.0/16(rw,sync,no_subtree_check,no_root_squash)`; `exportfs -ra` rc 0. Read back:
|
||
`exportfs -v` and `/proc/fs/nfsd/exports` show `10.42.0.0/16`; Longhorn's backup target synced at 15:20:10Z with
|
||
`available: true`; the Longhorn client is still attached. Backup of the old file: `/etc/exports.bak-2026-10-23-plex2shell`.
|
||
Undo: `sudo cp -p /etc/exports.bak-2026-10-23-plex2shell /etc/exports && sudo exportfs -ra`. **This closes the folder to
|
||
the home network and the tailnet. It does NOT close it to a hacked pod** — Longhorn connects from its pod address, so
|
||
the pod network must stay allowed; limiting it to DooPlex itself would break Longhorn. Root squash is not possible either
|
||
(the store is root-owned 0700 and Longhorn writes as root). `p1-export-change.txt`.
|
||
|
||
## Part 2 — what the hacked Plex could reach without a login
|
||
|
||
Measured from probe pod `default/cc-p2s-probe` (uid 1000, CapEff 0, no service-account token, no NetworkPolicy in
|
||
`default` — the Plex pod's position before the isolation). Created 15:19:46Z, removed 15:33:44Z
|
||
(`p2-probe-lifecycle.txt`, before/after listings). 186 of 196 TCP Service ports answered (`p2-service-sweep.txt`).
|
||
|
||
| target | no-login result | what a no-login caller gets |
|
||
|---|---|---|
|
||
| Longhorn manager API `longhorn-backend:9500`, UI `longhorn-frontend:80` | 200 | every volume (234 names), backup targets, backup volumes; actions `snapshotCreate`, `snapshotBackup`, `attach`, `detach`, `pvcCreate`, `activate` offered on `hub-data` |
|
||
| NFS on the node :2049, :111 | open | see Part 1 |
|
||
| Felhom hub `hub:8080` (bypasses the ingress geo filter) | `/healthz` 200, `/login` 200, every data path 401/403/302 | nothing (matches `13` §4: health, login, register, armed re-enroll) |
|
||
| kube-apiserver 10.43.0.1:443 / 192.168.0.180:6443 | 401 | nothing (k3s `config.yaml` unchanged since 2026-08-05) |
|
||
| kubelet :10250 | 401; :10255 closed | nothing |
|
||
| Prometheus, Grafana (anonymous Viewer), Alertmanager, node-exporter, kube-state-metrics | 200 | metrics and dashboards; Alertmanager also lets anyone create a silence. No secret values |
|
||
| Gitea | 200 | the public repos (R-925 ruling); registry `/v2/` 401 |
|
||
| Vaultwarden | `/alive`, `/api/config`, `/admin` login page 200 | nothing without the master password |
|
||
| ArgoCD, Headlamp, the filebrowsers | 401 on the API | nothing (Headlamp's own service account can read every Secret — behind OIDC) |
|
||
| `felhom-system/umami-db` 5432, other databases | open | password needed |
|
||
| LAN | router admin (.1), a TP-Link (.254), a printer (.20), **demo-felhom's Proxmox UI and SSH (.162)**, PBS UI on DooPlex :8007, Samba :445 | logins needed; `p2-lan-reach.txt` |
|
||
|
||
## Part 3 — evidence of use in the window, per path
|
||
|
||
| path | best record that exists | result |
|
||
|---|---|---|
|
||
| NFS | nfsd counters in Prometheus (connections, ops by method, bytes, RPC errors) — `p3-nfsd-*.json` | **unused** (verdict 1). RPC errors (auth/fmt/cInt) 0 |
|
||
| Longhorn API | `longhorn_backup_state`, `longhorn_volume_state` series; nfsd bytes written per night; Backup/Snapshot/BackupTarget/Volume `creationTimestamp` — `p3-longhorn-*` | **no sign of use**: 106 backups in the window = 53 daily (10-23 02:00Z) + 53 weekly (10-18 03:00Z), all to the local URL; 0 backups first seen off-schedule; no unexplained volume. Longhorn CRs carry **no managedFields**, so who changed what is not recorded there (`p3-longhorn-objects.txt`, CORRECTION) |
|
||
| Longhorn manager/UI logs | — | **no record: lost.** DooPlex was rebooted at 13:20Z by the maintenance session (`~/maintenance/2026-10-window`); the pre-reboot container logs of longhorn-manager, longhorn-ui, the hub, the ingress, Gitea, Vaultwarden, Grafana and Prometheus are gone. The incident session's earlier greps (before the reboot) found 10.42.0.52 in none of longhorn-manager, CNPG, Gitea (from 10-21 20:59), authentik; that is their record, not re-verifiable |
|
||
| hub | `/data/access-log` (R-1140), 10-18 10:25Z → now, 12 764 lines — `p3-hub-access-log.txt` | **no call from the Plex pod** (10.42.0.52: 0; positive control: the probe's 10.42.0.107 is logged). Credentialed requests come only from the boxes' LAN addresses, the home WAN and the operator's sessions. **Gap:** 10-17 04:00Z → 10-18 10:25Z no record (the log started then) — but no hub route gives a secret without a credential |
|
||
| SSH to DooPlex | journal (boot -1 survives) | 0 sshd lines from 10.42/10.43 in the window (control: 61 lines from 192.168.x) |
|
||
| Samba | per-client log files `log.%m` | no file for 10.42.0.52 (positive control: `log.10.42.0.107` appeared when the probe touched :445) — `p3-samba-clients.txt` |
|
||
| Alertmanager | silences API (keeps expired ones) | only two silences, both CC's, both outside the attack |
|
||
| DNS | CoreDNS keeps **no query log**; only upstream-timeout lines name a domain, without the client | names in `p3-coredns-error-names.txt`; `hand.genddos.st` looked up 10 times on 10-19 08:31–09:50Z (client unknown) |
|
||
| network bytes | cAdvisor per pod (no peer label) — `p3-network-bytes.txt` | **the Plex pod sent ~1.5 TB** (143 / 202 / 528 / 441 / 206 GiB per day, 10-18 → 10-22; before: 0.04 GiB/day) and received ≤ 7.9 GiB/day. DooPlex's NIC `eno1` sent the same extra out. Outbound-heavy: the shape of a flood tool or of streaming, not of a download. It did **not** come from the NFS store (nfsd reads flat) or from other pods (bridge-to-pod bytes flat) |
|
||
|
||
## Part 4 — Felhom's secrets on DooPlex
|
||
|
||
Exposure = what a path could read, judged with Part 3. „Possible path" means the door existed; it does not mean it was used.
|
||
|
||
| secret | where it lives | path that could read it (Part 2) | evidence of use | exposure |
|
||
|---|---|---|---|---|
|
||
| hub DB: Cloudflare API tokens, tunnel tokens, git tokens (**unsealed**, R-975) | `hub.db` on Longhorn `hub-data`; **unencrypted nightly copies in the NFS store** | NFS export (direct, no escape needed); Longhorn API backup to an outside target | none (Part 3) | **none** |
|
||
| hub DB: box API keys, controller keys, owner passphrases, PBS-DR tokens, break-glass passwords, Storage Box sub-account passwords (**sealed**, AES-GCM, key `OFFSITE_SECRET_KEY`) | same | same — but the key is a k8s Secret, not in the DB or its backups | none | **none** (sealed values useless without the key) |
|
||
| escrow blobs `host_escrow.blob` / `identity_blob` | same | same | none | **none** — sealed means: wrapped by each household's recovery code, which the hub never holds (`03-host-agent.md`, escrow retention: „the hub never decrypts either table") |
|
||
| hub tokens (`api_tokens.token_hash`), self-bind / claim codes | same, hashed | same | none | **none** (hashes) |
|
||
| k8s Secrets in `felhom-system` (report-api global key, offsite-secret-key, storagebox = main Storage Box/Hetzner login, resend-api, tenantsync, agent-poke, wg-endpoint-ssh, gitea-creds, umami, TLS keys) | etcd/k3s datastore (`/var/lib/rancher`, sdb1); also the nightly GPG secrets export on sda1 (`/mnt/5_hdd/backup/secrets/exports`, symmetric, passphrase file on sdb1) | k8s API anonymous: 401. NFS escape → the `.gpg` files only (key not on sda1) | none | **none** |
|
||
| hub session / signing material | k8s Secret / hub DB | as above | none | **none** |
|
||
| **operator signing keys** (operational + recovery + `felhom_op_ed25519`, no passphrase) | `/mnt/5_hdd/felhom.eu/` on **sda1** | NFS export → file-handle escape (documented, not exercised) | none (no NFS connection at all) | **none** — but the door is still open to any pod (row filed) |
|
||
| `sec` store | `cc-secrets/store/cc.sops.yaml` on sda1, age key `~/.config/sops/age/keys.txt` on sdb1 | escape → ciphertext only | none | **none** |
|
||
| registry login, Gitea tokens | `~/.docker/config.json`, `~/.git-credentials` (sdb1, 0600) | none from the pod network (registry 401, Gitea API anonymous = public repos only) | — | **none** |
|
||
| Vaultwarden (operator's password manager, R-923) | Longhorn volume; nightly unencrypted Longhorn copy in the NFS store (vault items stay encrypted by the master password) | login page; NFS export → encrypted vault DB | none for NFS; login logs lost | **none** (no record of login attempts — logs lost, and the vault is encrypted at rest) |
|
||
| off-site ciphertext (ep0 copy, demo-felhom backups) | sda1 | escape → ciphertext only | none | **none** |
|
||
|
||
## Part 5 — plan
|
||
|
||
Verdict NOT REACHED → **no rotation plan.** The open doors are filed instead (rows below). Nothing was rotated.
|
||
|
||
## Observations
|
||
|
||
- **A hacked pod on DooPlex can still read the hub's unencrypted nightly backups and, by the file-handle escape, the
|
||
operator signing keys through the NFS export.** → FILED: R-1241 (P1).
|
||
- **The Felhom demo NAS export `/mnt/5_hdd/felhom-data → 192.168.0.162` is a sub-directory of the same disk with
|
||
`no_subtree_check`**: a compromised demo-felhom host could walk all of sda1 as uid 101000 (world-readable files only,
|
||
e.g. the `.gpg` secrets exports and every repo). → FILED: R-1242 (P3).
|
||
- The hub answers pods directly, without the ingress geo filter. → NOT-A-FINDING: the geo filter is not an
|
||
authentication control; every no-token path refused (measured).
|
||
- demo-felhom's Proxmox UI and SSH are reachable from any DooPlex pod (masqueraded as 192.168.0.180). → NOT-A-FINDING:
|
||
same door as R-982 (root password SSH from the LAN), now with „a hacked DooPlex pod" as one more LAN attacker; added
|
||
to `13` §3.
|
||
- Pre-reboot pod logs were erased by the 13:20Z reboot. → NOT-A-FINDING for Felhom: the hub's own access log
|
||
persisted (R-1140 did its job). Homelab item, below.
|
||
|
||
## For the homelab session (no HM numbers minted; evidence paths in this folder)
|
||
|
||
1. **The NFS export is still open to every pod**, `no_root_squash`, `no_subtree_check`, on the disk that holds the
|
||
Felhom signing keys. `secure` does nothing in pods (containerd sets `ip_unprivileged_port_start=0`). Options: an
|
||
egress NetworkPolicy so only `longhorn-system` reaches :2049/:111, and/or move the store to its own filesystem
|
||
(LV or loop) so the subtree escape has nothing else to reach. Not managed by Ansible — the narrowed line persists.
|
||
2. **Longhorn manager API and UI need no login** from any pod (create backups/targets/volumes, delete volumes).
|
||
3. **The old Plex volume was deleted through Longhorn's API at 11:24:15Z** by nobody identified (`INCIDENT.md`
|
||
EVIDENCE NOTE). Ask the operator whether it was them; if not, it is an unattributed write on an open API.
|
||
4. **The Plex pod sent ~1.5 TB out on 10-18 → 10-22** (not in INCIDENT.md); and CoreDNS saw `hand.genddos.st`
|
||
lookups on 10-19. Possibly a flood tool riding the same compromise.
|
||
5. **The maintenance reboot at 13:20Z erased the window's pod logs** (longhorn-manager, ingress, hub, Gitea, …): no log
|
||
shipping, kubelet keeps one previous container log. Evidence rule R-320 applies to the homelab too.
|
||
6. Prometheus, Alertmanager (silences!), Grafana (anonymous Viewer), kube-state-metrics, node-exporter, MinIO console,
|
||
Headlamp (its service account can read every Secret and write RBAC — behind OIDC) all answer every pod.
|
||
7. A user process `python3 main.py -d -i` (kisfenyo) listens on 0.0.0.0:8443 on DooPlex.
|