night 2026-10-23 so far: D30 live proof (re-key → new-key handshake; ep0-started handshakes heal a short clock jump by themselves), D4 patches v2 after review, test-night install/fixture evidence, DooPlex survival survey (read only) + PLAN, 03/05/07 + CONTEXT, total-loss runbook: MAIL-HOLD shipped

2026-10-22 22:24:48 +02:00
parent 84751cb0ce
commit 7585443227
32 changed files with 734 additions and 2 deletions
@@ -131,7 +131,20 @@ the one trust root; the agent's API token is untouched; a box gets the new rule

 **[DESIGN, ruled 2026-10-22 15:47 (D4, R-982) — built as a ready patch, NOT shipped] Root SSH from the LAN by key only; the console password stays.** The config bundle gains one fixed file, `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`), checked and reloaded by `felhom-os-apply`; the installer's uninstall removes it. Why a drop-in: Debian's `sshd_config` Includes `sshd_config.d/*` FIRST and sshd keeps the first value it reads, so the drop-in wins over the stock line; the out-of-band operator sshd (`FELHOM_SSHD`, port 8822) has its own config with no Include and is not touched. Proven on a fresh ISO 1.30.0 install: password SSH refused after, key SSH and the PVE web password login (200) and the console login still work (`audits/night-2026-10-22/P2-D4/`). It ships with the next agent release + installer tag.

-**[FACT, agent main `a0c88e5`, unreleased] The heartbeat reports a WireGuard handshake from the future (R-1204).** WireGuard refuses an initiation whose TAI64N time is not newer than the last one accepted from that peer, so a box that once ran with its clock ahead is locked out of ep0 until real time passes that moment. The agent used to clamp the negative age to 0 (it looked healthy); `wgtunnel.handshakeFuture` now sets `handshake_future_s` when the handshake is more than 60 s ahead (`TestR1204_AFutureHandshakeIsReported`). The hub alarms on it (`05` §15.4b). The repair is STATUS decision D30.
+**[FACT, agent main `a0c88e5`, unreleased] The heartbeat reports a WireGuard handshake from the future (R-1204).** WireGuard refuses an initiation whose TAI64N time is not newer than the last one accepted from that peer, so a box that once ran with its clock ahead is locked out of ep0 until real time passes that moment. The agent used to clamp the negative age to 0 (it looked healthy); `wgtunnel.handshakeFuture` now sets `handshake_future_s` when the handshake is more than 60 s ahead (`TestR1204_AFutureHandshakeIsReported`). The hub alarms on it (`05` §15.4b).
+
+**[DESIGN, operator ruling D30 2026-10-22; agent main `ae69ad4`, unreleased] The tunnel heals itself by a new key.** When
+the latest handshake stays stamped in the future for 15 minutes, the agent makes a new WireGuard key pair (the old
+private key kept as `wg/private.key.prev`, 0600 — the escrow may hold it) and registers the new public key on the
+existing path, `POST /hosts/<id>/wg`: the hub re-keys the peer in place (same /32, generation bump) and its full-list
+push to ep0 (`wgsync`, `wg syncconf`) drops the old key and adds the new one. A new key is a new peer for ep0, with no
+stored timestamp. Once an hour at most; not while the unit is down; never during a re-register already in flight. The
+interface switches at the agent's next desired-state fetch (up to its report interval). **`[FACT]` 2026-10-23 night
+(test box Tester4-16fdef, `audits/night-2026-10-23/P2-D30/`):** with ep0 free to start a handshake of its own, a 2-hour
+clock jump healed by itself in ~3 minutes (ep0 had traffic for the box; the box accepts ep0's timestamps) and the agent
+correctly did not re-key; with ep0's handshake starts dropped on the box (the rehearsal's condition), the tunnel stayed
+dead and the agent re-keyed after 15 minutes. **Known gap:** a reboot after the clock was corrected forgets the
+future stamp (R-1223).

 ## 4. Control model — reconcile + signed destructive ops

@@ -371,6 +371,7 @@ token id, fingerprint, datastore, tunnel IP, `secret_generation`) with its consu
 | **re-issue, descriptor ABSENT, DR flag on** (R-511, v0.114.0) | **adopted**: re-keyed | **rebuilt** from the endpoint's answer; `pbsdr_adopted` audit row |
 | host delete | **kept** (tenancy survives) | goes with the host |
 | host delete acknowledged through escrow-ack, then a new box | re-keyed automatically (F-14) | rebuilt |
+| …and a later install deleted before it made a key, then a new box (D31, operator 2026-10-22; hub 0.167.0 source) | re-keyed automatically — the gate reads the newest deletion of a host that HELD A KEY (`LatestKeyHoldingHostDeletion`) | rebuilt |
 | RESET / customer delete | **deprovisioned** — namespace, every backup group AND token | purged |

 **Why adopt exists:** a rebuilt box's WG hook refused („the endpoint already holds a PBS token … use
@@ -459,6 +460,12 @@ date taken) are shown on its own app page, on its box, until it dismisses them

 **[FACT, hub `e267c5e2`]** The host report carries the agent's `wireguard` block (`hostWireguard`). `checkWGTunnel` (`api/host_tunnel.go`, called after `checkHostSwap`) calls the tunnel stale when the box reports `handshake_future_s` > 0 (the clock-ahead lock-out) or a last handshake older than 6 h (healthy boxes measured at most 113–124 s over 24 h). It is edge-triggered against the host's latest tunnel event (`store.LatestHostTunnelEvent`): `wg_tunnel_stale` once on the way down, `wg_tunnel_ok` once on the way back; both are allow-listed and operator-only (`notify.operatorOnlyEvents`, `wg_tunnel_operator_only_test.go`) — the household cannot repair a tunnel. A box with no `wireguard` block (an agent before the field, or no tunnel) raises nothing.

+**[FACT, hub `84751cb0`, 0.167.0 source] The repair clears it (D30).** An agent with D30 (`03` §R-1204) registers a new key
+after 15 minutes of a future-stamped handshake; the new key's first handshake makes the report fresh, and the alarm
+records `wg_tunnel_ok` once (`TestD30_TheAlarmClearsWhenTheNewKeyShakesHands`). `wg_tunnel_ok` is an info event:
+stored, not mailed. **Known gaps (rows):** a reboot after the clock was corrected sends no handshake at all and is not
+judged (R-1223); the operator's mail cooldown is per customer, not per host (R-1224).
+
 ### 15.5 How "the Hungarian did not change" is known

 56 goldens captured from v0.117.0 before any string moved, in
@@ -1590,6 +1590,21 @@ or pattern. **Why:** the data-loss hunt of 2026-10-20 proved two nights that del
 (R-1175: one share group's reconcile removed the other groups' copies, and a dead drive's copy; R-1181: a missing or
 emptied folder took its copy within one night). Controller: `internal/backup/tier2_kept.go`.

+### 8.2b The second-drive copy runs in a memory-capped container that never outlives the copy [DESIGN + FACT, R-1214, 2026-10-22/23 nights; rides controller 0.340.0]
+
+The copy (`rsync -a --delete`) runs in a container of the controller's own image, capped at 512 MB, so its page cache
+is reclaimed inside the cap instead of from the apps (R-1214: full stall 5.7 s → 0.37 s on 9202). Only for paths the
+controller and the Docker daemon see alike (`/mnt/…`, no symlink); the daemon must see the path itself (`--mount`
+refuses a missing one — then the in-process copy runs). **The container must never outlive the copy** (the stray-copy
+fix, 2026-10-23 night): a fixed name and the label `felhom.tier2-copy=1`; on any error, the timeout included, it is
+removed by name; at controller start, before the scheduler and the web server, every labelled container is removed;
+and a restore is refused while any second-drive copy runs or a leftover may still run (the other direction of R-1180 —
+a `--delete` beside a restore removes what the restore brings back). **`[FACT]` 9202, rc.5
+(`audits/night-2026-10-23/P1-straycopy/`):** a controller restart in the middle of a 16 GB copy → the new controller
+removed the old container at start; a restore probe during the copy was refused; the redone copy equals the source; a
+15-second test limit → the container removed, the partial copy stopped growing. Controller:
+`internal/backup/tier2_copy_lifecycle.go`.
+
 ### 8.1 The blank cells, listed explicitly

 Per the rule that a blank is a finding, here they are:
@@ -0,0 +1,70 @@
+# DooPlex survival survey — night 2026-10-22 → 23 (read only)
+
+Night brief Part 5. **Nothing on DooPlex was changed.** Every number below was read on DooPlex tonight (unit state,
+journal lines, `du`, `kubectl get`, DNS, HTTP status) — raw readings in `readings/`. Secrets are named, never read:
+no key file, token file or password was opened. ep0 was not contacted (the nightly jobs' own journal lines are the
+evidence of what reached it). Read with `runbooks/total-loss-of-dooplex.md` (the order of steps) and the 2026-10-09
+plan (`audits/dooplex-survival-2026-10-09/PLAN.md`, which built the nightly copy).
+
+## 1. What lives on DooPlex, and what saves it
+
+| Thing | Where it lives | What backs it up | Where the copy is | Last copy (measured) | Who can restore | How long |
+|---|---|---|---|---|---|---|
+| **Gitea** — 12 repositories (845 MB), DB in CNPG, `app.ini` | PVC `gitea-data` (Longhorn, `sdb1`) | (1) 6-hourly `pg_dump` + Longhorn backup, both on `sda1` (same box); (2) `felhom-dooplex-offsite` nightly 00:20 | ep0, namespace `operator`, `host/dooplex-gitea`, encrypted (key **S1**) | 2026-10-22 00:22, 736 MiB (425 MiB new), 42 s, success | the operator with S1 + the read token S7 (or ep0 root) | restore ~1 min + start ~1 min on a ready machine (walked 2026-10-09 Part C); on new hardware add k3s/Docker install — hours |
+| **Weekly proof of that copy** | DooPlex timer, Sunday 05:30 | — | — | 2026-10-18: 27 739 files match, **11** repositories pass `git fsck`, `gitea.dump` readable, Vaultwarden 1 user / 810 items | — | — |
+| **Container registry + generic packages** (controller/hub images, agent packages) | `/data/gitea/packages` in `gitea-data`, **38.6 GB** | Longhorn backup on `sda1` only | **nowhere off DooPlex** (left out on purpose 2026-10-09: „rebuilt from the code") | — | rebuild from the restored code on a build machine | never timed |
+| **Hub + its DB** (`hub.db` ~493 MiB) | PVC `hub-data` (Longhorn) | Longhorn backup (same box); `felhom-hub-db-backup` nightly 02:30 | ep0 `operator`, `host/dooplex-hub`, encrypted (key **S2**); seal key **S4** needed to open its sealed passwords | 2026-10-22 02:31, 493 MiB (24 MiB compressed), 6 s, success | the operator with S2 + S4 + S7 | ~5 min on a bench with k3s ready (2026-10-09 Part D) |
+| **Weekly proof of that copy** | Sunday 04:31 | — | — | 2026-10-18: integrity ok, 4 hosts, 4 sealed console passwords, 0 readable | — | — |
+| **Operator signing keys** (3 files, 0600, no passphrase) | `/mnt/5_hdd/felhom.eu/` on `sda1` (the same disk as the local backup tree) | inside the nightly `host/dooplex-gitea` copy (`signing/`; a missing key refuses the push) | ep0, behind S1; paper (S6) **if printed** | with the Gitea copy (2026-10-22) | the operator with S1, or the printout | minutes |
+| **Password manager** (Vaultwarden, 1 user, 810 items) | PVC `vaultwarden-data` (Longhorn) | Longhorn backup (same box); nightly copy (its own `vaultwarden backup` + `rsa_key.pem`) | ep0, behind S1; the vault itself opens with the master password S3 | with the Gitea copy (2026-10-22) | the operator with S1 + S3 | started once on a throwaway (2026-10-09) |
+| **k8s Secrets** (every Secret, GPG-encrypted with S5) | nightly export `/mnt/5_hdd/backup/secrets` (136 MB, 30 days) | the same file rides the nightly copy | ep0 behind S1, then S5 | 2026-10-22 | the operator with S1 + S5 | minutes |
+| **CI** (`act-runner` pod, PVC `act-runner-data`) | k3s | nothing needed: re-registered from Gitea | — | — | the operator | ~15 min |
+| **Website, installer script, `hub.felhom.eu`, `gitea.dooplex.hu`** | DooPlex, behind the home line (all four resolve to the home address; measured) | the website and installer are code in the Gitea copy | ep0 (as code) | — | the operator + a DNS change at Cloudflare (S11) | after Gitea + k3s |
+| **The ISO** | Cloudflare R2 (`iso.felhom.eu`) | — | not on DooPlex | — | — | survives |
+| **ArgoCD** | k3s; Applications from `homelab-manifests` (in the Gitea copy) | its Secrets in the export | ep0 behind S1 + S5 | — | the operator | ~30 min; **the per-app auto-sync settings are set on the live objects, not in git** (several apps differ) |
+| **DooPlex's own local backups** (R-232) | `/mnt/5_hdd/backup` on `sda1`: Longhorn 117 GB, `data` 53 GB, PostgreSQL dumps 2.1 GB, `ep0-copy` 21 GB, … | `dooplex-backup` nightly 03:00 | **the same box** | 2026-10-22 03:16, „All backup components completed successfully" | — | — |
+
+**Restore tests:** both weekly tests passed on 2026-10-18 (their next run is Sunday 2026-10-25). Since then the pod
+lists **12** repositories against the test's 11; the nightly push refuses a copy with fewer repositories than the pod
+lists, and it succeeded, so the twelfth is in the copy — the next Sunday test will show it.
+
+## 2. `total-loss-of-dooplex.md`, step by step — works today / stale / never tried
+
+| Step | Verdict | Why |
+|---|---|---|
+| 1. A machine and the sheet | **depends** | the sheet's printing is the operator's (R-923, R-924 „LEFT"); CC cannot see whether it exists |
+| 2. Reach ep0's PBS | works **from DooPlex** (the nightly jobs use the tunnel); **never tried** from new hardware (WireGuard peer, Hetzner console) | |
+| 3. A read token (S7) | works today | the weekly restore tests use it |
+| 4. Restore `host/dooplex-gitea` with S1 | works today | the weekly test restores and checks it (2026-10-18) |
+| 5. Vaultwarden first | works (once) | started on a throwaway 2026-10-09; its rows are checked weekly |
+| 6. The Secrets export with S5 | works (once) | opened 2026-10-09; **the form of S5's offline copy is unknown to CC** |
+| 7. Gitea, then rebuild the images | Gitea: works (once, 2026-10-09). Image rebuild: **never tried** | the registry is not copied |
+| 8. The hub with S2 + S4, mail held | works (once, 2026-10-09 Part D) | **stale text:** it says the mail hold comes „from the next hub release" — it shipped in hub 0.145.0 (2026-10-10); fixed tonight |
+| 9. DNS at Cloudflare | **never tried** | |
+| 10. Signing | keys readable (the weekly test derives each public half) — **signing from the restored keys never tried** | |
+| 11. Backups again | **never tried** on new hardware | `install.sh` exists in the code |
+
+## 3. The gaps — two options each, with CC's pick
+
+| # | Gap | Option A (cost: money / operator hours / risk) | Option B (cost) | CC's pick |
+|---|---|---|---|---|
+| G1 | **The paper sheet** — S1 opens every off-site copy (Gitea, Vaultwarden, signing keys, Secrets); without it no copy opens (R-923, R-924) | print `break-glass-sheet.md` filled (S1, S2, S4, S5, S6, S7, S8, S11, ep0 fingerprint) and store it outside the home, sealed — 0 € / ~30 min / paper can be lost or read | the key files on an encrypted USB stick kept outside the home + its passphrase in the head — ~10 € / ~1 h / one more secret to remember, a stick can die | **A** — it is the written plan; nothing else breaks the circle |
+| G2 | **Signing keys without a passphrase** (R-924) | keep as the custody ruling (2026-09-16) says until the first sale; they ride the S1-encrypted copy — 0 € / 0 h / a DooPlex break-in can sign | a passphrase on the operational key and the recovery key off DooPlex — 0 € / ~2 h / every signed job then needs the operator; no unattended signing | **A** now; revisit at the first sale (the ruling stands) |
+| G3 | **The password manager holds keys to DooPlex's own copies** (R-923) | the sheet (G1) breaks the circle — no extra cost | a monthly encrypted export to the operator's laptop — 0 € / 10 min a month / easy to forget | **A** (same act as G1) |
+| G4 | **DooPlex's backups sit inside the same box** (R-232) — the Felhom parts are off-site now; the homelab apps, Authentik, Prometheus are not | accept for the homelab (not Felhom) — 0 € / 0 h / a homelab loss is total | restic of the chosen `/mnt/5_hdd/backup` sets to a Storage Box sub-account — ~4 €/month / ~2 h / ~170 GB first upload | **A** for Felhom's scope; the homelab is the operator's call |
+| G5 | **Hub DB copy is up to 24 h old** — a box that enrolled or re-escrowed after 02:30 has no escrow row in a restored hub | keep nightly — 0 € / 0 h / up to a day of hub changes lost | push every 6 h (same job, same token, same prune) — 0 € / ~30 min (a DooPlex change, operator's yes) / ~4 × 24 MB a day on ep0 | **B** |
+| G6 | **Gitea repositories off-site** — covered nightly, tested weekly | keep — 0 € | a second off-site mirror (Codeberg/GitHub private) — 0 € / ~1 h / code at a third party | **A** |
+| G7 | **The registry and the agent packages exist only on DooPlex** — every box update, every reinstall and every new install pull from them | copy only what a box can be offered (the released controller/hub tags, the agent packages, the infra images) nightly to ep0 `operator` — 0 € / ~2 h CC + the operator's yes for a DooPlex change / a few GB on ep0 | a timed rebuild-from-code drill, then accept the outage — 0 € / ~3 h once / a rebuilt image is not the same digest (the „a version is pushed once" rule) | **A** |
+| G8 | **Website, installer script and the hub all on the home line** — a DooPlex loss stops new installs and the public site | accept: Gitea and the hub come back first, then the site — 0 € / hours on the day / no installs meanwhile | serve the static site and the installer tag from Cloudflare (as the ISO already is) — 0 € / ~2 h / a second publish path to keep true | **B**, after the release week |
+| G9 | **ArgoCD's per-app sync settings live only on the cluster** | write them into `homelab-manifests` — 0 € / ~30 min / none | re-set by hand after a loss from the memory notes — 0 € / ~1 h on the day | **A** |
+| G10 | **S5's offline copy: form unknown to CC** | put S5 on the sheet (G1) — 0 € | re-encrypt the Secrets export with an S1-derived key — 0 € / ~1 h / a DooPlex change | **A** |
+
+**The three biggest:** G1 (no sheet → nothing opens), G7 (no registry copy → no update or install after a loss),
+G8 (the website, installer and hub on one home line → new installs stop for hours or days).
+
+## 4. What tomorrow's hour decides (also in `STATUS.md`)
+
+1. G1 — print the sheet now? (CC: yes)
+2. G7 — copy the release images and agent packages to ep0 nightly? (CC: yes)
+3. G5 — hub DB every 6 hours? (CC: yes)
+4. G8 — the site and installer on Cloudflare after the release week? (CC: yes, later)
@@ -0,0 +1,7 @@
+=== felhom-dooplex-offsite: success Thu 2026-10-22 00:22:53 CEST 0 
+=== felhom-hub-db-backup: success Thu 2026-10-22 02:31:31 CEST 0 
+=== dooplex-backup: success Thu 2026-10-22 03:16:26 CEST 0 
+=== felhom-hub-db-restore-test: success Sun 2026-10-18 04:31:19 CEST 0 
+=== felhom-dooplex-offsite-restore-test: success Sun 2026-10-18 05:33:45 CEST 0 
+=== backup-freshness: success Thu 2026-10-22 21:30:01 CEST 0 
+=== felhom-ep0-copy-gc: success Thu 2026-10-22 08:00:00 CEST 0 
@@ -0,0 +1,43 @@
+=== felhom-dooplex-offsite
+2026-10-22T00:22:09+02:00 dooplex felhom-dooplex-offsite[1390185]: Downloading previous manifest (Wed Oct 21 00:21:51 2026)
+2026-10-22T00:22:09+02:00 dooplex felhom-dooplex-offsite[1390185]: Upload directory '/var/lib/felhom-dooplex-offsite/stage/root' to 'dooplex-hub@pbs!push@127.0.0.1:18007:felhom-offsite' as dooplex.pxar.didx
+2026-10-22T00:22:49+02:00 dooplex felhom-dooplex-offsite[1390185]: dooplex.pxar: had to backup 424.942 MiB of 736.086 MiB (compressed 415.906 MiB) in 40.05 s (average 10.611 MiB/s)
+2026-10-22T00:22:49+02:00 dooplex felhom-dooplex-offsite[1390185]: dooplex.pxar: backup was done incrementally, reused 311.144 MiB (42.3%)
+2026-10-22T00:22:49+02:00 dooplex felhom-dooplex-offsite[1390185]: Uploaded backup catalog (1.402 MiB)
+2026-10-22T00:22:51+02:00 dooplex felhom-dooplex-offsite[1390185]: Duration: 42.46s
+2026-10-22T00:22:51+02:00 dooplex felhom-dooplex-offsite[1390185]: End Time: Thu Oct 22 00:22:51 2026
+2026-10-22T00:22:51+02:00 dooplex felhom-dooplex-offsite[1375323]: felhom-dooplex-offsite: pushed to ep0 (ns operator, host/dooplex-gitea) in 42 s
+2026-10-22T00:22:51+02:00 dooplex felhom-dooplex-offsite[1375323]: felhom-dooplex-offsite: success signal written
+2026-10-22T00:22:53+02:00 dooplex systemd[1]: felhom-dooplex-offsite.service: Deactivated successfully.
+2026-10-22T00:22:53+02:00 dooplex systemd[1]: Finished felhom-dooplex-offsite.service - Felhom: push Gitea + DooPlex secrets to ep0, encrypted (R-232).
+2026-10-22T00:22:53+02:00 dooplex systemd[1]: felhom-dooplex-offsite.service: Consumed 36.202s CPU time, 1.6G memory peak.
+=== felhom-hub-db-backup
+2026-10-22T02:31:18+02:00 dooplex felhom-hub-db-backup[2573155]: Encryption key fingerprint: b2:19:bf:36:3b:97:3d:6c
+2026-10-22T02:31:18+02:00 dooplex felhom-hub-db-backup[2573155]: Downloading previous manifest (Wed Oct 21 02:30:55 2026)
+2026-10-22T02:31:18+02:00 dooplex felhom-hub-db-backup[2573155]: Upload directory '/var/lib/felhom-hub-backup/stage' to 'dooplex-hub@pbs!push@127.0.0.1:18007:felhom-offsite' as hubdb.pxar.didx
+2026-10-22T02:31:24+02:00 dooplex felhom-hub-db-backup[2573155]: hubdb.pxar: had to backup 492.879 MiB of 492.879 MiB (compressed 23.873 MiB) in 5.17 s (average 95.306 MiB/s)
+2026-10-22T02:31:24+02:00 dooplex felhom-hub-db-backup[2573155]: Uploaded backup catalog (56 B)
+2026-10-22T02:31:24+02:00 dooplex felhom-hub-db-backup[2573155]: Duration: 5.76s
+2026-10-22T02:31:24+02:00 dooplex felhom-hub-db-backup[2573155]: End Time: Thu Oct 22 02:31:24 2026
+2026-10-22T02:31:24+02:00 dooplex felhom-hub-db-backup[2570694]: felhom-hub-db-backup: pushed hub-20261022T000000Z.db to ep0 (ns operator) in 6 s
+2026-10-22T02:31:24+02:00 dooplex felhom-hub-db-backup[2570694]: felhom-hub-db-backup: success signal written
+2026-10-22T02:31:31+02:00 dooplex systemd[1]: felhom-hub-db-backup.service: Deactivated successfully.
+2026-10-22T02:31:31+02:00 dooplex systemd[1]: Finished felhom-hub-db-backup.service - Felhom: push the hub DB snapshot to ep0 (R-173).
+2026-10-22T02:31:31+02:00 dooplex systemd[1]: felhom-hub-db-backup.service: Consumed 16.683s CPU time, 766M memory peak.
+=== dooplex-backup
+2026-10-22T03:16:26+02:00 dooplex backup-all.sh[2932973]: 115G        /mnt/5_hdd/backup/longhorn-pvc
+2026-10-22T03:16:26+02:00 dooplex backup-all.sh[2932973]: 15G        /mnt/5_hdd/backup/minecraft-tlauncher
+2026-10-22T03:16:26+02:00 dooplex backup-all.sh[2932973]: 35M        /mnt/5_hdd/backup/paperless
+2026-10-22T03:16:26+02:00 dooplex backup-all.sh[2932973]: 2.1G        /mnt/5_hdd/backup/postgresql
+2026-10-22T03:16:26+02:00 dooplex backup-all.sh[2932973]: 992K        /mnt/5_hdd/backup/romm
+2026-10-22T03:16:26+02:00 dooplex backup-all.sh[2932973]: 136M        /mnt/5_hdd/backup/secrets
+2026-10-22T03:16:26+02:00 dooplex backup-all.sh[2932973]: 4.8M        /mnt/5_hdd/backup/sparkyfitness
+2026-10-22T03:16:26+02:00 dooplex backup-all.sh[2970506]: [2026-10-22 03:16:26] [INFO] All backup components completed successfully!
+2026-10-22T03:16:26+02:00 dooplex backup-all.sh[2970509]: [2026-10-22 03:16:26] [INFO] ========================================================
+2026-10-22T03:16:26+02:00 dooplex systemd[1]: dooplex-backup.service: Deactivated successfully.
+2026-10-22T03:16:26+02:00 dooplex systemd[1]: Finished dooplex-backup.service - Dooplex Cluster Backup.
+2026-10-22T03:16:26+02:00 dooplex systemd[1]: dooplex-backup.service: Consumed 5min 2.211s CPU time, 1.9G memory peak.
+=== felhom-hub-db-restore-test
+-- No entries --
+=== felhom-dooplex-offsite-restore-test
+-- No entries --
@@ -0,0 +1,31 @@
+=== felhom-hub-db-restore-test (last run)
+2026-10-18T04:31:08+02:00 dooplex felhom-hub-db-restore-test[3899842]: Using encryption key from '/etc/felhom-hub-backup/enc.key'..
+2026-10-18T04:31:08+02:00 dooplex felhom-hub-db-restore-test[3899842]: Fingerprint: b2:19:bf:36:3b:97:3d:6c
+2026-10-18T04:31:13+02:00 dooplex felhom-hub-db-restore-test[3899842]: restore complete (458.028 MiB processed in 4.9s, average 93.178 MiB/s)
+2026-10-18T04:31:13+02:00 dooplex felhom-hub-db-restore-test[3899712]: felhom-hub-db-restore-test: checked: integrity ok, 4 host(s), 4 sealed console password(s), 0 readable
+2026-10-18T04:31:13+02:00 dooplex felhom-hub-db-restore-test[3899712]: felhom-hub-db-restore-test: success signal written
+2026-10-18T04:31:19+02:00 dooplex systemd[1]: felhom-hub-db-restore-test.service: Deactivated successfully.
+2026-10-18T04:31:19+02:00 dooplex systemd[1]: Finished felhom-hub-db-restore-test.service - Felhom: restore-test the hub DB copy on ep0 (R-173).
+2026-10-18T04:31:19+02:00 dooplex systemd[1]: felhom-hub-db-restore-test.service: Consumed 5.907s CPU time, 821.2M memory peak.
+=== felhom-dooplex-offsite-restore-test (last run)
+2026-10-18T05:32:13+02:00 dooplex felhom-dooplex-offsite-restore-test[269573]: progress 46% (291.015 MiB of 625.417 MiB in 36.5s, 5.911 MiB/s)
+2026-10-18T05:32:53+02:00 dooplex felhom-dooplex-offsite-restore-test[269573]: progress 77% (485.113 MiB of 625.417 MiB in 1m 17.3s, 4.763 MiB/s)
+2026-10-18T05:33:21+02:00 dooplex felhom-dooplex-offsite-restore-test[269573]: restore complete (625.417 MiB processed in 1m 45.4s, average 5.935 MiB/s)
+2026-10-18T05:33:44+02:00 dooplex felhom-dooplex-offsite-restore-test[269413]: felhom-dooplex-offsite-restore-test: checked: 27739 files match the manifest, 11 repositories pass git fsck, gitea.dump readable, 3 secrets file(s), Vaultwarden 1 user(s) / 810 item
+2026-10-18T05:33:44+02:00 dooplex felhom-dooplex-offsite-restore-test[269413]: felhom-dooplex-offsite-restore-test: success signal written
+2026-10-18T05:33:45+02:00 dooplex systemd[1]: felhom-dooplex-offsite-restore-test.service: Deactivated successfully.
+2026-10-18T05:33:45+02:00 dooplex systemd[1]: Finished felhom-dooplex-offsite-restore-test.service - Felhom: restore-test the Gitea + secrets copy on ep0 (R-232).
+2026-10-18T05:33:45+02:00 dooplex systemd[1]: felhom-dooplex-offsite-restore-test.service: Consumed 35.494s CPU time, 931.3M memory peak.
+=== offsite stage (names + sizes, no contents)
+du: cannot access '/var/lib/felhom-dooplex-offsite/stage/root/*': No such file or directory
+ls: cannot access '/var/lib/felhom-dooplex-offsite/stage/root/signing': No such file or directory
+felhom-backup-failmail
+felhom-backup-failmail@.service
+felhom-dooplex-offsite
+felhom-dooplex-offsite-restore-test
+felhom-dooplex-offsite-restore-test.service
+felhom-dooplex-offsite-restore-test.timer
+felhom-dooplex-offsite.service
+felhom-dooplex-offsite.timer
+install.sh
+test_dooplex_offsite.py
@@ -0,0 +1,11 @@
+=== felhom-ep0-copy-gc last run
+2026-10-22T08:00:00+02:00 dooplex felhom-ep0-copy-gc[1347525]: ep0-copy-gc: done: 6 copy namespace(s), 6 on ep0, 0 absent, 0 to delete, 0 failed
+2026-10-22T08:00:00+02:00 dooplex systemd[1]: Finished felhom-ep0-copy-gc.service - Felhom: remove a deleted customer's namespace from ep0-copy (R-901, decision 181).
+=== gitea repos (pod listing)
+pod gitea-system/gitea-5dbcbf7c7b-4j8qp
+12
+845.4M	/data/git/repositories
+38.6G	/data/gitea/packages
+=== backup-freshness unit
+Description=Backup freshness textfile metrics for localhost
+ExecStart=/usr/local/sbin/backup-freshness.sh
@@ -0,0 +1,32 @@
+=== /mnt/5_hdd/backup (names, sizes)
+53G	/mnt/5_hdd/backup/data
+25G	/mnt/5_hdd/backup/demo-felhom
+21G	/mnt/5_hdd/backup/ep0-copy
+69M	/mnt/5_hdd/backup/homelab-manifests
+41M	/mnt/5_hdd/backup/jarrs.eu
+178M	/mnt/5_hdd/backup/k3s
+2.5M	/mnt/5_hdd/backup/logs
+117G	/mnt/5_hdd/backup/longhorn-pvc
+15G	/mnt/5_hdd/backup/minecraft-tlauncher
+35M	/mnt/5_hdd/backup/paperless
+2.1G	/mnt/5_hdd/backup/postgresql
+992K	/mnt/5_hdd/backup/romm
+136M	/mnt/5_hdd/backup/secrets
+4.8M	/mnt/5_hdd/backup/sparkyfitness
+=== longhorn-pvc sets (names)
+backupstore 
+=== signing key files (names, modes, no contents)
+-rw------- kisfenyo /mnt/5_hdd/felhom.eu/felhom-op-operational
+-rw-rw-r-- kisfenyo /mnt/5_hdd/felhom.eu/felhom-op-operational.pub
+-rw------- kisfenyo /mnt/5_hdd/felhom.eu/felhom-rec-recovery
+-rw-rw-r-- kisfenyo /mnt/5_hdd/felhom.eu/felhom-rec-recovery.pub
+-rw------- kisfenyo /mnt/5_hdd/felhom.eu/felhom_op_ed25519
+-rw-rw-r-- kisfenyo /mnt/5_hdd/felhom.eu/felhom_op_ed25519.pub
+=== same disk? backup tree vs sources
+Filesystem     Mounted on
+/dev/sda1      /mnt/5_hdd
+/dev/sda1      /mnt/5_hdd
+/dev/sdb1      /
+/dev/sdb1      /
+
+[exited with code 0]
@@ -0,0 +1,8 @@
+felhom.eu -> 37.191.56.193 | 200 server=
+www.felhom.eu -> 37.191.56.193 | 200 server=
+hub.felhom.eu -> 37.191.56.193 | 302 server=
+iso.felhom.eu -> 2a06:98c1:3121::1 | 404 server=cloudflare
+gitea.dooplex.hu -> 37.191.56.193 | 200 server=
+=== where the installer script is fetched from (installer docs / website)
+https://iso.felhom.eu/felhom-installer-1.30.0-pve9.2-1.iso
+https://iso.felhom.eu/felhom-installer-1.30.0-pve9.2-1.iso.sha256
@@ -0,0 +1,63 @@
+=== pods: runner / argocd / vaultwarden / hub / website / registry
+actualbudget-system/actualbudget-7fb44cd948-2xqjc Running
+argocd/argocd-application-controller-0 Running
+argocd/argocd-applicationset-controller-746fdcd449-s87sk Running
+argocd/argocd-server-b7bb9755c-s96pt Running
+felhom-system/contact-mailer-5bb869b85b-pqtjt Running
+felhom-system/felhom-webpage-7f5c7968b5-g62t6 Running
+felhom-system/filebrowser-59ff87cd88-c8qqm Running
+felhom-system/hub-676bcc45cc-gnbzp Running
+felhom-system/umami-d84d978f8-95rsv Running
+felhom-system/umami-db-5fd98f59c5-slhcw Running
+gitea-system/act-runner-84ff4656c4-t2857 Running
+gitea-system/gitea-5dbcbf7c7b-4j8qp Running
+vaultwarden-system/vaultwarden-7d967778cf-5mkr5 Running
+=== PVCs of those namespaces (name, size, storageclass)
+felhom-system/filebrowser-db 100Mi longhorn
+felhom-system/filebrowser-files 1Gi longhorn
+felhom-system/hub-data 4Gi longhorn
+felhom-system/umami-db-data 2Gi longhorn
+gitea-system/act-runner-data 5Gi longhorn
+gitea-system/gitea-data 50Gi longhorn
+vaultwarden-system/vaultwarden-data 5Gi longhorn
+=== ArgoCD applications (names, sync policy auto?)
+actualbudget auto={"enabled":true}
+admin-tools auto={"prune":true,"selfHeal":true}
+adventurelog auto={"enabled":true}
+arcade auto={"enabled":true}
+audiobookshelf auto={"enabled":true}
+authentik auto={"prune":true,"selfHeal":true}
+booking auto={"enabled":true}
+bookstack auto={"enabled":true}
+calibre auto={"enabled":true}
+cnpg-operator auto={"enabled":false,"prune":true,"selfHeal":true}
+code-server auto={"enabled":true}
+controller auto={"enabled":true}
+crafty auto={"enabled":true}
+database-system auto=
+external-dns-pihole auto={"enabled":true}
+felhom auto={"enabled":false}
+fileshare auto={"enabled":true}
+gitea auto=
+glance auto={"enabled":true}
+headlamp auto={"enabled":true}
+homepage auto={"enabled":true}
+immich auto={"enabled":true}
+infrastructure auto=
+jarr auto={"enabled":true,"prune":true,"selfHeal":true}
+jarr-nightly auto={"enabled":true,"prune":true,"selfHeal":true}
+kisfenyo auto={"enabled":true}
+monitoring auto=
+nextcloud auto={"enabled":true}
+office auto={"enabled":true}
+opengist auto={"enabled":true}
+orsi auto={"enabled":true}
+outline auto={"enabled":true}
+paperless auto={"enabled":true}
+pihole auto={"enabled":false}
+plantit auto={"enabled":true}
+plex auto={"enabled":true}
+privatebin auto={"enabled":true}
+root-apps auto={"prune":true,"selfHeal":true}
+servarr auto={"enabled":true}
+storage auto={"enabled":true}
@@ -0,0 +1,4 @@
+2026-10-22T19:45:26Z real-ish | wg pub 3q1B4aHW… | latest_handshake=1792698270 (now 1792698326, diff -56 s) | prev key file: absent
+== clock set +2 h: 2026-10-22T21:45:26Z
+== a handshake under the ahead clock at 21:46:32 (epoch 1792705591)
+2026-10-22T21:46:32Z real-ish | wg pub 3q1B4aHW… | latest_handshake=1792705591 (now 1792705592, diff -1 s) | prev key file: absent
@@ -0,0 +1,3 @@
+== clock back + chrony: 2026-10-22T19:46:43Z synced=yes
+2026-10-22T19:46:43Z real-ish | wg pub 3q1B4aHW… | latest_handshake=1792705591 (now 1792698403, diff 7188 s) | prev key file: absent
+tunnel ping: ok
@@ -0,0 +1,15 @@
+2026-10-22T19:50:56Z real-ish | wg pub 3q1B4aHW… | latest_handshake=1792698580 (now 1792698657, diff -77 s) | prev key file: absent
+interface: wg-felhom
+  public key: 3q1B4aHW…
+  listening port: 50300
+
+peer: f3d1ZI7/…
+  endpoint: 167.233.158.164:443
+  allowed ips: 10.77.0.1/32, 10.77.0.250/32
+  latest handshake: 1 minute, 17 seconds ago
+  transfer: 4.14 MiB received, 2.44 MiB sent
+  persistent keepalive: every 25 seconds
+tunnel-ping-ok
+2026-10-22T23:46:09+02:00 night1023 felhom-agent[35760]: time=2026-10-22T23:46:09.963+02:00 level=WARN msg="wgtunnel: tunnel handshake stale but endpoint IP unchanged — endpoint may be down; not restarting" endpoint_ip=167.233.158.164
+2026-10-22T21:47:09+02:00 night1023 felhom-agent[35760]: time=2026-10-22T21:47:09.791+02:00 level=WARN msg="wgtunnel: the latest handshake is stamped in the future — this box's clock ran ahead, and the endpoint refuses its newer handshakes; re-keying if it s
+2026-10-22T21:50:09+02:00 night1023 felhom-agent[35760]: time=2026-10-22T21:50:09.778+02:00 level=INFO msg="wgtunnel: the handshake is no longer stamped in the future — no re-key needed"
@@ -0,0 +1,5 @@
+-A INPUT -s 167.233.158.164/32 -p udp -m udp --sport 443 -m u32 --u32 "0x0>>0x16&0x3c@0x8=0x1000000" -m comment --comment d30-test -j DROP
+2026-10-22T19:52:01Z real-ish | wg pub 3q1B4aHW… | latest_handshake=1792698706 (now 1792698721, diff -15 s) | prev key file: absent
+== clock set +2 h: 2026-10-22T21:52:01Z
+== a handshake under the ahead clock at 21:54:50 (epoch 1792706086)
+2026-10-22T21:54:50Z real-ish | wg pub 3q1B4aHW… | latest_handshake=1792706086 (now 1792706090, diff -4 s) | prev key file: absent
@@ -0,0 +1,3 @@
+== clock back + chrony: 2026-10-22T19:55:01Z synced=yes
+2026-10-22T19:55:01Z real-ish | wg pub 3q1B4aHW… | latest_handshake=1792706086 (now 1792698901, diff 7185 s) | prev key file: absent
+tunnel ping: ok
@@ -0,0 +1,10 @@
+2026-10-22T20:10:49Z real-ish | wg pub 3q1B4aHW… | latest_handshake=1792706086 (now 1792699849, diff 6237 s) | prev key file: present
+tunnel ping: FAILS
+2026-10-22T22:10:09+02:00 night1023 felhom-agent[35760]: time=2026-10-22T22:10:09.972+02:00 level=WARN msg="wgtunnel: RE-KEYED after a handshake stamped in the future (D30) — registering the new key; the hub re-keys the peer in place and the endpoint takes the new key" old_pubkey="3q1B4aHW7EohL7Y8
+2026-10-22T22:10:10+02:00 night1023 felhom-agent[35760]: time=2026-10-22T22:10:10.772+02:00 level=INFO msg="wgtunnel: registered with hub" pubkey="HAYwc1CxA7Qb19sd6tlX1Y5YxHM8FeN0Tp8VCa2K8lY=" assigned_ip=10.77.0.5/32 existed=false generation=3 sync=ok
+total 20
+-rw------- last-applied.sha
+-rw------- private.key
+-rw------- private.key.prev
+-rw------- registered.json
+-rw------- wg-felhom.conf
@@ -0,0 +1,9 @@
+2026-10-22T20:16:04Z real-ish | wg pub HAYwc1Cx… | latest_handshake=1792700136 (now 1792700164, diff -28 s) | prev key file: present
+tunnel ping: ok
+2026-10-22T22:10:09+02:00 night1023 felhom-agent[35760]: time=2026-10-22T22:10:09.972+02:00 level=WARN msg="wgtunnel: RE-KEYED after a handshake stamped in the future (D30) — registering the new key; the hub re-keys the peer in place and the endpoint takes the new key" old_pubkey="3q1B4aHW7EohL7Y8qbE5Z0lbo21kqT3mocvigZdPrBg=" 
+2026-10-22T22:10:10+02:00 night1023 felhom-agent[35760]: time=2026-10-22T22:10:10.772+02:00 level=INFO msg="wgtunnel: registered with hub" pubkey="HAYwc1CxA7Qb19sd6tlX1Y5YxHM8FeN0Tp8VCa2K8lY=" assigned_ip=10.77.0.5/32 existed=false generation=3 sync=ok
+2026-10-22T22:11:09+02:00 night1023 felhom-agent[35760]: time=2026-10-22T22:11:09.867+02:00 level=WARN msg="wgtunnel: the latest handshake is stamped in the future — this box's clock ran ahead, and the endpoint refuses its newer handshakes; re-keying if it stays" future_s=6217 rekey_after=15m0s
+2026-10-22T22:15:36+02:00 night1023 felhom-agent[35760]: time=2026-10-22T22:15:36.774+02:00 level=INFO msg="wgtunnel: tunnel conf applied" endpoint=167.233.158.164:443 dns_name=ep0.felhom.eu assigned_ip=10.77.0.5/32 action=restart
+2026-10-22T22:15:36+02:00 night1023 felhom-agent[35760]: time=2026-10-22T22:15:36.849+02:00 level=INFO msg="wgtunnel: the handshake is no longer stamped in the future — no re-key needed"
+(re-registers while the served block still named the old key, once a minute, existed=true: 5)
+rules left: 0
@@ -0,0 +1,11 @@
+== hub DB read (named non-secret columns), host Tester4-16fdef
+wg_peers cols: ['pubkey', 'assigned_ip', 'host_id', 'note', 'created_at', 'updated_at']
+wg_peer ('Tester4-16fdef', 'HAYwc1Cx…', '10.77.0.5', '2026-10-22 20:10:10')
+host ('Tester4-16fdef', 3, '2026-10-22 20:15:36')
+host_reports cols: ['id', 'host_id', 'customer_id', 'received_at', 'report_json', 'agent_version', 'cpu_percent', 'memory_percent', 'disk_percent', 'guest_total', 'guest_running', 'cloudflared_status']
+2026-10-22 19:13:31 wg: {'pubkey': '', 'active': False, 'last_handshake_age_s': None, 'handshake_future_s': None}
+2026-10-22 19:28:36 wg: {'pubkey': '3q1B4aHW…', 'active': True, 'last_handshake_age_s': 14, 'handshake_future_s': None}
+2026-10-22 19:30:14 wg: {'pubkey': '3q1B4aHW…', 'active': True, 'last_handshake_age_s': 112, 'handshake_future_s': None}
+2026-10-22 19:45:20 wg: {'pubkey': '3q1B4aHW…', 'active': True, 'last_handshake_age_s': 50, 'handshake_future_s': None}
+2026-10-22 20:00:35 wg: {'pubkey': '3q1B4aHW…', 'active': True, 'last_handshake_age_s': 0, 'handshake_future_s': 6851}
+2026-10-22 20:15:36 wg: {'pubkey': 'HAYwc1Cx…', 'active': True, 'last_handshake_age_s': 0, 'handshake_future_s': 5950}
@@ -0,0 +1,18 @@
+# Part 2 — D30 live: clock ahead → (alarm input) → re-key → handshake
+
+Box: test box Tester4-16fdef (nested VM 9344 on demo-hp, installed tonight from ISO 1.30.0), agent fixture
+`0.163.0-night.1` = felhom-agent `ae69ad4` (D30). Hub: the live 0.165.0 (no alarm yet — the alarm and its clearing are
+pinned by hub tests, `84751cb0`, not deployed). Tool: `d30-box.sh` (in the scratchpad; state lines print 8 characters
+of a PUBLIC key, never a private key).
+
+| File | What |
+|---|---|
+| `01a-run1-clock-ahead.txt`, `02a-run1-clock-back.txt`, `03a-run1-ep0-initiated-heal.txt` | Run 1: clock +2 h → a handshake stamped +2 h; clock back → the agent saw the future stamp (21:47) and, 3 min later, the stamp was in the past: **ep0 started a handshake of its own** (it had traffic for the box; the box accepts ep0's timestamps), the tunnel healed, the agent correctly did not re-key |
+| `04-run2-block-and-ahead.txt`, `05-run2-clock-back.txt` | Run 2, the rehearsal's condition: ep0's handshake starts dropped on the box (iptables `u32` on WireGuard message type 1, comment `d30-test`); clock +2 h, back |
+| `06-run2-after-rekey.txt` | the tunnel dead; at 22:10:09 local **RE-KEYED** (old `3q1B4aHW…` → new `HAYwc1Cx…`), the hub: `existed=false generation=3 sync=ok`, same 10.77.0.5; the old key kept as `private.key.prev` |
+| `07-run2-new-key-handshake.txt` | 22:15:36 local the desired state carried the new key → conf applied → **a handshake with the new key**, tunnel ping ok; the test rule removed (rules left: 0) |
+| `08-hub-db-read.txt` | the hub (named columns): `wg_peers` holds the new key for the host since 20:10:10Z; the stored reports show `handshake_future_s=6851` at 20:00:35Z (the alarm's input) |
+
+Timeline (UTC): clock back 19:55:01 → future stamp seen 19:55:09 → re-key 20:10:09 → hub took it 20:10:10 → the box
+switched 20:15:36 (its next desired-state fetch) → handshake 20:15:36. Outage after the clock was right: ~20 min (the
+rehearsal's: until 2028 or a hand reset on ep0).
@@ -0,0 +1,222 @@
+diff --git a/configs/felhom-os-apply b/configs/felhom-os-apply
+index 81565fb..ee8429f 100755
+--- a/configs/felhom-os-apply
+++ b/configs/felhom-os-apply
+@@ -165,7 +165,8 @@ KERNEL_ACTIVE = ("staged", "oneshot", "judging", "reverting")
+ #   dest: (source file under configs/, mode, check, policy)
+ # check:  the content check BEFORE anything is written (R18): sudoers → visudo -cf; sh / bash → -n; python → compile;
+ #         unit → a [Unit]/[Service]/[Timer] section; unit-nort → that, and never RuntimeDirectory= (the G1 incident);
+-#         agent-unit → User=felhom-agent; dropin → a [Unit] section (SF-3); nft → nft -c -f; plain → none.
+#         agent-unit → User=felhom-agent; dropin → a [Unit] section (SF-3); nft → nft -c -f; plain → none;
+#         sshd → `sshd -t -f` on the snippet alone (R-982).
+ # policy: replace → always written when it differs; if-absent → only when the box has none (a setting the operator may
+ #         have tuned); oob → only on a box with the OOB belt (/etc/felhom-sshd exists — the installer's --no-oob leaves none).
+ # Order matters: the sudoers files come LAST, so a referenced wrapper is in place before the line that allows it.
+@@ -184,6 +185,9 @@ BUNDLE_KEEP = 3
+ BUNDLE_MAX_BYTES = 4 * 1024 * 1024
+ BUNDLE_RE = re.compile(r"^bundle-[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?\.json$")
+ OOB_DIR = "/etc/felhom-sshd"
+# R-982: the host's own sshd drop-in (after_install reloads sshd for it; undo reloads after putting it back).
+SSHD_DROPIN = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
+
+ BUNDLE_FILES = [
+     ("/usr/local/sbin/felhom-mkfs-guarded", "felhom-mkfs-guarded.sh", 0o755, "bash", "replace"),
+     ("/usr/local/sbin/felhom-selfupdate-guarded", "felhom-selfupdate-guarded", 0o755, "sh", "replace"),
+@@ -221,6 +225,9 @@ BUNDLE_FILES = [
+     ("/etc/systemd/system/felhom-sshd.service", "felhom-sshd.service", 0o644, "unit-nort", "oob"),
+     ("/etc/felhom-oob.nft", "felhom-oob.nft", 0o644, "nft", "oob"),
+     ("/etc/systemd/system/felhom-oob-nft.service", "felhom-oob-nft.service", 0o644, "unit", "oob"),
+    # R-982, operator ruling D4 (2026-10-22): the host's own sshd (port 22, the LAN): root by key only. The console
+    # password stays; felhom-sshd (the operator's door) reads nothing from sshd_config.d. Reloaded in after_install.
+    ("/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf", "felhom-root-ssh-key-only.conf", 0o644, "sshd", "replace"),
+     ("/etc/sudoers.d/felhom-op", "felhom-op.sudoers", 0o440, "sudoers", "oob"),
+     ("/etc/sudoers.d/felhom-agent", "felhom-agent.sudoers", 0o440, "sudoers", "replace"),
+ ]
+@@ -434,7 +441,8 @@ class Runner:
+     def check_content(self, kind, data):
+         """Run an external syntax check on a root-only temp copy. Returns (rc, message)."""
+         import tempfile
+-        cmd = {"sudoers": ["visudo", "-cf"], "sh": ["sh", "-n"], "bash": ["bash", "-n"], "nft": ["nft", "-c", "-f"]}[kind]
+        cmd = {"sudoers": ["visudo", "-cf"], "sh": ["sh", "-n"], "bash": ["bash", "-n"], "nft": ["nft", "-c", "-f"],
+               "sshd": ["/usr/sbin/sshd", "-t", "-f"]}[kind]
+         with tempfile.TemporaryDirectory(prefix="felhom-bundle-") as d:
+             p = os.path.join(d, "f")
+             with open(p, "wb") as f:
+@@ -1864,6 +1872,7 @@ class Bundle:
+     def __init__(self, apply):
+         self.a, self.r = apply, apply.r
+         self.report = apply.report
+        self.sshd_dropin_not_in_force = False
+ 
+     # ---------- reading and checking ----------
+     def load_file(self, path):
+@@ -1932,10 +1941,17 @@ class Bundle:
+             text = content.decode("utf-8")
+         except UnicodeDecodeError:
+             raise Refused("R18", f"{dest} is not UTF-8 text")
+-        if check in ("sudoers", "sh", "bash", "nft"):
+        if check in ("sudoers", "sh", "bash", "nft", "sshd"):
+             rc, msg = self.r.check_content(check, content)
+             if rc != 0:
+                 raise Refused("R18", f"{dest} fails its {check} check: {msg}")
+        if check == "sshd":
+            # R-982: the drop-in's PermitRootLogin must be exactly `prohibit-password`. `no` would also shut the root key
+            # out (Proxmox's own node-to-node SSH and the LAN key route), `yes` would undo the ruling — both refused.
+            # Only that line is pinned: the file is signed like every bundle file (review A7).
+            vals = re.findall(r"^\s*PermitRootLogin\s+(\S+)\s*$", text, re.M | re.I)
+            if vals != ["prohibit-password"]:
+                raise Refused("R18", f"{dest} must set exactly `PermitRootLogin prohibit-password` (found {vals})")
+         elif check == "python":
+             try:
+                 compile(text, dest, "exec")
+@@ -2044,6 +2060,8 @@ class Bundle:
+             self.report["failed"] = {"rc": 3, "step": "bundle", "reason": reason[:300]}
+             return 3
+         rep["signers_created"] = False
+        if self.sshd_dropin_not_in_force:
+            rep["sshd_dropin_not_in_force"] = True  # R-982 A2: written and reloaded, but sshd -T shows root passwords allowed
+         if bootstrap and not self.r.lexists(TRUST_SIGNERS):
+             self.r.put_file(TRUST_SIGNERS, pinned_signers_line().encode(), 0o644)
+             rep["signers_created"] = True
+@@ -2073,6 +2091,23 @@ class Bundle:
+                       "enable the crash guard")
+         if "/etc/systemd/system/felhom-mgmt-watchdog.timer" in dests:
+             self.must(["systemctl", "enable", "--now", "felhom-mgmt-watchdog.timer"], "enable the mgmt watchdog timer")
+        # R-982: the host's sshd takes the root-key-only drop-in. The WHOLE config is tested first (a failure is a
+        # refusal, and the bundle undo puts the old files back); `try-reload-or-restart` keeps every open session and
+        # does nothing on a box whose ssh.service is not running (a socket-activated or stopped sshd reads the file
+        # at its next start) — a failure there must not freeze the bundle route (review 2026-10-22 night, A1).
+        # felhom-sshd is not touched. Then the POSITIVE check: sshd's own effective view for root. A box whose
+        # sshd_config has no Include of sshd_config.d, an earlier drop-in or a Match block can keep the password route
+        # — that is said loudly in the log and the report, never silently recorded as done (A2).
+        if SSHD_DROPIN in written:
+            self.must(["/usr/sbin/sshd", "-t"], "sshd -t (the host's whole sshd config)")
+            self.must(["systemctl", "try-reload-or-restart", "ssh"], "systemctl try-reload-or-restart ssh")
+            rc, out, err = self.r.host(["/usr/sbin/sshd", "-T", "-C", "user=root,host=felhom-check,addr=192.0.2.1"], 60)
+            eff = re.findall(r"^permitrootlogin\s+(\S+)", out, re.M | re.I)
+            if rc != 0 or eff != ["without-password"]:
+                self.r.log(f"os-apply: BUNDLE WARNING the root-key-only drop-in is NOT in force: sshd -T says "
+                           f"permitrootlogin={eff or '?'} (rc={rc}) — check sshd_config's Include, an earlier drop-in "
+                           f"or a Match block (R-982)")
+                self.sshd_dropin_not_in_force = True
+ 
+     def must(self, argv, what):
+         rc, out, err = self.r.host(argv, 120)
+@@ -2133,6 +2168,9 @@ class Bundle:
+                 self.r.log(f"os-apply: BUNDLE UNDO could not restore {dest}: {e}")
+         if any(d.startswith("/etc/systemd/system/") for d in back):
+             self.r.host(["systemctl", "daemon-reload"], 120)
+        if SSHD_DROPIN in back:
+            # A3: after_install may already have reloaded sshd with the drop-in; the restored state takes effect now.
+            self.r.host(["systemctl", "try-reload-or-restart", "ssh"], 120)
+         rc, _, _ = self.r.host(["visudo", "-c"], 60)
+         self.r.log(f"os-apply: BUNDLE UNDO restored={len(back)} visudo-after={'ok' if rc == 0 else 'FAILED'}")
+         return back
+diff --git a/configs/felhom-root-ssh-key-only.conf b/configs/felhom-root-ssh-key-only.conf
+new file mode 100644
+index 0000000..d92eea9
+--- /dev/null
+++ b/configs/felhom-root-ssh-key-only.conf
+@@ -0,0 +1,7 @@
+# MANAGED BY FELHOM (the agent's config bundle) — R-982, operator ruling D4 (2026-10-22).
+# The Proxmox host's OWN sshd — the listener the household network reaches on port 22: root logs in with a KEY only.
+# The root password still works on the console (the break-glass password: felhom.eu runbooks/break-glass.md). The
+# operator's door (felhom-sshd, port 8822, /etc/felhom-sshd/sshd_config) is a separate daemon whose config includes
+# nothing from here, so it is not changed. Debian's sshd_config reads this folder BEFORE its own settings, and for
+# sshd the first value wins; the bundle checks the effective value with `sshd -T` after the reload.
+PermitRootLogin prohibit-password
+diff --git a/configs/test_felhom_config_bundle.py b/configs/test_felhom_config_bundle.py
+index 1c5a6ab..58b5238 100644
+--- a/configs/test_felhom_config_bundle.py
+++ b/configs/test_felhom_config_bundle.py
+@@ -429,6 +429,90 @@ class SelfCheckUndo(unittest.TestCase):
+         self.assert_restored(box, before, rep)
+ 
+ 
+
+class RootKeyOnly(unittest.TestCase):
+    """R-982, operator ruling D4 (2026-10-22): the bundle gives the host's own sshd a root-key-only drop-in, tests the
+    whole config and reloads sshd — only when the file was written; a drop-in that says anything but
+    `prohibit-password` is refused before anything is written.
+    RED-PROOF: with the after_install block removed, test_fresh_box_takes_the_dropin_and_reloads fails (no reload);
+    with the content rule removed, test_a_dropin_that_says_yes_or_no_is_refused fails — seen failing 2026-10-22."""
+    DEST = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
+
+    def test_fresh_box_takes_the_dropin_and_reloads(self):
+        data, sha = real_bundle()
+        box = Box(data, signed_job(sha))
+        rc, rep = run(box)
+        self.assertEqual(rc, 0, rep)
+        self.assertIn(b"PermitRootLogin prohibit-password", box.files[self.DEST])
+        self.assertEqual(box.modes[self.DEST], 0o644)
+        self.assertIn(["/usr/sbin/sshd", "-t"], box.calls)
+        self.assertIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
+        self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
+        self.assertFalse(any("felhom-sshd" in " ".join(c) for c in box.calls), "the operator's door must not be touched")
+
+    def test_an_unchanged_dropin_is_not_reloaded(self):
+        data, sha = real_bundle()
+        box = Box(data, signed_job(sha))
+        box.put(self.DEST, (REPO / "configs" / "felhom-root-ssh-key-only.conf").read_bytes(), 0o644)
+        rc, rep = run(box)
+        self.assertEqual(rc, 0, rep)
+        self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
+
+    def test_a_dropin_that_says_yes_or_no_is_refused(self):
+        for bad in (b"PermitRootLogin yes\n", b"PermitRootLogin no\n", b"# nothing\n",
+                    b"PermitRootLogin prohibit-password\nPermitRootLogin yes\n"):
+            def edit(files, bad=bad):
+                for e in files:
+                    if e["path"] == self.DEST:
+                        e["content_b64"] = base64.b64encode(bad).decode()
+            data, sha = edited_bundle(edit)
+            box = Box(data, signed_job(sha))
+            rc, rep = run(box)
+            self.assertNotEqual(rc, 0, bad)
+            self.assertNotIn(self.DEST, box.files, bad)
+
+    # Review 2026-10-22 night. A2: the effective value is read back with `sshd -T`; a drop-in that is not in force is
+    # said in the log and the report (not refused — a refusal would freeze the bundle route). A3: a bundle undone
+    # after the reload reloads sshd again with the old state. RED-PROOF: with the sshd -T check removed,
+    # test_a_dropin_not_in_force_is_reported fails; with the undo reload removed, test_an_undone_bundle_reloads_sshd
+    # fails — seen failing 2026-10-22 night.
+    def _sshd_T(self, box, value):
+        orig = box.host
+        def host(argv, timeout=600, stdin=None):
+            if argv[:2] == ["/usr/sbin/sshd", "-T"]:
+                box.calls.append(argv)
+                return 0, f"port 22\npermitrootlogin {value}\npasswordauthentication yes\n", ""
+            return orig(argv, timeout, stdin)
+        box.host = host
+
+    def test_the_effective_value_is_checked(self):
+        data, sha = real_bundle()
+        box = Box(data, signed_job(sha))
+        self._sshd_T(box, "without-password")
+        rc, rep = run(box)
+        self.assertEqual(rc, 0, rep)
+        self.assertTrue(any(c[:2] == ["/usr/sbin/sshd", "-T"] for c in box.calls))
+        self.assertNotIn("sshd_dropin_not_in_force", rep["bundle"])
+
+    def test_a_dropin_not_in_force_is_reported(self):
+        data, sha = real_bundle()
+        box = Box(data, signed_job(sha))
+        self._sshd_T(box, "yes")   # e.g. no Include of sshd_config.d
+        rc, rep = run(box)
+        self.assertEqual(rc, 0, rep)
+        self.assertTrue(rep["bundle"].get("sshd_dropin_not_in_force"), rep)
+        self.assertTrue(any("NOT in force" in l for l in box.logs), box.logs)
+
+    def test_an_undone_bundle_reloads_sshd(self):
+        data, sha = real_bundle()
+        box = Box(data, signed_job(sha))
+        box.visudo_fail = True     # the self-check fails AFTER the install → undo
+        rc, rep = run(box)
+        self.assertEqual(rc, 3, rep)
+        self.assertNotIn(self.DEST, box.files)
+        reloads = [i for i, c in enumerate(box.calls) if c == ["systemctl", "try-reload-or-restart", "ssh"]]
+        self.assertEqual(len(reloads), 2, box.calls)   # after_install, then the undo
+
+ class TrustBootstrap(unittest.TestCase):
+     def test_missing_signers_verifies_against_the_pinned_key_and_creates_it(self):
+         data, sha = real_bundle()
@@ -0,0 +1,29 @@
+diff --git a/scripts/felhom-host-install.sh b/scripts/felhom-host-install.sh
+index 970109fd..349da0c5 100644
+--- a/scripts/felhom-host-install.sh
+++ b/scripts/felhom-host-install.sh
+@@ -1417,6 +1417,12 @@ run_uninstall() {
+     if [[ -f /etc/default/grub.d/zz-felhom-kernel-default.cfg ]]; then
+         log_info "  kept /etc/default/grub.d/zz-felhom-kernel-default.cfg (the kernel this box booted healthily stays the default)"
+     fi
+    # R-982 (operator ruling D4, 2026-10-22; agent bundle after v0.162.0): the host sshd's root-key-only drop-in. The
+    # uninstall gives the host back as Proxmox set it up, so the drop-in goes and sshd is reloaded (open sessions stay).
+    if [[ -f /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf ]]; then
+        run rm -f /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf
+        run systemctl try-reload-or-restart ssh 2>/dev/null || log_warn "  sshd was not reloaded — the drop-in is gone, sshd reads it at its next start (R-982)"
+    fi
+     # 1.31.0 (R-840): the bundle's previous copies and the wrapper's nonce record.
+     if [[ -d /var/lib/felhom-os-apply ]]; then run rm -rf /var/lib/felhom-os-apply; fi
+     if [[ -f /var/lib/vz/snippets/felhom-guest-hook.sh ]]; then run rm -f /var/lib/vz/snippets/felhom-guest-hook.sh; fi
+diff --git a/scripts/test_hostinstall.py b/scripts/test_hostinstall.py
+index 3f7088a6..e9cf9ad9 100644
+--- a/scripts/test_hostinstall.py
+++ b/scripts/test_hostinstall.py
+@@ -60,6 +60,7 @@ BUNDLE_DESTS = [
+     "/etc/systemd/system/felhom-sshd.service",
+     "/etc/felhom-oob.nft",
+     "/etc/systemd/system/felhom-oob-nft.service",
+    "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf",   # R-982, operator ruling D4 (2026-10-22)
+     "/etc/sudoers.d/felhom-op",
+     "/etc/sudoers.d/felhom-agent",
+ ]
@@ -0,0 +1,4 @@
+GET HTTP/2 200 
+fields: ['pairing_code', 'passphrase', 'robots', 'viewport']
+POST HTTP/2 200  
+answer: Felhom — Doboz összekötése Felhom doboz összekötése Sikeres összekötés. A doboz kb. egy percen belül folytatja a telepítést. Ezt az oldalt bezárhatod — a beállítás a háttérben befejeződik, és a vezérlőpultod hamarosan elérhető lesz. Felhom.eu
@@ -0,0 +1,3 @@
+GET /claim HTTP/2 200 
+csrf_len 64 cookie_present yes
+POST /claim HTTP/2 302  location: /
@@ -0,0 +1,17 @@
+== 19:30:07 agent fixture
+felhom-agent 0.163.0-night.1
+active
+ac39dadd162bc74f
+== 19:30:11 D4 drop-in by hand (what the bundle after_install does: sshd -t -f, install, sshd -t, try-reload-or-restart, sshd -T)
+snippet check rc=0
+sshd -t ok
+reloaded
+permitrootlogin without-password
+passwordauthentication yes
+== 19:30:28 D4 checks from DooPlex (LAN)
+root PASSWORD over SSH: Permission denied, please try again.
+root KEY over SSH: root
+web page password login (/api2/json/access/ticket): HTTP 200
+== 19:30:43 controller fixture
+gitea.dooplex.hu/admin/felhom-controller:0.340.0-rc.6 Up 15 seconds (healthy)
+2026/10/22 19:30:45 [INFO] [tier2] start-up: no leftover second-drive copy container
@@ -0,0 +1,10 @@
+== 19:32:04 POST /api/storage/init device=/dev/sdb fstype=ext4 mount_name=hdd_1
+200
+{
+ "data": {
+  "phase": "formatting",
+  "started": true
+ },
+ "ok": true
+}
+200{"data":{"device":"/dev/sdb","durable_id":"","error":"","opsign":"","phase":"done","reason":"","started_at":"2026-10-22T19:32:09.922168725Z","updated_at":"2026-10-22T19:32:12.640993967Z","where":"/mnt/felhom-drives/hdd_1"},"ok":true}
@@ -0,0 +1,17 @@
+19:32:39Z nextcloud deploy 202 Telepítés elindítva – az állapot a kártyán követhető
+19:34:09Z nextcloud → running 97 s
+19:34:10Z immich deploy 202 Telepítés elindítva – az állapot a kártyán követhető
+19:36:31Z immich → running 141 s
+19:36:31Z paperless-ngx deploy 202 Telepítés elindítva – az állapot a kártyán követhető
+19:38:41Z paperless-ngx → running 130 s
+19:38:41Z vaultwarden deploy 202 Telepítés elindítva – az állapot a kártyán követhető
+19:39:01Z vaultwarden → running 20 s
+19:39:02Z mealie deploy 202 Telepítés elindítva – az állapot a kártyán követhető
+19:40:32Z mealie → running 90 s
+19:40:33Z actualbudget deploy 202 Telepítés elindítva – az állapot a kártyán követhető
+19:41:03Z actualbudget → running 30 s
+19:41:03Z radicale deploy 202 Telepítés elindítva – az állapot a kártyán követhető
+19:41:33Z radicale → running 30 s
+19:41:33Z bookstack deploy 202 Telepítés elindítva – az állapot a kártyán követhető
+19:42:43Z bookstack → running 70 s
+19:42:43Z memory: {"data": {"vmid": 9201, "allocated_mb": 4096, "usage_mb": 2126, "host_total_mb": 7939, "min_mb": 2048, "max_mb": 5891, "floor_mb": 2638, "running": true, "host_swap_mb": 7935, "guest_swap_mb": 4096}, "ok": true}
@@ -0,0 +1,7 @@
+WARN: Systemd 257 detected. You may need to enable nesting.
+Task finished with 1 warning(s)!
+hostname: night1023-loader
+memory: 2048
+rootfs: nvme-scratch:9345/vm-9345-disk-0.raw,size=24G
+192.168.0.152 
+19:39:56
@@ -0,0 +1,22 @@
+Formatting '/mnt/hdd_1/images/9344/vm-9344-disk-0.raw', fmt=raw size=274877906944 preallocation=off
+scsi0: successfully created disk 'nvme-scratch:9344/vm-9344-disk-0.raw,size=256G'
+Formatting '/mnt/hdd_1/images/9344/vm-9344-disk-1.raw', fmt=raw size=214748364800 preallocation=off
+scsi1: successfully created disk 'nvme-scratch:9344/vm-9344-disk-1.raw,size=200G'
+update VM 9344: -boot order=scsi0;ide2
+18:46:17
+boot: order=scsi0;ide2
+cores: 4
+cpu: host
+ide2: local:iso/felhom-installer-1.30.0-pve9.2-1.iso,media=cdrom,size=1665356K
+memory: 8192
+meta: creation-qemu=11.0.3,ctime=1792694775
+name: night1023-soak
+net0: virtio=BC:24:11:AB:3A:21,bridge=vmbr0
+ostype: l26
+scsi0: nvme-scratch:9344/vm-9344-disk-0.raw,size=256G
+scsi1: nvme-scratch:9344/vm-9344-disk-1.raw,size=200G
+scsihw: virtio-scsi-single
+smbios1: uuid=cbc4fbdd-86b6-4050-9f10-7f4319c9f86b
+vga: std
+vmgenid: aa1b5848-cd99-43d0-ae66-f222698ee0ab
+Install pressed 18:51:58
@@ -0,0 +1,4 @@
+bind POST 19:12:17
+claim 19:29:40
+env written: IRISZ ODON COOKIE
+soak started 19:55:59
@@ -69,7 +69,7 @@ local restic repos; Prometheus history; DooPlex's other homelab apps (not Felhom
 7. **Gitea:** `gitea-restore.md` (on a normal network this time). Then rebuild the images from the code into the new
   registry: hub, controller, agent (`RUNBOOK-manual-build.md`).
 8. **The hub:** k3s, then `RUNBOOK-hub-db-offsite-backup.md` §3 with S2 and S4 — **start it with mail held**
-   (§3 step 1: the `MAIL-HOLD` marker, from the next hub release; until then: no network until the pending notices are understood).
+   (§3 step 1: the `MAIL-HOLD` marker — shipped in hub 0.145.0, 2026-10-10).
 9. **DNS:** in Cloudflare (S11) point `hub.felhom.eu` (today a CNAME to `dooplex.hopto.org`) and `gitea.dooplex.hu` at
   the new place. The boxes reconnect by themselves: their API keys are in the restored hub DB.
 10. **Signing:** the keys are in the restored copy (`signing/`) and on the sheet (S6) — put them back and sign as before. Without it, no box accepts an agent update or a