night 2026-10-23 so far: D30 live proof (re-key → new-key handshake; ep0-started handshakes heal a short clock jump by themselves), D4 patches v2 after review, test-night install/fixture evidence, DooPlex survival survey (read only) + PLAN, 03/05/07 + CONTEXT, total-loss runbook: MAIL-HOLD shipped
gates / gates (push) Successful in 15m49s

This commit is contained in:
2026-10-22 22:24:48 +02:00
parent 84751cb0ce
commit 7585443227
32 changed files with 734 additions and 2 deletions
+14 -1
View File
@@ -131,7 +131,20 @@ the one trust root; the agent's API token is untouched; a box gets the new rule
**[DESIGN, ruled 2026-10-22 15:47 (D4, R-982) — built as a ready patch, NOT shipped] Root SSH from the LAN by key only; the console password stays.** The config bundle gains one fixed file, `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`), checked and reloaded by `felhom-os-apply`; the installer's uninstall removes it. Why a drop-in: Debian's `sshd_config` Includes `sshd_config.d/*` FIRST and sshd keeps the first value it reads, so the drop-in wins over the stock line; the out-of-band operator sshd (`FELHOM_SSHD`, port 8822) has its own config with no Include and is not touched. Proven on a fresh ISO 1.30.0 install: password SSH refused after, key SSH and the PVE web password login (200) and the console login still work (`audits/night-2026-10-22/P2-D4/`). It ships with the next agent release + installer tag.
**[FACT, agent main `a0c88e5`, unreleased] The heartbeat reports a WireGuard handshake from the future (R-1204).** WireGuard refuses an initiation whose TAI64N time is not newer than the last one accepted from that peer, so a box that once ran with its clock ahead is locked out of ep0 until real time passes that moment. The agent used to clamp the negative age to 0 (it looked healthy); `wgtunnel.handshakeFuture` now sets `handshake_future_s` when the handshake is more than 60 s ahead (`TestR1204_AFutureHandshakeIsReported`). The hub alarms on it (`05` §15.4b). The repair is STATUS decision D30.
**[FACT, agent main `a0c88e5`, unreleased] The heartbeat reports a WireGuard handshake from the future (R-1204).** WireGuard refuses an initiation whose TAI64N time is not newer than the last one accepted from that peer, so a box that once ran with its clock ahead is locked out of ep0 until real time passes that moment. The agent used to clamp the negative age to 0 (it looked healthy); `wgtunnel.handshakeFuture` now sets `handshake_future_s` when the handshake is more than 60 s ahead (`TestR1204_AFutureHandshakeIsReported`). The hub alarms on it (`05` §15.4b).
**[DESIGN, operator ruling D30 2026-10-22; agent main `ae69ad4`, unreleased] The tunnel heals itself by a new key.** When
the latest handshake stays stamped in the future for 15 minutes, the agent makes a new WireGuard key pair (the old
private key kept as `wg/private.key.prev`, 0600 — the escrow may hold it) and registers the new public key on the
existing path, `POST /hosts/<id>/wg`: the hub re-keys the peer in place (same /32, generation bump) and its full-list
push to ep0 (`wgsync`, `wg syncconf`) drops the old key and adds the new one. A new key is a new peer for ep0, with no
stored timestamp. Once an hour at most; not while the unit is down; never during a re-register already in flight. The
interface switches at the agent's next desired-state fetch (up to its report interval). **`[FACT]` 2026-10-23 night
(test box Tester4-16fdef, `audits/night-2026-10-23/P2-D30/`):** with ep0 free to start a handshake of its own, a 2-hour
clock jump healed by itself in ~3 minutes (ep0 had traffic for the box; the box accepts ep0's timestamps) and the agent
correctly did not re-key; with ep0's handshake starts dropped on the box (the rehearsal's condition), the tunnel stayed
dead and the agent re-keyed after 15 minutes. **Known gap:** a reboot after the clock was corrected forgets the
future stamp (R-1223).
## 4. Control model — reconcile + signed destructive ops
@@ -371,6 +371,7 @@ token id, fingerprint, datastore, tunnel IP, `secret_generation`) with its consu
| **re-issue, descriptor ABSENT, DR flag on** (R-511, v0.114.0) | **adopted**: re-keyed | **rebuilt** from the endpoint's answer; `pbsdr_adopted` audit row |
| host delete | **kept** (tenancy survives) | goes with the host |
| host delete acknowledged through escrow-ack, then a new box | re-keyed automatically (F-14) | rebuilt |
| …and a later install deleted before it made a key, then a new box (D31, operator 2026-10-22; hub 0.167.0 source) | re-keyed automatically — the gate reads the newest deletion of a host that HELD A KEY (`LatestKeyHoldingHostDeletion`) | rebuilt |
| RESET / customer delete | **deprovisioned** — namespace, every backup group AND token | purged |
**Why adopt exists:** a rebuilt box's WG hook refused („the endpoint already holds a PBS token … use
@@ -459,6 +460,12 @@ date taken) are shown on its own app page, on its box, until it dismisses them
**[FACT, hub `e267c5e2`]** The host report carries the agent's `wireguard` block (`hostWireguard`). `checkWGTunnel` (`api/host_tunnel.go`, called after `checkHostSwap`) calls the tunnel stale when the box reports `handshake_future_s` > 0 (the clock-ahead lock-out) or a last handshake older than 6 h (healthy boxes measured at most 113–124 s over 24 h). It is edge-triggered against the host's latest tunnel event (`store.LatestHostTunnelEvent`): `wg_tunnel_stale` once on the way down, `wg_tunnel_ok` once on the way back; both are allow-listed and operator-only (`notify.operatorOnlyEvents`, `wg_tunnel_operator_only_test.go`) — the household cannot repair a tunnel. A box with no `wireguard` block (an agent before the field, or no tunnel) raises nothing.
**[FACT, hub `84751cb0`, 0.167.0 source] The repair clears it (D30).** An agent with D30 (`03` §R-1204) registers a new key
after 15 minutes of a future-stamped handshake; the new key's first handshake makes the report fresh, and the alarm
records `wg_tunnel_ok` once (`TestD30_TheAlarmClearsWhenTheNewKeyShakesHands`). `wg_tunnel_ok` is an info event:
stored, not mailed. **Known gaps (rows):** a reboot after the clock was corrected sends no handshake at all and is not
judged (R-1223); the operator's mail cooldown is per customer, not per host (R-1224).
### 15.5 How "the Hungarian did not change" is known
56 goldens captured from v0.117.0 before any string moved, in
@@ -1590,6 +1590,21 @@ or pattern. **Why:** the data-loss hunt of 2026-10-20 proved two nights that del
(R-1175: one share group's reconcile removed the other groups' copies, and a dead drive's copy; R-1181: a missing or
emptied folder took its copy within one night). Controller: `internal/backup/tier2_kept.go`.
### 8.2b The second-drive copy runs in a memory-capped container that never outlives the copy [DESIGN + FACT, R-1214, 2026-10-22/23 nights; rides controller 0.340.0]
The copy (`rsync -a --delete`) runs in a container of the controller's own image, capped at 512 MB, so its page cache
is reclaimed inside the cap instead of from the apps (R-1214: full stall 5.7 s → 0.37 s on 9202). Only for paths the
controller and the Docker daemon see alike (`/mnt/…`, no symlink); the daemon must see the path itself (`--mount`
refuses a missing one — then the in-process copy runs). **The container must never outlive the copy** (the stray-copy
fix, 2026-10-23 night): a fixed name and the label `felhom.tier2-copy=1`; on any error, the timeout included, it is
removed by name; at controller start, before the scheduler and the web server, every labelled container is removed;
and a restore is refused while any second-drive copy runs or a leftover may still run (the other direction of R-1180 —
a `--delete` beside a restore removes what the restore brings back). **`[FACT]` 9202, rc.5
(`audits/night-2026-10-23/P1-straycopy/`):** a controller restart in the middle of a 16 GB copy → the new controller
removed the old container at start; a restore probe during the copy was refused; the redone copy equals the source; a
15-second test limit → the container removed, the partial copy stopped growing. Controller:
`internal/backup/tier2_copy_lifecycle.go`.
### 8.1 The blank cells, listed explicitly
Per the rule that a blank is a finding, here they are: