From 6bf4bef2beb56eea58d180974a7c4e83dce650c6 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 21 Jul 2026 10:24:52 +0200 Subject: [PATCH] docs(report): hub v0.68.0 (R-39 fleet fix + R-50b(a)); STOP-2/3 need the operator --- REPORT.md | 169 ++++++++++++++++++++---------------------------------- 1 file changed, 61 insertions(+), 108 deletions(-) diff --git a/REPORT.md b/REPORT.md index 0852adb..bd6850e 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,123 +2,76 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## Tailscale on DooPlex + felhom-pve — make the N100 location-independent — 2026-07-19 +# TASK-B — R-39 fleet fix + R-50b(a) · hub **v0.67.0 → v0.68.0** -**Not a code change.** Network/ops on two production hosts (DooPlex, felhom-pve). Run supervised, -normal permission prompts, one host at a time, verify-after-every-mutation. Baseline `felhom.eu` -`a771cda`. +**Date:** 2026-07-21 · **Baseline:** `c35da9d` (clean, == `origin/main`) → `54a4644`. +Companion: agent **v0.91.2** (`felhom-agent`, see that repo's `REPORT.md` for the agent half, +STOP-1 evidence and the four red-proofs). -### 1. Part 0 audit (verbatim highlights) + dependency classification +## Status -``` -# felhom-pve host — STATIC, not DHCP (contradicted the operator's expectation) -iface vmbr0 inet static / address 192.168.0.162/24 / gateway 192.168.0.1 -resolv.conf: search local / nameserver 192.168.0.250 (pi-hole) +| Leg | Status | +|---|---| +| Hub v0.68.0 | **SHIPPED + DEPLOYED** — GitOps manifest bump `0.67.0→0.68.0`, ArgoCD `Synced/Healthy`, rollout complete, pod 1/1 | +| Agent v0.91.2 | shipped + published + deployed to felhom-pve | +| STOP-1 | done + verified | +| **STOP-2 / STOP-3** | **BLOCKED — need the operator.** I cannot authenticate to the hub UI (login → HTTP 401); the password is a bcrypt hash I do not hold, by design. | -# PBS is OFFSITE, not on 192.168.0.180 (contradicted the task premise) -storage.cfg: pbs felhom-pbs server 10.77.0.1 datastore felhom-offsite namespace demo-felhom -ip route get 10.77.0.1 -> dev wg-felhom src 10.77.0.2 -wg-felhom endpoint = 167.233.158.164:443 (Hetzner, public); handshake fresh, 3.75 GiB sent +## What shipped hub-side -# guest 9201 — already DHCP; grep 192.168.0.x found the pinned control-plane endpoint -pct config 9201 net0: ...,ip=dhcp -9201:/etc/felhom-bootstrap/bootstrap.json "endpoint": "192.168.0.162:8443" -felhom-agent listen_addr = 192.168.0.162:8443 (LISTEN 192.168.0.162:8443 users:(("felhom-agent"))) +- **`host_pbs_secrets.generation`** — a monotonic per-host counter advanced by every fresh MINT and + by nothing else, stamped into the descriptor as `secret_generation`. Since the agent re-applies on + the descriptor's CONTENT HASH and a re-key returns byte-identical `token_id` / `fingerprint` / + `datastore` / `namespace`, this is the only field that moves — and therefore the thing that + re-arms a converged agent. + - A **re-stage** deliberately does not advance it (same secret, unchanged descriptor content). + - `omitempty` is load-bearing: emitting a zero would shift every pre-existing descriptor's hash at + once — a fleet-wide spurious re-apply. + - **Deviation from spec, deliberate:** the brief said to reuse "the new row's id … no schema + change". There is no row id — the table is `host_id PRIMARY KEY`, UPSERTed last-write-wins — and + `created_at` collides for two mints in one second. An additive counter column (existing + idempotent `ALTER TABLE` idiom) is the only monotonic source. **Verified applied on the live DB + after deploy.** +- **`pbsdrheal` gains an `auth_failed` trigger** — a new trigger in the existing machine, escalating + to a fresh mint (never a re-stage, which would re-feed the secret PBS just rejected) through the + **existing damper**, so a 401 flap cannot become a secret-minting chain. +- **`consumed_at` honesty gauge** — an unconsumed secret past a 15-minute grace under a box reporting + `applied` is the exact 2026-07-18 fingerprint and a disagreement **no single tier can detect + alone**. Surfaced with its own event, deliberately as a SURFACE not a heal: auto-re-issuing would + mint a second secret on top of an unconsumed one, which is the mint/consume race R-39(a) recorded. +- **Corrected a comment that stated a falsehood** — `ReissuePBSDR` claimed it refreshed the + descriptor "with the NEW token_id/fingerprint". False for a re-key, and believing it is why nobody + expected the descriptor to come back identical. +- **R-50b(a)** — `ArtifactManifest.WrapperSHA256` + operator field + host-page drift surface. **An + unknown on either side reads as quiet, never as drift.** -# other -grep 192.168.0.x in /etc/pve /etc/cron /etc/systemd (excl .162): (none) -pve-firewall: disabled/running (no Tailscale allow-rule needed) -DooPlex ipv4 ip_forward = 1 (k3s), ipv6 = 0 ; both hosts Debian 13 trixie -``` +## Method notes worth keeping -| Dep | Where | Address | Survives the move? | -|---|---|---|---| -| PBS backup | host storage.cfg | `10.77.0.1` via `wg-felhom`→Hetzner:443 | **Yes** — offsite tunnel, internet-only | -| Default gateway | host | `192.168.0.1` | Yes — DHCP-derived once host is DHCP | -| DNS (pi-hole) | host + guest | `192.168.0.250` | Yes for general DNS; `gitea.dooplex.hu` split-horizon won't resolve off-LAN → **finding** | -| **Agent listen + guest endpoint** | agent.json + guest bootstrap.json | **`192.168.0.162:8443`** | **NO** — hard-pinned to host LAN IP; L2-bridged → subnet routes don't help → **finding, record-not-fix** | -| `*.demo-felhom.eu` LAN forward | home pi-hole → box | inbound | LAN-local demo breaks off-LAN; Cloudflare Tunnel external access unaffected → note only | +- **P2 confirmed GitOps-only, and the trap is real:** `build.sh` itself prints + `kubectl set image …` as its deploy hint, contradicting `CLAUDE.md`. Not used. Worth fixing in the + script — it will mislead exactly the session that trusts tool output over the runbook. +- **P4 read the fleet from a temporary copy of the hub DB**, which carries live credentials + (`api_key`, `host_pbs_secrets.value`). Copy shredded immediately after each read. Result: **one + enrolled host**, so the MinAgent raise strands nobody. +- Tests include a **flow-level** `ReissuePBSDR` test against a fake that models a real re-key + (identical token/fingerprint, rotated secret only). Its red-proof fails on the assertion with both + byte-identical blocks printed — the July-18 defect reproduced in a unit test. -### 2. Tailscale nodes +## For the operator -| Node | Tailnet IPv4 | Route advertised | Key expiry | How installed | -|---|---|---|---|---| -| `dooplex` | `100.107.87.53` | `192.168.0.0/24` (approved) | disabled | **pre-existing** k3s pod `admin-system/tailscale` (hostNetwork, GitOps) — *not* installed by this task | -| `felhom-pve` | `100.70.170.35` | — | disabled (operator-confirmed) | host apt package (trixie repo), `--accept-dns=false` | +**STOP-2 (one click):** press **Re-issue PBS credentials** for the demo customer. Expected: +fresh secret row → `secret_generation` **0 → 1** (the live descriptor has no such key today) → poke → +agent re-applies with **no short-circuit** → fresh secret consumed → reconcile rc-0 → probe 200 → tier +`active`. The July-18 negative — the same click doing nothing — is the historical red-proof. -`tailscale status` from the N100: both nodes listed/online. resolv.conf on felhom-pve **byte-identical** -after `tailscale up` (`CorpDNS:false`). +**STOP-3 (manifest save):** Agent `0.91.2` / sha256 `34d309be429473f3f0ab34e3185e17b22463a341b30bf46e162306bff4aec22a` / +**PBS wrapper sha256** `104db0a4401f65bbc476e82bfb1796433bcb36f8f8cce69efb3bb5c40fcb16b3` / +MinAgent `0.91.2`. Superseded, do not vouch: 0.91.0 (inert probe leg), 0.91.1. `0.90.1` correctly +stays 404. -**Course correction:** the task assumed Tailscale had to be *installed* on DooPlex and that PBS lived -on `.180`. Both were wrong — DooPlex already ran Tailscale as a GitOps k3s pod (the operator was -right; my `which tailscale`/systemd probe was too narrow and missed it), and PBS is offsite. The -redundant host-level Tailscale I briefly installed on DooPlex was **fully purged** (package + repo + -keyring + sysctl file), verified: k3s `tailscale0` and the `dooplex` node stayed up, `ip_forward` -still 1. +## Residual -### 3. accept-routes spike (§3.1) — CONFIRMED degradation, rolled back - -Enabling `--accept-routes` on felhom-pve *while on `192.168.0.0/24`*: - -``` -PRE : ip route get 192.168.0.180 -> dev vmbr0 src 192.168.0.162 ; ping .180 avg 0.3 ms -POST: ip route get 192.168.0.180 -> dev tailscale0 table 52 src 100.70.170.35 ; ping .180 avg 68 ms (max 136) - table 52 contains: 192.168.0.0/24 dev tailscale0 ; ip rule 5270: from all lookup 52 (outranks main) - -> local subnet hijacked onto the tunnel; inbound SSH went asymmetric and STALLED - PBS (10.77.0.1 via wg-felhom) UNAFFECTED -ROLLBACK (via the tailnet path 100.70.170.35, which was immune): tailscale set --accept-routes=false - -> route to .180 back on vmbr0 direct -``` - -Verdict: keep `--accept-routes=false` at home; it is a **travel-only opt-in** (safe on a foreign -subnet). Documented with the reasoning in `documentation/operations/tailscale.md`. - -### 4. SSH chain over the tailnet (§3.2) - -`~/.ssh/config` on DooPlex: `Host felhom-pve` → `HostName 100.70.170.35`; added `Host felhom-pve-lan` -→ `192.168.0.162`. Proof: - -``` -ssh felhom-pve -> pve-manager/9.2.2 ; pct exec 9201 docker ps -> felhom-controller:0.148.0 Up (healthy) -tailscale ping 100.70.170.35 -> pong via 192.168.0.162:41641 in 1ms (DIRECT over LAN, not DERP) -ssh felhom-pve-lan -> OK (LAN fallback works) -``` - -### 5. Part 4 — host static→DHCP: **DONE** (operator at console, reservation set) - -Reservation MAC `68:1d:ef:5d:a6:64`→`192.168.0.162` set by operator first (so the LAN-pinned agent -keeps working at home). Applied detached (`systemd-run … ifreload -a`); result: - -``` -ifreload rc=0 ; vmbr0 inet dhcp -> got 192.168.0.162 back (reservation) -default via 192.168.0.1 ; felhom-agent active, still bound 192.168.0.162:8443 (no restart) -guest 9201: 12 containers ; Tailscale still direct 1ms -``` - -Exact revert staged on the box: `/root/interfaces.static-revert-20260719`. resolv.conf is now -DHCP-managed (`.250` + `.1`) instead of the old static `search local` — expected (Tailscale didn't -touch it; accept-dns rule intact). - -### 6. Off-LAN legs NOT validated at home → vacation-day checklist - -Cannot be tested until the box is on a foreign LAN: PBS from a foreign subnet, DERP-relay fallback, -off-LAN `gitea.dooplex.hu` resolution. The step-by-step is in `documentation/operations/tailscale.md` -(§ *Vacation-day checklist*) — notably `ssh felhom-pve "pvesm status | grep -i pbs"` is the real -off-LAN PBS test. - -### 7. Findings recorded, NOT fixed - -1. **Control plane pinned to `192.168.0.162`** (HIGH for travel): `felhom-agent` `listen_addr` and - guest `bootstrap.json` `endpoint` both = `192.168.0.162:8443`. Off-LAN the agent can't bind → - agent down → controller can't reach it. Subnet routes don't help (L2 bridge). Needs a design - change (bind `0.0.0.0` + stable guest→host path). **The N100 is reachable and backs up off-LAN, - but its local operator plane is not yet portable.** -2. **Off-LAN gitea DNS**: `gitea.dooplex.hu` (pi-hole split-horizon) won't resolve on a foreign LAN; - image pulls while travelling need the travel `--accept-routes` toggle + a temporary `/etc/hosts` - entry. Contingency ("Plan B") documented, not armed. - -### 8. Commit - -Docs commit: `83c47ea` (felhom.eu). Files: `documentation/operations/tailscale.md` (new), -`CLAUDE.md` + `skills/felhom-build-deploy/SKILL.md` env-table notes, `CONTEXT.md` decision entry, -this `REPORT.md`. No secrets committed (tunnel keys/tokens/PBS fingerprint referenced out-of-band). +R-50b **(b)/(c) remain open** — the wrapper is still fetched unversioned from `raw/branch/main`; this +release makes drift visible, it does not fix the channel. The 0440 sudoers file is not agent-readable, +so its drift stays invisible. The DR-tier capability-map row is deliberately **not** upgraded to +PROVEN-LIVE until STOP-2 supplies the evidence.