admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(report): hub v0.68.0 (R-39 fleet fix + R-50b(a)); STOP-2/3 need the operator

Browse Source
This commit is contained in:
authentik Default Admin
2026-07-21 10:24:52 +02:00
parent 54a4644721
commit 6bf4bef2be
1 changed files with 61 additions and 108 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+61 -108
View File
@@ -2,123 +2,76 @@
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
## Tailscale on DooPlex + felhom-pve — make the N100 location-independent — 2026-07-19
# TASK-B — R-39 fleet fix + R-50b(a) · hub **v0.67.0 → v0.68.0**
**Not a code change.** Network/ops on two production hosts (DooPlex, felhom-pve). Run supervised,
normal permission prompts, one host at a time, verify-after-every-mutation. Baseline `felhom.eu`
`a771cda`.
**Date:** 2026-07-21 · **Baseline:** `c35da9d` (clean, == `origin/main`) → `54a4644`.
Companion: agent **v0.91.2** (`felhom-agent`, see that repo's `REPORT.md` for the agent half,
STOP-1 evidence and the four red-proofs).
### 1. Part 0 audit (verbatim highlights) + dependency classification
## Status
```
# felhom-pve host — STATIC, not DHCP (contradicted the operator's expectation)
iface vmbr0 inet static / address 192.168.0.162/24 / gateway 192.168.0.1
resolv.conf: search local / nameserver 192.168.0.250 (pi-hole)
| Leg | Status |
|---|---|
| Hub v0.68.0 | **SHIPPED + DEPLOYED** — GitOps manifest bump `0.67.0→0.68.0`, ArgoCD `Synced/Healthy`, rollout complete, pod 1/1 |
| Agent v0.91.2 | shipped + published + deployed to felhom-pve |
| STOP-1 | done + verified |
| **STOP-2 / STOP-3** | **BLOCKED — need the operator.** I cannot authenticate to the hub UI (login → HTTP 401); the password is a bcrypt hash I do not hold, by design. |
# PBS is OFFSITE, not on 192.168.0.180 (contradicted the task premise)
storage.cfg: pbs felhom-pbs server 10.77.0.1 datastore felhom-offsite namespace demo-felhom
ip route get 10.77.0.1 -> dev wg-felhom src 10.77.0.2
wg-felhom endpoint = 167.233.158.164:443 (Hetzner, public); handshake fresh, 3.75 GiB sent
## What shipped hub-side
# guest 9201 — already DHCP; grep 192.168.0.x found the pinned control-plane endpoint
pct config 9201 net0: ...,ip=dhcp
9201:/etc/felhom-bootstrap/bootstrap.json "endpoint": "192.168.0.162:8443"
felhom-agent listen_addr = 192.168.0.162:8443 (LISTEN 192.168.0.162:8443 users:(("felhom-agent")))
- **`host_pbs_secrets.generation`** — a monotonic per-host counter advanced by every fresh MINT and
by nothing else, stamped into the descriptor as `secret_generation`. Since the agent re-applies on
the descriptor's CONTENT HASH and a re-key returns byte-identical `token_id` / `fingerprint` /
`datastore` / `namespace`, this is the only field that moves — and therefore the thing that
re-arms a converged agent.
- A **re-stage** deliberately does not advance it (same secret, unchanged descriptor content).
- `omitempty` is load-bearing: emitting a zero would shift every pre-existing descriptor's hash at
once — a fleet-wide spurious re-apply.
- **Deviation from spec, deliberate:** the brief said to reuse "the new row's id … no schema
change". There is no row id — the table is `host_id PRIMARY KEY`, UPSERTed last-write-wins — and
`created_at` collides for two mints in one second. An additive counter column (existing
idempotent `ALTER TABLE` idiom) is the only monotonic source. **Verified applied on the live DB
after deploy.**
- **`pbsdrheal` gains an `auth_failed` trigger** — a new trigger in the existing machine, escalating
to a fresh mint (never a re-stage, which would re-feed the secret PBS just rejected) through the
**existing damper**, so a 401 flap cannot become a secret-minting chain.
- **`consumed_at` honesty gauge** — an unconsumed secret past a 15-minute grace under a box reporting
`applied` is the exact 2026-07-18 fingerprint and a disagreement **no single tier can detect
alone**. Surfaced with its own event, deliberately as a SURFACE not a heal: auto-re-issuing would
mint a second secret on top of an unconsumed one, which is the mint/consume race R-39(a) recorded.
- **Corrected a comment that stated a falsehood** — `ReissuePBSDR` claimed it refreshed the
descriptor "with the NEW token_id/fingerprint". False for a re-key, and believing it is why nobody
expected the descriptor to come back identical.
- **R-50b(a)** — `ArtifactManifest.WrapperSHA256` + operator field + host-page drift surface. **An
unknown on either side reads as quiet, never as drift.**
# other
grep 192.168.0.x in /etc/pve /etc/cron /etc/systemd (excl .162): (none)
pve-firewall: disabled/running (no Tailscale allow-rule needed)
DooPlex ipv4 ip_forward = 1 (k3s), ipv6 = 0 ; both hosts Debian 13 trixie
```
## Method notes worth keeping
| Dep | Where | Address | Survives the move? |
|---|---|---|---|
| PBS backup | host storage.cfg | `10.77.0.1` via `wg-felhom`→Hetzner:443 | **Yes** — offsite tunnel, internet-only |
| Default gateway | host | `192.168.0.1` | Yes — DHCP-derived once host is DHCP |
| DNS (pi-hole) | host + guest | `192.168.0.250` | Yes for general DNS; `gitea.dooplex.hu` split-horizon won't resolve off-LAN → **finding** |
| **Agent listen + guest endpoint** | agent.json + guest bootstrap.json | **`192.168.0.162:8443`** | **NO** — hard-pinned to host LAN IP; L2-bridged → subnet routes don't help → **finding, record-not-fix** |
| `*.demo-felhom.eu` LAN forward | home pi-hole → box | inbound | LAN-local demo breaks off-LAN; Cloudflare Tunnel external access unaffected → note only |
- **P2 confirmed GitOps-only, and the trap is real:** `build.sh` itself prints
`kubectl set image …` as its deploy hint, contradicting `CLAUDE.md`. Not used. Worth fixing in the
script — it will mislead exactly the session that trusts tool output over the runbook.
- **P4 read the fleet from a temporary copy of the hub DB**, which carries live credentials
(`api_key`, `host_pbs_secrets.value`). Copy shredded immediately after each read. Result: **one
enrolled host**, so the MinAgent raise strands nobody.
- Tests include a **flow-level** `ReissuePBSDR` test against a fake that models a real re-key
(identical token/fingerprint, rotated secret only). Its red-proof fails on the assertion with both
byte-identical blocks printed — the July-18 defect reproduced in a unit test.
### 2. Tailscale nodes
## For the operator
| Node | Tailnet IPv4 | Route advertised | Key expiry | How installed |
|---|---|---|---|---|
| `dooplex` | `100.107.87.53` | `192.168.0.0/24` (approved) | disabled | **pre-existing** k3s pod `admin-system/tailscale` (hostNetwork, GitOps) — *not* installed by this task |
| `felhom-pve` | `100.70.170.35` | — | disabled (operator-confirmed) | host apt package (trixie repo), `--accept-dns=false` |
**STOP-2 (one click):** press **Re-issue PBS credentials** for the demo customer. Expected:
fresh secret row → `secret_generation` **0 → 1** (the live descriptor has no such key today) → poke →
agent re-applies with **no short-circuit** → fresh secret consumed → reconcile rc-0 → probe 200 → tier
`active`. The July-18 negative — the same click doing nothing — is the historical red-proof.
`tailscale status` from the N100: both nodes listed/online. resolv.conf on felhom-pve **byte-identical**
after `tailscale up` (`CorpDNS:false`).
**STOP-3 (manifest save):** Agent `0.91.2` / sha256 `34d309be429473f3f0ab34e3185e17b22463a341b30bf46e162306bff4aec22a` /
**PBS wrapper sha256** `104db0a4401f65bbc476e82bfb1796433bcb36f8f8cce69efb3bb5c40fcb16b3` /
MinAgent `0.91.2`. Superseded, do not vouch: 0.91.0 (inert probe leg), 0.91.1. `0.90.1` correctly
stays 404.
**Course correction:** the task assumed Tailscale had to be *installed* on DooPlex and that PBS lived
on `.180`. Both were wrong — DooPlex already ran Tailscale as a GitOps k3s pod (the operator was
right; my `which tailscale`/systemd probe was too narrow and missed it), and PBS is offsite. The
redundant host-level Tailscale I briefly installed on DooPlex was **fully purged** (package + repo +
keyring + sysctl file), verified: k3s `tailscale0` and the `dooplex` node stayed up, `ip_forward`
still 1.
## Residual
### 3. accept-routes spike (§3.1) — CONFIRMED degradation, rolled back
Enabling `--accept-routes` on felhom-pve *while on `192.168.0.0/24`*:
```
PRE : ip route get 192.168.0.180 -> dev vmbr0 src 192.168.0.162 ; ping .180 avg 0.3 ms
POST: ip route get 192.168.0.180 -> dev tailscale0 table 52 src 100.70.170.35 ; ping .180 avg 68 ms (max 136)
table 52 contains: 192.168.0.0/24 dev tailscale0 ; ip rule 5270: from all lookup 52 (outranks main)
-> local subnet hijacked onto the tunnel; inbound SSH went asymmetric and STALLED
PBS (10.77.0.1 via wg-felhom) UNAFFECTED
ROLLBACK (via the tailnet path 100.70.170.35, which was immune): tailscale set --accept-routes=false
-> route to .180 back on vmbr0 direct
```
Verdict: keep `--accept-routes=false` at home; it is a **travel-only opt-in** (safe on a foreign
subnet). Documented with the reasoning in `documentation/operations/tailscale.md`.
### 4. SSH chain over the tailnet (§3.2)
`~/.ssh/config` on DooPlex: `Host felhom-pve` → `HostName 100.70.170.35`; added `Host felhom-pve-lan`
→ `192.168.0.162`. Proof:
```
ssh felhom-pve -> pve-manager/9.2.2 ; pct exec 9201 docker ps -> felhom-controller:0.148.0 Up (healthy)
tailscale ping 100.70.170.35 -> pong via 192.168.0.162:41641 in 1ms (DIRECT over LAN, not DERP)
ssh felhom-pve-lan -> OK (LAN fallback works)
```
### 5. Part 4 — host static→DHCP: **DONE** (operator at console, reservation set)
Reservation MAC `68:1d:ef:5d:a6:64`→`192.168.0.162` set by operator first (so the LAN-pinned agent
keeps working at home). Applied detached (`systemd-run … ifreload -a`); result:
```
ifreload rc=0 ; vmbr0 inet dhcp -> got 192.168.0.162 back (reservation)
default via 192.168.0.1 ; felhom-agent active, still bound 192.168.0.162:8443 (no restart)
guest 9201: 12 containers ; Tailscale still direct 1ms
```
Exact revert staged on the box: `/root/interfaces.static-revert-20260719`. resolv.conf is now
DHCP-managed (`.250` + `.1`) instead of the old static `search local` — expected (Tailscale didn't
touch it; accept-dns rule intact).
### 6. Off-LAN legs NOT validated at home → vacation-day checklist
Cannot be tested until the box is on a foreign LAN: PBS from a foreign subnet, DERP-relay fallback,
off-LAN `gitea.dooplex.hu` resolution. The step-by-step is in `documentation/operations/tailscale.md`
(§ *Vacation-day checklist*) — notably `ssh felhom-pve "pvesm status | grep -i pbs"` is the real
off-LAN PBS test.
### 7. Findings recorded, NOT fixed
1. **Control plane pinned to `192.168.0.162`** (HIGH for travel): `felhom-agent` `listen_addr` and
guest `bootstrap.json` `endpoint` both = `192.168.0.162:8443`. Off-LAN the agent can't bind →
agent down → controller can't reach it. Subnet routes don't help (L2 bridge). Needs a design
change (bind `0.0.0.0` + stable guest→host path). **The N100 is reachable and backs up off-LAN,
but its local operator plane is not yet portable.**
2. **Off-LAN gitea DNS**: `gitea.dooplex.hu` (pi-hole split-horizon) won't resolve on a foreign LAN;
image pulls while travelling need the travel `--accept-routes` toggle + a temporary `/etc/hosts`
entry. Contingency ("Plan B") documented, not armed.
### 8. Commit
Docs commit: `83c47ea` (felhom.eu). Files: `documentation/operations/tailscale.md` (new),
`CLAUDE.md` + `skills/felhom-build-deploy/SKILL.md` env-table notes, `CONTEXT.md` decision entry,
this `REPORT.md`. No secrets committed (tunnel keys/tokens/PBS fingerprint referenced out-of-band).
R-50b **(b)/(c) remain open** — the wrapper is still fetched unversioned from `raw/branch/main`; this
release makes drift visible, it does not fix the channel. The 0440 sudoers file is not agent-readable,
so its drift stays invisible. The DR-tier capability-map row is deliberately **not** upgraded to
PROVEN-LIVE until STOP-2 supplies the evidence.