admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
02d63ed0702e4336ef25d6fc16b889ac810f2f72
felhom.eu/CONTEXT.md
T
admin 02d63ed070 GL-2: --mode appliance|byo install profile (host-install v1.10.0) + harness 
Mandatory install profile (no default): byo = a host the operator does not
own. Break-glass gated OFF at its call site in byo (root@pam never touched),
mandatory --cores/--memory, argv-time refusals (--enable-oob/--rotate-recovery,
non-9.x PVE, missing --acl-storages), host-mutation disclosure + typed-hostname
ack, byo config asserts (lan_resolver/wg_tunnel/oob off; byo flips the
lan_resolver write default to off), pool+ACL verify asserts in BOTH modes (R2),
--preflight-only (no state, PASS/FAIL verdict), resume mode-mismatch refusal,
FELHOM_INSTALL_STATE_DIR harness override.

NEW scripts/hostinstall-mode-harness.sh: static refusal matrix C1-C4 + grep
invariants + PVE tier (C5 + A/B dry transcripts). 16/16 PASS on felhom-pve
(C5 live); red-proofs RP-1..RP-3 run->fail->revert. shellcheck clean at
severity=warning. Docs: day0-install SC.5 byo section + trust model; REUSE row;
CONTEXT + REPORT. Live drill = GL-6 (supervised); STOP honored (no non-dry run).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-07 20:26:32 +02:00

156 lines
14 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# CONTEXT — felhom.eu working state
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
- **2026-07-07 — TASK GL-2 SHIPPED: `--mode appliance|byo` install profile (host-install v1.10.0 +
harness)** — go-live items G2/G4/G5 implemented, **awaiting the supervised GL-6 drill** for live
provision/uninstall + C7 verify-drift. DECISIONS: the flag is MANDATORY (no default; the old
`--mode provision` value retired with a specific die, `dr` stub kept); break-glass gated at its
CALL SITE (byo never touches root@pam); byo requires `--cores`+`--memory`, refuses
`--enable-oob`/`--rotate-recovery`/non-9.x-PVE/missing `--acl-storages`; disclosure + typed-hostname
ack; **the config write's `lan_resolver` default flips to OFF in byo** (the spec assumed
default-off — it was default-ON; appliance unchanged) + post-write asserts refuse
lan_resolver/wg_tunnel/oob=true; pool+ACL verify asserts in BOTH modes (R2 lesson);
`--preflight-only` (no state, PASS/FAIL verdict, proven live on felhom-pve). NEW
`scripts/hostinstall-mode-harness.sh` (static C1–C4 + invariants everywhere; PVE tier: C5 +
A/B dry transcripts) — 16/16 PASS on felhom-pve incl. C5 live; red-proofs RP-1..3 run→fail→revert.
FINDING: demo-felhom's controller.yaml has EMPTY git.username/git.token → any step-5
(re)install for that customer dies until the operator sets real Gitea read creds (relates to the
standing scope-down+rotate follow-up). `documentation/pilot/GO-LIVE-PACKAGE.md` is NOT in the
repo and no operator copy was provided — the G2/G4/G5 status flip is recorded here instead;
add the doc when the operator supplies it. Next: **GL-6 supervised drill** (appliance + byo
end-to-end, uninstall/re-provision, C7, the ack prompt interactively).
- **2026-07-05 — TASK H1 SHIPPED: OOB operator access (hub v0.35.0 + installer + endpoint; agent
v0.72.0)** — merged E1+H1. Hub: operator OOB peer (`store/wg_operator.go`, `PUT/GET
/admin/wg/operator-peer` global key) + `oob_peer_ip`/`oob_operator_ssh_key` in the desired-state
wireguard block + `monitor/host_oob` (oob_degraded/recovered). Installer: `--enable-oob`/`install_oob`
(static felhom-sshd + belt). Doc 06 §4.5/§4.6 amended (forwarding per-pair allow-listed, box↔box
drop explicit, oob health stanza); endpoint runbook §9. Live-validated: operator→box SSH as
felhom-op with scoped sudo; OF-1 /32 self-heal survival; coexistence (:22 PID unchanged); belt
drops LAN + box↔box; PBS unaffected. Operator key = hub report_api_key.
- **2026-07-05 — TASK G1 SHIPPED: management-plane break-glass (hub v0.34.1 + installer; agent
v0.71.0)** — prereq for felhom-sshd (H1). Hub: `store.host_recovery` vault (per-host root@pam
console password; `PUT /hosts/{id}/recovery-credential` self-scoped + `GET /admin/hosts/{id}/
recovery-credential` global-only, secret never logged) + `host_mgmtplane` monitor raising
`mgmt_plane_healed`. Installer: `step_break_glass` (generate+set+vault root@pam, `--rotate-recovery`)
+ `install_mgmt_watchdog` (the 3 host artifacts, RuntimeDirectory-guarded). Live-validated:
agent-independent `/run/sshd` auto-heal in 30.0s; warning fired end-to-end; break-glass
vault→retrieve→PVE-ticket(200). felhom-pve root@pam is now the vaulted value.
- **2026-07-04 — S3 SHIPPED: agent WG tunnel (felhom-agent v0.64.0; docs-only here)** — the
doc-06 §3.3 handshake is now END-TO-END LIVE on demo: agent keygen → `POST /hosts/{id}/wg` →
desired-state `wireguard` block → agent-managed `wg-quick@wg-felhom` → PBS page over
10.77.0.1:8007 in 3 s. **`wg_tunnel.enabled` defaults FALSE agent-side** (rollouts to other
boxes are no-ops until the production endpoint exists). Revocation semantics completed (§3.5):
removed peer → agent disables + keeps its marker + NEVER re-registers; operator re-adds via the
pubkey the heartbeat report now carries. Escrow: `IdentityBundle.WGPrivateKey` auto-injected on
new escrow-create runs (pre-S3 blobs can't be retrofitted — honest fallback = fresh-key
re-register, /32 kept). OPEN: CGNAT/mobile-hotspot smoke (operator-assisted appendix, deferred
non-blocking); S4 = PBS over the tunnel; hub-side S6 = tunnel-health alerting on the new report
stanza. Detail: felhom-agent CHANGELOG v0.64.0 + REPORT.
- **2026-07-04 — S2 SHIPPED: box-facing WG registration + wireguard desired-state block + /offsite UI (hub v0.33.0)** —
doc-06 row S2. DECISIONS: the WG assignment is **hub-owned state, merged into served
desired-state at READ time** — stored `desired_json` stays a pure operator blob; admin PUT with
a top-level `wireguard` key → 400 (no copy-paste clobber); **re-key-in-place keeps the /32**
(stable tunnel addressing across rotation/DR); one bound peer per host (partial unique index);
generation bumps ONLY on real change (register/re-key/adopt/bound-delete — idempotent paths
move nothing). `POST /hosts/{id}/wg` self-scoped per-host key; NEW golden
`desired-state-wireguard.golden.json` = the S3 cross-repo contract (byte-identical agent copy
rule). peersync **v1.0.1** (strip out of process substitution — the S1 exit-swallow finding;
sandbox red-proofed) redeployed to the dev endpoint. Live-proven on the REAL demo-felhom-01
record via the real per-host key: gen 2→3, served block = golden shape, endpoint `wg show`
carried the peer, v0.63.0 agent tolerated the unknown key (fetch clean, guests untouched),
then removed → gen 4, all baselines restored. `/offsite` page unit-proven (live view is
operator-password-gated — eyeball after next login). Next: **S3** (agent keygen +
registration + `wg-felhom` service + escrow join + CGNAT smoke — risky/supervised).
- **2026-07-04 — S1 SHIPPED: offsite endpoint live + hub peer-sync (hub v0.32.1)** — the doc-06
roadmap's first slice, done-criterion met live. Peer-sync design point RESOLVED: **hub pushes
over SSH** to a forced-command `felhom-peersync` script (pull rejected: weakens immediate
revocation; HTTPS API rejected: new binary + third public port). Endpoint = the dev
`felhom-hetzner` box, provisioned by the NEW runbook `documentation/runbooks/offsite-endpoint.md`
(WG on **443/udp**, subnet 10.77.0.0/24, PBS datastore `felhom-offsite` at `/srv/pbs-felhom`,
nftables drop-all + wg0-only 8007, `ep0.felhom.eu` grey-cloud). Hub: `wg_endpoints`/`wg_peers` +
/32 allocator + `/admin/wg/*` (global-key, pubkey-in-body) + `internal/wgsync` (FixedHostKey pin
+ **pinned-type HostKeyAlgorithms — live-caught bug**: multi-hostkey sshd presented ECDSA vs the
ed25519 pin; fixed v0.32.1 + regression test) + full-list declarative reconciler (5-min drift
repair). Live: add/remove→wg show, malformed→state-untouched, reboot→persist+converge, client
tunnel via ep0:443→PBS login page, public 8007 closed. INCIDENT handled: first WG server key
leaked via `wg show dump` into the session log → rotated immediately (warning now in the
runbook). OPERATOR follow-up: **fix the AAAA record** (currently `…7aa1::`, must be `…7aa1::1`).
Next: **S2** (hub schema host-join + `wireguard` block on WireDesiredState + box-facing
registration + UI).
- **2026-07-03 — OFFSITE-CONNECTIVITY DESIGN-OF-RECORD WRITTEN (architecture Part 06)** —
`documentation/architecture/06-offsite-connectivity.md`: records the six settled decisions
(plain WG / host-side agent-managed `wg-felhom` as the agent-managed-unit PILOT / one shared
hub-driven endpoint VM running WG+PBS, NO agent on it / hub = source of truth, `wireguard`
block rides `WireDesiredState` on the existing DesiredGeneration channel / one datastore +
per-customer namespaces / PBS ON the VM, relay-through-DooPlex REJECTED as non-scaling),
the Day-0 join handshake, robustness set (WG-roaming-not-DynDNS, DNS re-resolve watchdog, MTU
1420, per-/32 isolation, tunnel-health via storage-target model), trust conformance (outbound
only; operator-inbound + game-ingress out of scope), the honest ledger (CGNAT still unmeasured
— mobile-hotspot smoke closure; peer-sync push-vs-pull = slice-1 design point), and the S1–S6
slice roadmap (**MVP = S1 endpoint + S2 hub schema/desired-state + S3 agent WG service, then S4
PBS-over-tunnel**; S5 merges with DR-completeness). day0-install backlog line now cites spike +
design doc. Docs-only; slices are separate future tasks.
- **2026-07-03 — OFFSITE-BACKUP TRANSPORT DECIDED (connectivity spike, GO at the lowest rung)** —
plain WireGuard, host-side, box-dials-out to a small public endpoint; **no Headscale/DERP needed
for backup** (separable fleet layer, adoptable later). Live-proven on demo-felhom → throwaway
Hetzner: 11.4-min idle hold, PBS API through the tunnel with the exact-cert pin intact
(positive + negative), real 2 GiB worst-case backup at 4.26 MiB/s (= the home uplink; tunnel
overhead ~5%), UDP 51820 *and* 443 both pass. P0 honesty: the operator's line is plain-NAT
(fixed public IP, One Hungary), NOT CGNAT, and has zero IPv6 — CGNAT confirmation deferred to
Peti's VM 110 (second ISP). P7 surprise: kernel WG *works* inside the unprivileged guest (no TUN
needed) — host placement stands on architecture (backups originate host-side; cloudflared
precedent), not infeasibility. Evidence + architecture recommendation (endpoint VM shapes, key
custody via IdentityBundle escrow, per-peer /32 isolation):
`documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md`. **This unblocks the
DR-completeness work: next task = production connectivity spec** (endpoint VM, agent-managed WG
service, hub peer registry, escrow join). Throwaway Hetzner box parked powered-off (secrets
shredded); demo-felhom verified residue-free.
- **2026-07-03 — 4th skill: `felhom-app-catalog`** (catalog authoring workflow; points at
app-catalog REUSE.md §1–2, no duplicated tables; auto-discovered by `install_skills.py`, 4/4
found fresh-session). Proven by finalizing SparkyFitness live on demo (image-inspected
healthchecks, probe-container naming rule discovered + REUSE row added, UI-applied, controller
probe healthy). App-count drift recorded: website says "45+", per-category spans sum 49, real
catalog 53 — website fix is a separate task.
- **2026-07-03 — GOLDEN 0.98.3 LIVE + VOUCHED; guide D.1b RETIRED (drill B1+B5 FIXED)** —
`build-golden.sh` v2.0.0 (mandatory controller tag + baked bootstrap `.path` unit); golden baked
with controller 0.98.3, clean-room validated (drill VM) through all four scenarios incl. the
published-artifact `--force-gitea-golden` path, published + operator-vouched (Day-0 manifest now:
agent 0.63.0 + golden 0.98.3). Fresh installs land CURRENT and self-manage — D.1b is a one-line
check; the prior "vouch agent 0.63.0" operator follow-up is DONE. Evidence:
`documentation/audits/DRILL-golden-098-2026-07-03.md`. NEW operator follow-ups: (1) SECURITY —
the customer-config `git.token` is a Gitea admin token with package-WRITE (proven during the
bake); scope down + rotate. (2) optionally raise the global controller floor to 0.98.3.
- **2026-07-03 — drill findings B2+B3 FIXED (agent v0.63.0, live on felhom-pve + published)** —
token reload-on-miss (no more fresh-install 401/manual restart) + guesthook snippets-dir mkdir.
Guide D.1b restart step narrowed to "agent < v0.63.0"; drill ledger updated. **OPERATOR: vouch
agent 0.63.0 in the Day-0 manifest UI** (until then fresh installs fetch 0.62.0).
- **2026-07-03 — GO-LIVE PACKAGE SHIPPED (host-install v1.9.1 + day0-install runbook + clean-room drill)** —
`documentation/runbooks/day0-install.md` is the operator Day-0 guide, **every command executed
verbatim** on a throwaway nested PVE 9.2.2 (build server): install → verify → app deploy →
uninstall → **zero-residue diff** → reinstall. v1.9.1 = residue-free uninstall (agent config w/
hub api_key, shared-parent unit/script/mount, mkfs wrapper, hook snippet, dnsmasq snippets) +
post-provision guest reboot (without it the controller never deploys on slower hardware) + header
sync. Evidence + go/no-go for Peti (**GO**, 3 install-day items):
`documentation/audits/DRILL-day0-cleanroom-2026-07-03.md`. Key OQ-2 finding: golden bakes a
pre-floor controller 0.85.1 → **fresh installs need the one-time D.1b update** (agent restart →
update button → 0.98.3); follow-ups recorded: rebuild golden ≥0.86.0 + re-vouch (operator), agent
`mkdir -p` for the guesthook snippet dir, fresh-install local-API 401 (pre-existing), hub has no
host-delete path. Drill VM parked virgin-snapshotted on 180 (`~/drill/`).
- **2026-07-03 — A1 CLOSED (host-install v1.9.0 + agent v0.62.0)** — `Pool.Audit` added to
`FelhomAgentGuest`; the agent's stale-lock reaper now intersects its scan with
`GET /pools/felhom` membership (fail-safe skip on read failure). Spike:
`documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md`. Deploy order on any box:
rescope ACL first, then the agent. STILL PENDING (operator): rotate + de-git the 3 committed
secrets in `manifests/felhom.secret.yaml` (spike appendix; REUSE §2 errata added).
- **2026-07-03 — hub v0.31.0 LIVE** (critical-severity ingest fix + UI badges; ArgoCD Synced/Healthy;
live probe: critical event stored + operator email delivered).
- **2026-07-03 — Felhom skills exist**: `skills/{felhom-build-deploy,felhom-ui-design,felhom-testing}`,
installed via `scripts/install_skills.py` — **junction mode** (repo edits live immediately; no
re-install needed after editing). Fresh-session discovery verified.
- **2026-07-03 — CLAUDE.md refreshed** (all four repos): no version-pinned state in CLAUDE.md —
current state lives here / CHANGELOG / REUSE.md.
- **2026-07-03 — `REUSE.md` exists at the repo root** (hub helpers / website conventions / gate scripts / manifest patterns, code-verified); maintenance rule active: update it in the same commit that changes a shared helper. `scripts/reuse_refs_check.py` machine-checks its citations across all four felhom repos.
Reference in New Issue View Git Blame Copy Permalink