admin
9bd1a54d71
A census of all thirteen gate scripts across the four felhom repos on 2026-08-02 found one clean correlation: every check a CLAUDE.md tells a person to run was passing, and two of the four nobody is told to run were failing — one since 14 July. Neither failure was harmful in effect (checked line by line); nothing would have said so if they had been. The fix is not more gates, it is one place to run them from. repo_gates.py runs site + hostinstall + hub-confirm + manifest-bearer + reuse-refs, streams each gate's own output, and exits worst-wins non-zero. A missing gate script is a FAILURE and prints the path tried — fail-closed, because a runner that quietly skips a gate is the inert-seam failure this project has shipped four times. It copies catalog_gates.py (R-161), NOT site_gates.py, which is a gate and not a runner. .githooks/pre-push runs it with --fast and refuses the push. Honest limits are written into the hook itself: per-clone (core.hooksPath is local config), and --no-verify bypasses it on purpose. Any manual run WARNS when the clone is unarmed. Measured on git 2.47.3: a relative core.hooksPath resolves correctly and the hook's cwd is the repo root from any subdirectory. test_repo_gates.py is a SEAM test — it asserts each member gate's own distinctive stdout, not the runner's summary line, which an inert runner prints while calling nothing. Red-proofed: replacing run_gate's body with 'return 0' still prints 'all felhom.eu gates OK' and exits 0, and turns the seam test red.
214 lines
14 KiB
Markdown
214 lines
14 KiB
Markdown
# CLAUDE.md — Project Instructions for Claude Code (`felhom.eu`)
|
||
|
||
> Read automatically when Claude Code works in this repo. Stable orientation only — **current state
|
||
> lives in `CONTEXT.md` and the tops of `hub/CHANGELOG.md` / `scripts/CHANGELOG.md` /
|
||
> `website/CHANGELOG.md`**, never here. Cross-repo orientation (the felhom system, artifact
|
||
> taxonomy, access): workspace-root `/mnt/5_hdd/felhom.eu/git/CLAUDE.md`; this file is
|
||
> `felhom.eu`-specific. A versioned copy of that workspace file lives at
|
||
> `documentation/runbooks/workspace-CLAUDE.md`.
|
||
|
||
## Project overview
|
||
|
||
This repo contains:
|
||
- **Website** (`website/`) — static HTML at felhom.eu, served via k3s nginx + git-sync sidecar.
|
||
- **Hub** (`hub/`) — Go application (felhom-hub), the **operator backend**, on k3s at `hub.felhom.eu`.
|
||
- **K8s manifests** (`manifests/`) — k3s deployment manifests for felhom-system services.
|
||
- **Architecture docs** (`documentation/`) — the **authoritative design home for the whole Felhom
|
||
system**: `architecture/01..05-*.md`, `proxmox-platform.md`, `tests/phase*-findings.md`,
|
||
runbooks, audits. Read these before designing.
|
||
- **Skills** (`skills/`) — the versioned source of the Claude Code skills
|
||
(`felhom-build-deploy`, `felhom-ui-design`, `felhom-testing`, `felhom-app-catalog`);
|
||
install/update with `python3 scripts/install_skills.py` (symlink into `~/.claude/skills/` on
|
||
POSIX, junction on Windows — either way repo edits are live immediately).
|
||
|
||
See `README.md` for full architecture/DNS/email/SEO docs. See `TASK.md` for the current task (if any).
|
||
See `REUSE.md` before writing new code.
|
||
|
||
## The Felhom system (so the hub's role is in context)
|
||
|
||
Felhom is **Proxmox-based**, with a locked **three-component model**:
|
||
- **Hub** (this repo, `hub/`) — operator backend. Authors operator *intent*; mirrors box *reality*;
|
||
holds **no data-plane role** and never connects inbound to a box.
|
||
- **Host agent** (repo `felhom-agent/`) — one per Proxmox host; owns all Proxmox interaction.
|
||
- **In-guest controller** (repo `felhom-controller/`) — one per customer LXC; Docker-only.
|
||
|
||
## Hub — architecture (version-free; current version = `manifests/hub.yaml` image tag)
|
||
|
||
The hub ingests two report streams — the agent's host-domain report (`POST /api/v1/host-report`, the
|
||
heartbeat/dead-man's-switch) and the legacy controller report (`POST /api/v1/report`, **frozen until
|
||
the slice-10 cutover — do not modify**) — plus structured controller events (`POST /api/v1/event`,
|
||
gated by `allowedEventTypes`). Around them: staleness/disk/storage-fill/leaf/capability monitor
|
||
checkers, the two-tier notification dispatcher (operator English / customer Hungarian, Resend,
|
||
cooldowns), the app-mail relay, customer-config + Day-0 artifact-manifest management (the checksum
|
||
trust root the host bootstrap verifies against), assets serving, and the password-gated operator web
|
||
UI. Package map, helpers, seams, extension points: **`REUSE.md`** (e.g. new event types must enter
|
||
`allowedEventTypes` + `customerMessages` together).
|
||
|
||
## Code quality rules
|
||
|
||
- Always double-check generated code for bugs, logic issues, syntax errors.
|
||
- Handle edge cases without overcomplicating.
|
||
- Add debug capabilities (logging, verbose output).
|
||
- If you need more input or troubleshooting output, **ask first — don't guess**.
|
||
- Testing doctrine (non-hollow tests, red-proofs, seams): use the `felhom-testing` skill.
|
||
- **Seam-wiring rule — and it covers TEMPLATE GATES (fourth inert seam, hub v0.70.1):** a feature
|
||
is not shipped until its entry point is reachable. For UI, any conditional affordance
|
||
(`{{if .Flag}}` around a button/form/script) ships with a render test per branch of the gate —
|
||
handler tests that POST directly prove nothing about reachability. The v0.70.0 ghost-delete was
|
||
fully implemented server-side and fully dead UI because the button sat inside the wrong gate.
|
||
- **A health check issues no block I/O.** A probe that touches a wedged device enters uninterruptible
|
||
sleep, survives `SIGKILL`, and cannot be recovered until the device returns or the host reboots — so
|
||
`systemctl restart` hangs too. A timeout protects the caller's control flow and nothing else: the
|
||
blocked thread remains. Liveness is decided from `/proc` and the kernel's own state, never by reading
|
||
or writing the filesystem. Measured, R-117 spike §6.3
|
||
(`documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md`): a probe stayed in `D` state 3m50s
|
||
after `kill -9`; a buffered write with no `fsync` blocked too (`O_CREAT` needs journal access); and
|
||
`statfs`/`getdents` returned **healthy** on a namespace that `EIO`s every byte — fast, and wrong.
|
||
- UI/design work (tokens, gates, copy rules): use the `felhom-ui-design` skill.
|
||
- **Logging**: levels/English/no-secrets rules per `documentation/runbooks/logging-conventions.md`
|
||
(DEBUG = flow detail, INFO = state change + duration; logs are operator-tier English; keys never
|
||
values — the hub's bundle secret-gate blocks violating pulls fail-closed).
|
||
|
||
## Workflow & artifacts
|
||
|
||
The planning/architecture assistant ("project Claude", in claude.ai) writes specs and validates
|
||
pushes; **you (Claude Code) implement**. A file being open in the editor is NOT an instruction.
|
||
|
||
- **`TASK.md` / `TASK-*.md`** — a spec for you to implement. Then push and update `hub/CHANGELOG.md`
|
||
and root `REPORT.md` per the convention below.
|
||
- **`RUNBOOK-*.md`** — an operational procedure. CC executes the steps it has access and capability
|
||
for, including live validation on the demo nodes and the demo Proxmox host (CC has root@felhom-pve
|
||
SSH + the felhom-agent token). Mark a step HUMAN only when it genuinely needs physical presence, a
|
||
real-world decision, or credentials CC truly lacks.
|
||
- Validation of a push against a spec's criteria is project Claude's job, not yours, unless asked.
|
||
- **Browser automation is NOT available** in the DooPlex environment (`claude-in-chrome` was a
|
||
Windows-workstation capability). Validate at the endpoint level — invoke the exact endpoint the UI
|
||
invokes — and via render tests; say which method was used. The hub UI is operator-password-gated
|
||
anyway, so render tests were already the method for UI changes. Strict end-to-end UI coverage is a
|
||
manual click-through by the operator.
|
||
|
||
> **In every repository where you make a change, update both files in that repo:**
|
||
> - **`CHANGELOG.md`** — cumulative log, newest on top (here: per-area `hub/`, `scripts/`, `website/`).
|
||
> - **`REPORT.md`** — **overwrite** with the most recent implementation/validation summary only.
|
||
> **Parallel sessions:** `REPORT.md` is overwritten, so two sessions working in this repo at once
|
||
> will clobber each other. The second session writes **`REPORT-<topic>.md`** instead and never
|
||
> touches the shared `REPORT.md`.
|
||
>
|
||
> **Never write secrets** into any committed file — reference them as "stored out-of-band".
|
||
|
||
- Update `REUSE.md` if you added/changed/deprecated a shared helper or pattern (same commit).
|
||
- **Never `git add -A` in this repo** — parallel sessions share the clone and it sweeps foreign
|
||
WIP (the v0.47.0 `146d165` incident: a red-proof-mutated guard got swept to `main`). Stage
|
||
explicit paths only, `git pull --rebase` before every push, and do not run two writing
|
||
sessions on one clone (use `git worktree` if truly needed).
|
||
|
||
## End-of-session checklist
|
||
|
||
- **`CHANGELOG.md` + `REPORT.md`** per the rule above, in every repo touched.
|
||
- **`REUSE.md`**, if a shared helper or pattern moved (same commit).
|
||
- **The capability map** (`documentation/architecture/00-capability-map.md`), if a capability's status
|
||
changed — with its new evidence citation.
|
||
- **The architecture doc** that owns any changed contract (S-1, `CONTEXT.md`).
|
||
- **Root `STATUS.md`** — *update it at the end of every session in which something shipped, broke, or
|
||
was decided. It is a view of `documentation/backlog/OPEN-ITEMS.md` — **nothing may exist only
|
||
there**. One screen; cut items rather than extending it.* It is written for the **operator** in
|
||
plain language, and is deliberately **not** `CONTEXT.md` — do not consolidate the two.
|
||
- **A finding goes in `OPEN-ITEMS.md` first**, never only in a report, an audit or `STATUS.md`. Four
|
||
items in this project were minted in a spike doc and lost (R-153/154/155, R-156/157).
|
||
|
||
## Tech stack (Hub)
|
||
|
||
- **Language:** Go (stdlib `net/http` + `html/template`, no frameworks). **DB:** SQLite via
|
||
`modernc.org/sqlite` (pure Go). **Auth:** bcrypt + Bearer tokens + session cookies + CSRF.
|
||
- **Deploy:** Docker on k3s (`felhom-system` ns). **Storage:** Longhorn PVC at `/data/` (SQLite DB).
|
||
- **Config:** YAML via ConfigMap at `/etc/felhom-hub/hub.yaml`. Secrets via out-of-band
|
||
`secretKeyRef` (never inline stringData — REUSE.md §3).
|
||
|
||
## Environment & access
|
||
|
||
Claude Code runs **on DooPlex (192.168.0.180, Debian 13, user `kisfenyo`)** — the k3s node itself.
|
||
Repos in `/mnt/5_hdd/felhom.eu/git/`, build dirs in `/mnt/5_hdd/felhom.eu/build/`. `kubectl` and the
|
||
image build/push are local commands; felhom-pve is one SSH hop.
|
||
|
||
| Host | Access | Role | Blast radius |
|
||
|------|--------|------|--------------|
|
||
| **DooPlex (this host)** | local — `/mnt/5_hdd/felhom.eu/{git,build}/` | Build + push images, `sudo kubectl` | **Tier 2 — precious.** It *is* the recovery chain (hub, Gitea, registry, PBS, k3s+Longhorn). **Never a drill target** |
|
||
| Demo Proxmox host (N100) | `ssh felhom-pve` — via Tailscale `100.70.170.35` (location-independent); `felhom-pve-lan` = LAN `192.168.0.162` fallback | pveum/pct + live Proxmox validation | **Tier 0 — disposable** |
|
||
| Demo Proxmox host (HP t740) | `ssh demo-hp` — via Tailscale `100.76.96.79`; `demo-hp-lan` = LAN `192.168.0.87` (ProxyJump `felhom-pve`). **No baked SSH key** — G1 break-glass password vaulted in the hub | **The designated drill + build VM host** (operator ruling 2026-07-25) | **Tier 0 — disposable. Reach here first** |
|
||
|
||
**Which box do I break?** → **`documentation/runbooks/target-selection.md`** — the tiers, and per
|
||
machine what is freely permitted / needs care / forbidden, each with its reason. Read it before picking
|
||
a machine for a drill, a destructive test or a throwaway VM.
|
||
|
||
The `felhom-pve` transport is Tailscale (the N100 is travel-portable) — topology, the accept-dns
|
||
rule, the accept-routes spike result, rollback, and the vacation-day checklist live in
|
||
`documentation/operations/tailscale.md`.
|
||
|
||
> **Legacy: Windows workstation.** Until 2026-07-19 CC ran on Windows 11 with repos in `E:\git\`,
|
||
> and every remote command needed `SSH=/c/Windows/System32/OpenSSH/ssh.exe` (Git Bash's ssh fails
|
||
> silently). Retained in case that environment is revived.
|
||
|
||
## Build & deploy — Hub (GitOps via ArgoCD)
|
||
|
||
**Full runbook: use the `felhom-build-deploy` skill.** The load-bearing rules:
|
||
|
||
The whole cluster is GitOps via a **single ArgoCD app `felhom`** syncing this repo's `manifests/`
|
||
to `felhom-system`. **Auto-sync is OFF** — deploys are a deliberate manual sync. ArgoCD's source of
|
||
truth is the **manifest**:
|
||
|
||
- **A code change + CHANGELOG bump deploys NOTHING.** The running image changes only when
|
||
`manifests/hub.yaml`'s `image:` tag changes in git and the app is synced.
|
||
- **Pin explicit versions, never `:latest`.** Never bare `kubectl set image`/`kubectl apply`
|
||
(reverted on next sync).
|
||
- **The live image can lag the CHANGELOG** when a bump was committed but the manifest/sync step never
|
||
happened — reconcile via the manifest, not the changelog.
|
||
- Green gate before any hub commit: `go build ./... && go vet ./... && go test ./...` in `hub/`.
|
||
|
||
> **Clean-tree gate before any build:** `git status --porcelain` must be empty and
|
||
> `git rev-parse HEAD` must equal `git rev-parse origin/main` in the repo being built. An unpushed
|
||
> change does not exist — never build a dirty or unpushed tree. The `git pull` in the build step
|
||
> stays (it is a no-op when you work in this tree, and load-bearing if anything was pushed from
|
||
> elsewhere).
|
||
|
||
Steps: commit+push code → `cd /mnt/5_hdd/felhom.eu/build/felhom-hub && ./build.sh <VER> --push`
|
||
(local) → bump `manifests/hub.yaml` tag + push → ArgoCD hard-refresh + sync (kubectl-patch method in
|
||
the skill, now local `sudo kubectl`) → verify Synced/Healthy + rollout + image + startup log.
|
||
|
||
## Gates — ONE entry point
|
||
|
||
**Run `python3 scripts/repo_gates.py` after ANY change in this repo.** It is the one entry point
|
||
and runs every gate — `site_gates.py`, `hostinstall_gates.py`, `hub_confirm_gate.py`,
|
||
`manifest_bearer_gate.py` and `reuse_refs_check.py` on this root — streaming each gate's own output
|
||
and exiting non-zero if any fails. `--fast` selects only the gates that touch no network and no
|
||
container runtime; today that is all of them. A missing gate script is a FAILURE, never a skip.
|
||
|
||
**Why a runner and not five invocations** (2026-08-02, R-29): a census of all thirteen gates across
|
||
the four repos found that every check a `CLAUDE.md` names was passing, and two of the four nobody
|
||
is told to run were failing — one since 14 July. The single-entry-point shape is the only one that
|
||
demonstrably gets run here; `app-catalog-felhom.eu/scripts/catalog_gates.py` is the canonical
|
||
version of it (R-161) and `repo_gates.py` copies it. `site_gates.py` is a *gate*, not a runner —
|
||
do not model new work on it.
|
||
|
||
**The pre-push hook.** `.githooks/pre-push` runs `repo_gates.py --fast` and refuses the push if it
|
||
fails. It is **per-clone** and switched on once with `git config core.hooksPath .githooks` — a
|
||
clone does not carry it, and any manual `repo_gates.py` run WARNS when this clone is unarmed.
|
||
`git push --no-verify` bypasses it deliberately; **say so in the session report when you use it**.
|
||
Both facts are why continuous integration is still owed (`OPEN-ITEMS.md` R-168) — this hook is
|
||
local and skippable, and only CI is neither.
|
||
|
||
## Build & deploy — Website / Manifests
|
||
|
||
- **Website** auto-deploys via git-sync; just push to `main` (live in 1–2 min). Website changes go
|
||
through `repo_gates.py` above (it runs `site_gates.py`); new pages go into that gate's `PAGES`
|
||
list. Emergency edits: https://files.felhom.eu. All `website/` HTML is **UTF-8 with BOM** — preserve it.
|
||
- **Manifests** are GitOps via the `felhom` app — commit to `main`, then deliberate sync.
|
||
|
||
## Key patterns
|
||
|
||
- Hub status logic: OK (report < 30m), WARN (30m–1h or health=warn), DOWN (> 1h or health=fail);
|
||
host liveness thresholds shared between UI and checker (never invent a second definition).
|
||
- SQLite timestamps vary in format — always `parseSQLiteTime()`.
|
||
- Dashboard/detail auto-refresh every 60s via meta refresh. Geo-restricted to Hungary via nginx
|
||
ingress annotation.
|
||
- Helpers, seams, extension points, traps: **`REUSE.md`** — the map is maintained same-commit.
|