felhom.eu/CLAUDE.md

# CLAUDE.md — Project Instructions for Claude Code (`felhom.eu`)

> Read automatically when Claude Code works in this repo. Stable orientation only — **current state
> lives in `CONTEXT.md` and the tops of `hub/CHANGELOG.md` / `scripts/CHANGELOG.md` /
> `website/CHANGELOG.md`**, never here. Cross-repo orientation (the felhom system, artifact
> taxonomy, access): workspace-root `/mnt/5_hdd/felhom.eu/git/CLAUDE.md`; this file is
> `felhom.eu`-specific. A versioned copy of that workspace file lives at
> `documentation/runbooks/workspace-CLAUDE.md`.

## Project overview

This repo contains:
- **Website** (`website/`) — static HTML at felhom.eu, served via k3s nginx + git-sync sidecar.
- **Hub** (`hub/`) — Go application (felhom-hub), the **operator backend**, on k3s at `hub.felhom.eu`.
- **K8s manifests** (`manifests/`) — k3s deployment manifests for felhom-system services.
- **Architecture docs** (`documentation/`) — the **authoritative design home for the whole Felhom
  system**: `architecture/01..05-*.md`, `proxmox-platform.md`, `tests/phase*-findings.md`,
  runbooks, audits. Read these before designing.
- **Skills** (`skills/`) — the versioned source of the Claude Code skills
  (`felhom-build-deploy`, `felhom-ui-design`, `felhom-testing`, `felhom-app-catalog`);
  install/update with `python3 scripts/install_skills.py` (symlink into `~/.claude/skills/` on
  POSIX, junction on Windows — either way repo edits are live immediately).

See `README.md` for full architecture/DNS/email/SEO docs. See `TASK.md` for the current task (if any).
See `REUSE.md` before writing new code.

## The Felhom system (so the hub's role is in context)

Felhom is **Proxmox-based**, with a locked **three-component model**:
- **Hub** (this repo, `hub/`) — operator backend. Authors operator *intent*; mirrors box *reality*;
  holds **no data-plane role** and never connects inbound to a box.
- **Host agent** (repo `felhom-agent/`) — one per Proxmox host; owns all Proxmox interaction.
- **In-guest controller** (repo `felhom-controller/`) — one per customer LXC; Docker-only.

## Hub — architecture (version-free; current version = `manifests/hub.yaml` image tag)

The hub ingests two report streams — the agent's host-domain report (`POST /api/v1/host-report`, the
heartbeat/dead-man's-switch) and the legacy controller report (`POST /api/v1/report`, **frozen until
the slice-10 cutover — do not modify**) — plus structured controller events (`POST /api/v1/event`,
gated by `allowedEventTypes`). Around them: staleness/disk/storage-fill/leaf/capability monitor
checkers, the two-tier notification dispatcher (operator English / customer Hungarian, Resend,
cooldowns), the app-mail relay, customer-config + Day-0 artifact-manifest management (the checksum
trust root the host bootstrap verifies against), assets serving, and the password-gated operator web
UI. Package map, helpers, seams, extension points: **`REUSE.md`** (e.g. new event types must enter
`allowedEventTypes` + `customerMessages` together).

## Code quality rules

- Always double-check generated code for bugs, logic issues, syntax errors.
- Handle edge cases without overcomplicating.
- Add debug capabilities (logging, verbose output).
- If you need more input or troubleshooting output, **ask first — don't guess**.
- Testing doctrine (non-hollow tests, red-proofs, seams): use the `felhom-testing` skill.
- **Seam-wiring rule — and it covers TEMPLATE GATES (fourth inert seam, hub v0.70.1):** a feature
  is not shipped until its entry point is reachable. For UI, any conditional affordance
  (`{{if .Flag}}` around a button/form/script) ships with a render test per branch of the gate —
  handler tests that POST directly prove nothing about reachability. The v0.70.0 ghost-delete was
  fully implemented server-side and fully dead UI because the button sat inside the wrong gate.
- **A `go test -run` pattern that matches no test prints `ok` and exits 0.** Found 2026-08-02 while
  red-proofing: `-run TestCustomerUnified` matched nothing in the target file and reported
  `ok … 0.062s`, which was read as a passing red-proof. **A red-proof that uses `-run` must first
  prove the filter matched something** (`-v` and look for `=== RUN`). This is the "an absent line is
  not evidence" rule aimed at the one place a false green costs most — the proof itself. The same
  class bit twice that day: a `| tail -5` inside a census query silently dropped rows and looked
  exactly like a real finding. **An instrument that can drop results silently is not a measurement.**
- **A health check issues no block I/O.** A probe that touches a wedged device enters uninterruptible
  sleep, survives `SIGKILL`, and cannot be recovered until the device returns or the host reboots — so
  `systemctl restart` hangs too. A timeout protects the caller's control flow and nothing else: the
  blocked thread remains. Liveness is decided from `/proc` and the kernel's own state, never by reading
  or writing the filesystem. Measured, R-117 spike §6.3
  (`documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md`): a probe stayed in `D` state 3m50s
  after `kill -9`; a buffered write with no `fsync` blocked too (`O_CREAT` needs journal access); and
  `statfs`/`getdents` returned **healthy** on a namespace that `EIO`s every byte — fast, and wrong.
- UI/design work (tokens, gates, copy rules): use the `felhom-ui-design` skill.
- **Logging**: levels/English/no-secrets rules per `documentation/runbooks/logging-conventions.md`
  (DEBUG = flow detail, INFO = state change + duration; logs are operator-tier English; keys never
  values — the hub's bundle secret-gate blocks violating pulls fail-closed).

## Workflow & artifacts

The planning/architecture assistant ("project Claude", in claude.ai) writes specs and validates
pushes; **you (Claude Code) implement**. A file being open in the editor is NOT an instruction.

- **`TASK.md` / `TASK-*.md`** — a spec for you to implement. Then push and update `hub/CHANGELOG.md`
  and root `REPORT.md` per the convention below.
- **`RUNBOOK-*.md`** — an operational procedure. CC executes the steps it has access and capability
  for, including live validation on the demo nodes and the demo Proxmox host (CC has root@felhom-pve
  SSH + the felhom-agent token). Mark a step HUMAN only when it genuinely needs physical presence, a
  real-world decision, or credentials CC truly lacks.
- Validation of a push against a spec's criteria is project Claude's job, not yours, unless asked.
- **Browser automation is NOT available** in the DooPlex environment (`claude-in-chrome` was a
  Windows-workstation capability). Validate at the endpoint level — invoke the exact endpoint the UI
  invokes — and via render tests; say which method was used. The hub UI is operator-password-gated
  anyway, so render tests were already the method for UI changes. Strict end-to-end UI coverage is a
  manual click-through by the operator.

> **In every repository where you make a change, update both files in that repo:**
> - **`CHANGELOG.md`** — cumulative log, newest on top (here: per-area `hub/`, `scripts/`, `website/`).
> - **`REPORT.md`** — **overwrite** with the most recent implementation/validation summary only.
>   **Parallel sessions:** `REPORT.md` is overwritten, so two sessions working in this repo at once
>   will clobber each other. The second session writes **`REPORT-<topic>.md`** instead and never
>   touches the shared `REPORT.md`.
>
> **Never write secrets** into any committed file — reference them as "stored out-of-band".

- Update `REUSE.md` if you added/changed/deprecated a shared helper or pattern (same commit).
- **Never `git add -A` in this repo** — parallel sessions share the clone and it sweeps foreign
  WIP (the v0.47.0 `146d165` incident: a red-proof-mutated guard got swept to `main`). Stage
  explicit paths only, `git pull --rebase` before every push, and do not run two writing
  sessions on one clone (use `git worktree` if truly needed).

## End-of-session checklist

- **`CHANGELOG.md` + `REPORT.md`** per the rule above, in every repo touched.
- **`REUSE.md`**, if a shared helper or pattern moved (same commit).
- **The capability map** (`documentation/architecture/00-capability-map.md`), if a capability's status
  changed — with its new evidence citation.
- **The architecture doc** that owns any changed contract (S-1, `CONTEXT.md`).
- **Root `STATUS.md`** — *update it at the end of every session in which something shipped, broke, or
  was decided. It is a view of `documentation/backlog/OPEN-ITEMS.md` — **nothing may exist only
  there**. One screen; cut items rather than extending it.* It is written for the **operator** in
  plain language, and is deliberately **not** `CONTEXT.md` — do not consolidate the two.
- **A finding goes in `OPEN-ITEMS.md` first**, never only in a report, an audit or `STATUS.md`. Four
  items in this project were minted in a spike doc and lost (R-153/154/155, R-156/157).
- **Confirm your own last push's CI run went green, by run ID.** CI emails on failure, which is a
  PUSH signal — this is the PULL check that catches a lost, filtered or unread mail. Quote the run
  id and its conclusion in the session report, e.g.
  `curl -s "https://gitea.dooplex.hu/api/v1/repos/admin/<repo>/actions/tasks?limit=3"` → match the
  `head_sha` to your commit. An unchecked green is an assumption, not an observation.

## Tech stack (Hub)

- **Language:** Go (stdlib `net/http` + `html/template`, no frameworks). **DB:** SQLite via
  `modernc.org/sqlite` (pure Go). **Auth:** bcrypt + Bearer tokens + session cookies + CSRF.
- **Deploy:** Docker on k3s (`felhom-system` ns). **Storage:** Longhorn PVC at `/data/` (SQLite DB).
- **Config:** YAML via ConfigMap at `/etc/felhom-hub/hub.yaml`. Secrets via out-of-band
  `secretKeyRef` (never inline stringData — REUSE.md §3).

## Environment & access

Claude Code runs **on DooPlex (192.168.0.180, Debian 13, user `kisfenyo`)** — the k3s node itself.
Repos in `/mnt/5_hdd/felhom.eu/git/`, build dirs in `/mnt/5_hdd/felhom.eu/build/`. `kubectl` and the
image build/push are local commands; felhom-pve is one SSH hop.

| Host | Access | Role | Blast radius |
|------|--------|------|--------------|
| **DooPlex (this host)** | local — `/mnt/5_hdd/felhom.eu/{git,build}/` | Build + push images, `sudo kubectl` | **Tier 2 — precious.** It *is* the recovery chain (hub, Gitea, registry, PBS, k3s+Longhorn). **Never a drill target** |
| Demo Proxmox host (N100) | `ssh felhom-pve` — via Tailscale `100.70.170.35` (location-independent); `felhom-pve-lan` = LAN `192.168.0.162` fallback | pveum/pct + live Proxmox validation | **Tier 0 — disposable** |
| Demo Proxmox host (HP t740) | `ssh demo-hp` — via Tailscale `100.76.96.79`; `demo-hp-lan` = LAN `192.168.0.87` (ProxyJump `felhom-pve`). **No baked SSH key** — G1 break-glass password vaulted in the hub | **The designated drill + build VM host** (operator ruling 2026-07-25) | **Tier 0 — disposable. Reach here first** |

**Which box do I break?** → **`documentation/runbooks/target-selection.md`** — the tiers, and per
machine what is freely permitted / needs care / forbidden, each with its reason. Read it before picking
a machine for a drill, a destructive test or a throwaway VM.

The `felhom-pve` transport is Tailscale (the N100 is travel-portable) — topology, the accept-dns
rule, the accept-routes spike result, rollback, and the vacation-day checklist live in
`documentation/operations/tailscale.md`.

> **Legacy: Windows workstation.** Until 2026-07-19 CC ran on Windows 11 with repos in `E:\git\`,
> and every remote command needed `SSH=/c/Windows/System32/OpenSSH/ssh.exe` (Git Bash's ssh fails
> silently). Retained in case that environment is revived.

## Build & deploy — Hub (GitOps via ArgoCD)

**Full runbook: use the `felhom-build-deploy` skill.** The load-bearing rules:

The whole cluster is GitOps via a **single ArgoCD app `felhom`** syncing this repo's `manifests/`
to `felhom-system`. **Auto-sync is OFF** — deploys are a deliberate manual sync. ArgoCD's source of
truth is the **manifest**:

- **A code change + CHANGELOG bump deploys NOTHING.** The running image changes only when
  `manifests/hub.yaml`'s `image:` tag changes in git and the app is synced.
- **Pin explicit versions, never `:latest`.** Never bare `kubectl set image`/`kubectl apply`
  (reverted on next sync).
- **The live image can lag the CHANGELOG** when a bump was committed but the manifest/sync step never
  happened — reconcile via the manifest, not the changelog.
- Green gate before any hub commit: `go build ./... && go vet ./... && go test ./...` in `hub/`.

> **Clean-tree gate before any build:** `git status --porcelain` must be empty and
> `git rev-parse HEAD` must equal `git rev-parse origin/main` in the repo being built. An unpushed
> change does not exist — never build a dirty or unpushed tree. The `git pull` in the build step
> stays (it is a no-op when you work in this tree, and load-bearing if anything was pushed from
> elsewhere).

Steps: commit+push code → `cd /mnt/5_hdd/felhom.eu/build/felhom-hub && ./build.sh <VER> --push`
(local) → bump `manifests/hub.yaml` tag + push → ArgoCD hard-refresh + sync (kubectl-patch method in
the skill, now local `sudo kubectl`) → verify Synced/Healthy + rollout + image + startup log.

## Gates — ONE entry point

**Run `python3 scripts/repo_gates.py` after ANY change in this repo.** It is the one entry point
and runs every gate — `site_gates.py`, `hostinstall_gates.py`, `hub_confirm_gate.py`,
`manifest_bearer_gate.py` and `reuse_refs_check.py` on this root — streaming each gate's own output
and exiting non-zero if any fails. `--fast` selects only the gates that touch no network and no
container runtime; today that is all of them. A missing gate script is a FAILURE, never a skip.

**Why a runner and not five invocations** (2026-08-02, R-29): a census of all thirteen gates across
the four repos found that every check a `CLAUDE.md` names was passing, and two of the four nobody
is told to run were failing — one since 14 July. The single-entry-point shape is the only one that
demonstrably gets run here; `app-catalog-felhom.eu/scripts/catalog_gates.py` is the canonical
version of it (R-161) and `repo_gates.py` copies it. `site_gates.py` is a *gate*, not a runner —
do not model new work on it.

**The pre-push hook.** `.githooks/pre-push` runs `repo_gates.py --fast` and refuses the push if it
fails. It is **per-clone** and switched on once with `git config core.hooksPath .githooks` — a
clone does not carry it, and any manual `repo_gates.py` run WARNS when this clone is unarmed.
`git push --no-verify` bypasses it deliberately; **say so in the session report when you use it**.
Both facts are why continuous integration is still owed (`OPEN-ITEMS.md` R-168) — this hook is
local and skippable, and only CI is neither.

## Build & deploy — Website / Manifests

- **Website** auto-deploys via git-sync; just push to `main` (live in 1–2 min). Website changes go
  through `repo_gates.py` above (it runs `site_gates.py`); new pages go into that gate's `PAGES`
  list. Emergency edits: https://files.felhom.eu. All `website/` HTML is **UTF-8 with BOM** — preserve it.
- **THE INSTALLER DOES NOT (R-110, 2026-08-03).** `manifests/webpage.yaml` runs **two** git-syncs:
  the website from `main` as above, and `/scripts/` from the tag **`installer-v<SCRIPT_VERSION>`**.
  Pushing `scripts/felhom-host-install.sh` therefore changes nothing that any machine downloads —
  which it used to, within thirty seconds, for the one artifact that runs as **root on a virgin box**.
  - **To publish:** cut `installer-v<new SCRIPT_VERSION>`, bump the `--ref` in `webpage.yaml`
    (both the sidecar and the init container), commit, and sync. `hostinstall_gates.py` gate 6
    fails if the manifest stops naming an `installer-v…` tag or if the website stops tracking `main`.
  - **To roll back:** move the tag back to the previous commit and wait ~30 s. **No ArgoCD sync and
    no deploy** — git-sync picks up a moved tag on its next period, measured live on 2026-08-03 in
    both directions. That is the emergency lever; fix forward with a new version afterwards.
  - **Do NOT pin the website to the tag.** The sparse-checkout used to cover `/website/` and
    `/scripts/` in one sync, and pinning that would turn every copy edit into a release.
  - The **URL never carries a ref** (`https://felhom.eu/scripts/felhom-host-install.sh`), so
    `felhom-bootstrap.sh` and the hub's day-0 command follow the tag with no edit — do not add one.
  - The installer's own sixteen run-time fetches are pinned separately, to `raw/tag/v$ART_AGENT_VER`
    in the **agent** repo (R-183) — they are the agent's configs, not this repo's.
- **Manifests** are GitOps via the `felhom` app — commit to `main`, then deliberate sync.

## Key patterns

- Hub status logic: OK (report < 30m), WARN (30m–1h or health=warn), DOWN (> 1h or health=fail);
  host liveness thresholds shared between UI and checker (never invent a second definition).
- SQLite timestamps vary in format — always `parseSQLiteTime()`.
- Dashboard/detail auto-refresh every 60s via meta refresh. Geo-restricted to Hungary via nginx
  ingress annotation.
- Helpers, seams, extension points, traps: **`REUSE.md`** — the map is maintained same-commit.