# CLAUDE.md — `/mnt/5_hdd/felhom.eu/git` workspace root (DooPlex)

## What this workspace is

A parent folder holding the felhom sibling repos. Most are one logical product — **Felhom**, a
managed home-server service for Hungarian households. (Any non-felhom repo is unrelated; ignore
unless asked.)

**Claude Code runs HERE, on DooPlex (192.168.0.180), as `kisfenyo`.** Builds are local; the Proxmox
host is one SSH hop. Run CC inside tmux so sessions survive SSH drops: **`tmux new -A -s cc`**.

- **Hub** — operator backend on k3s (`hub.felhom.eu`), in `felhom.eu/hub/`.
- **Host agent** — one per Proxmox host, operator-tier, owns all Proxmox interaction: `felhom-agent/`.
- **In-guest controller** — one per customer LXC, Docker-only: `felhom-controller/`.
- Also: `app-catalog-felhom.eu/` (app templates), `homelab-manifests/` (DooPlex k3s).

Each repo's own `CLAUDE.md` and `.claude/rules/` load when you touch files there. The four Felhom
skills are installed from `felhom.eu/skills/` with `python3 felhom.eu/scripts/install_skills.py`
(symlink — repo edits are live immediately).

## This host is production infrastructure

DooPlex runs Gitea, the container registry, k3s + Longhorn, PBS, and the hub. Treat it accordingly:

- NEVER run `docker system prune`, `docker image prune -a`, or any global Docker cleanup here.
- NEVER touch k3s data dirs, Longhorn mounts, PBS datastores, or Gitea storage. Workspace is
  `/mnt/5_hdd/felhom.eu/` — stay inside it plus `~/build` symlinks/dirs.
- Destructive disk/guest operations belong to felhom-pve via the agent — never on this host.
- Do not run Claude Code with permission prompts disabled on this host.
- Watch disk headroom before large builds: `df -h /mnt/5_hdd /` — abort if either is >90%.

## Artifact taxonomy (it prevents the "what do I do?" stall)

The planning/architecture assistant (in claude.ai, "project Claude") produces files with distinct
roles. **A file being open in the editor is NOT an instruction. If no task is stated, ask.**

- **`TASK.md` / `TASK-*.md`** — a spec for **you (Claude Code) to implement**. Implement it when it is
  placed as `TASK.md` at a repo root, or when explicitly told "implement <file>". Then push, update
  `CHANGELOG.md`, and write the repo's `REPORT.md`.
- **`RUNBOOK-*.md`** — an operational procedure. CC executes every step it has access and capability
  for, live hosts included (CC has root@felhom-pve SSH + the felhom-agent token). Mark a step HUMAN
  only when it genuinely needs physical presence, a real-world decision, or credentials CC lacks.
  **Do not decline a whole procedure because it touches a live host or a privileged token.** Confirm
  before irreversible ops on real customer data; demo scratch guests are fair game.
- **Validation/review** — checking a push against a spec's criteria is **project Claude's** job, not
  yours, unless asked.

## Standing rules — each earned by a real failure (R-96)

1. **Never combine a test run and a commit in one command.** A combined command has ONE exit code and
   the interesting one gets swallowed. Run the suite, read `rc`, *then* commit.
2. **A "no access" claim must list what was tried.** "No access" is unfalsifiable unless it names its
   attempts.
3. **An absent log line is not evidence of correct behaviour.** Verify with a POSITIVE observable —
   something that MUST appear when the system is healthy. An empty log is equally consistent with
   "working" and "stopped entirely".
4. **A recommendation that is not followed gets one line saying why.** Silence reads as agreement and
   the disagreement is lost.
5. **Evidence is copied off the machine at the end of the phase that produced it — before any revert,
   snapshot restore or teardown. Not at the end of the session.** The intermediate revert is the one
   that gets forgotten; both losses were the *middle* teardown, never the final one. **The mechanism,
   because a rule without one is a wish:** the last act of a phase that ran on a machine is `scp`/`pct
   pull` its logs to the evidence directory on DooPlex — the same act that ends the phase, not a
   separate step to remember later. **And when a session notices the evidence is already gone: say so
   plainly in the report and REPRODUCE it independently.** That is the documented expectation, not an
   improvisation to be invented under pressure — it is what both sessions did, and it is the only
   reason two sets of conclusions survived.

<!--
R-96 incident record (committed 2026-07-27) — rationale, not directives.
1. Three recorded occurrences; the worst pushed a red suite because `packages ok: 28` was read while
   rc=1 was not.
2. Two wrong verdicts on 2026-07-27 alone: ep0 (declared unreachable after trying exactly one route
   — felhom-pve -> 10.77.0.1; DooPlex -> 167.233.158.164 worked and the project memory said so), and
   the storage-box API (api.hetzner.cloud 404s for every storage-box endpoint; api.hetzner.com/v1 is
   the real one, and the hub's own hetznerapi.go:3 records it).
3. Earned twice on 2026-07-27: the R-88 watcher (an empty quiesce log could not distinguish a healthy
   loop from a dead one — retired in favour of the per-tier /backup/due polls in pveproxy/access.log),
   and a hub DB copy whose write had silently failed, returning a confident "0 events in window" from
   a file a day stale until its mtime was checked.
4. Twice in the R-88/R-97 arc a review point was absorbed rather than argued: R-84 was folded into
   R-82 without a word, and R-97a's operator-only guard was dropped while the claim it was meant to
   enforce got committed as a comment — which is how a false guarantee shipped and survived a release.
5. Twice in three days, on the SAME machine and at the SAME point. 2026-08-12, the retained-key drill:
   the Phase A logs lived on `drill-r50`'s disk and were destroyed by the revert to `virgin` between
   Phase A and Phase B (`audits/DRILL-retained-key-2026-08-12.md` §11.5). 2026-08-13, R-316: the Part 1
   logs, same disk, same revert, between Part 1 and Part 2 (`audits/REPORT-r316-installer-v1.28.0-2026-08-13.md`
   §9 — "the same mistake as Tuesday, in the same place"; preserved out of `REPORT.md`, which is
   overwritten every session). Both times the golden-bake runbook's existing "scp the log OUT first"
   was applied to the FINAL teardown and not the intermediate one. Both times the conclusions survived
   only because the quotations had been read live and an independent reproduction happened to exist —
   that is luck, and the second occurrence is what makes it procedural rather than another apology.
   Filed as R-320.
-->

## Shared conventions

- **Push to `main` directly** — no feature branches.
- **Versioning** via build-time ldflags (`-X main.version`/`-X main.Version`); bump on meaningful
  changes + a CHANGELOG entry.
- Code quality: double-check for bugs/edge cases; add debug logging; **ask rather than guess** when
  you'd otherwise need to invent input or output.

> **Clean-tree gate before any build:** `git status --porcelain` must be empty and
> `git rev-parse HEAD` must equal `git rev-parse origin/main` in the repo being built. An unpushed
> change does not exist — never build a dirty or unpushed tree. The `git pull` in the build step
> stays (it is a no-op when you work in this tree, and load-bearing if anything was pushed from
> elsewhere).

> **In every repository where you make a change, update both files in that repo:**
> - **`CHANGELOG.md`** — a cumulative log of **all** changes; newest entry on top.
> - **`REPORT.md`** — **overwrite** with a summary of the **most recent** implementation (or
>   significant validation/operational run) only; not cumulative.
>
> **Never write secrets** — tokens, passwords, private keys, API keys — into `CHANGELOG.md`,
> `REPORT.md`, or any committed file. Reference them as "stored out-of-band" instead.

## Live validation — no browser here

**`claude-in-chrome` is NOT available on DooPlex.** The standard method is endpoint-level: invoke the
exact endpoint the UI invokes (no server logic is skipped, only rendering) and say which method was
used. Strict end-to-end UI coverage is a manual click-through by the operator.

## Access

Local (this host): repos `/mnt/5_hdd/felhom.eu/git/<repo>`, build dirs
`/mnt/5_hdd/felhom.eu/build/felhom-{controller,hub,agent}`, `sudo kubectl`, Go toolchain, Docker
build+push to `gitea.dooplex.hu/admin/`.

**Host addresses, routes, break-glass and per-node facts:**
`felhom.eu/documentation/operations/nodes.md` — the single home. Do not restate them elsewhere.

**Which box do I break?** → `felhom.eu/documentation/runbooks/target-selection.md` — the tiers, and
per machine what is freely permitted / needs care / forbidden, each with its reason. Read it before
picking a machine for a drill, a destructive test or a throwaway VM. **A task that needs a victim
names one; an absent fence is not permission.** DooPlex is **Tier 2 — precious**: it *is* the recovery
chain, and never a drill target.

**Component versions are not recorded in any inventory doc** — agent/controller/hub versions change
several times a day and the fleet is not uniform. Ask the hub's `/hosts` + `/configs`, or
`felhom-agent --version` / `pct exec <vmid> -- docker ps` on the box.

## Memory

Project memory lives at `/mnt/5_hdd/felhom.eu/git/.claude-memory/`, surfaced via
`~/.claude/projects/-mnt-5-hdd-felhom-eu-git/memory` (symlink); `MEMORY.md` is the index. Memories
reflect what was true when written — **verify a named file/flag still exists before acting on it.**

## Presence is not success

A timestamp recording an **attempt** must never be read as evidence of a **result**. Where a status
field travels alongside a timestamp, the verdict consults both — or the timestamp records only
successes. Ask of any timestamp: *what exactly must have happened for this to be set?* If the answer
is "we tried", it cannot answer "did it work".

**Corollary:** when a verdict changes which field it counts from, the alarm text has to change with
it. Leaving the message reading `last run 8h ago` while alarming on a six-day-old success turns a true
alarm into one the operator dismisses.

<!--
Two instances. F-CRIT-2: a phantom snapshot's ctime set tier freshness — an aborted 1-byte upload
made the tier look backed up. R-100: LastRun is written on failure, so a nightly-failing offsite tier
kept the staleness clock fresh forever. Both found by asking of a timestamp what must have happened
for it to be set.
-->

## A comment asserting an invariant needs a test pinning it, or it is a wish

**Nine instances in this project have shipped guarantees the code did not provide** — each survived
review because the comment read as settled, and three were caught only on live hardware. The case
table is in the **`felhom-testing`** skill, which loads when you write or review a test, harden a
guard, or fix a bug.

- If a comment states an invariant, **name the test that pins it**, or write one.
- If an invariant has a stated dependency (*"if either invariant changes, revisit this"*), that is not
  a safeguard — nobody revisits. Pin it with a test that fails when the dependency moves.
- Prefer a test that asserts the **consequence** (does the alarm fire?) over one that asserts the
  **mechanism** (does suppression expire?). R-97b's Scenario F proved the mechanism and the
  consequence was still broken.

<!--
LEGACY: WINDOWS WORKSTATION — kept so the old environment can be revived; not the current setup.
- Repos were in E:\git\ (/e/git/ in Git Bash); this file lived at E:\git\CLAUDE.md.
- SSH binary had to be SSH=/c/Windows/System32/OpenSSH/ssh.exe — Git Bash's /usr/bin/ssh lacks
  access to the Windows SSH Agent and fails silently. Every remote command was
  $SSH kisfenyo@192.168.0.180 "..."; details in felhom-controller/docs/vscode-ssh-fix.md.
- pct exec over SSH needed export MSYS_NO_PATHCONV=1 (MSYS mangled /-paths).
- Agent deploy was a two-hop copy: build on 180 -> scp to the Windows box (local path needed
  cygpath -w) -> scp on to felhom-pve. Beware CRLF when scp-ing config files through Windows.
- Skills were installed as Windows junctions (mklink /J) rather than POSIX symlinks.
- claude-in-chrome browser automation WAS available there (attaching only to sessions started after
  the bridge connected).

THIS FILE'S SHAPE (2026-08-06, instruction-trim task): core + path-scoped rules. Removed here and
rehomed, not lost — the per-repo guidance list (those files load on their own), the skills roster
(already resident in the skill listing), the host table (nodes.md is the single home), the "(119
files)" memory count (derivable and wrong — 158), and the nine-row invariant table (felhom-testing
skill). Full accounting: felhom.eu/documentation/audits/LEDGER-instruction-trim-2026-08-06.md
An HTML comment is invisible to Claude and costs no context — verified 2026-08-06 with a control
(both markers plain -> both seen) and a treatment (one marker commented -> not seen), twice.
-->
