f65ea89a24
gates / gates (push) Successful in 8s
install_workspace.py lays down the two things that shaped every session while existing on one host only. Unlike install_skills.py the targets are LIVE CONFIG, so: timestamped backup before every write, settings.json MERGED (this script owns exactly one key), a diverged CLAUDE.md reported rather than silently resolved, and an unparseable settings.json refused outright. Proven: all 7 top-level settings keys survived byte-identically, and run 2 wrote nothing. rules_report.py surfaces the column that matters -- rules that have NEVER fired, which are mis-globbed or dead. 6 of 9 on first run. The hook now self-rotates at 5 MB. The memory store is BACKED UP, NOT COMMITTED (auto-written, may name hosts/paths): added to dooplex-backup.service's User Data component. /opt/backup/scripts/ is itself unversioned host state -- filed, not fixed here.
1613 lines
121 KiB
Markdown
1613 lines
121 KiB
Markdown
## install_workspace.py + rules_report.py — the workspace survives the machine (2026-08-06, R-229)
|
||
|
||
The workspace-root `CLAUDE.md` and the `InstructionsLoaded` hook shape every session and existed on
|
||
**one host** — the one that cannot be rebuilt from anything else. `install_workspace.py` lays both
|
||
down from versioned copies.
|
||
|
||
**Canonical-path decision:** `documentation/runbooks/workspace-CLAUDE.md` stays the source.
|
||
`workspace/` carries only the hook fragment. A `workspace/CLAUDE.md` would be a *third* copy of a
|
||
file whose entire problem is that copies drift, and check 5 already enforces byte-identity against
|
||
the runbooks path.
|
||
|
||
**Where this deliberately differs from `install_skills.py`:** that script's targets are disposable —
|
||
a skill dir can be deleted and re-linked losing nothing. These are **live configuration**.
|
||
`~/.claude/settings.json` holds permissions, plugins and effort level this repo knows nothing about.
|
||
So every write is preceded by a timestamped backup; `settings.json` is **merged**, this script owning
|
||
exactly one key (`hooks.InstructionsLoaded`); a diverged `CLAUDE.md` is backed up and *reported*, not
|
||
silently resolved; and an unparseable `settings.json` is **refused**, never overwritten — a malformed
|
||
settings file disables every setting in it, and overwriting would destroy whatever was mid-fix.
|
||
|
||
Proven: merge preserved all 7 top-level keys byte-identically (`sha256` of the file minus `.hooks`
|
||
unchanged across the write), and a second run wrote nothing — *idempotent* means "changed nothing the
|
||
second time", not "ran twice without erroring".
|
||
|
||
**`rules_report.py`** answers what the hook log exists for: which rules fired, how often, and **which
|
||
never have**. The empty column is the point — a never-fired rule is mis-globbed or dead, which is the
|
||
built-but-never-wired class applied to instructions. It prints each silent rule's `paths:` beside it
|
||
so "wrong glob" is distinguishable from "quiet month", and it judges neither. First run: 6 of 9 rule
|
||
files had never fired. **Caveat: the log only covers since the hook was armed** — `gates.md` shows
|
||
silent despite having fired earlier the same day, before installation.
|
||
|
||
The hook now **self-rotates at 5 MB**, one generation, since the log is otherwise unbounded.
|
||
|
||
## instructions_gate.py — check 6: the auto-memory index (2026-08-06, R-229)
|
||
|
||
**The bigger half of what loads was watched by nothing.** Measured at the workspace root, the
|
||
hand-written root `CLAUDE.md` is 6.6k tokens and `MEMORY.md` is 8.4k — and `MEMORY.md` is the one
|
||
instruction file nobody hand-edits, because Claude writes it.
|
||
|
||
Check 6 asserts the index is within its line and byte ceilings and that every top-level topic file is
|
||
referenced. **Its three outcomes are deliberately different**, and each has a test proving the
|
||
difference is the intended one:
|
||
|
||
- **Over the ceiling FAILS.** Content past the auto-memory limit is dropped with **no error** — a
|
||
silent truncation with no observable at all.
|
||
- **An orphan WARNS.** The store lives outside git and changes between sessions; a hard failure
|
||
would block pushes for something no commit can fix.
|
||
- **An absent store PASSES *and prints why*.** A deliberate exception to check 1's "a missing input
|
||
is a FAILURE, never a skip" — the store is machine-local by design, so a clone on any other host
|
||
legitimately has none. The reason is printed in the tally, and a test asserts on that text rather
|
||
than on `rc == 0`, because **a pass with no reason is indistinguishable from a gate that stopped
|
||
running.** That printed reason is the only thing that made the exception safe to grant.
|
||
|
||
Two load-bearing negatives: a file reachable only via `[[wikilink]]` is **indexed, not orphaned**
|
||
(counting it would push someone to add a duplicate row), and `archive/` contents are set aside on
|
||
purpose and are never orphans.
|
||
|
||
Suite: **39 assertions, 0 failures** (was 20). Companion red-proof against the **real** store, not a
|
||
fixture — ceiling lowered 200 → 100, the gate went red naming the real file and count
|
||
(`.claude-memory/MEMORY.md: 150 lines, ceiling 100`), ceiling restored, gate and suite green again.
|
||
|
||
## repo_gates.py — `instructions` registered, and the repo that owns the gate now runs it (2026-08-06, R-229)
|
||
|
||
`instructions_gate.py` **lives in this repo's `scripts/`** and was registered in `controller_gates.py`
|
||
and `agent_gates.py` on the day it was written — but not in `repo_gates.py`. The reason was
|
||
mechanical: `felhom.eu/CLAUDE.md` was 227 effective lines against a 200 ceiling, and a
|
||
registered-but-failing gate refuses **every** push through `.githooks/pre-push`. So the one repo that
|
||
owns the check was the one place it did not run — the exact failure the R-29 gate census was created
|
||
to find.
|
||
|
||
Fixed in the required order: **trim first, register second.**
|
||
|
||
- `CLAUDE.md` **227 → 115 effective lines**, restructured into a core plus
|
||
`.claude/rules/{hub,website,manifests,docs}.md`, all `paths:`-scoped, all ≤60 effective lines.
|
||
- `repo_gates.py` gains gate 6, `--fast` safe. Six gates, all OK.
|
||
|
||
**Two placements deviate from the spec's sketch, both to avoid rebuilding a failure class:**
|
||
- **Register discipline (`OPEN-ITEMS.md` first, `STATUS.md` is a view) stayed in the core**, not in
|
||
`docs.md`. It applies to every session that ships or decides something; behind a
|
||
`documentation/**` glob it would be invisible in exactly the code-shipping sessions where findings
|
||
get lost — which is how R-153/154/155 and R-156/157 were minted and lost.
|
||
- **The R-110 installer fence stayed in the core.** Its trigger is editing
|
||
`scripts/felhom-host-install.sh`, which none of the four fixed globs covers; behind `website/**` it
|
||
would never load for the one artifact that runs as root on a virgin box. Compressed to its
|
||
actionable clauses rather than relocated.
|
||
|
||
**Scoping proven from the hook log, not the frontmatter** (`InstructionsLoaded`,
|
||
`~/.claude/instructions-loaded.jsonl`), in two fresh sessions so the negative control is clean:
|
||
reading `website/index.html` produced `website.md path_glob_match` and **no** `hub.md` line; reading
|
||
`hub/internal/api/handler.go` produced `hub.md path_glob_match` and **no** `website.md` line.
|
||
|
||
**Trap worth keeping:** creating `.claude/rules/` mid-session does **not** arm it. The in-session
|
||
reads that followed produced no hook line at all — a directory whose instructions were already seeded
|
||
is not re-scanned. Same class as the settings-watcher caveat. A new rule file is only live in a
|
||
session started after it exists, which is why the proof above needed `claude -p`.
|
||
|
||
## instructions_gate.py 1.0.0 — instruction files cannot silently regrow (2026-08-06, R-229)
|
||
|
||
New shared gate, registered in `controller_gates.py` and `agent_gates.py`, never copied into a
|
||
sibling repo (the `reuse_refs_check.py` precedent). `--fast` safe.
|
||
|
||
**It is a consistency gate, not a budget gate, and the failure message says so.** A `/context`
|
||
reading on 2026-08-06 measured the instruction files at 15k tokens against **869k free** in a 1M
|
||
window — space is not the constraint, and a future reader must not re-derive the wrong reason. The
|
||
200-line ceiling is Anthropic's *adherence* guidance, and a file nobody can hold in their head is
|
||
where contradictions hide: four were found in this project on the same day, two of which decided
|
||
where a destructive drill runs.
|
||
|
||
Checks, all against **effective** text (HTML comments stripped, because they are stripped before
|
||
injection): the line ceiling; every `.claude/rules/*.md` declares `paths:` or an explicit
|
||
`unconditional: true`; no component version literal; no TEMPORARY block carrying a past date; and the
|
||
workspace-root `CLAUDE.md` is byte-identical to its versioned copy — the live file sits outside any
|
||
git repo, so that copy is its only version-controlled record.
|
||
|
||
**Two traps found while building it, recorded so they are not reintroduced:**
|
||
- A bare `\d+\.\d+\.\d+` matches the first three octets of **every IPv4 address**. The gate
|
||
excludes dotted quads; without that it fails on `192.168.0.180` in the agent's own file.
|
||
- `unconditional: true` is **not** a Claude Code feature — it is this project's marker, asserting
|
||
that always-loading was deliberate. The docstring says so, so nobody looks for it in the product.
|
||
|
||
`test_instructions_gate.py`: 20 fixture assertions, all on the **effect** (exit code *and* that the
|
||
message names the file and the reason). Includes the load-bearing negatives — 400 commented lines
|
||
must not trip the ceiling, and a version literal inside a comment must be allowed — because the gate
|
||
must not punish the very move it exists to encourage. Companion red-proof recorded in the ledger:
|
||
ceiling temporarily set to 100 against the real trimmed files, `felhom-agent` (173) FAILED and was
|
||
named, `felhom-controller` (92) still passed; threshold restored, suite re-run green.
|
||
|
||
## 1.25.0 — the off-site tier stops asking to prune (2026-08-04, R-191)
|
||
|
||
**A backup that worked must not report failure.** The off-site tier was written with `keep_last: 2`,
|
||
so every weekly run uploaded its snapshot successfully and then failed the whole job on a prune the
|
||
box's token is deliberately refused: `prune 'ct/9201': permission check failed - missing
|
||
Datastore.Modify|Datastore.Prune` → `TASK ERROR: job errors` → `whole_guest_backup_failed` in the
|
||
operator's inbox. Every week, on both boxes, about a backup that had already succeeded.
|
||
|
||
**R-89 moved off-site pruning SERVER-SIDE** — ep0 runs a per-namespace prune job and box tokens stay
|
||
write-only, so a box can never delete its own off-site history. The 2026-07-26 "two weeks" ruling was
|
||
not reversed; where it is ENFORCED moved, and this value did not follow. The tier now writes
|
||
`keep_last: 0`, which the agent's existing guard (`allowPBSPrune = !primary && keep_last > 0`) already
|
||
reads as "never prune from the box" — no agent change needed.
|
||
|
||
**VERIFIED BEFORE CHANGING IT** (read-only on ep0, 2026-08-04): prune jobs `prune-demo-felhom` and
|
||
`prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, schedule 03:30, keep-last 2,
|
||
and have run **every day since 2026-07-27 — 18 tasks, all `status=OK`**, the newest showing
|
||
`retention options: --ns demo-felhom --max-depth 0 --keep-last 2` and keeping exactly two. Disabling
|
||
the client-side prune without that check would have traded a weekly false alarm for unbounded growth.
|
||
|
||
A gate now asserts the off-site tier carries no client-side prune, so the value cannot drift back
|
||
quietly. The local tier's retention is untouched.
|
||
|
||
## 1.24.0 — a pre-existing backup target is granted too (2026-08-03, R-185)
|
||
|
||
**`configure_backup_target` has two arms and only one of them granted.** The Case A arm creates the
|
||
storage and calls `felhom-backup-target-apply grant` in the same breath — a box that builds its own
|
||
target has always been fine. The **Scenario-F arm** — *"the target already exists, leave it exactly as
|
||
it is"* — returned without ever granting.
|
||
|
||
So a box whose `felhom-backup` pre-dated the install (created by the vzdump-target-move runbook, or
|
||
surviving a reinstall) ended up with `local_backup_target: felhom-backup` while its token held
|
||
`FelhomAgentStore` on only `local`, `local-lvm` and `felhom-pbs`. Measured on **both** demo boxes
|
||
2026-08-03: the content API answers `{"data":[]}` through the agent's token while root lists three
|
||
archives. That tier was invisible to the agent and never restore-tested — and nothing said so,
|
||
because an empty listing is also what a brand-new tier returns.
|
||
|
||
The reuse arm now ensures the ACL through the same guarded wrapper, so both arms leave the box in the
|
||
same state. **Scenario F is unviolated:** the storage DEFINITION is still untouched — granting the
|
||
role the agent is supposed to have on the target this script is about to write into `agent.json` is
|
||
finishing the job, not retargeting the box. `pveum acl modify` is idempotent, so a box that already
|
||
has the grant is unchanged and a box whose token was rotated gets it back.
|
||
|
||
`$BACKUP_TARGET_ID` is deliberately **not** added to `PVE_STORAGES`, and the comment now says why: that
|
||
list is granted in step 4/5, before the target has been resolved in step 6, and `--acl-storages`
|
||
entries are preflight-checked for existence. The grant belongs with the resolution, which is where it
|
||
already was for a newly created target.
|
||
|
||
**A gate now asserts it** (`hostinstall_gates.py`): every arm of `configure_backup_target` that
|
||
resolves the target must also grant on it. Red-proved by reverting the reuse arm — `resolves the
|
||
backup target in 2 place(s) but grants in only 1`.
|
||
|
||
## v1.23.0 — the installer is published, not pushed (2026-08-03, R-110 + R-183)
|
||
|
||
**Two channels moved off `main` in the same change, because either one left behind makes the other
|
||
cosmetic.**
|
||
|
||
**Channel 1 — the served script.** `manifests/webpage.yaml` git-synced `/scripts/` from
|
||
`--branch=main` on a 30 s period and nginx served that working tree, so **pushing this file WAS
|
||
publishing it**: within half a minute it was what every new machine downloaded and ran as root, with
|
||
no staging and no rollback but another push. The sync is now **split in two**: the website keeps
|
||
tracking `main` at the same cadence (a copy edit must never need a release), and `/scripts/` tracks
|
||
the tag **`installer-v<SCRIPT_VERSION>`**. Publishing is moving that tag; rolling back is moving it
|
||
back.
|
||
|
||
**PROVEN, not assumed:** git-sync v4.4.0 follows a tag *and* notices a **moved** one — measured on a
|
||
throwaway sync against this repo, `update required … local:<old> remote:<new>` → `updated
|
||
successfully`, within one period (~20 s). The moved-tag half is what the whole publish model rests
|
||
on, so it was measured before the manifest was touched.
|
||
|
||
**Channel 2 — the sixteen files the installer fetches while it runs.** `fetch_raw` pulled from
|
||
`$AGENT_REPO/raw/branch/main`. It now pulls from **`raw/tag/v$ART_AGENT_VER`** — the agent version the
|
||
hub has vouched and whose binary sha this script already verifies.
|
||
|
||
**That is a correctness fix, not only a publish-channel one (→ R-183).** These are the AGENT's
|
||
configs — its systemd unit, its sudoers, its guarded wrappers — and a fresh install was fetching the
|
||
**vouched binary** while taking its configs from **whatever `main` held**. Two refs, one install, and
|
||
nothing compared them. The right ref for them was never this script's `SCRIPT_VERSION`: they do not
|
||
live in this repo and have no relationship to its version line.
|
||
|
||
**No fallback to a branch.** A vouched version whose tag is missing fails loudly rather than quietly
|
||
serving `main` — a silent fallback is the appearance of control with none of it. `felhom-agent`
|
||
carries `v<version>` tags from now on, `release-agent.sh` creates them, and `agent_gates.py` fails if
|
||
the vouched version is not downloadable.
|
||
|
||
**Channel 3 — the URL — needed no change, and that is worth recording rather than leaving as a
|
||
silence.** `https://felhom.eu/scripts/felhom-host-install.sh` never carried a ref: the ref lives in
|
||
the manifest. So both producers of that URL (`scripts/iso/felhom-bootstrap.sh`, the hub's day-0
|
||
command) follow the tag with no edit — **and no hub change, so no hub version bump.**
|
||
|
||
**Gate 6 in `hostinstall_gates.py`** pins all three structurally, with no network so it stays in
|
||
`--fast` and runs in CI on every push: no `raw/branch/` ref anywhere in the installer; `fetch_raw`
|
||
still pins to `$ART_AGENT_VER`; the manifest still syncs `/scripts/` from an `installer-v…` tag and
|
||
the website still from `main`.
|
||
|
||
**It deliberately does NOT assert "a tag exists for the current SCRIPT_VERSION".** That gate would go
|
||
red on the very push that bumps the version, before publishing — and publishing being a separate
|
||
deliberate act is the entire ruling. A gate that fails on the normal path is one people learn to
|
||
ignore.
|
||
|
||
## docs — v1.22.0 exercised end to end on two real reinstalls (2026-08-03, R-178) — **no script change**
|
||
|
||
**Nothing shipped.** `felhom-host-install.sh` stayed at **v1.22.0**; the published copy at
|
||
`https://felhom.eu/scripts/felhom-host-install.sh` was confirmed byte-identical to the repo copy
|
||
(`sha256 ed02acb2da46c8d2b5c486ce99d5b9a2747e8786c6eb03652cf755ed1abdd9f4`) before use. Both demo
|
||
boxes were uninstalled and reinstalled with it, by **two deliberately different supply paths**:
|
||
demo-hp with `--golden <local volid>` (the `:2584` alternative), demo-felhom with
|
||
`--force-gitea-golden` (the canonical C.3 customer command). The merge-aware `step_grows` produced
|
||
`data +46G (->70G, ONE volume)` and `+226G (->250G)` respectively, and `fetch_verify` was observed
|
||
succeeding against the vouched manifest for **both** artifacts on demo-felhom
|
||
(`verified sha256 a7763d31b55b5ce7…` agent, `verified sha256 54e2a4c431daf580…` golden).
|
||
|
||
**Two script-side findings, filed not fixed** (the session was a runbook; §7 forbade code):
|
||
|
||
- **R-180** — `--archive-storage` is validated for existence (`:1583`) and for golden resolution
|
||
(`:1661`), but never against the ACL storage set it is about to grant (the fixed default
|
||
`local local-lvm felhom-pbs`). Staging the golden on `felhom-backup` therefore passed every
|
||
pre-flight gate and died at **step 8/8**: `HTTP 403: permission denied at /storage/felhom-backup
|
||
(missing privilege Datastore.AllocateSpace)` — *after* step 2 minted the token, step 4b **rotated
|
||
and vaulted root@pam**, and step 5 installed the agent. `ARCHIVE_STORAGE ∈ PVE_STORAGES` is a
|
||
one-line assertion over two variables both known at `:1583`.
|
||
- **R-179** — `--uninstall` leaves the NAS network-storage systemd units behind
|
||
(`mnt-felhom\x2ddrives-<share>.{mount,automount}`; automount left `failed`, parent bind left
|
||
mounted). The Part E residue-diff provenance is from **v1.9.1**, which predates the feature — and
|
||
demo-felhom, which never had a share configured, left nothing, which is exactly why a diff on such
|
||
a box reported clean.
|
||
|
||
Full evidence: root `REPORT.md`.
|
||
|
||
## host-install: one data volume, derived from the disk (2026-08-03, R-165)
|
||
|
||
**Forced by a census, not planned.** `felhom-agent` v0.120.0 merges the appliance's two data volumes
|
||
into one (decision D-a). `step_grows` computed **two** numbers and the install call passed both, so
|
||
this script had to change with the agent or every install would have provisioned a half-sized box.
|
||
|
||
- **`step_grows` computes ONE total.** The old 80/20 docker-vs-sysdata split is summed: `226` where it
|
||
was `184 + 42`, `106` where it was `84 + 22`, `46` where it was `34 + 12`. **A standard appliance
|
||
keeps exactly the capacity it had — 250 G — it is simply no longer split by a wall.**
|
||
- **The size still comes from the physical disk.** `step_grows` already read the thin pool's real free
|
||
space (`lvs /dev/pve/data`); the merge only collapsed its two outputs into one. This is what makes
|
||
the merge safe to ship: an unflagged install does **not** get the golden's 24 G base.
|
||
- **`--sysdata-grow` is DEPRECATED but still honoured.** It is no longer auto-computed (set to 0), and
|
||
a hand-passed value still counts because the agent **folds** it into the single volume's grow rather
|
||
than dropping it — so an operator reproducing an old command line gets the same total.
|
||
|
||
## CI — a Gitea Actions runner, and a red run that reaches a person (2026-08-02, R-168)
|
||
|
||
**No version bump anywhere: nothing in the product repos is compiled, built or deployed by this.**
|
||
Recorded explicitly so the omission reads as a decision rather than a miss.
|
||
|
||
**What this closes.** Session 1 (same day) gave every repo one gate entry point and a
|
||
`.githooks/pre-push` that refuses a failing push. That hook is per-clone and `--no-verify` skips it,
|
||
so nothing independent of the person pushing ever saw whether the gates passed. This is the
|
||
independent half, and with it **R-29 CLOSES** — on the demonstrated alarm, not on a green run.
|
||
|
||
**`.gitea/workflows/gates.yml` (new)** — triggers on `push`, `runs-on: felhom-gates`, obtains the
|
||
source with a shallow `git fetch` of the **exact pushed SHA** from the in-cluster Gitea Service, and
|
||
runs `scripts/repo_gates.py --fast` and nothing else. **No `uses:` step anywhere** — JavaScript
|
||
actions need a node runtime the host-mode runner does not have, and probe P3 measured that a plain
|
||
`git fetch` is sufficient and lands on the pushed commit. No `|| true`; the entry point's exit code
|
||
IS the job's result.
|
||
|
||
**The alarm, which is the half that matters.** Probe P5 measured that a failed run produces **no
|
||
mail, no notification row and no log line** from Gitea. A red tick in a web UI nobody watches is
|
||
exactly the defect R-29 filed, rebuilt one layer up — so the workflow sends its own email on failure
|
||
via Resend (the hub's existing transactional path) and **prints the provider's accepted id**, making
|
||
"a message left the machine" an observable. **Demonstrated, not asserted:** a deliberately broken
|
||
commit pushed with `--no-verify` produced run #6 `failure` and
|
||
`RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`.
|
||
|
||
Two traps found while building it, both worth keeping because each looks like something else:
|
||
the runner image has **no `curl`** on purpose (python3 and git only — so the step uses `urllib`
|
||
rather than growing the image), and `api.resend.com` sits behind **Cloudflare, which 403s the default
|
||
`Python-urllib` User-Agent with error 1010** — a failure that reads exactly like an auth failure and
|
||
is not one.
|
||
|
||
**The standing limit, written into the workflow itself: it REPORTS, it cannot REFUSE.** Every repo
|
||
pushes straight to `main` with no pull request, so there is no merge for a status check to stand at.
|
||
That is not a gap in the runner; there is no gate in the road. Making it blocking needs branch
|
||
protection plus a PR workflow, which changes how the operator works → **R-169**, waiting on them.
|
||
|
||
**`documentation/audits/SPIKE-ci-runner-2026-08-02.md` (new)** — all six probes, method, measurement
|
||
and ruling; none produced a STOP. Also records a near-miss worth more than the probes: a `| tail -5`
|
||
inside my own census query silently dropped rows and looked exactly like a baseline drift big enough
|
||
to change the task. **An instrument that can drop results silently is not a measurement.**
|
||
|
||
**`CLAUDE.md`** gains the matching rule from session 1's red-proofing: a `go test -run` pattern that
|
||
matches no test prints `ok` and exits 0, so a red-proof using `-run` must first prove the filter
|
||
matched something.
|
||
|
||
**`CONTEXT.md`** gains S-8 (CI detects, does not block, and why that is structural), S-9 (a detector
|
||
that tells no one is not finished), S-10 (the runner is unprivileged because DooPlex is Tier 2), and
|
||
S-11 (CI reproduces the workspace's sibling layout, because two entry points depend on it).
|
||
|
||
## Gate enforcement — one entry point per repo, and a pre-push hook (2026-08-02)
|
||
|
||
**No version bump: `scripts/` carries no version, and this is tooling.** Recorded explicitly so the
|
||
omission reads as a decision rather than a miss.
|
||
|
||
**The census that started it.** Thirteen gate scripts exist across the four felhom repos. A full run
|
||
on 2026-08-02 found one clean correlation: **every check a `CLAUDE.md` tells a person to run was
|
||
passing, and two of the four nobody is told to run were failing** — `hostinstall_gates.py` since
|
||
14 July, and `reuse_refs_check.py` on all four repos. Both failures were harmless in effect, which
|
||
was checked line by line; nothing would have said so if they had not been.
|
||
|
||
**`scripts/repo_gates.py` (new)** — THE entry point for this repo. Runs `site_gates`,
|
||
`hostinstall_gates`, `hub_confirm_gate`, `manifest_bearer_gate` and `reuse_refs_check` on this root,
|
||
streams each gate's own output, exits worst-wins non-zero, and reports exit 2 distinctly as
|
||
INCONCLUSIVE. **A missing gate script is a FAILURE and prints the path tried** — fail-closed, because
|
||
a runner that quietly skips a gate is the inert-seam failure this project has shipped four times. It
|
||
copies `app-catalog-felhom.eu/scripts/catalog_gates.py` (R-161), **not** `site_gates.py`, which is a
|
||
gate and not a runner — copying that would have produced a ninth monolith.
|
||
|
||
**`scripts/reuse_refs_check.py` — resolution taught, not loosened.** RED on all four repos with 13
|
||
findings, of which a hand audit found **zero** genuine drift: twelve were package shorthand whose
|
||
file sits a couple of directories deeper, and `wgsync/reconciler.go`, cited by the controller, lives
|
||
in the hub. `REUSE.md` cites by package shorthand and across repos on purpose; the tool was wrong.
|
||
New order, first hit wins: exact → suffix → ambiguous (real citation, imprecise shorthand — not a
|
||
failure) → sibling repo (as-is, or with the sibling's own name stripped off the front) → FAIL.
|
||
**Every non-exact hit is printed** and every root prints a per-rule tally, because "0 failures" alone
|
||
cannot tell a working checker from a blind one. A failure lists every resolution attempted. Evidence
|
||
trees (`audits/`, `documentation/tests/`) are excluded from the suffix index — a copy of a file is
|
||
not the file. An absent sibling is never a failure; an unreadable parent says so and continues.
|
||
Result: 13/13 resolve, all four roots exit 0.
|
||
|
||
**`scripts/test_reuse_refs_check.py` (new, 13 tests)** — one per resolution row plus the kill
|
||
condition. Red-proof: making `resolve()` return `exact` for an unresolvable token turns four of them
|
||
red. **`scripts/test_repo_gates.py` (new, 3 tests)** — a SEAM test asserting each member gate's own
|
||
distinctive stdout, never the runner's summary line; red-proofed with an inert `run_gate` that still
|
||
prints "all felhom.eu gates OK" and exits 0.
|
||
|
||
**`.githooks/pre-push` (new)** — runs `repo_gates.py --fast` and refuses the push. Its honest limits
|
||
are written into the hook itself: it is **per-clone** (`core.hooksPath` is local config; arm with
|
||
`git config core.hooksPath .githooks`, and any manual entry-point run WARNS when a clone is unarmed)
|
||
and **`git push --no-verify` bypasses it on purpose** — an escape hatch that cannot be reached is one
|
||
that gets removed the first time it is inconvenient; using it must be stated in the session report.
|
||
Measured on git 2.47.3: a relative `core.hooksPath` resolves correctly and the hook's cwd is the repo
|
||
root whether `git push` is issued from the root or any subdirectory. The half that is neither
|
||
per-clone nor skippable is CI — now tracked as R-168.
|
||
|
||
## ISO v1.26.1 — the PUBLIC installer ISO, PUBLISHED (2026-07-31)
|
||
|
||
**Live at `https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso`**
|
||
sha256 `f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6`, 1 705 322 496 bytes,
|
||
with its `.sha256` and manifest beside it. Round-trip verified: the bytes downloaded from the public
|
||
URL checksum to the same value. Bucket `Public Access` stays disabled — an unauthenticated GET to the
|
||
S3 endpoint returns 400, and the custom domain has no index (404 on `/`).
|
||
|
||
**What it is.** Write it to a USB stick, boot it, choose your own disk and your own root password in
|
||
the standard Proxmox installer. The box sets itself up on first boot and shows a pairing code. The
|
||
image carries **no answer file, no root password, no SSH key, no customer identity and no disk
|
||
profile** — the entire Spike 1-2 problem space is deleted rather than mitigated.
|
||
|
||
**How day-0 is delivered.** A `felhom-bootstrap` .deb in `/proxmox/packages/`, which the PVE installer
|
||
unpacks on EVERY install path including the interactive one (`Install.pm:1343-1372`, `:1378`). The
|
||
answer file's `[first-boot]` hook does NOT reach an interactive install — measured in
|
||
SPIKE-universal-iso-3 with a same-image control.
|
||
|
||
**PROVEN ON BOTH MENU ENTRIES** before publication, four observables each:
|
||
graphical (`spikegfx.felhom.eu`) package installed, unit enabled, unit fired at first boot,
|
||
pairing code `J7N-2DA`, token 0600
|
||
Terminal UI (`spikesix.felhom.eu`) same, pairing code `ZY5-YY4`
|
||
|
||
**One defect was caught by that gate and fixed before publication:** the package did not ship
|
||
`/etc/felhom/`, so the first build registered at the hub and then polled `HTTP 401` forever with no
|
||
claim code ever shown. G9 proved the payload script was byte-identical to HEAD and said nothing about
|
||
the directory the payload writes into. Fixed, plus gate criterion **G13** (every directory the payload
|
||
writes into is in the package), red-proofed.
|
||
|
||
**A second false claim was caught before publication:** the generated manifest — the file a tester
|
||
reads — described a single automated entry with a 5s timeout and listed Graphical/Terminal UI as
|
||
'menu-removed'. It now describes the image it ships.
|
||
|
||
Menu: two INTERACTIVE entries, graphical default, **timeout 15s** (Spike 2 lost a probe to a
|
||
1-second menu). No automated entry exists at all: without `auto-installer-mode.toml` the stock
|
||
grub.cfg does not emit it.
|
||
|
||
R-128 FIXED (the ISO/SCRIPT_VERSION 'alignment' was a comment nothing evaluated; the coupling does not
|
||
exist, so the claim was corrected rather than asserted). R-155 RESOLVED (guard narrowed to
|
||
`FELHOM_MENU=single`, not deleted). Release gate: `documentation/runbooks/iso-release-gate.md`.
|
||
|
||
## SPIKE 4 (no code change) — a `.deb` in the ISO DOES deliver on an interactive install (2026-07-31)
|
||
|
||
**No script, profile or build file was modified.** Findings:
|
||
`documentation/audits/SPIKE-universal-iso-4-2026-07-31.md`. **No new R-rows; R-153 RETRACTED.**
|
||
|
||
**MEASURED, with a control, and the negative control is in the same box.** One ISO (15 GRUB entries),
|
||
a trivial probe `.deb` injected into `/proxmox/packages/`, two VMs on demo-hp created with `qm` (400
|
||
interactive / 401 automated control) on a scratch dir storage at the `/mnt/nvme-1tb` mount ROOT.
|
||
|
||
On the **Terminal-UI (interactive)** install: the package is installed (`ii felhom-spike4-probe`), its
|
||
postinst ran, it enabled a systemd unit, and **that unit FIRED on first boot** (uptime 7.98s,
|
||
pid1=systemd) — while on the *same machine* `proxmox-first-boot` is not installed and
|
||
`/var/lib/proxmox-first-boot` does not exist, reproducing Spike 3's negative rather than assuming it.
|
||
The automated control delivered both mechanisms.
|
||
|
||
**The postinst environment (identical on both paths):** `pid1=unconfigured.sh`, **no running systemd**,
|
||
but `systemctl enable` SUCCEEDS; `/proc` + `/sys` mounted; network and DNS happened to be up (inherited
|
||
from the installer's DHCP — **must not be relied on**). Constraints for a real postinst: never
|
||
`systemctl start`/`daemon-reload`, never require network, never fail (`dpkg --configure -a` surfaces it
|
||
as an install error), do the real work in the unit at first boot.
|
||
|
||
**Repack preserves it** — but a naive `xorriso -boot_image any replay` fails with "Overlapping MBR
|
||
partition entries"; `iso-repack.sh:270-292` already documents that exact failure and its fix, and
|
||
mirroring it produced a working image (19 El Torito entries, probe `.deb` byte-identical out of the
|
||
finished ISO).
|
||
|
||
**Q3:** `iso-repack.sh:100-106` refuses an ISO without `auto-installer-mode.toml` — a guard, not a
|
||
structural requirement, already filed as R-155 and cited exactly here. With no mode file the stock
|
||
grub.cfg does not emit the Automated entry at all; with a mode file but no answer, the automated entry
|
||
aborts safely and loudly.
|
||
|
||
**R-153 is RETRACTED** into R-94 leg (b): `OPEN-ITEMS.md:15` already carries it verbatim at
|
||
`READY (XS)`, and R-29 (the class) says explicitly "do not mint a new ID for a new instance".
|
||
**Spike 3's claim that the drift leaves the install-command generator "three minor versions stale" was
|
||
also FALSE and is corrected** — R-94 retracts exactly that reading: the constant selects no script, the
|
||
served script is always `main`, so it is a wrong number on a screen and nothing more.
|
||
|
||
## SPIKE 3 (no code change) — `[first-boot]` does NOT fire on an interactive install (2026-07-31)
|
||
|
||
**No script, profile or build file was modified.** Findings:
|
||
`documentation/audits/SPIKE-universal-iso-3-2026-07-31.md`. Opens R-153..R-155.
|
||
|
||
**MEASURED, with a control from the same image.** One probe ISO (15 GRUB entries: Automated,
|
||
Graphical, Terminal UI). Automated entry -> the first-boot hook fires: ttyS0 marker, marker file,
|
||
`/var/lib/proxmox-first-boot/proxmox-first-boot` (0700) and the activation symlink all present.
|
||
Terminal-UI entry, normal manual install -> **every one of those is absent, and the
|
||
`proxmox-first-boot` PACKAGE is not installed at all** (`dpkg-query: no packages found`). A
|
||
whole-filesystem grep for the marker returns nothing.
|
||
|
||
Mechanism, cited: `Config.pm:118` defaults `first_boot.enabled => 0` and `set_first_boot_opt` is never
|
||
called in the Perl tree; `Install.pm:746` returns early unless the flag is set; `Install.pm:1360` skips
|
||
the package itself. `proxinstall` (the graphical installer) contains **zero** occurrences of
|
||
`first-boot`. **`[first-boot]` is an automated-installer feature — unavailable on every interactive
|
||
path, by construction.** R-154.
|
||
|
||
**A delivery mechanism DOES exist, and it is untested.** `Install.pm:1343-1372` unpacks EVERY `.deb`
|
||
in the ISO's `/proxmox/packages/` into the target on every install path (fixed skip-list of known
|
||
names), then `dpkg --configure -a` runs postinsts (`:1378`). That is how PVE ships first-boot itself.
|
||
Read from source, NOT measured — proving it is the top follow-up.
|
||
|
||
Also: the interactive installer shows the target disk and a summary listing `Bootdisk(s) /dev/sda`
|
||
before erasing — the operator's ruling needs no Felhom work for that warning.
|
||
|
||
**Incidental: a working gate is red and nobody runs it.** `hub/internal/web/configs.go:28` has
|
||
`hostInstallVersion = "1.19.0"` vs `SCRIPT_VERSION = "1.22.0"`; `scripts/hostinstall_gates.py` detects
|
||
it and exits 1. The comment claiming drift "is now gated" is true of the gate and false of its
|
||
enforcement. R-153.
|
||
|
||
Q3 (the real stub at `before-network`) was NOT reached and is recorded as not reached.
|
||
|
||
## SPIKE 2 (no code change) — locked root closes the PVE web UI; before-network gives a measured zero window (2026-07-31)
|
||
|
||
**No script, profile or build file was modified.** Findings:
|
||
`documentation/audits/SPIKE-universal-iso-2-2026-07-31.md`. Opens R-148..R-152.
|
||
|
||
Both Tier 0 boxes went offline mid-session (provider cable fault; four routes tried) and returned;
|
||
all three scenarios then ran to completion on real PVE, each signalled by `reboot-mode="power-off"`.
|
||
|
||
- **A locked root CLOSES the PVE web interface.** Measured against the exact endpoint the UI logs in
|
||
through (`POST /api2/json/access/ticket`, `root@pam`) **with a working control**: the known-password
|
||
install returns **HTTP 200 + ticket**; the locked install returns 401 for every password, and no
|
||
password exists. `passwd -S root` = `L`, shadow = `literal-asterisk`. PVE uses the stock PAM stack
|
||
(`pve_pam_realm: 0`).
|
||
- **GRUB recovery mode is ALSO closed**: `Cannot open access to console, the root account is locked.`
|
||
— the container mechanism result confirmed verbatim on real PVE.
|
||
- **But `init=/bin/bash` still works** — an unauthenticated `root@(none):/#`. So a locked-root box is
|
||
recoverable, operator-only, at the physical console. The installed GRUB has **no password**, so
|
||
locking root is NOT a physical-security measure. **R-152.**
|
||
- **`before-network` measured, A/B, same image:** the hook runs (positive marker, uptime 6.58s) with
|
||
entropy 256, writable /etc, all binaries and `openssl_rand_len=32`, while `ip_global` is EMPTY and
|
||
`listen_22_8006` is **0**. The `fully-up` column is the converse proof: sshd + pveproxy already
|
||
active, 3 listening. Zero credential window, measured.
|
||
- **R-148:** `answer.toml.tmpl:27` justifies `fully-up` with a `pvesh`/`pct` dependency the stub does
|
||
not have (grep rc=1) — it blocked the ordering that is now measured as the fix.
|
||
- **R-149** three ordering values, not two · **R-150** Condition-guarded hooks skip silently ·
|
||
**R-151** demo-felhom built from an uncommitted profile.
|
||
- Q4 from source: four gates in `iso-repack.sh` enforce the single-entry menu; `default`/`timeout`
|
||
already settable in `grub/grub.cfg.tmpl`.
|
||
|
||
Three probes failed and are recorded as failed, not massaged: a container probe that ran as uid 0
|
||
(so `su` never invoked pam_unix), a GRUB probe that missed the **1-second** menu timeout and typed into
|
||
the login prompt, and a kernel-line edit that landed one line off — caught by a pre-typing
|
||
verification screendump.
|
||
|
||
## SPIKE (no code change) — a universal ISO needs a different disk strategy and a locked root (2026-07-31)
|
||
|
||
**No script, profile or build file was modified.** Findings only:
|
||
`documentation/audits/SPIKE-universal-iso-2026-07-31.md`. Opened R-139..R-147; restated R-128.
|
||
|
||
Three results that constrain any future universal-ISO spec:
|
||
|
||
- **A disk filter matching more than one device does not fail safe.** Observed in a nested VM: with
|
||
`filter.ID_SERIAL_SHORT = "SPIKE*"` matching two disks and `filesystem = "ext4"`, the installer
|
||
silently picked one and wiped it (`disk0` hash changed, `disk1` byte-identical). `validate-answer`
|
||
accepts such an answer. The `filter did not match any devices` guard covers the ZERO-match case
|
||
only. **R-139.**
|
||
- **No udev property distinguishes an internal system disk from external media.** Measured on
|
||
`demo-felhom` with its 1 TB external drive attached: `ID_BUS` is `ata` for *both* (the external is a
|
||
SATA drive in a USB caddy), `lsblk RM` is `0` for both, and `device-info` exposes no removability
|
||
property at all. On `demo-hp` the NVMe carries no `ID_BUS` or `ID_TYPE` whatsoever. Nothing
|
||
expresses *role*. **R-141 context.**
|
||
- **A root credential is mandatory in the answer schema, but a usable one is not.**
|
||
`root-password-hashed = "*"` validates and installs to completion (guest powered off after 200 s
|
||
with `reboot-mode = "power-off"`). `[first-boot].ordering` also accepts `before-network`, which is
|
||
the only ordering that could close the exposure window structurally — today's bootstrap unit is
|
||
`After=network-online.target pveproxy.service` and the pairing poll waits indefinitely. **R-141.**
|
||
|
||
Also: `prepare-iso` leaves `grub.cfg` byte-identical to stock (15 menu entries, automated **and**
|
||
interactive), so a two-entry boot menu is purely a change to Felhom's own `grub.cfg.tmpl` — the
|
||
single-entry menu is Felhom's repack, not a constraint of ISO preparation.
|
||
|
||
## v1.22.0 — E-2 Part 2: new boxes get a real backup target, or are told they do not (2026-07-29)
|
||
|
||
Every box installed before this got `local_backup_target: "local"` — the vzdump target on the SAME
|
||
physical device as the guest, so a drive failure took the guest and its only local backup together.
|
||
E-1 fixed two machines by hand; this fixes the installer.
|
||
|
||
**Case A** — an eligible secondary drive is already mounted: create `felhom-backup` on that drive's
|
||
own mountpoint through the new `felhom-backup-target-apply` wrapper (`create` + `grant`), and point
|
||
the primary tier at it.
|
||
|
||
**Case B** — system drive only: the target stays on the system drive and this is **recorded as
|
||
DEGRADED, not as normal** — two warning lines saying it protects against corruption but not against
|
||
a disk failure, and what to do about it. **The install still succeeds**: a single-drive appliance is
|
||
a valid product, it just cannot survive drive loss.
|
||
|
||
**Phase 0 reality check, which inverts the emphasis:** the installer has **no drive-enrollment
|
||
step** — `/mnt/felhom-drives` is agent-installed at runtime and drives are formatted/mounted through
|
||
the dashboard later. So on a fresh appliance **Case A almost never fires**; the common case is
|
||
Case B, with the drive arriving months later through the wizard (E-2 Part 3). Case A exists for the
|
||
reinstall/recovery shape, where an agent-generated `.mount` unit already brings the drive up by
|
||
fs-UUID.
|
||
|
||
**Eligibility suggests and refuses the absurd; it never decides by transport.** The reference
|
||
hardware forbids that: demo-felhom's backup drive is an external **USB HDD**, and **both** demo
|
||
boxes' drives report `removable=0`. A transport rule would disqualify the reference drive; a
|
||
removable rule would find no candidate at all. The filter is: a real mounted `ext4/xfs/btrfs` under
|
||
`/mnt`, on a `/dev` source, not the root device, ≥ 64 GB.
|
||
|
||
**SCENARIO F — an already-configured box is never "corrected".** `configure_backup_target` returns
|
||
early, changing nothing, when the storage id already exists; `setdefault` independently preserves an
|
||
existing box's whole `backup` block. Both guards are load-bearing and both were exercised.
|
||
|
||
Also installs (and uninstalls) `/usr/local/sbin/felhom-backup-target-apply`, 0755 root:root, with
|
||
`bash -n` before install — the pbs-apply pattern. The wrapper is the single place the F-1/F-2 laws
|
||
live, so the installer and the runtime move cannot drift.
|
||
|
||
**Proofs (harness against the extracted functions, stubbed pvesm/wrapper — installer-logic-tested,
|
||
NOT install-tested; no reinstall was performed):**
|
||
|
||
| Scenario | Result |
|
||
|---|---|
|
||
| A — eligible drive | `create felhom-backup /mnt/bigdisk` then `grant felhom-backup`; resolved `felhom-backup` |
|
||
| B — single drive | DEGRADED warnings, resolved `local`, **rc=0 — install not failed** |
|
||
| F — configured box | skipped, **0 wrapper calls** |
|
||
| F red-proof (guard removed) | **2 wrapper calls** — it would have "corrected" a correct box |
|
||
|
||
## felhom-host-install 1.21.0 — F-LEAK: the restore-test scratch band gets its own path-scoped grant (2026-07-28)
|
||
|
||
Campaign 8's F-LEAK: a restore-test whose restore **fails** leaves a scratch guest the agent cannot
|
||
destroy — `403 missing privilege VM.Allocate` — so a half-restored guest holds its disks until a human
|
||
removes it. The cause is structural, not a missing privilege in the role: `FelhomAgentGuest` is granted
|
||
at `/pool/felhom`, and **a guest only joins that pool when its restore completes**. A failed restore
|
||
therefore produces a guest that exists, is in no pool, and is out of the token's reach.
|
||
|
||
`apply_scoped_acl` now also grants `FelhomAgentGuest` at each `/vms/<id>` in
|
||
`PVE_SCRATCH_VMID_MIN..PVE_SCRATCH_VMID_MAX` (990000–990009, the band the restore-test already picks
|
||
from), to **both** the user and the token — the privsep intersection rule applies here as everywhere.
|
||
|
||
**This is not a widening.** PVE ACLs are path-scoped: ten explicit `/vms/<id>` grants authorise the
|
||
agent on exactly those ten IDs. Granting at `/vms` was rejected — that would authorise destroying every
|
||
guest on the box, including a co-tenant's. Verified live:
|
||
|
||
| path | `VM.Allocate` |
|
||
|---|---|
|
||
| `/vms/990000` (scratch band) | **yes** — 13 privs |
|
||
| `/vms/100`, `/vms/9999` | no — 3 privs (base only) |
|
||
| `/vms/990010` (one past the band) | no — 3 privs |
|
||
|
||
Two supporting changes, both load-bearing:
|
||
- `remove_scoped_acl` deletes the band grants **before** the role delete. PVE refuses to delete a role
|
||
still referenced by any ACL, so omitting this would have broken the uninstall.
|
||
- `step_verify` asserts the band grants. A missing one is otherwise invisible until a restore-test
|
||
*fails*, which is precisely the case that leaked a guest.
|
||
|
||
**The alternative that does not work, recorded so it is not retried:** adopting the stranded guest into
|
||
the pool first. It was implemented (agent v0.107.0), shipped, and refuted live on 2026-07-28 —
|
||
`PUT /pools/felhom` with `vms=<id>` **also** requires `VM.Allocate` on `/vms/<id>`, so `Pool.Allocate`
|
||
cannot bootstrap its own membership. That code was removed in agent v0.108.0.
|
||
|
||
## felhom-host-install 1.20.0 — R-82: a fresh box defaults to local-daily + offsite-weekly (2026-07-26)
|
||
|
||
The `backup` defaults gain `backup_targets: [{target_id: "felhom-pbs", cadence_seconds: 604800,
|
||
keep_last: 2}]` — the offsite tier, weekly, two weeks retained (operator ruling 2026-07-26).
|
||
|
||
**`setdefault` semantics preserved, and PROVEN both ways** against the exact line:
|
||
|
||
```
|
||
FRESH -> backup_targets: [{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":2}]
|
||
UPGRADE -> preserved verbatim, no backup_targets
|
||
```
|
||
|
||
That second line is the load-bearing one: **an in-place upgrade can never silently start writing to
|
||
an offsite datastore.** Existing boxes are migrated explicitly (R-82 Slice D.2), which is what keeps
|
||
a routine host-install re-run from enrolling a box into offsite backups nobody asked for.
|
||
|
||
**Why the tier is written before its storage exists:** `felhom-pbs` only appears when the hub
|
||
provisions the DR tier. The agent (v0.102.0) DEFERS a tier whose target storage is absent, so the
|
||
tier stays silent until it is real and goes live with no restart the moment it is provisioned.
|
||
Writing it here means a box is never left with a DR tier that is provisioned but unscheduled — the
|
||
exact "applied and empty" state R-82 exists to end.
|
||
|
||
# Felhom scripts — Changelog
|
||
|
||
## felhom-host-install v1.19.0 — R-50 island control plane is the appliance default (2026-07-25)
|
||
|
||
Implements Phase A of the R-50 island bridge (spike GO, `documentation/audits/SPIKE-island-bridge-2026-07-25.md`).
|
||
A fresh **appliance** install is now born immune to F1 (a LAN/DHCP/site move can no longer take the
|
||
control plane down). **byo is unchanged**; an explicit `--bridge-ip` or `--no-island` keeps the LAN bind.
|
||
|
||
- New `ensure_island_bridge()` creates a portless host-internal bridge `vmbr9` (`169.254.253.1/30`),
|
||
idempotently, before the agent binds it. Portless = no physical port, so the /30 cannot collide with
|
||
any customer LAN and survives any renumber. vmbr0 untouched.
|
||
- `step_agent_config` (appliance default): `local_api.listen_addr` → `169.254.253.1:8443`; writes
|
||
`local_api.island_bridge`/`island_guest_addr` (the agent ≥ 0.96.0 attaches the guest's `net1` from
|
||
these); and pins `lan_resolver.host_ip` to the **LAN** IP explicitly — **Finding-1**: without it the
|
||
agent derives the DNS listen-addr from `listen_addr` and silently moves LAN DNS onto the island.
|
||
- `--no-island` opt-out; `--bridge-ip` still overrides. `configs/felhom-localapi-firewall.example`
|
||
(agent repo) rewritten — the portless island bind IS the LAN close.
|
||
- **Coupling:** this install REQUIRES agent ≥ 0.96.0 (older agents ignore the island fields → no net1 →
|
||
broken control plane). Vouch 0.96.0 before island installs go live. Hub `hostInstallVersion` synced to
|
||
1.19.0 (F-1). Existing boxes migrate via `documentation/runbooks/RUNBOOK-island-migration.md`.
|
||
|
||
## ISO train v1.25.0 — the belt goes fleet-wide, the repo goes free, the screen learns ő, fresh boxes boot at the floor (2026-07-23)
|
||
|
||
Four rulings from 2026-07-23. host-install **1.17.0 → 1.18.0**, build-felhom-iso **1.24.0 → 1.25.0**,
|
||
felhom-bootstrap R-63 font.
|
||
|
||
- **Part 1 (F9) — the OOB belt is a DEFAULT appliance leg.** `felhom-host-install.sh`: appliance mode
|
||
now installs the H1 belt (felhom-sshd + static `inet felhom_oob` table + felhom-op) unless `--no-oob`;
|
||
byo still refuses `--enable-oob` (owner-consented, deliberate exclusion — flagged, unchanged). The
|
||
config writer now sets `oob.enabled=true` when the belt is installed, so the agent activates it (claims
|
||
a port, renders the sshd config, fills the belt sets) instead of the units sitting inert — the
|
||
pre-v1.25.0 state that left the belt unusable on both fleet boxes (07-22 power-outage audit F9). byo
|
||
config asserts unchanged (they already refuse `oob.enabled`). The belt is lockout-SAFE with no operator
|
||
peer yet (empty sets + no key = no access granted, `:22` untouched). **Live: installed + validated
|
||
end-to-end on felhom-pve** (operator WG peer → wg-felhom → belt → felhom-op login PROVEN) **and
|
||
installed on demo-hp** (same infra; login is the same mechanism).
|
||
- **Part 2 (apt) — enterprise → no-subscription on fresh appliance installs.** New idempotent
|
||
`align_apt_repos` (appliance-only; runs before the first apt op): disables the PVE/ceph **enterprise**
|
||
`.sources` (deb822 `Enabled: no`) or legacy `.list`, adds `pve-no-subscription.sources` (suite+keyring
|
||
derived from the enterprise file), `apt-get update` must exit clean. Scoped to the Proxmox-managed repos
|
||
only; the owner's lists are never touched; no upgrades run.
|
||
- **Part 3 (R-63) — the console learns ő/ű.** `felhom-bootstrap.sh` loads a Latin-2 console font
|
||
(`Lat2-Terminus16` → `Lat2-Fixed16` → `Lat2-Terminus14`) ONCE before the first paint (idempotent,
|
||
best-effort — never blocks boot). Fixes the R-59 network screen (élő/telepítő/ellenőrizze) and the
|
||
pairing banner (képernyő/teendő) rendering ő as blanks. Lat2 fonts ship in the trixie/PVE base, so no
|
||
copy rewording was needed.
|
||
- **Part 4 (R-71 mitigation) — every ISO build asserts golden ≥ managed floor.** `build-felhom-iso.sh`
|
||
gains `ver_ge` + `assert_golden_ge_floor` (runs before workspace/mint; `die`s with both versions on
|
||
`golden < floor`; `FELHOM_ASSERT_GOLDEN`/`FELHOM_ASSERT_FLOOR` inputs; unset warns loudly, never a
|
||
silent pass). Red-proofed both directions (floor>golden → exit 1; golden≥floor → OK). Standing rule 5
|
||
added to `publish-train-rules.md`. Closes the day-0 update-vs-bridge race window at its source
|
||
(DIAG-f10 / R-71). ISO_VERSION → 1.25.0.
|
||
|
||
**Golden lift EXECUTED (2026-07-23):** golden republished at controller 0.161.0 (`build-golden.sh` on felhom-pve), published to Gitea (618 MB, sha `77624408…`, served-copy verified), operator-vouched (golden 0.153.0→0.161.0; floor stays 0.156.0) — the LIVE golden<floor gap is CLOSED. **v1.25.0 ISO built** (`felhom-pve-9.2-1-v1.25.0-nested-vm-generic-mkimage.iso`, sha `99b38420…`) with the gate passing live; rootpw sibling + R-63 font present. Nested drill = the remaining validation leg.
|
||
|
||
## ISO train v1.24.0 — the box that can never call home learns to say so, heal itself, and let you in (R-59 + R-60 + R-61) (2026-07-22)
|
||
|
||
All three findings are from the HP t740 install (2026-07-21). Proven end-to-end on a nested
|
||
wrong-NIC install (spike + virgin-ISO drill: `documentation/audits/SPIKE-firstboot-nic-sweep-2026-07-22.md`).
|
||
|
||
- **R-59 — the silent unreachable box gets a legible screen.** A no-DHCP install bakes the
|
||
installer's **192.168.100.2 fallback as a static `vmbr0` config** (with a dead
|
||
`gateway 192.168.100.1`) and completes; the box then waited silently forever. The first-boot
|
||
bootstrap now runs a **network gate before mode dispatch**
|
||
(`scripts/iso/felhom-bootstrap.sh:251` `network_gate`, called at `:510`): hub unreachable →
|
||
physical-NIC diagnosis (name/MAC/carrier/speed; NICs are raised first — unused ports sit
|
||
admin-DOWN with unreadable carrier), the fallback signature named when present, and a calm
|
||
Hungarian console screen (`paint_network_screen`, `:230` — the `print_pairing_banner` pattern)
|
||
with the NIC table and the one-line remedy; then retry every 60 s with the unit held in
|
||
`activating` (waiting is not failing — the v1.21.0 lesson, unit `TimeoutStartSec=infinity`).
|
||
The trigger is always **hub-unreachable, never the fallback signature**: the drill proved a
|
||
second wrong-NIC variant (the installer picks the NIC that LEASES and bakes the lease as
|
||
static — a later cable move leaves a plausible-looking dead config with no signature).
|
||
*Recorded deviation:* R-59 ships as this first-boot refuse-loudly gate, not an install-time
|
||
abort — the fallback is baked inside the Proxmox auto-installer itself, unreachable without an
|
||
installer-initrd hook (out of scope by operator ack, noted on the ROADMAP row).
|
||
- **R-60 — first-boot NIC sweep self-heal.** While the hub is unreachable AND no install attempt
|
||
has begun (`state.json` gate; the unit's `ConditionPathExists=!done-flag` confines the whole
|
||
script to pre-install anyway), the gate sweeps (`sweep_nics`,
|
||
`scripts/iso/felhom-bootstrap.sh:178`): per candidate NIC, re-point `vmbr0`'s `bridge-ports`
|
||
(atomic tmp+mv, always rendered from the pristine copy) → `ifreload` → **flush vmbr0's
|
||
addresses/routes** → bounded `dhclient` (20 s) → hub probe (any HTTP status counts — the hub
|
||
answers `/` with 302). The flush is load-bearing: the baked fallback default route survives a
|
||
successful lease otherwise (dhclient-script never replaces an existing default route) and the
|
||
probe rides the dead gateway — found live in the drill. First success persists
|
||
`bridge-ports <winner>` + `iface vmbr0 inet dhcp` (original kept as
|
||
`interfaces.felhom-bak`) and proceeds; total failure restores the config **byte-identically**.
|
||
Drill: cable moved → healed + registered at the hub in well under a minute, unaided.
|
||
- **R-61 slice 1 — the baked root password becomes knowable.** `build-felhom-iso.sh` writes the
|
||
minted throwaway plaintext to a **0600 sibling `<iso>.rootpw.txt`** (password + ISO name +
|
||
build date; `scripts/iso/build-felhom-iso.sh:254`) — the single record of truth. Never
|
||
stdout/logs; the manifest carries only a pointer (`:383`). A fixed well-known password stays
|
||
REJECTED (operator ruling 2026-07-21). Drill: the emitted password's hash matches the
|
||
installed box's `/etc/shadow` — the console lockout is closed.
|
||
- **Harness** (`scripts/iso/test/bootstrap-modes.sh`): caught up with the v1.21.0 in-script wait
|
||
(the old pairing scenarios would hang), plus four gate scenarios — hub-reachable = ZERO new
|
||
behavior (no ip/ifreload/dhclient/sleep call, fixture untouched), sweep-success (persist +
|
||
bak + proceed), sweep-fail (screen + byte-identical config), state-file-present (sweep NEVER
|
||
invoked). New `test/rootpw-emission.sh` for R-61 (emission, 0600, plaintext↔answer-hash
|
||
cross-check, manifest guard). **Four red-proofs run and restored:** manifest-plaintext leak,
|
||
restore-on-failure dropped, state gate removed, hub-reachable early-return removed — each
|
||
failed exactly its guarding assertion.
|
||
- Assistant image gains `python3` (the bootstrap's JSON parsing needs it; PVE ships it on real
|
||
boxes — the harness runs the bootstrap in the image). `build-felhom-iso.sh` gains the
|
||
`FELHOM_ISO_KEEP_WORK=1` debug/test escape and resolves `OUT_ISO` before the mint (the rootpw
|
||
sibling is named after the ISO).
|
||
|
||
## build-hub.sh v1.23.0 — the hub build script enters the repo (2026-07-21)
|
||
|
||
It never was in one. `/mnt/5_hdd/felhom.eu/build/felhom-hub/build.sh` lived only in a DooPlex build
|
||
directory — unversioned, unreviewed, un-diffable and gone with the box, while being the only way the
|
||
hub image gets built. Recorded as a follow-up in `felhom.eu/REPORT.md` §6; adopted here.
|
||
|
||
Adopted verbatim — the build logic is byte-identical (verified by diff), and the only additions are
|
||
a `SCRIPT_VERSION` constant echoed in the banner and a header naming the canonical/operative split.
|
||
The build-dir path is now a **symlink** to this file, so there is one source of truth and a repo edit
|
||
is live immediately (the `install_skills.py` pattern). The previous copy is retained beside it as
|
||
`build.sh.pre-adopt-2026-07-21`.
|
||
|
||
It already carried the GitOps deploy hint fixed earlier in the week (manifest bump → hard-refresh →
|
||
deliberate sync, and the named trap that `kubectl set image` / `kubectl apply` are silently reverted
|
||
by the next ArgoCD sync) — that wording now lives in git along with the rest of it.
|
||
|
||
## install_skills.py — cross-platform (POSIX symlink / Windows junction) (2026-07-19)
|
||
|
||
Claude Code now runs on DooPlex (Debian 13), where `mklink /J` does not exist — the script would
|
||
have fallen through to COPY mode on every run, silently breaking the "repo edits are live
|
||
immediately" property that makes `felhom.eu/skills/` the source of truth.
|
||
|
||
- Link creation is now platform-dispatched behind `os.name == "nt"`: `os.symlink(...,
|
||
target_is_directory=True)` on POSIX, the existing `mklink /J` on Windows. Copy-mode fallback,
|
||
idempotency, and the re-run warning are unchanged.
|
||
- **`remove_existing()` had a real hazard on POSIX**: `os.rmdir()` fails on a symlink-to-directory,
|
||
which would have fallen through to `shutil.rmtree()` — and rmtree following a symlink into
|
||
`skills/` would have deleted the repo's own skill sources. It now unlinks symlinks explicitly
|
||
before any rmdir/rmtree path is reached. Existence checks use `os.path.lexists()` so a broken
|
||
link is seen and replaced rather than ignored.
|
||
- Human-facing output says "symlink" or "junction" per platform.
|
||
|
||
## build-felhom-iso.sh v1.22.0 — the boot screen is ours, and it offers exactly one thing (R-38) (2026-07-19)
|
||
|
||
**A boot menu is a product surface, and ours was Proxmox's.** Every ISO is now repacked after
|
||
`prepare-iso` to carry a Felhom boot screen and — the half that actually matters — a menu with
|
||
**exactly one entry**.
|
||
|
||
- **The safety half.** The stock PVE menu offers *Install Proxmox VE (Graphical)*, *(Terminal UI)*, a
|
||
serial variant, and an **Advanced Options** submenu holding two `nomodeset` entries, three debug
|
||
entries, *Rescue Boot*, memtest and *UEFI Firmware Settings*. Every one of those reaches the
|
||
**manual** installer, whose first question is which disk to wipe. A customer — or their helpful
|
||
nephew — must not be able to get there from a boot menu they are staring at. Those entries are not
|
||
hidden and not password-gated: **they are not emitted**. What ships is „Felhom telepítés", default,
|
||
5 s.
|
||
- **Boot behavior is unchanged.** The `linux`/`initrd` lines are lifted **verbatim at repack time**
|
||
from the ISO's own *Install Proxmox VE (Automated)* entry rather than frozen into a copy in this
|
||
repo, so a PVE bump that moves the kernel path or edits the append line tracks automatically
|
||
instead of silently diverging. The build **fails** if they cannot be found, if the append line has
|
||
lost `proxmox-start-auto-installer`, or if `auto-installer-mode.toml` is absent — that last one
|
||
because without it the single Felhom-labelled entry would boot a *manual* installer, which is the
|
||
precise outcome this change exists to prevent.
|
||
- **Gates, and then a re-check against the shipped artifact.** The rendered menu is asserted to have
|
||
exactly 1 `menuentry`, 0 `submenu`s and no live reference to
|
||
`proxtui`/`proxdebug`/`nomodeset`/`Rescue Boot`/`memtest`/`fwsetup` (comments are stripped first —
|
||
the template's header names the dropped entries on purpose). Then the menu and the theme background
|
||
are read back **out of `final.iso`**, not out of the extract tree, so "the branding landed in the
|
||
image we ship" is proven rather than assumed.
|
||
- **The boot card.** `grub/generate-grub-background.sh` letterboxes `website/assets/og-image_2.png`
|
||
onto a 1024×768 gfxterm canvas at repack time (ImageMagick, added to the assistant image), so the
|
||
boot screen has **one source** — the website asset — and not a second pre-rendered PNG in the repo
|
||
to drift. The card's own subtle background grid is continued across the letterbox fill
|
||
**phase-locked** to where the card's grid lands, so the fill is seamless instead of a 500 px square
|
||
of grid floating in flat navy. The generator refuses a source whose geometry no longer matches the
|
||
measured grid constants — a silently swapped asset would misplace every line, and that only shows
|
||
up on a boot screen nobody re-checks.
|
||
- **Menu positioning needs a theme.** Plain `background_image` cannot move the menu off the wordmark,
|
||
so `grub/felhom-theme.txt` (gfxmenu) puts it in the lower third the layout deliberately leaves
|
||
empty, with the entry optically centered under the card (measured off a canary screenshot — the
|
||
comment records the measurement). No "e: edit entry" hint label: advertising the editor works
|
||
against a single-entry menu.
|
||
- **`mkimage-surgery.sh` → `iso-repack.sh`.** Branding and the slice-B loader swap need the same
|
||
extract → modify → re-master cycle, so they now share **one** pass instead of re-mastering twice.
|
||
The mkimage recipe is untouched. The embedded module list is still derived from the **stock**
|
||
`grub.cfg` (snapshotted before branding rewrites it, so branding cannot shrink the embedded set),
|
||
plus gfxmenu's `bitmap`/`bitmap_scale`/`trig` renderer dependencies.
|
||
- `--no-brand` exists for debugging the stock menu and says so loudly in the build log and the
|
||
manifest. It is not a shipping mode.
|
||
|
||
**Live-validated (nested canary, UEFI/OVMF, PVE 9.2-1).** GRUB renders the card; the single entry is
|
||
selected and counts down in Hungarian (accents render correctly under gfxterm); it auto-fires at 0;
|
||
the automatic installer runs and reaches the **match-nothing abort exactly as in v1.21.0**
|
||
(`ERROR: Installation failed: filter did not match any device` → `Installation aborted`), touching no
|
||
disk. Both shipping ISOs rebuilt (see `REPORT-iso-branding-2026-07-19.md` for shas).
|
||
|
||
**Not touched (deferred):** the installer squashfs/initrd theme — the post-GRUB screens are still
|
||
Proxmox-branded. GRUB's `e`/`c` keys also remain reachable (suppressing them needs a superuser
|
||
password, which is a separate decision); the menu no longer advertises them.
|
||
|
||
## build-felhom-iso.sh v1.21.0 — the pairing wait stops looking like a failure (R-33) (2026-07-18)
|
||
|
||
**Waiting is not failing, and must not be reported as failure.** Until now the PAIRING poll loop WAS
|
||
systemd's `Restart=on-failure`/`RestartSec=30`: each invocation did register-if-needed plus exactly
|
||
one poll, then exited **non-zero** so the unit would be restarted 30s later. Functionally fine — but
|
||
every one of those exits made systemd print `Failed to start Felhom host bootstrap …` on the physical
|
||
console **the customer is watching**. The 2026-07-18 N100 rehearsal measured **52 FAILED lines in
|
||
~11 minutes** while nothing whatsoever was wrong: the box had registered correctly and was waiting to
|
||
be bound, which is the normal state of a freshly installed appliance. That is the customer's literal
|
||
first impression of the product
|
||
(`documentation/tests/VALIDATION-n100-rehearsal-2026-07-18.md` finding F6, ROADMAP **R-33**).
|
||
|
||
- **`felhom-bootstrap.sh` — the wait moved INSIDE the script.** `run_pairing()` is now a
|
||
`while true` loop that sleeps `POLL_INTERVAL` (30s — the hub-side rate is **unchanged**) between
|
||
polls, so the unit sits in `activating` instead of failing. Registration was split out into
|
||
`register_appliance()`, which returns non-zero for a *transient* problem (no network yet, identity
|
||
not gatherable, hub returned no token) and is simply retried by the loop rather than taking the
|
||
whole unit down. Three cadence constants at the top of the file: `POLL_INTERVAL=30`,
|
||
`BANNER_EVERY=10` (console banner every 5 min), `HEARTBEAT_EVERY=20` (journal heartbeat every
|
||
10 min).
|
||
- **The journal got quiet without going dark.** A `204 not bound yet` is logged **once** on entry
|
||
(worded so nobody reads it as an error) and then only on the 10-minute heartbeat with elapsed
|
||
minutes. `404` and unexpected HTTP codes likewise degrade to the heartbeat instead of a line every
|
||
30s. **`410` still exits non-zero on purpose** — the delivery was consumed but no local env exists,
|
||
a genuine crash window where handing the box back to systemd for a clean restart is the right move.
|
||
- **The console banner got calmer and less repetitive.** Printed every 5 min instead of every cycle
|
||
(a customer may walk up at any time, but a banner every 30s is its own noise). Dropped the
|
||
belt-and-braces `parositasra var / párosításra vár` double-spelling for a single accented line, and
|
||
added the reassurance the rehearsal showed was missing: „Ez a képernyő magától frissül — nincs
|
||
teendő a doboznál, és nyugodtan itt hagyhatod bekapcsolva."
|
||
- **`felhom-bootstrap.service` — `TimeoutStartSec=infinity` (load-bearing, not cosmetic).** A
|
||
`Type=oneshot` `ExecStart` is killed at `DefaultTimeoutStartSec` (90s). Without this line systemd
|
||
would kill the new in-script wait after 90 seconds and `Restart=on-failure` would **silently
|
||
reinstate the exact console spam this change removes** — a regression that would look like the
|
||
feature working for the first three polls. `Restart=on-failure`/`RestartSec=30` are deliberately
|
||
kept: they still cover the DIRECT path, a failed host-install, and the 410 crash window.
|
||
- **Behaviour verified, not assumed.** Driven in a throwaway Debian container against a stub hub that
|
||
answers `204` five times and then delivers: the script logged the wait **once** plus one heartbeat,
|
||
**never exited between polls**, then consumed the delivery, wrote the 0600 env, fell through to the
|
||
direct install in the same invocation and exited **0**. Under the old design the same sequence
|
||
produced five separate unit invocations and five `Failed to start` console lines.
|
||
|
||
**Compatibility:** hub-side polling rate, endpoints, payloads and the one-shot delivery semantics are
|
||
all unchanged; this is purely how the box waits and what it says while waiting. An older ISO against
|
||
any hub keeps working exactly as before.
|
||
|
||
|
||
## build-felhom-iso.sh v1.20.0 — console pairing-code banner for customer self-bind (R-27 slice 1) (2026-07-17)
|
||
|
||
Supports the hub's customer self-bind flow (hub v0.66.0). In PAIRING mode, `felhom-bootstrap.sh` now
|
||
reads the additive `pairing_code` from the `POST /api/v1/appliance/register` response, persists it at
|
||
`/etc/felhom/appliance-pairing-code`, and prints a Hungarian **console banner** (to `/dev/console`,
|
||
stdout fallback) each pairing cycle so the customer can read the code off the physical screen and type
|
||
it — together with their retrieval passphrase — on the hub's public `/bind/<token>` page. The code is
|
||
**non-secret** (possession proof only; the passphrase is the second factor), so it is safe on the
|
||
console. **Graceful degradation both ways:** a hub older than v0.66.0 omits `pairing_code` → the banner
|
||
prints nothing and register/poll are unchanged; an old ISO against a v0.66.0 hub simply ignores the new
|
||
field. No change to DIRECT mode. Green: `bash -n` clean on both scripts.
|
||
|
||
## build-felhom-iso.sh v1.19.0 — the universal secret-free ISO: `--pairing` mode (R-21 slice C) (2026-07-17)
|
||
|
||
The scripts half of the universal ISO. `felhom-bootstrap.sh` gains a PAIRING mode — **one unit, two
|
||
modes**, decided by the env:
|
||
|
||
- **DIRECT** (env has `FELHOM_CUSTOMER_ID` + `FELHOM_RETRIEVAL_PASSPHRASE`): the slice-A path,
|
||
**byte-identical** — only branched around. Scenario D regression proves the pairing code is provably
|
||
NOT entered (a fake hub records ZERO `/appliance/*` calls).
|
||
- **PAIRING** (generic ISO, no customer/passphrase baked in): gather identity (SMBIOS uuid + physical
|
||
MAC set + SSH host pubkeys + hw), `POST /api/v1/appliance/register` → persist the appliance token
|
||
(0600), then ONE `GET /api/v1/appliance/poll` per invocation (the existing
|
||
`Restart=on-failure`/`RestartSec=30` IS the poll timer — no long-running-oneshot timeout). On the
|
||
bind's 200 delivery, WRITE the delivered credentials into the env (0600) and fall through to the
|
||
DIRECT path — so every later retry is a plain direct install (the delivery is one-shot; a second poll
|
||
→ 410). Delivery-received secrets + the appliance token are shredded on host-install success.
|
||
- **`build-felhom-iso.sh --pairing`** builds the GENERIC ISO: no `--bootstrap-env`, a secret-free env
|
||
carrying only the hub URL, `-generic` filename marker, manifest `mode: pairing` + `secret-bearing:
|
||
no`. Direct mode (secret-bearing) is unchanged. **New `profiles/generic.profile`.**
|
||
- Validated: `bash -n` + shellcheck clean; the `test/bootstrap-modes.sh` harness (Scenario D + pairing
|
||
register/poll + the delivery→env→host-install handoff) all green in a debian container.
|
||
|
||
## build-felhom-iso.sh v1.18.0 — firmware loader option `--loader shim|mkimage` (R-21 slice B, F1) (2026-07-17)
|
||
|
||
Closes N100 finding **F1 (HIGH):** cheap AMI (`AN3PLUS 0.01`-class) UEFI firmware can't relocate the
|
||
ISO's stock signed GRUB from USB (`relocation 0x0 is not implemented yet`; SB-off + shim-bypass don't
|
||
help). The run's live `grub-mkimage` workaround is now a first-class pipeline mode.
|
||
|
||
- **`--loader shim|mkimage`** (default **shim**; profile-settable via `FELHOM_LOADER`, `--loader`
|
||
wins). `shim` = the stock output, **byte-for-byte unchanged** (the surgery is skipped) — keeps the
|
||
MS-signed shim→GRUB chain and Secure Boot on compliant firmware (S2b). `mkimage` replaces the ISO's
|
||
UEFI boot path with a **monolithic `grub-mkimage` loader** built from the ISO's own GRUB modules
|
||
(module set from the ISO's `grub.cfg`; embedded `search --fs-uuid` → `configfile` the real menu).
|
||
The recipe is reproduced from the N100 run evidence, not re-derived.
|
||
- **`mkimage-surgery.sh`** (new): runs in the assistant container **after** `prepare-iso`, so the
|
||
answer/first-boot payload is provably untouched except the loader. Swaps the loader into the ISO9660
|
||
tree (real lowercase Rock-Ridge path) **and** inside the `efi.img` ESP (FAT), then re-masters with
|
||
`xorriso` preserving the hybrid BIOS boot (grub2-mbr + El Torito) and the GPT EFI System Partition,
|
||
dropping only the irrelevant Apple HFS+/APM map (which trips xorriso's overlap check on re-emit).
|
||
- **Loud + unmistakable (rule 4):** build banner, manifest `loader:` + `grub-mkimage:` fields, and a
|
||
**`-mkimage`** filename suffix when non-default. mkimage ISOs are **unsigned → Secure Boot must be
|
||
OFF** on the target board.
|
||
- **Dockerfile.assistant:** adds `grub-common` + `grub-efi-amd64-bin` + `mtools` + `dosfstools` (grub
|
||
2.12 == the PVE 9.x ISO's `2.12-9+pmx2` generation). **New `profiles/n100.profile`** (cheap-board,
|
||
`FELHOM_LOADER="mkimage"`, SB-off note).
|
||
- **Validated (RUNBOOK-B legs, nested VM 311 on felhom-pve):** leg 1 shim boots + zero-touch installs
|
||
under OVMF **Secure Boot enforcing** (+ SeaBIOS boot — hybrid path survived); leg 2 `--loader
|
||
mkimage` boots + installs under OVMF **SB-off** (the monolith's embedded fs-uuid search found the
|
||
ISO + chained its menu); **leg 3 (red-proof)** the mkimage ISO under **SB-enforcing FAILS** —
|
||
firmware `Access Denied`, no boot; leg 4 the surgery is surgical (only the loader + regenerated El
|
||
Torito artifacts differ; the full install payload is byte-identical). `bash -n` + shellcheck clean.
|
||
Physical N100 closure folds into the supervised rehearsal (an `n100-safety` match-nothing ISO built
|
||
+ sha-recorded, unbooted, for an optional zero-risk pre-flight).
|
||
|
||
## felhom-tenantsync.sh v1.1.0 — deprovision op (customer RESET teardown) (2026-07-17)
|
||
|
||
Adds the `{"op":"deprovision","customer_id":"<id>"}` op the slice-1 header explicitly reserved
|
||
("namespace/data deletion is a deliberate, separate decision"). It is exactly that deliberate,
|
||
hub-side ack-gated decision (the customer RESET, hub v0.61.0): delete the token (its ACLs purge with
|
||
it) → delete the residual namespace ACLs → **destroy the namespace AND all its backup groups**
|
||
(`proxmox-backup-client namespace delete <ns> --delete-groups true`, via the transient admin token).
|
||
IDEMPOTENT — a missing token / namespace is success (`deleted:false`), so a re-run after a partial
|
||
reset converges. The shared `felhom@pbs` user is NEVER touched (co-tenants ride it). Returns
|
||
`{"status":"ok","namespace","datastore","deleted":<bool>}`. Secret hygiene unchanged (no secrets in
|
||
this path). Client seam: `tenantsync.Deprovision(ctx, customerID) (existed bool, err error)`.
|
||
|
||
## felhom-host-install.sh v1.17.0 — appliance guest auto-sizing (F5) + doc-drift fix (2026-07-17)
|
||
|
||
Closes `VALIDATION-n100-baremetal-2026-07-16.md` **F5 (MEDIUM):** appliance mode provisioned the
|
||
golden default (2 GB RAM) on a 16 GB host and never surfaced the `--memory`/`--cores` caps. Now, in
|
||
**appliance** mode with no explicit cap, the guest is auto-sized from the host (LXC limits are cheap →
|
||
err generous): RAM = `clamp(host-4096, min 4096, max host-2048)` then a hard ceiling `host-1024`
|
||
(never over-commit); cores = `host-1, min 2`. An explicit `--memory`/`--cores` ALWAYS wins untouched;
|
||
byo mode still requires explicit caps (never auto-sized). Host reads (`MemTotal`/`nproc`) are
|
||
overridable via `FELHOM_FAKE_MEMTOTAL_MIB`/`FELHOM_FAKE_NPROC` for the mode harness. Sizing table:
|
||
8 GB→4096, 16 GB→12288, 32 GB→28672 MiB; a 4 GB host→3072 (min capped at host-1024).
|
||
Red-proof: `hostinstall-mode-harness.sh` new F5 section (8/16/32 GB + small-host edge + explicit-flag
|
||
precedence) — pre-fix (auto-size disabled) FAILs, post-fix PASSes. **Same commit (R-16 doc-drift
|
||
leftover):** the operator-signing-keys "EMPTY by default" comment was stale (the keys are PINNED to
|
||
the real `felhom-op-1`/`felhom-rec-1` ceremony keypairs) — corrected, comment-only, no behavior.
|
||
|
||
## felhom-poke.sh v1.0.0 — NEW (agent-plane immediate-sync, Direction-2a, 2026-07-16)
|
||
|
||
The offsite endpoint's THIRD hub forced-command surface (`documentation/runbooks/offsite-endpoint.md`
|
||
§11): a NON-root (`felhom-peersync`, no sudoers grant — a datagram needs no privilege) forced command
|
||
that reads a box's WireGuard /32 from `$SSH_ORIGINAL_COMMAND`, validates it to `10.77.0.0/24`, and
|
||
sends ONE contentless (empty) UDP datagram from wg0 to `<ip>:51822`. That is the last hop of the
|
||
hub→ep0→box "sync now" poke; the box's felhom-agent (v0.89.0) poke listener then runs an immediate
|
||
desired-state cycle. Contentless + confined (the WG kernel independently refuses non-peer /32s —
|
||
`SPIKE-immediate-sync-transport-2026-07-16` P1 EKEYREJECTED). Port **51822** is a shared cross-repo
|
||
constant (felhom-agent `internal/poke.Port`, `hub/internal/poke`, this script's `POKE_PORT`). Hub
|
||
client: `hub/internal/poke`. Installed + live-validated on ep0 2026-07-16 (`poke-fired`; non-WG target
|
||
and arbitrary SSH command both `refused non-WG target` — the forced command never executes client
|
||
input). Peersync/tenantsync surfaces untouched.
|
||
|
||
## felhom bare-metal ISO pipeline v1.16.0 — NEW (R-21 slice A, 2026-07-16)
|
||
|
||
New operator tooling under `scripts/iso/`: a reproducible DooPlex pipeline that turns the official
|
||
PVE ISO into a Felhom auto-install ISO whose first-boot chain unattended-fetches-and-runs
|
||
`felhom-host-install.sh` until the host is enrolled + a guest provisioned. `felhom-host-install.sh`
|
||
is **unchanged** (invoked, not modified). Tagged **v1.16.0** to align the ISO with the current
|
||
host-install release (they must stay coherent; the ISO's baked invocation targets that flag surface).
|
||
|
||
- `Dockerfile.assistant` (assistant paired to the ISO's PVE generation by Debian codename + shellcheck),
|
||
`build-felhom-iso.sh` (render answer from `answer.toml.tmpl` + a profile, mint a fresh THROWAWAY root
|
||
hash per build, **gate validate-answer on OUTPUT TEXT not `$?`** — spike S1 exit-0-on-failure trap,
|
||
`prepare-iso --fetch-from iso --on-first-boot`, emit ISO + sha256 + manifest).
|
||
- In-ISO chain: `stub-first-boot.sh` (dumb, exactly-once, from-iso/fully-up) installs
|
||
`felhom-bootstrap.sh` + `felhom-bootstrap.service` (retry-forever `Type=oneshot`+`Restart=on-failure`)
|
||
+ a 0600 `bootstrap.env`. Retry-vs-resume encoded once (plain first; `--resume` when
|
||
`/var/lib/felhom-install/state.json` exists — safe per host-install v1.11.3).
|
||
- **Secret-bearing:** the unattended install needs the customer retrieval passphrase, so the ISO embeds
|
||
it (the distribution URL itself carries no secret). Supervised/single-use; the bootstrap scrubs the
|
||
env on success. Slice C removes the secret from the *distributed* ISO via claim-pairing.
|
||
- Validated on nested VM 310: build gate + red-proof; disk-filter fail-safe through the pipeline;
|
||
stub→unit→real public-channel fetch→invocation→retry; resume decision; exactly-once; no-net retry +
|
||
recovery. Terminal host-install success (hub-enrolled + guest-provisioned) is operator-gated (drill
|
||
customer needs the password-gated create-UI; live-DB insert unsafe). Full report: root `REPORT.md`,
|
||
`documentation/audits/SPIKE-baremetal-iso-2026-07-16.md`.
|
||
|
||
## felhom-host-install v1.16.0 — FELHOM_ESCROW rides the canonical sudoers (2026-07-13)
|
||
|
||
Companion to agent v0.88.0 + controller v0.127.0 (the controller-driven escrow ceremony wizard).
|
||
No new installer STEP: the `FELHOM_ESCROW` sudoers alias (the wizard's fixed-argv sudo
|
||
self-invocation grant) ships via the existing canonical-sudoers fetch
|
||
(`configs/felhom-agent.sudoers` from the agent repo, `visudo -cf`-gated before install) — the
|
||
bump documents that every ≥1.16.0 install/update carries it. Hub `hostInstallVersion` bumped in
|
||
the same commit (hostinstall_gates.py asserts equality); the hub deploy rides the next hub train
|
||
(display-only lag, accepted). EXISTING hosts get the refreshed sudoers with the agent-0.88.0
|
||
deploy (ship-configs step), not from this script.
|
||
|
||
## manifest_bearer_gate.py — new gate: no bearer-shaped literals in manifests/ (2026-07-13)
|
||
|
||
Part of the hub v0.53.0 bearer de-git. Greps every `manifests/*.yaml` for 64-hex literals
|
||
(the `openssl rand -hex 32` shape all felhom bearer/API keys use), comments included; exit 1 on
|
||
any hit. `felhom.secret.yaml`'s pre-existing committed secrets (umami APP_SECRET is 64-hex)
|
||
print as a visible non-fatal KNOWN-BACKLOG line until that file's own de-git (secrets.md
|
||
backlog). Run after ANY manifests/ change. Named "bearer" not "secret" — the repo `.gitignore`
|
||
`*secret*` pattern silently un-tracks any "secret"-named file (learned the hard way: the gate's
|
||
first landing never reached git). Red-proven: reintroducing a 64-hex literal → exit 1.
|
||
|
||
## felhom-host-install v1.15.0 — DR-tier-by-default: uniform DR plumbing on every install (2026-07-12)
|
||
|
||
Sweeps the DRILL-day0-vm-2026-07-12 installer findings so the next Day-0 runs the drill's §5
|
||
sequence with zero fix-and-continue stops. Operator decisions 2026-07-12: capability is BAKED
|
||
uniformly; activation is a hub flag; WG is base infrastructure.
|
||
|
||
- **F-7** — ships `/usr/local/sbin/felhom-pbs-apply` (fetch `configs/felhom-pbs-apply` from the
|
||
agent repo, `bash -n`, install 0755 — the mkfs/selfupdate wrapper pattern). The FELHOM_PBSDR
|
||
sudoers alias already shipped; the 3 `pbsdr-*` capabilities are no longer born DEGRADED.
|
||
Uninstall removes it (disclosure↔uninstall parity kept, harness GL4-D green).
|
||
- **F-10** — installs the `age` package (escrow-ceremony identity-wrap dependency; a fresh-box
|
||
ceremony died on it live).
|
||
- **F-9 / decision 5** — the rendered agent.json now defaults `wg_tunnel.enabled: true` in BOTH
|
||
modes (WG is base infrastructure like cloudflared, also the future OOB transport; setdefault —
|
||
a deliberately-disabled preserved config is respected). The byo wg-off config assert is
|
||
RETIRED; byo consent = the typed disclosure ack (the WG tunnel + pbs-apply wrapper + age are
|
||
now on the disclosed list) + the tester-agreement disclosure line.
|
||
- **ACL** — the default `--acl-storages` set keeps `felhom-pbs` (drill-proven: narrowing it 403s
|
||
the PBS-DR apply-bridge at `/storage/felhom-pbs`); a missing felhom-pbs storage is now an
|
||
expected-pre-tier INFO, and no longer a byo refusal (the tier creates the storage later).
|
||
- **F-1** — single version source: `SCRIPT_VERSION` alone feeds `-h`, the run banners, and (via a
|
||
gate cross-check) the hub Setup-tab const `hostInstallVersion` (bumped 1.12.0→1.15.0 in the
|
||
same commit). The header no longer carries its own version literal.
|
||
- **F-2** — dry-run fetch lines print the REAL auth mode (`curl (anonymous …)` vs `curl -u <git>`)
|
||
instead of the hardcoded `-u <git>` lie on the anonymous branch.
|
||
- **F-8** — step 4b + the final summary now say out loud that root@pam was rotated + vaulted and
|
||
where to retrieve it (pointer only, never the credential).
|
||
- **New gate: `scripts/hostinstall_gates.py`** — mechanical grep-assertions (version single-source
|
||
+ hub-const sync, age, pbs-apply ship+uninstall, wg default-on + byo-assert retirement,
|
||
felhom-pbs in the default ACL). All red-proofed; the age assertion was hardened TWICE by its
|
||
red-proof (a prefix regex matched `agekit`, then a loose one matched the dry-run print line).
|
||
- F-3 (root-owned `guests/` parents) is fixed at its creator — the AGENT (v0.86.0); this pass
|
||
applied the drill's live-fix (chown the two parent dirs) to felhom-pve, which had the same
|
||
latent state. Peti's host: unreachable, check deferred.
|
||
|
||
## felhom-host-install v1.14.0 — deployment_mode from --mode (appliance self-heal gate) (2026-07-12)
|
||
|
||
- The agent-config writer now templates the top-level **`deployment_mode`** field from `--mode`
|
||
(`appliance` | `byo`), authoritative even over a `--preserve-from` config. It gates the agent's
|
||
node self-heal (host networking recovery at boot — F12-class defense in depth, agent v0.85.0):
|
||
only `deployment_mode:"appliance"` unlocks the remedy; anything else, including a stale preserved
|
||
value or a typo, is byo (the check runs + WARNs, the remedy is unreachable). The byo config assert
|
||
additionally refuses `deployment_mode=appliance` (a byo box must never carry a host-service
|
||
self-heal). Companion to CAMPAIGN-3 Task A (`documentation/audits/CAMPAIGN-3-2026-07-11.md`).
|
||
|
||
## felhom-host-install v1.13.0 — systemd-journal group for the agent user (NAS verify) (2026-07-11)
|
||
|
||
- The agent-user setup now adds `felhom-agent` to the **`systemd-journal` group** (idempotent
|
||
`usermod -aG`): the NAS verify-before-commit pipeline (agent v0.81.0) classifies mount failures
|
||
from an UNPRIVILEGED `journalctl -u <unit>` read — group membership instead of a sudoers grant
|
||
(rule: no journalctl via sudo). Existing installs get the one-liner
|
||
`usermod -aG systemd-journal felhom-agent && systemctl restart felhom-agent` (applied to the
|
||
demo host during this task's live validation; Peti's box only when the NAS feature reaches him).
|
||
- Version-header drift fixed: the file header still said v1.11.0 while `SCRIPT_VERSION` was
|
||
1.12.0 (the v1.12.0 bump shipped without a changelog entry or header sync); both now 1.13.0.
|
||
|
||
## felhom-host-install v1.11.0 — uninstaller gap-closure + operator-key day-0 fold (TASK GL-4, go-live G6 + G1-follow-up) (2026-07-08)
|
||
|
||
- **Uninstall gap-closure (G6):**
|
||
- NEW 4b4 block removes the self-update artifacts the install lays down but v1.10.0 never
|
||
removed: `/usr/local/sbin/felhom-selfupdate-guarded`, the A/B slot files next to the live
|
||
binary (`felhom-agent.prev` + orphaned `.new.*` temps), `felhom-agent-rollback.service`, and
|
||
the `felhom-agent.service.d/felhom-agent-limits.conf` drop-in (+dir). Paths derived from the
|
||
authoritative `felhom-agent/configs/felhom-selfupdate-guarded`; `pending.json` was already
|
||
covered by the state-dir removal. Tolerate-absent, 4b2 shape.
|
||
- Enrolled/network drives mounted under `/mnt/felhom-drives/<name>` are now unmounted (deepest
|
||
first) before the root self-bind — **plain `umount` ONLY, never `-l`/`-f`**: a busy mount gets
|
||
a warning + "eject via the dashboard or stop the apps and retry" and the root bind is then left
|
||
alone (previously a child mount made the root umount die mid-teardown). The data is NEVER
|
||
touched — no wipe/format path exists anywhere near `/mnt/felhom-drives`.
|
||
- Both modes now END with an explicit **KEPT-vs-WIPED statement**: WIPED mirrors what the mode
|
||
actually ran; KEPT names the drives + their data (physically removable; busy ones listed as
|
||
retry), PBS backups + customer namespace (conditional on a pbs storage), the hub host/customer
|
||
record, the escrow blob, and — when step 4b had vaulted one — the root@pam recovery credential.
|
||
- Guest-only mode prints the drives the agent's bind store records for that vmid (best-effort
|
||
from `/var/lib/felhom-agent/guest-binds.json`, generic note otherwise) + "eject BEFORE
|
||
uninstall" guidance; no umounts there (remaining guests may use the drives).
|
||
- **Operator-key day-0 fold (the GL-1/G1 key-pin follow-up):** `OPERATOR_KEY_{OPERATIONAL,RECOVERY}_{ID,LINE}`
|
||
constants (EMPTY until the operator's offline pin ceremony) + `--operator-pubkey-file PATH`
|
||
(lines `operational|recovery <authorized_keys line>`; the comment field IS the key_id, required;
|
||
malformed/empty file dies at argv time naming the line; file OVERRIDES the constants with a
|
||
notice). Resolved keys are written to `authz.signers` in the agent config (exact
|
||
`config.go` SignerKey schema; `nonce_store_path` left to the agent default). **Preserve rule:**
|
||
script/file keys are authoritative ONLY when non-empty — a reinstall with no keys KEEPS a
|
||
preserved config's signers (never silently clobber a manually-pinned box; replacing preserved
|
||
signers logs a pin-rotation notice). byo disclosure names the operational key_id (or "NONE —
|
||
dormant"); verify reports `authz signers: N` (armed) or a dormant WARN (dormant = safe default).
|
||
- Harness: +13 static cases (GL4-C1..C5 incl. 5 live key-file refusals + grep shapes, GL4-D
|
||
disclosure↔uninstall parity, GL4-INV no-forced-unmount/no-format invariant) + PVE-tier **GL4 H-U**
|
||
(full uninstall DRY transcript against the live guest, state-override protected). 28/28 PASS on
|
||
felhom-pve; red-proofs RP-1 (4b4 dropped → parity FAILs), RP-2 (preserve guard dropped → C4
|
||
FAILs), RP-3 (role-die dropped → C3a FAILs) all run→fail→revert on scratch copies.
|
||
- NOT live-validated (GL-6): the real (non-dry) uninstall with mounted/busy drives, the statement
|
||
on a real teardown, and an armed key-pin install end-to-end.
|
||
|
||
## felhom-host-install v1.10.0 — --mode appliance|byo install profile (TASK GL-2, go-live G2/G4/G5) (2026-07-07)
|
||
|
||
- **`--mode appliance|byo` is now REQUIRED** for a fresh install / `--resume` (no default — the
|
||
build-golden v2.0.0 precedent). `appliance` = the historical behavior, dry-transcript-preserved.
|
||
`byo` = a Proxmox host the operator does NOT own (the pilot): **`step_break_glass` is gated OFF at
|
||
its call site** (root@pam never reset, nothing vaulted — the one place the skip is auditable),
|
||
`--cores` + `--memory` both mandatory (the only noisy-neighbor protection there),
|
||
`--enable-oob`/`--rotate-recovery` refused at argv time, PVE-major gate (die on non-9.x unless
|
||
`--force`), every `--acl-storages` entry must exist on the box (die naming ALL missing; appliance
|
||
warns), informational host-DNS `:53` line, full host-mutation disclosure + typed-hostname
|
||
acknowledgement (`/dev/tty`, dry-run-aware), and post-write config asserts that refuse
|
||
`lan_resolver.enable`/`wg_tunnel.enabled`/`oob.enabled` = true (byo also flips the config write's
|
||
lan_resolver DEFAULT to off; appliance keeps default-on).
|
||
- The old `--mode provision` value is retired (specific die message); the `dr` stub seam is kept.
|
||
The mode is persisted in state.json; a `--resume` under the other mode refuses (C4).
|
||
- **Verify asserts (BOTH modes — campaign-2 R2 lesson):** the verify step now asserts pool
|
||
membership of the provisioned vmid + every scoped ACL grant (user AND token: Base@/,
|
||
Guest@/pool/felhom, Store@each storage); any miss → error + "WITH WARNINGS" verdict.
|
||
- **`--preflight-only`**: runs the full mode-aware step-1 checks, prints an explicit
|
||
`PRE-FLIGHT PASS/FAIL` verdict, exits 0/1, writes NO state marks (never lets a later `--resume`
|
||
skip the real preflight). Existing-guest count line added to preflight (both modes).
|
||
- `FELHOM_INSTALL_STATE_DIR` env override (test-harness only) so harness cases can never touch a
|
||
live `state.json`; `usage()` switched to a marker-based range (the numeric one had drifted).
|
||
- **NEW `scripts/hostinstall-mode-harness.sh`**: static tier (bash -n, shellcheck, C1–C4 refusal
|
||
matrix asserting exact die messages, grep-invariants: one gated break-glass call site, chpasswd
|
||
unreachable outside it, --mode in usage) + PVE tier (C5 live-refusal + Scenario A/B dry
|
||
transcripts via `FELHOM_TEST_CUSTOMER`/`FELHOM_TEST_PASSFILE`). Red-proofs RP-1..RP-3 executed
|
||
(gate removed / caps requirement dropped / mode-mismatch check dropped → matching case FAILs).
|
||
- Live-validated read-only on felhom-pve: full static tier + C5 + byo dry transcript (no
|
||
4b/8/chpasswd/recovery-credential; disclosure+ack+DNS+storage lines present) + `--preflight-only`
|
||
PASS and FAIL verdicts with zero state entries. Full A/B rc=0 transcripts + C7 await the
|
||
supervised GL-6 drill (blocked today by demo-felhom's EMPTY git.username/git.token — any step-5
|
||
re-run for that customer dies until the operator sets real Gitea read creds).
|
||
|
||
## felhom-host-install — --enable-oob installs the dedicated felhom-sshd OOB instance + belt (TASK H1) (2026-07-05)
|
||
|
||
- `--enable-oob` + `install_oob` (in step 5): lays down the STATIC OOB parts — `/etc/felhom-sshd` tree
|
||
+ a dedicated ed25519 host key + the `felhom-op` login user + its scoped `/etc/sudoers.d/felhom-op`
|
||
(visudo-validated) + the `felhom-sshd.service` unit (**RuntimeDirectory-guarded** — refuses the G1
|
||
incident cause) enabled-not-started (the agent renders the config) + the static `inet felhom_oob`
|
||
nft table (`nft -c`-checked) + its boot loader `felhom-oob-nft.service`. The agent reconciles the
|
||
DYNAMIC parts (port claim, config, operator authorized_keys, belt set elements) once `oob.enabled`.
|
||
- Uninstall (step 4b3): stops/disables felhom-sshd + the belt loader, removes both units + the config
|
||
tree + belt table + `felhom-op` user/sudoers; NEVER touches the stock sshd/:22/`/etc/ssh`.
|
||
|
||
## felhom-host-install — break-glass credential + management-plane watchdog install (TASK G1) (2026-07-05)
|
||
|
||
- `step_break_glass` (new step 4b): generates a strong root@pam console password (`openssl rand`,
|
||
NEVER logged/filed — stdin→chpasswd + stdin→curl), sets it, and vaults it to the hub over the
|
||
enroll-authenticated host-key channel (`PUT /hosts/{id}/recovery-credential`). Idempotent (state
|
||
marker) unless `--rotate-recovery`. The human break-glass path for the PVE web console.
|
||
- `install_mgmt_watchdog` (in step 5): installs the G1 host artifacts from the agent repo configs —
|
||
`felhom-privsep.tmpfiles` (boot-persistent `/run/sshd`), `felhom-mgmt-watchdog.{sh,service,timer}`
|
||
(agent-independent ~60s privsep-dir auto-heal). **HARD GUARD:** refuses any fetched unit declaring
|
||
`RuntimeDirectory=` (the SPIKE-felhom-sshd §8 incident cause). Non-fatal on a repo that predates the
|
||
artifacts. Uninstall (step 4b2) stops+disables the timer and removes all four artifacts + the marker.
|
||
|
||
## docs — 06-doc S3 row SHIPPED + agent-side revocation semantics (2026-07-04)
|
||
|
||
Docs-only companion to **felhom-agent v0.64.0** (the S3 slice — keygen, registration,
|
||
agent-managed `wg-quick@wg-felhom`, escrow join; live-validated on felhom-pve incl. revocation
|
||
drill, reboot persistence, 30-min soak). 06-doc §3.5 now records: register-once marker,
|
||
revoked-stays-revoked, re-add via the registration endpoint (the raw registry add doesn't bump
|
||
the host generation — live finding), `wg_tunnel.enabled` default-FALSE rollout gate. S6 backlog
|
||
notes added (hub poll constant configurable + first-adoption log; registry-add bump-or-label).
|
||
CGNAT/mobile-hotspot appendix deferred (operator-assisted; §7's open validation stands).
|
||
|
||
## felhom-peersync.sh v1.0.1 — strip out of process substitution (exit-swallow fix) (2026-07-04)
|
||
|
||
The S1 REPORT's exit-swallow class, fixed: `wg syncconf wg0 <(wg-quick strip "$tmp")` hid the
|
||
strip exit code — a corrupt `wg0.conf.head` could feed syncconf empty/partial input that WIPES
|
||
the live peer set while the script exits 0 (and the bad conf is then persisted). v1.0.1 runs
|
||
strip as its own step into `$tmpdir/stripped`; a strip failure aborts BEFORE `wg` is invoked.
|
||
Sandbox red-proof (stub `wg-quick` exiting 1 after partial output + recording stub `wg`):
|
||
pre-fix shape invoked wg and returned rc=0; fixed shape errors first, wg never called.
|
||
Redeployed to the dev endpoint (runbook step 5 install); shellcheck clean.
|
||
|
||
## felhom-peersync.sh v1.0.0 — the offsite endpoint's WG reconcile script (2026-07-04)
|
||
|
||
S1 (doc 06 §5): the forced-command target the hub's wgsync pushes to (runbook
|
||
`offsite-endpoint.md` step 5 installs it as `/usr/local/bin/felhom-peersync`, root:root 0755,
|
||
invoked via a one-line sudoers grant from the `felhom-peersync` user's `restrict,command=`
|
||
authorized_keys entry). Validate-FIRST design: jq contract check (version 1, interface wg0,
|
||
44-b64 pubkeys, `10.77.0.x/32` allowed_ips, never the endpoint's own .1) rejects on stderr with
|
||
exit 1 before touching anything; then head-file + generated `[Peer]` blocks into a same-fs tmp,
|
||
`wg syncconf <(wg-quick strip …)` from the TMP (exact-match: adds/removes without bouncing the
|
||
interface), and only on success the atomic `mv` to `/etc/wireguard/wg0.conf` — runtime and boot
|
||
config can never diverge in the failure direction. Zero-peer payload = valid wipe. Never reads
|
||
or prints the private key; no `wg-quick save`; no second mode. shellcheck-clean. Live-proven on
|
||
felhom-hetzner incl. the negatives (malformed JSON / bad pubkey / own-IP peer → exit 1, wg state
|
||
byte-identical) and reboot persistence.
|
||
|
||
## docs — architecture Part 06: offsite connectivity design-of-record (2026-07-03)
|
||
|
||
`documentation/architecture/06-offsite-connectivity.md` — the settled offsite-backup-transport
|
||
design, authored from the spike verdict + operator-resolved forks (recorded, not re-litigated):
|
||
plain WG (D1), host-side **agent-managed** `wg-felhom` as the agent-managed-unit pilot on the
|
||
sudoers `*.mount` install pattern (D2), one shared hub-driven endpoint VM running WG + the
|
||
offsite PBS with no agent (D3, CF-token pattern), hub source-of-truth with a `wireguard` block
|
||
riding the existing `WireDesiredState`/DesiredGeneration channel (D4), one datastore +
|
||
per-customer namespaces (D5), and PBS **on** the VM — relay-through-DooPlex rejected as
|
||
non-scaling (D6). Includes the Day-0 join handshake, the robustness set (NOT-DynDNS roaming,
|
||
endpoint DNS re-resolve watchdog, MTU 1420, per-/32 topological isolation, tunnel-health through
|
||
the storage-target reachability model), trust conformance, the honest open ledger (CGNAT
|
||
unmeasured → mobile-hotspot smoke test; peer-sync push-vs-pull = slice-1 design point), and the
|
||
S1–S6 slice roadmap (MVP = S1→S2→S3, then S4; S5 merges with DR-completeness). All claims cited
|
||
at file:line against felhom.eu @ bf099f6 + felhom-agent @ 4ba1b14. `day0-install.md` backlog line
|
||
now points at spike + design doc. Docs-only.
|
||
|
||
## docs — SPIKE: offsite-backup connectivity — plain WireGuard WINS the ladder; Headscale = separable fleet layer (2026-07-03)
|
||
|
||
`documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` — the offsite-backup transport
|
||
decision, empirically grounded on both real ends (demo-felhom PVE host ⟷ throwaway Hetzner box,
|
||
end state: powered off, secrets shredded). Headline results: the operator's line is **plain-NAT
|
||
with a fixed public IP, not CGNAT, and has zero IPv6** (P0 honesty — CGNAT confirmation deferred
|
||
to Peti's VM 110); plain outbound WG held an 11.4-min idle window and carried a **real 2 GiB
|
||
worst-case PBS backup at 4.26 MiB/s = the full home uplink** (~5% tunnel overhead), TLS pin
|
||
intact through the tunnel (positive + negative proof); UDP 51820 **and** 443 both pass; kernel WG
|
||
surprisingly *works* inside the unprivileged guest (P7 — host placement stands on architecture,
|
||
not infeasibility). Recommendation: host-side agent-managed WG (cloudflared pattern), key in the
|
||
escrowed IdentityBundle, small public endpoint VM (PBS-on-VM vs rendezvous-relay deferred to the
|
||
spec). `runbooks/day0-install.md` backlog line resolved to point here; `CONTEXT.md` notes the
|
||
DR-completeness task is unblocked (next: the production connectivity spec).
|
||
|
||
## skills — NEW: felhom-app-catalog (4th skill) + SparkyFitness as its worked example (2026-07-03)
|
||
|
||
`skills/felhom-app-catalog/SKILL.md` — the catalog **authoring workflow** (research → inspect the
|
||
image for the healthcheck family → write compose/.felhom.yml → deploy live through the dashboard →
|
||
verify healthy → reconcile the app count). Deliberately points at app-catalog `REUSE.md` §1–2 +
|
||
`README.md` §format for every field table (one-fact-one-place; no duplication). Unique content:
|
||
the never-guess-the-healthcheck rule with the per-tool image-inspection loop (BusyBox `ash`
|
||
`command -v` gotcha: it silently ignores all but its first argument — verified), the
|
||
probe-container naming rule (controller probes the container named exactly like the stack —
|
||
verified in `felhom-controller/internal/stacks/healthprobe.go`, row added to app-catalog REUSE.md),
|
||
the Hungarian-quote YAML kill, and the deploy-is-the-test doctrine. No installer change needed —
|
||
`install_skills.py` auto-discovers `skills/*/SKILL.md`; fresh-session discovery probe listed all 4.
|
||
Proven by finalizing `sparkyfitness` end-to-end on demo (see app-catalog-felhom.eu CHANGELOG).
|
||
|
||
## docs — golden 0.98.3 live: D.1b RETIRED, drill ledger B1/B5 → FIXED, backlog note resolved (2026-07-03)
|
||
|
||
Companion to felhom-agent's `build-golden.sh` v2.0.0 (@ `ceca355`): the golden now bakes the CURRENT
|
||
controller (0.98.3, mandatory-tag convention — B5) and a `felhom-controller-bootstrap.path` unit
|
||
(controller deploys on the bootstrap-mount hot-plug, no reboot — B1). Clean-room validated end-to-end
|
||
BEFORE publish (bake → isolated hot-plug proof → local-golden Day-0 install → publish → vouch →
|
||
`--force-gitea-golden` install); evidence: `documentation/audits/DRILL-golden-098-2026-07-03.md`.
|
||
|
||
- `documentation/runbooks/day0-install.md`: **D.1b reduced to a one-line version check** (fresh boxes
|
||
land current + self-manage); old manual-update procedure moved to Part F troubleshooting keyed on
|
||
"golden older than 0.86.0 was vouched"; header validated-versions line → script v1.9.1 / agent
|
||
v0.63.0 / golden v0.98.3; A.3 records the drilled-known-good pair + "vouch ≥ 0.98.3"; A.4 floor
|
||
text rewritten (fresh boxes now floor-honoring; recommend raising the global floor after a golden
|
||
rebuild — operator, 1 min).
|
||
- `documentation/audits/DRILL-day0-cleanroom-2026-07-03.md` ledger: **B1, B5 → FIXED** (pointers);
|
||
R6 note: the installer's post-provision reboot is retained as a belt, removal recorded as a
|
||
candidate cleanup (not done).
|
||
- `documentation/backlog/FOLLOWUP-golden-default-controller-tag.md` + `backlog/README.md`:
|
||
**RESOLVED** per the M18/M19 convention (file kept + annotated; README entry marked FIXED).
|
||
- New evidence doc: `documentation/audits/DRILL-golden-098-2026-07-03.md` (A–D transcripts, unit
|
||
states, resolution-order + fetch/sha proofs, cleanup, observations — incl. the SECURITY
|
||
observation that the customer `git.token` has package-WRITE rights → scope-down + rotate
|
||
follow-up).
|
||
|
||
## docs — day0-install D.1b narrowed + drill ledger B2/B3 → FIXED agent v0.63.0 (2026-07-03)
|
||
|
||
Agent v0.63.0 fixed both drill findings (B3 token reload-on-miss + B2 guesthook snippets-dir
|
||
mkdir; red-proofed, deployed on felhom-pve, Gitea-published sha256 b4a89c81…). Guide follow-through:
|
||
the D.1b "restart the agent first" step is now CONDITIONAL (only for an installed agent < v0.63.0 —
|
||
the Day-0 manifest still vouches 0.62.0, so today's fresh installs still hit it); the 401
|
||
troubleshooting row records the fix version; the drill ledger + go/no-go item 8 marked FIXED.
|
||
Operator follow-up unchanged: vouch agent 0.63.0 in the Day-0 manifest UI, then the step is dead.
|
||
|
||
## felhom-host-install.sh v1.9.1 — clean-room drill fixes: residue-free uninstall + post-provision reboot (2026-07-03)
|
||
|
||
Companion to the Day-0 go-live package (`documentation/runbooks/day0-install.md` +
|
||
`documentation/audits/DRILL-day0-cleanroom-2026-07-03.md`). Every fix was found by the clean-room
|
||
drill (virgin nested PVE 9.2.2) and re-verified there (v1.9.1 uninstall → **zero-Felhom-residue
|
||
diff vs the pre-install baseline**; v1.9.1 install → controller up with no manual intervention).
|
||
|
||
- **Header/version sync** (the header said v1.8.0 while `SCRIPT_VERSION` said 1.9.0); keep-in-sync
|
||
note on `SCRIPT_VERSION`; usage sed range follows the header (2,95).
|
||
- **Uninstall now removes the drill-found residue (R1–R5):** the agent **config**
|
||
(resolved from the unit's `-config` BEFORE the unit is removed — it holds the per-host hub
|
||
api_key), the `felhom-shared-parent` unit + wants links + `/usr/local/sbin/felhom-shared-parent.sh`
|
||
+ the `/mnt/felhom-drives` self-bind/dir, `/usr/local/sbin/felhom-mkfs-guarded`,
|
||
`/var/lib/vz/snippets/felhom-guest-hook.sh`, and `/etc/dnsmasq.d/felhom-*.conf`
|
||
(+ dnsmasq restart when touched). All tolerate-absent; summary lines updated (`sudo` AND
|
||
`dnsmasq` packages are the documented package remnants).
|
||
- **Post-provision guest reboot (R6):** the golden's `felhom-controller-bootstrap.service`
|
||
evaluates `ConditionPathExists=/etc/felhom-bootstrap/bootstrap.json` at BOOT, but the agent
|
||
back-half hot-plugs the mount into the running guest — on slower hardware the first boot loses
|
||
that race deterministically and the controller never deploys. `step_provision` now reboots the
|
||
guest once (the agent's own output says "next: reboot the guest"); `step_verify` waits bounded
|
||
(180 s) for the controller container instead of a momentary look.
|
||
|
||
## felhom-host-install.sh v1.9.0 — Pool.Audit for the stale-lock reaper (A1) (2026-07-03)
|
||
|
||
Companion to felhom-agent v0.62.0 (audit A1: pool-membership ownership check). `PVE_PRIVS_GUEST`
|
||
gains **`Pool.Audit`** (12 → 13 privs, granted at `/pool/felhom` via the existing FelhomAgentGuest
|
||
role) so the agent can read `GET /pools/felhom` — its stale-lock reaper's ownership registry.
|
||
`Pool.Allocate` does NOT satisfy the read (spike SPIKE-a1-pool-membership-read-2026-07-03 T2).
|
||
No structural change: `_ensure_role` already `role modify`s to the exact priv set, so re-running
|
||
`--rescope-acl` (or a fresh install) upgrades an existing box idempotently; `remove_scoped_acl`
|
||
deletes by role name and needs nothing. **Deploy order on a live box: rescope FIRST, then deploy
|
||
agent v0.62.0** — the added read priv is harmless to an older agent, while the new agent on an old
|
||
ACL fail-safes its reaper (skips) and reports `pve:pool-read` degraded until the rescope lands.
|
||
|
||
## docs — SPIKE: A1 pool-membership read for the stale-lock reaper (2026-07-03)
|
||
|
||
Findings doc `documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md`. Live-probed on
|
||
felhom-pve under the PRODUCTION scoped token vs root: LXC enumeration IS already pool-filtered
|
||
(token sees only 9201 of 4 guests); `GET /pools/felhom` 403s naming `Pool.Audit`; a throwaway
|
||
token with ONLY `Pool.Audit`@`/pool/felhom` reads members (minimal delta proven, fully torn down);
|
||
`/cluster/resources` withholds the `pool` field without `Pool.Audit`; local ownership records are
|
||
all partial. Recommendation for the A1 impl spec: add `Pool.Audit` to `PVE_PRIVS_GUEST` in
|
||
`felhom-host-install.sh` (L183) + a `GET /pools/felhom` cross-check in the agent's
|
||
`staleLockController.Guests()`, fail-safe skip on read failure. No script/agent change in this
|
||
commit — docs only. Appendix: committed-secrets (felhom.secret.yaml) rotation micro-runbook,
|
||
operator follow-up.
|
||
|
||
## install_skills.py — new: Claude Code skills installer (2026-07-03)
|
||
|
||
Installs `skills/*/SKILL.md` (felhom-build-deploy, felhom-ui-design, felhom-testing) into
|
||
`~/.claude/skills/` as Windows junctions (`mklink /J`) so repo edits are live immediately; falls
|
||
back to a full copy if junction creation fails or isn't followed (copy mode prints a re-run
|
||
reminder). Idempotent — re-runs detect a correct junction and leave it. Verified: junctions ARE
|
||
followed by Claude Code skill discovery (fresh-session probe found all three).
|
||
|
||
## reuse_refs_check.py — new gate: REUSE.md citation checker (2026-07-03)
|
||
|
||
Staleness defense for the new per-repo `REUSE.md` reuse maps. Takes repo roots as argv, extracts
|
||
every cited `*.go/*.py/*.html/*.css/*.yml/*.yaml/*.sh` path (slash-containing tokens only — bare
|
||
filenames are conventions, not citations), verifies each exists; prints offenders, non-zero exit on
|
||
any missing path. Symbols are spot-verified by the reviewer, not this script.
|
||
Usage: `python scripts/reuse_refs_check.py <repo-root> [...]`.
|
||
|
||
## felhom-host-install.sh v1.8.0 — install the guarded-mkfs wrapper (Impl-1 Part B) (2026-07-01)
|
||
|
||
Companion to felhom-agent v0.54.0 (format-safety foundation). During agent install, fetch + install the
|
||
guarded-mkfs wrapper so the agent's format path is safe on any box.
|
||
|
||
- **New step in `step_agent_install`:** fetch `configs/felhom-mkfs-guarded.sh` from Gitea, `bash -n`
|
||
validate, `install -m0755 -o root -g root` → `/usr/local/sbin/felhom-mkfs-guarded`. Installed BEFORE
|
||
the sudoers (which now allowlists ONLY the wrapper, not raw `mkfs.*`), so the ordering is gap-free.
|
||
- The agent v0.54.0 sudoers (fetched by the same step) drops the raw `mkfs.ext4 -F /dev/* / mkfs.xfs -f
|
||
/dev/*` allowlist and permits only `felhom-mkfs-guarded /dev/* *`, plus read-only `pvs`/`zpool` for
|
||
the agent's unclaimed-disk guard. No other host-install change.
|
||
- `bash -n` + `shellcheck` clean (0 new warnings). Live-validated on felhom-pve (agent v0.54.0 deploy):
|
||
wrapper refuses the OS disk + an LVM-PV partition, raw mkfs is sudo-denied, an unclaimed throwaway
|
||
disk formats; the agent guard's sudo reads (pvs/lsblk/zpool) all work as the felhom-agent user.
|
||
|
||
## felhom-host-install.sh v1.7.0 — 3b-fix: `Datastore.Audit` box-wide (restore drive visibility) (2026-07-01)
|
||
|
||
Fixes a regression the v1.6.0 pool-scoped ACL introduced: `Datastore.Audit` was placed in the
|
||
per-storage `Store` role (granted only on `local`/`local-lvm`/`felhom-pbs`), which **excluded the
|
||
enrolled removable drives** `felhom-usb`/`felhom-flash`. The agent enumerates storage via
|
||
`ListStorage`/`NodeStorage` (both gated by `Datastore.Audit` — `internal/storage/observe.go`), so it
|
||
could no longer SEE the drives → false "Meghajtó leválasztva" (drive detached) alerts + drives absent
|
||
from the agent-view. (The v1.6.0 swap's "felhom-usb → 403" was mis-read as blast-radius success;
|
||
felhom-usb is Felhom's OWN customer drive, not an out-of-scope object.)
|
||
|
||
- **`Datastore.Audit` moved from Store → Base** (`PVE_PRIVS_BASE` now `"Sys.Audit SDN.Use
|
||
Datastore.Audit"`; `PVE_PRIVS_STORE` now `"Datastore.Allocate Datastore.AllocateSpace"`). Audit is
|
||
read-only metadata, so box-wide Audit restores visibility of ALL storages (incl. dynamically-enrolled
|
||
drives — no per-drive grant ever needed) while the **write** privs (`Allocate`/`AllocateSpace`) stay
|
||
per-storage → write/allocate blast-radius containment is UNCHANGED. Confirmed at source: the agent
|
||
creates no PVE storage (no `POST /storage`/`pvesm add`); drives are dir-storages it observes + mounts
|
||
via host ops, so they need only Audit, never Allocate.
|
||
- **`apply_scoped_acl` reordered** Base-before-Store (role + grant) so a RE-APPLY on a live box adds
|
||
`Audit@/` before Store drops its per-storage Audit → gap-free (the agent never loses enumeration).
|
||
- `remove_scoped_acl` / `--uninstall` / `--rescope-acl` operate by role NAME and inherit the corrected
|
||
privs automatically (no other change).
|
||
- **Live-repaired felhom-pve** (two `pveum role modify`, Base first — no agent stop/restart): drives
|
||
reappeared (agent-view 3→5 storages), detach alerts cleared. Re-tested under the scoped token: drives
|
||
readable (was 403), write-containment intact (vzdump→felhom-usb still 403; out-of-pool guest 403),
|
||
PBS Store grant unchanged. `bash -n` + `shellcheck` clean (0 new warnings).
|
||
- **NOT physically run** (source-confirmed, no `Datastore.Allocate` in the path): a brand-new-drive UI
|
||
enrollment (needs a spare USB) — the host-ops/Audit path is unchanged from pre-3b.
|
||
|
||
## felhom-host-install.sh v1.6.0 — pool-scoped token ACL (3-role) + `--rescope-acl` retrofit (2026-07-01)
|
||
|
||
Colleague-safety batch #4 phase b (script half; agent half = v0.53.0). Moves the agent token's dangerous
|
||
privileges off `/` (which spanned every guest + storage) to `/pool/felhom` + `/storage/<targets>`, so on
|
||
a shared box the token can only touch Felhom's own guests + storages. Validated by
|
||
`documentation/audits/SPIKE-pool-scoped-acl-2026-07-01.md` (PASS) — implemented here.
|
||
|
||
- **3-role scoped ACL (`step_token` rewrite).** Replaces the single `FelhomAgent` role granted at `/`
|
||
with three roles, each granted to BOTH the user AND the token (privsep intersection): `FelhomAgentGuest`
|
||
(`VM.*` + `Pool.Allocate`) @ `/pool/felhom`; `FelhomAgentStore` (`Datastore.*`) @ each of
|
||
`PVE_STORAGES` (default `local local-lvm felhom-pbs` — the offsite PBS MUST be included, SPIKE
|
||
residual #1; `--acl-storages` overrides); `FelhomAgentBase` (`Sys.Audit SDN.Use`) @ `/`. Helpers
|
||
`apply_scoped_acl`/`remove_scoped_acl`/`_grant`/`_ensure_role`.
|
||
- **Pool before token.** `ensure_felhom_pool` runs at the top of `step_token` (always, incl.
|
||
`--skip-provision`) so `/pool/felhom` exists before it's granted on.
|
||
- **Re-install safety.** `step_token` also removes the pre-3b broad `/` grant + `FelhomAgent` role if
|
||
present (`remove_old_broad_acl`, tolerate-absent), so a re-install can't leave the old grant unioned
|
||
with the scoped one. The post-provision `pool_add_guest` is gone (the agent's `restore --pool` makes
|
||
the guest a member atomically — v0.53.0).
|
||
- **`--rescope-acl` retrofit** (new mode, mirrors `--adopt-pool`): migrate an existing install — ensure
|
||
the pool + guest membership, apply the scoped grants, THEN remove the old broad grant (add-before-
|
||
remove: the token is never grant-less mid-migration). Prints the "now deploy agent ≥ v0.53.0"
|
||
ordering reminder. Idempotent + dry-run-aware. **SUPERVISED** (run with the agent stopped — the scoped
|
||
ACL and the pool-param agent are mutually dependent; §13 of the task).
|
||
- **`--uninstall`** now removes the scoped grants + 3 roles AND the pre-3b broad grant/role (both
|
||
tolerate-absent → works on either shape), keeping the pool delete-if-empty (v1.5.0).
|
||
- **Validated on felhom-pve** (dry-run): T-A fresh install (pool-before-token, 3 roles once, scoped
|
||
grants incl. `/storage/felhom-pbs`), `--rescope-acl` (add scoped → remove old `FelhomAgent`), T-F
|
||
uninstall (old-shape cleanup + pool not-empty skip). `bash -n` + `shellcheck` clean (0 new warnings).
|
||
**The live rescope + agent swap is the supervised STOP** — not run here.
|
||
|
||
## felhom-host-install.sh v1.5.0 — `felhom` pool by default + `--adopt-pool` retrofit + uninstall teardown (2026-07-01)
|
||
|
||
Colleague-safety batch #4 phase a. Every Felhom-managed guest now joins a dedicated **`felhom` pool**
|
||
for fleet uniformity (and as the environment the later pool-scoped ACL — 3b — will spike against). All
|
||
pool ops run as `root@pam` from the installer, so there is **NO agent/token/ACL change** and zero
|
||
permission-model risk (`PVE_PRIVS` untouched; the `FelhomAgent` token stays scoped at `/`).
|
||
|
||
- **New `felhom` pool default.** `step_provision` calls `ensure_felhom_pool` (create if absent,
|
||
idempotent) and, after a successful provision, adds the guest via `pveum pool modify felhom -vms
|
||
<vmid>` (skip-if-already-member). New helpers `pool_exists` / `pool_members` / `ensure_felhom_pool` /
|
||
`pool_add_guest`; const `PVE_POOL="felhom"`. PVE 9 syntax + `/pools` JSON shape confirmed live before
|
||
wiring (`pveum pool add|delete|modify`; `pvesh get /pools` → `[{poolid,comment}]`, `/pools/<id>` →
|
||
`{members:[{vmid,…}]}`).
|
||
- **`--adopt-pool` retrofit mode.** Non-destructive: adds an EXISTING Felhom guest to the pool (creating
|
||
it if needed), resolving the guest from `--vmid` else the recorded `provisioned_vmid`. Reuses the
|
||
ours-check (`/etc/felhom-bootstrap` mount) — refuses a non-Felhom guest unless `--force`. Touches ONLY
|
||
pool membership: never reconfigures/restarts the guest, never contacts the hub. Idempotent
|
||
(skip-if-member).
|
||
- **`--uninstall` pool teardown (step 5b).** After the pveum removal, deletes the `felhom` pool **only
|
||
if empty** (a destroyed guest is auto-removed from its pool); a pool that still has members is left
|
||
with a `log_skip` naming them. Not reached on the Spec-1 safe-skip path (other Felhom guests remain).
|
||
- **Validated on felhom-pve** (dry-run + SAFE live): T-A fresh-install dry-run shows the pool create +
|
||
membership lines; T-B **live adopt of guest 9201** → `pvesh get /pools/felhom` lists 9201, guest still
|
||
running, config unchanged (the demo node is now pool-uniform); re-run = no-op; T-B' non-Felhom vmid →
|
||
refusal; T-C uninstall dry-run → "pool felhom not empty (members: 9201) — leaving it". `bash -n` +
|
||
`shellcheck` clean (0 new warnings; the 2 pre-existing SC2015 in `step_verify` unchanged).
|
||
- **NOT changed:** `PVE_PRIVS`, the ACL grants, the agent, the provision-call args. 3b (pool-scoped ACL
|
||
+ agent restore-into-pool under a scoped token) is the separate spike-gated task.
|
||
|
||
## felhom-host-install.sh v1.4.0 — appliance CPU/RAM cap passthrough (`--cores` / `--memory`) (2026-07-01)
|
||
|
||
Colleague-safety batch #3 (host-install half; the mechanism is agent v0.52.0). Lets an operator cap the
|
||
provisioned guest so a trial appliance on a SHARED production Proxmox doesn't pressure the colleague's
|
||
existing guests.
|
||
|
||
- **`--cores N` / `--memory M` (MiB)** — optional; passed through to the agent's `--selftest=provision`
|
||
as `-cores`/`-memory`. `0`/unset = keep the golden's baked sizes (unchanged behaviour). New vars
|
||
`CPU_CORES`/`MEM_MIB`; `usage()` header gains an "Appliance cap (optional)" group.
|
||
- **Conditional passthrough** — `step_provision` builds a `cap_args` array and appends the flags to BOTH
|
||
the dry-run log and the real agent call **only when set**. An agent < v0.52.0 would reject an unknown
|
||
flag, so the flags are never sent unless the operator opts in (see the deploy dependency below).
|
||
- **Pre-flight sanity WARN (soft, provision only)** — if `--cores` > host `nproc` or `--memory` > host
|
||
`MemTotal`, `log_warn` "the cap won't protect other guests"; never `die` (the operator may know better).
|
||
- **Deploy dependency:** a fresh install using `--cores`/`--memory` needs the hub artifact manifest to
|
||
serve **agent ≥ v0.52.0**.
|
||
- **Validated dry-run on felhom-pve:** `--cores 2 --memory 4096 --dry-run` → provision command shows
|
||
`-cores 2 -memory 4096`; without the flags → neither present; `--cores 64 --memory 65536` → both WARN
|
||
lines (host 4 cores / ~15771 MiB). `bash -n` + `shellcheck` clean (0 new warnings; the 2 pre-existing
|
||
SC2015 in `step_verify` unchanged).
|
||
|
||
## felhom-host-install.sh v1.3.0 — `--uninstall` (clean revert) + pre-flight guards (2026-07-01)
|
||
|
||
Colleague-safety batch #1+#2. Adds a first-class, guarded **`--uninstall`** teardown so an operator can
|
||
cleanly back out of a trial install, plus three provision pre-flight guards that stop common footguns.
|
||
Script-only; no agent/hub/controller change.
|
||
|
||
- **`--uninstall` (local host teardown — no hub contact, no passphrase).** Reverses an install in the
|
||
install-order's reverse: **guest → agent(unit/sudoers/binary/state/user) → pveum(ACL,token,user,role)
|
||
→ golden(opt-in) → state file.** Every mutation goes through `run()` so `--dry-run` prints the full
|
||
plan and executes nothing. Safety:
|
||
- **Ours-check:** refuses to destroy a guest that lacks the `/etc/felhom-bootstrap` bind mount (matched
|
||
by the constant guest *path*, not a hardcoded `mpN` slot — on the demo host it's `mp9`), unless
|
||
`--force`.
|
||
- **Typed confirmation:** must type the vmid to confirm PERMANENT destruction (read from `/dev/tty`;
|
||
skipped only under `--dry-run`, where nothing is destroyed).
|
||
- **Other-guests guard:** if any OTHER Felhom guest remains, destroys only the target and **leaves the
|
||
agent + PVE token + state in place** (re-run with `--force` to remove host-level anyway — orphans the
|
||
others).
|
||
- **Never removes the `sudo` package**; never contacts the hub (the host record intentionally persists).
|
||
- Presence-checked + idempotent: an already-absent guest/unit/sudoers/binary/user/ACL/token/role is a
|
||
tolerated skip, not an error. The `pveum role delete` runs only after its ACL grants are gone (PVE
|
||
refuses to delete a referenced role). Confirmed PVE 9 ACL-delete form:
|
||
`pveum acl delete / --users|--tokens <x> --roles FelhomAgent`.
|
||
- Target vmid resolves from `--vmid`, else the recorded `provisioned_vmid` (else dies). A `--vmid` that
|
||
disagrees with the recorded one needs `--force`.
|
||
- **`--remove-golden`:** with `--uninstall`, also delete the golden vzdump from the archive storage
|
||
(`pvesm free`); otherwise it is left in place.
|
||
- **Install state now records `customer_id` + `provisioned_vmid`** (new `_state_put`/`_state_get` helpers,
|
||
dry-run-guarded like `_state_mark`; the `completed[]` shape is untouched) so a later `--uninstall`
|
||
resolves its target automatically and safely.
|
||
- **Pre-flight guards (provision mode):**
|
||
- **Multi-node guard** — on a 2+-node cluster, `die` (naming the nodes) unless `--node` is explicit
|
||
(new `NODE_EXPLICIT`); single-node keeps the current auto-pick. No-op under `--skip-provision`.
|
||
- **Archive-storage-exists guard** — verify `--archive-storage` appears in `pvesm status` (else `die`);
|
||
no-op under `--skip-provision`.
|
||
- **RAM floor (WARN, never fatal)** — warn when `MemAvailable < 2048 MiB`.
|
||
All three run inside `step_preflight` (before any mutation) so they also fire under `--dry-run`.
|
||
- **Validated dry-run-only on felhom-pve** (single-node, live guest 9201): T-A full uninstall plan, T-C
|
||
not-ours refusal (red-proof), archive-missing `die`, RAM line, other-guests detector, state round-trip;
|
||
confirmed 9201 + agent + pveum + state untouched after all dry-runs. `bash -n` + `shellcheck` clean
|
||
(0 new warnings vs. baseline; the 2 pre-existing SC2015 in `step_verify` are unchanged). **NOT yet
|
||
live-validated (awaiting a supervised run):** a real live `--uninstall` (guest destroy + pveum removal)
|
||
and the multi-node guard on an actual cluster.
|
||
|
||
## felhom-host-install.sh v1.2.0 — /dev/tty passphrase read + vmid auto-detect (2026-07-01)
|
||
|
||
Two operator-experience fixes so a colleague can install online (via the hub's new "Option 1: Online
|
||
install" one-liner) and onto a host that already runs a guest at 9201.
|
||
|
||
- **Passphrase prompt reads from `/dev/tty`, not stdin** (`read_passphrase`). `read -rsp … < /dev/tty`
|
||
makes the no-echo prompt work regardless of how stdin is wired — both download-then-run **and**
|
||
`curl … | sudo bash` (where stdin is the pipe). Strictly more correct; the `--passphrase-file` path is
|
||
unchanged. The passphrase is still never on argv / in logs / in the state file.
|
||
- **VMID auto-detect (`--vmid` now optional-smart).** New `VMID_EXPLICIT` flag (set by `--vmid`). The
|
||
pre-flight vmid guard now determines "in use" against the **`pct list` + `qm list`** id-set (LXC and
|
||
VMs share the id space — more complete than the old `pct status`, which only knew LXC):
|
||
- **explicit `--vmid`** → unchanged deterministic behavior: die if the id is in use unless `--force`
|
||
(destructive over-provision).
|
||
- **default 9201, in use, no `--force`** → **auto-pick the next free id** (scan upward from 9201 over
|
||
the used-set) and **ask to confirm** from the terminal (`read … < /dev/tty`, `[y/N]`); proceed on
|
||
yes, `die "no free vmid confirmed"` otherwise. Never a silent auto-pick.
|
||
- **default 9201 + `--force`** → over-provision 9201 (destructive) without prompting, as before.
|
||
- New helpers `used_vmids` / `_vmid_in_use` / `next_free_vmid`. `--vmid` help text + `usage()` updated.
|
||
|
||
## felhom-host-install.sh v1.1.0 — self-install the agent + fetch the golden from Gitea (2026-06-28)
|
||
|
||
The script now **installs the agent itself** (the last big manual Day-0 prerequisite is gone). It
|
||
fetches the agent binary + golden from Gitea generic packages and **verifies each against the
|
||
hub-vouched artifact manifest** before installing/using it. BUNDLE slice; pairs with hub v0.16.0
|
||
(artifact manifest endpoint + operator UI) and felhom-agent v0.43.0 (canonical unit + publish).
|
||
|
||
- **New step `5/8 agent install`** (before agent-config): resolves the manifest
|
||
(`GET /api/v1/artifacts/{id}`, passphrase) + the git fetch token (from the customer's
|
||
`controller.yaml` via config-retrieve — **NO new credential**); fetches
|
||
`/api/packages/admin/generic/felhom-agent/<ver>/felhom-agent`, **verifies sha256 vs the hub
|
||
manifest** (aborts on mismatch — verify-before-use), backs up any existing binary, installs
|
||
`0755 /usr/local/bin/felhom-agent`; ensures the non-root `felhom-agent` system user; installs the
|
||
canonical sudoers (`0440`, `visudo -cf`-validated) + systemd unit; `daemon-reload` + enable. Idempotent:
|
||
same version already installed + service active → skip.
|
||
- **`--skip-provision`:** install + configure + verify the agent (incl. golden fetch+verify) but do NOT
|
||
provision a guest — the agent-only path for re-installing/upgrading the agent on a host that already
|
||
has live guests. Adds an agent-only `step_verify_agent` (binary + non-root service active + a
|
||
`--selftest=hub` collect-report).
|
||
- **New step `7/8 golden`:** local auto-discovery stays the default/fallback; otherwise fetches
|
||
`/api/packages/admin/generic/felhom-golden/<ver>/golden.tar.zst`, **verifies sha256**, and imports it
|
||
into the archive storage's dump dir for the restore. `--force-gitea-golden` forces the Gitea path.
|
||
- **Non-root agent model:** the agent now runs as `felhom-agent` with `privileged.mode: "sudo"` (was the
|
||
dev/CI `direct`+root shortcut). The config is `chown`ed to the service user (0600) so the daemon can
|
||
read it; `systemctl is-active` after restart is the real proof the non-root user can read the config.
|
||
- **Pre-flight relaxed:** a missing agent binary is no longer fatal (step 5 installs it); the local
|
||
golden requirement is deferred to step 7.
|
||
- **Trust model:** checksum **trust root = the hub** (manifest), not Gitea; the fetch credential is the
|
||
existing config-retrieve git token; artifacts are pinned to a version (never `:latest`).
|
||
- **Secrets:** the git token is a never-logged runtime carrier (cleared on EXIT alongside the passphrase
|
||
/ pve-token / hub api_key); the sudoers is `0440` and `visudo -cf`-validated before install.
|
||
- `bash -n` + `shellcheck` clean.
|
||
|
||
## felhom-host-install.sh v1.0.0 — Day-0 host bootstrap (provision mode) (2026-06-26)
|
||
|
||
First release. A single operator-run script that automates Day-0 on a freshly-PVE-installed
|
||
host: Proxmox API token → hub host enrollment (option C, single secret) → agent config →
|
||
guest provision → verify. Composes proven mechanisms (the `pveum` role/token sequence, hub
|
||
`POST /host-enroll`, `felhom-agent --selftest=provision`); grounded by
|
||
`documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md`.
|
||
|
||
- **7 steps, idempotent + resumable** via `/var/lib/felhom-install/state.json`: pre-flight →
|
||
Proxmox token → compute grows → host-enroll → agent config → provision → verify.
|
||
- **Single-secret** (the retrieval passphrase): read no-echo or from a 0600 file, never on
|
||
argv/logs/state. The global operator key never touches the box.
|
||
- **pveum automation:** 16-priv `FelhomAgent` role (create-or-modify), `felhom-agent@pve` user,
|
||
privsep token (reuse-if-working else rotate), and **both** ACL grants applied **after** the
|
||
token exists (token-remove purges the token ACL).
|
||
- **Auto-discovery:** golden archive (newest `vzdump-lxc-<golden-vmid>`), PVE node name, vmbr0
|
||
bridge IP for the local-api, and the served-leaf TLS fingerprint pin.
|
||
- **Safety:** pre-flight fails fast (root, PVE 9.x, local-lvm headroom, hub reachable,
|
||
customer+passphrase valid via read-only `GET /config/{id}`, golden resolvable); refuses to
|
||
clobber an existing `--vmid` without `--force`; `--dry-run` previews every mutation;
|
||
`--preserve-from` keeps operator infra (PBS/local_api/privileged/authz) on re-deploys.
|
||
- **`--mode dr`:** documented 10D stub (restore customer PBS snapshot instead of golden) — not
|
||
implemented.
|
||
- **Live-validated** end-to-end on `felhom-pve`: authorized wipe of demo guest 9201 →
|
||
re-provision from the golden → controller config-pull + public tunnel `HTTP 200` →
|
||
host-report of guest 9201 → idempotent `--resume` no-op. (One ordering bug — token ACL
|
||
applied before rotation — was found and fixed during the live run.)
|