Scenario A: pushed to main without moving the tag; the served bytes stayed
byte-identical (sha 2f859555…) and the marker never appeared, while the website
tree advanced to the new commit — both halves of the split in one observation.
Scenario B: moving the tag published it (~40s, sha ea2b4aa9…, marker present);
moving it back rolled it back to exactly the pre-publish sha with the marker
gone. The website returned 200 throughout.
A comment only. It exists to be LOOKED FOR at the served URL: if it appears
there, /scripts/ is still tracking main and the publish channel is still
floating. It must not appear until installer-v1.23.0 is moved.
Two channels moved off main in the same change, because either one left behind
makes the other cosmetic.
Channel 1 — the served script. webpage.yaml git-synced /scripts/ from
--branch=main every 30s and nginx served that tree, so pushing this file WAS
publishing it: within half a minute it was what every new machine downloaded and
ran as root, with no staging and no rollback but another push. The sync is now
SPLIT: the website keeps tracking main at the same cadence (a copy edit must
never need a release) and /scripts/ tracks the tag installer-v<SCRIPT_VERSION>.
PROVEN before the manifest was touched: git-sync v4.4.0 follows a tag AND
notices a MOVED one — measured on a throwaway sync against this repo,
"update required ... local:<old> remote:<new>" -> "updated successfully",
within one period. The moved-tag half is what the publish model rests on.
Channel 2 — the sixteen files fetched at run time. fetch_raw pulled from
$AGENT_REPO/raw/branch/main; it now pulls raw/tag/v$ART_AGENT_VER. That is a
correctness fix, not only a channel one (R-183): a fresh install fetched the
vouched agent BINARY while taking its unit file, sudoers and guarded wrappers
from whatever main held. Two refs, one install, nothing compared them. Their
correct ref was never SCRIPT_VERSION — they do not live in this repo.
No fallback to a branch: a vouched version whose tag is missing fails loudly
rather than quietly serving main.
Channel 3 — the URL — needed no change, recorded rather than left silent:
https://felhom.eu/scripts/felhom-host-install.sh never carried a ref, so both
producers follow the tag with no edit. No hub change, no hub version bump.
Gate 6 in hostinstall_gates.py pins all three structurally with no network, so
it stays in --fast and runs in CI. It deliberately does NOT assert "a tag exists
for the current SCRIPT_VERSION": that would go red on the very push that bumps
the version, before publishing — and publishing being separate is the ruling.
felhom-agent v0.120.0 merges the two data volumes into one, and step_grows
computed two numbers while the install call passed both — so this had to
change with the agent or every install would have provisioned a half-sized
box. The 80/20 split is summed (226 = 184+42), so a standard appliance keeps
exactly the 250 G it had, no longer split by a wall.
The size still comes from the physical disk: step_grows already read the
thin pool's free space, and the merge only collapsed its two outputs into
one. --sysdata-grow is deprecated but still honoured, because the agent
folds a hand-passed value in rather than dropping it.
The Setup tab said 'host-install 1.19.0' while the served script was 1.22.0, and had
been wrong since 2026-07-14. Deriving the number honestly is not possible: the Option-1
command downloads felhom-host-install.sh from the website at RUN TIME and the website
git-syncs main every 30s (R-110), so no build-time value in the hub can be true. R-94(a)
offered derive-or-delete; deleted, which removes the drift class instead of automating it.
- configs.go: hostInstallVersion const, pageData.ScriptVersion field and its assignment
all removed; a NOTE in their place records why there is no constant here.
- customer_unified.html: the sentence now says the command always fetches the current
installer, and renders no version.
- hostinstall_gates.py gate 1: the third assertion INVERTS — it used to require the hub
const to equal SCRIPT_VERSION, it now asserts the hub carries no host-install version
literal at all, matched in six code shapes across every .go/.html under hub/ (comments
are deliberately not stripped: a // inside a URL literal would blind the scan).
- render_test.go: the assertion 'html contains hostInstallVersion' compared the constant
to itself and passed at ANY value — demonstrated green with the const at 9.9.9 while the
script was 1.22.0. Deleted, not replaced: there is no longer a version to assert.
- felhom-host-install.sh: COMMENT ONLY (SCRIPT_VERSION untouched) — it claimed the gate
keeps the hub copy equal, an invariant that no longer exists.
Red-proofs: restoring the const fails the rewritten gate 1 (3 shapes hit); the old
render_test assertion passes at 9.9.9.
Every box installed before this got local_backup_target "local" -- the vzdump
target on the SAME physical device as the guest, so a drive failure took the
guest and its only local backup together. E-1 fixed two machines by hand; this
fixes the installer.
Case A: an eligible secondary drive is already mounted -> create felhom-backup on
that drive's own mountpoint via the felhom-backup-target-apply wrapper (create +
grant) and point the primary tier at it.
Case B: system drive only -> the target stays on the system drive and this is
RECORDED AS DEGRADED, not as normal. The install still succeeds: a single-drive
appliance is a valid product, it just cannot survive drive loss.
Phase 0 inverts the emphasis: the installer has NO drive-enrollment step, so on a
fresh appliance Case A almost never fires. The common case is Case B with the
drive arriving later through the wizard (Part 3). Case A covers the reinstall
shape where an agent-generated .mount unit already brings the drive up by fs-UUID.
Eligibility suggests and refuses the absurd, never decides by transport: the
reference backup drive is an external USB HDD and BOTH demo boxes report
removable=0, so a transport rule disqualifies the reference drive and a removable
rule finds no candidate at all.
Scenario F: an already-configured box is never corrected -- an early return plus
setdefault, both load-bearing.
Proofs (installer-logic-tested against extracted functions with stubbed
pvesm/wrapper; NOT install-tested, no reinstall was performed):
A -> create + grant, resolved felhom-backup
B -> DEGRADED warnings, resolved local, rc=0 (install not failed)
F -> skipped, 0 wrapper calls
F red-proof (guard removed) -> 2 wrapper calls, i.e. it would have "corrected"
a correct box
A failed restore-test's scratch guest never joins the felhom pool, so the pool-scoped
grant cannot reach it and teardown 403s. Ten path-scoped /vms/<id> grants reach exactly
the scratch band and nothing else. Removal path + verify step extended.
Slice D.1 — host-install 1.20.0: a FRESH box defaults to local-daily +
offsite-weekly (felhom-pbs, 604800s, keep_last=2). setdefault semantics proven
both ways: fresh gets the tier, an UPGRADE preserves the existing backup block
verbatim — so an in-place upgrade can never silently start writing to an
offsite datastore. Existing boxes are migrated explicitly.
Slice E:
- 07-backup-architecture.md: honest status header per CONTEXT ruling S-2, with
an explicit STALE-outside-the-PBS-tier verdict (the controller tiers were last
verified 41 controller versions ago). The PBS row claimed 'PBS on DooPlex'
(the retired spike store) with no cadence; it now names felhom-pbs ->
felhom-offsite on ep0 over wg-felhom, weekly, keep_last=2. NOT marked
ratified — that is Viktor's review of the section 10 list. Discharges R-83.
- 06-offsite-connectivity.md: the target-split remaining-work note collapsed
(shipped), and records HOW S4.1's tier-aware timeout silently regressed — the
mechanism was never removed, its INPUT changed when local_backup_target was
retargeted to 'local'. Also notes S4.1 already diagnosed the teardown 403 as a
phantom (a timeout consequence, not an ACL gap).
- capability map: new row for recurring offsite backups actually LANDING, as
distinct from the existing row proving ACTIVATION. IMPLEMENTED, not
PROVEN-LIVE — the restore round-trip has not completed under the fixed code.
- ROADMAP: R-82 SHIPPED with its remaining gate named, R-83 DISCHARGED, R-84
left open.
- CONTEXT + REPORT: the arc, including the mid-arc correction I had to make.
The NAS verify pipeline (agent v0.81.0) reads mount-unit journals unprivileged
— group membership, NO sudoers grant. Fixes the v1.11.0/1.12.0 header drift.
New authoritative feature doc documentation/controller/network-storage-nas.md
(verify pipeline, §8 truth table, Q4 error taxonomy, retry=0, Route A recipes
incl. the chmod-persists nuance).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
F3: the populated-host leaf guard now fires only when a FELHOM guest exists
(felhom_guests, not `pct list`) — a fresh byo install on a host running only
the OWNER's guests proceeds without --allow-new-leaf; the pin protection for
real Felhom guests is unchanged (guard still fires on a Felhom guest).
F6: byo preflight REFUSES (never mutates) when a foreign resolver is bound
to :53, with remediation — Felhom needs the guest reachable by name and will
not stop/mask the owner's DNS (the dnsmasq analog of the break-glass rule).
Viktor's ruling: refuse+instruct only, no stop/mask path. ss failure degrades
to a warn. Appliance mode untouched (the agent's lan_resolver owns :53).
F1: uninstall now purges the agent config's .bak* siblings (one GL-6 residue
held a live hub api_key), not just agent.json; WIPED statement updated.
Harness +GL8-F3/F6/F1 static + GL8-F1b behavioural + GL8-F6 live (byo
preflight refuses on felhom-pve's leftover dnsmasq — the real F6 scenario).
Red-proofs RP-F3/F6/F1 each fail their invariant. 36/36 on felhom-pve;
shellcheck clean; GL-2 Scenario-A + GL-4 Scenario-D regression green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Live-found in the GL-6 drill: `should_skip X || step_X` fully skips a
completed step on --resume, but token/enroll/grows produce IN-MEMORY
outputs (pve token; hub host_id/api_key; volume grows) that later steps
consume — agent_config writes them into the config, provision passes the
grows as flags. A resume that had completed token/enroll but not
agent_config wrote a config missing hub.host_id/proxmox.token (daemon
crash-loop, "hub.host_id is required"); a resume past grows passed
`-rootfs-grow ""` (flag parse error). step_token even had an internal
resume-guard the `|| step_token` dispatch defeated.
Fix: token/enroll/grows now run every pass (all idempotent — token
reuses-or-rotates from the on-disk config, enroll is mint-once-reuse,
grows is a pure recompute); the guard uses _state_has (no misleading
SKIP log). golden's GOLDEN_VOLID is re-derived from the local archive in
the resume block so provision never gets an empty -archive. Harness
+GL6-F4 invariant; 27/27; shellcheck clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Operator ruling at the GL-6 drill's Gate 0: the Felhom artifacts are
world-readable by design; an EMPTY git.username/git.token in the customer
config now WARNS and fetches anonymously instead of dying at step 5/8
(the hub-vouched sha256 stays the integrity root; a configured credential
is still used when present; curl auth args are conditional because -u with
an empty token 401s even on public content). All 12 installer fetch
targets validated 200 anonymously before shipping. Harness: +GL6-ANON
shape case; GL4-C2 assertion updated for the v1.11.1 pinned constants
(the benign overrides-notice is not a die). Drill record carries the
deviation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Uninstall: NEW 4b4 removes the self-update artifacts (guarded wrapper,
.prev/.new.* A/B slots, rollback unit, limits drop-in — derived from
configs/felhom-selfupdate-guarded); enrolled drives under /mnt/felhom-drives
unmounted plain-umount-only (busy = warn+guidance, never -l/-f; root bind
guarded); both modes end with a KEPT-vs-WIPED statement (drives/PBS/hub
record/escrow/vaulted recovery credential live on); guest-only mode prints
the vmid's bind-store drives.
Key-pin: OPERATOR_KEY_* constants (empty until the operator ceremony) +
--operator-pubkey-file (argv-validated; comment=key_id required; file
overrides constants) -> authz.signers per the agent SignerKey schema;
no-keys-resolved KEEPS preserved signers; verify reports armed/dormant WARN.
Harness: +13 static cases + GL4-D parity + GL4-INV + PVE-tier GL4 H-U
(full-uninstall dry transcript vs live 9201). 28/28 on felhom-pve;
red-proofs RP-1..3 run->fail->revert. GO-LIVE-PACKAGE.md ABSENT AGAIN ->
status in CONTEXT.md. Live teardown/armed-pin = GL-6.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
felhom-host-install.sh installs felhom-selfupdate-guarded (sh -n), the rollback
unit + the [Unit] start-limit drop-in (daemon-reload) so day-0 boxes get
operator-signed self-update from birth; non-fatal on pre-D1 agent repos.
03-host-agent.md §11 updated to the shipped mechanism (signed op, A/B wrapper,
OnFailure rollback, tuned start-limit backstop).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Companion to felhom-agent v0.62.0: the stale-lock reaper reads GET /pools/felhom
as its ownership registry. Pool.Allocate does NOT satisfy the read (spike T2).
Idempotent upgrade via --rescope-acl (_ensure_role modifies to the exact set).
Rescope FIRST, agent second. + REUSE §2 errata: gitea-creds is NOT out-of-band
(committed in felhom.secret.yaml, live-consumed — rotation pending).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
step_agent_install now fetches configs/felhom-mkfs-guarded.sh (bash -n validated)
and installs it 0755 root to /usr/local/sbin/felhom-mkfs-guarded BEFORE the
sudoers (which — from agent v0.54.0 — allowlists only the wrapper, not raw mkfs,
plus read-only pvs/zpool). Live-validated on felhom-pve. bash -n + shellcheck clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
step_token now grants 3 scoped roles (Guest@/pool/felhom incl. Pool.Allocate,
Store@each PVE_STORAGES incl. felhom-pbs, Base=Sys.Audit+SDN.Use@/) to both user
and token, pool-before-token, and removes any pre-3b broad / grant. New
--rescope-acl migrates existing installs (add-before-remove; supervised).
--uninstall removes both scoped + old shapes. Post-provision pool_add_guest
dropped (agent restore --pool owns membership, v0.53.0). Validated dry-run on
felhom-pve; bash -n + shellcheck clean. LIVE rescope+swap is the supervised STOP.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every managed guest joins a dedicated `felhom` pool (fleet uniformity; foundation
for the later pool-scoped ACL). All pool ops run as root@pam from the installer —
NO agent/token/ACL change (PVE_PRIVS untouched). step_provision creates the pool
+ adds the guest; new --adopt-pool retrofits existing guests non-destructively
(ours-checked, membership-only); --uninstall deletes the pool only if empty.
Confirmed pveum pool / /pools syntax live; validated dry-run + SAFE live adopt of
9201 on felhom-pve. bash -n + shellcheck clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Optional --cores N / --memory M (MiB) passed through to the agent's
--selftest=provision as -cores/-memory ONLY when set (0/unset = golden default;
avoids unknown-flag death on an agent < v0.52.0). Pre-flight soft WARN when a cap
exceeds host nproc/MemTotal. Validated dry-run on felhom-pve. bash -n + shellcheck
clean. Deploy dependency: hub artifact manifest must serve agent >= v0.52.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Colleague-safety batch #1+#2 (script-only). Adds a guarded, dry-run-aware
--uninstall local host teardown (guest -> agent -> pveum(ACL,token,user,role)
-> golden(opt-in) -> state file), with ours-check on the /etc/felhom-bootstrap
mount, typed vmid confirm, other-Felhom-guests safe-skip, and idempotent
tolerate-absent. Records customer_id + provisioned_vmid into the install state
(_state_put/_state_get) so uninstall resolves its target automatically. Adds
three provision pre-flight guards: multi-node (--node required), archive-storage
exists, and a RAM-floor WARN. Validated dry-run-only on felhom-pve.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Part A (hub): move the global-floor + Day-0-artifacts cards from the Customers
page to the Configuration tab; routes → /configuration/{global-floor,artifacts};
redirects + flashes to /configuration. Customers page back to list + Add.
Part B: online setup command on the customer page (download-then-run, passphrase
at prompt, not templated); serve /scripts/ from the website (sparse-checkout +
nginx location) so felhom.eu/scripts/felhom-host-install.sh resolves; script
passphrase prompt reads < /dev/tty (works for pipe-to-bash too).
Part C (script): --vmid auto-detect — default 9201 in use + no --force → pick the
next free id from pct+qm and confirm; explicit --vmid stays die-unless-force.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
--preserve-state-from DIR carries the prior local-api.{crt,key}+local-tokens.log into the agent
state dir (validates the leaf parses) so the pinned fp stays stable across a reinstall — no
re-bootstrap. Populated-host guard: refuse to proceed leaf-less when the host already has guests
unless --preserve-state-from or --allow-new-leaf is given (converts the 2026-06-28 silent footgun
into a hard stop). bash -n clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
A host that previously ran the agent as root+direct has no sudo package, so visudo
and runtime sudo -n are missing. step_agent_install now apt-get installs sudo before
the sudoers/unit, and resolves visudo by absolute path (non-login SSH PATH gap).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Install + configure + verify the agent (incl. golden fetch+verify) without
provisioning a guest — for re-installing/upgrading the agent on a host with live
guests, and the agent-only live test. Adds step_verify_agent (binary + non-root
service active + --selftest=hub).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator-run Day-0 automation for a fresh PVE host: pveum token -> hub
POST /host-enroll (single secret, option C) -> agent config -> felhom-agent
--selftest=provision -> verify. Idempotent/resumable (state.json), --dry-run,
--resume, --force, --preserve-from; --mode dr is a documented 10D stub.
Live-validated on felhom-pve: authorized wipe of demo guest 9201 + full
re-provision from the golden -> controller config-pull + public tunnel HTTP 200
+ host-report of 9201; idempotent --resume no-op. Found+fixed one ordering bug
during the run (token ACL must be applied AFTER token rotation — token-remove
purges the ACL).
Adds scripts/README.md (operator runbook) + scripts/CHANGELOG.md; REPORT
overwritten; spike doc pointer added. No service code changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtXesNa2LGbMmE4DNL6SE7
admin
Claude Code