SPIKE-ci-runner-2026-08-02.md: all six probes with method, measurement and ruling; none
STOPped. P2 (stock image has git but no python3) and P6 (a runner that loses its state
re-registers and orphans the old record) changed the design; P5 (a failed run signals
NOTHING) is why the alarm exists at all.
R-168 SHIPPED with its evidence. R-29 CLOSED — on the demonstrated alarm and not on a green
run, as required: the class it opened is answered at both ends, the hook refusing locally and
CI catching a --no-verify bypass and emailing. R-161 noted: its automatic half now exists for
the STATIC gate, while its original scope, the runtime gate, is deliberately still not
automatic and should stay that way.
NEW R-169 (grep established R-168 was the highest in use): CI can only report, because there
is no gate in the road. Making it blocking needs branch protection plus a PR workflow, both
of which change how the operator works — so it is theirs to decide, and the row states the
cost honestly rather than recommending it.
CONTEXT gains S-8 (CI detects, does not block, and why that is structural), S-9 (a detector
that tells no one is not finished, plus the curl and Cloudflare-1010 traps), S-10 (the runner
is unprivileged because DooPlex is Tier 2), S-11 (CI reproduces the sibling layout).
CLAUDE.md gains the rule earned by red-proofing: a go test -run pattern that matches no test
prints ok and exits 0, and an instrument that can silently drop results is not a measurement.
hub/CHANGELOG v0.87.0 + scripts/CHANGELOG gate-enforcement entry. CONTEXT gains S-6 (the
hub renders no host-install version and the gate pins its absence) and S-7 (gates run from
one entry point per repo; reuse_refs_check was fixed rather than the REUSE.md convention,
with both rejected alternatives recorded).
OPEN-ITEMS: R-94 CLOSED all three legs, leg (a) by DELETION with its reason; R-29 leg (a)
CLOSED and leg (b) HALF-SHIPPED with the census result written into the row (13 gates; every
gate a CLAUDE.md names was green, two of the four unnamed were red); R-161 gains its
successor pointer. NEW R-168 (grep established R-167 was the highest in use): Gitea Actions
runner — measured 2026-08-02 as Gitea 1.26.2, Actions enabled on all four repos, 0 runners,
0 workflow runs, 0 branch protections, and the consequence that trunk-based direct-to-main
pushes leave no merge for a status check to gate, so CI here can detect but not block.
BLOCKED on a spike over host-mode vs privileged DinD on DooPlex and whether the workflow can
avoid JavaScript actions.
ROADMAP: R-94 collapsed to its one-liner, R-29 updated, R-168 added.
A census of all thirteen gate scripts across the four felhom repos on 2026-08-02 found one
clean correlation: every check a CLAUDE.md tells a person to run was passing, and two of the
four nobody is told to run were failing — one since 14 July. Neither failure was harmful in
effect (checked line by line); nothing would have said so if they had been. The fix is not
more gates, it is one place to run them from.
repo_gates.py runs site + hostinstall + hub-confirm + manifest-bearer + reuse-refs, streams
each gate's own output, and exits worst-wins non-zero. A missing gate script is a FAILURE and
prints the path tried — fail-closed, because a runner that quietly skips a gate is the
inert-seam failure this project has shipped four times. It copies catalog_gates.py (R-161),
NOT site_gates.py, which is a gate and not a runner.
.githooks/pre-push runs it with --fast and refuses the push. Honest limits are written into
the hook itself: per-clone (core.hooksPath is local config), and --no-verify bypasses it on
purpose. Any manual run WARNS when the clone is unarmed. Measured on git 2.47.3: a relative
core.hooksPath resolves correctly and the hook's cwd is the repo root from any subdirectory.
test_repo_gates.py is a SEAM test — it asserts each member gate's own distinctive stdout, not
the runner's summary line, which an inert runner prints while calling nothing. Red-proofed:
replacing run_gate's body with 'return 0' still prints 'all felhom.eu gates OK' and exits 0,
and turns the seam test red.
RED on all four repos with 13 findings, and a hand audit of all 13 on 2026-08-02 found
ZERO genuine drift: twelve were package shorthand whose file sits a couple of directories
deeper, and one (wgsync/reconciler.go, cited by the controller) lives in the hub. REUSE.md
cites by package shorthand and across repos on purpose; the tool was what was wrong.
Resolution order, first hit wins, every non-exact hit PRINTED so a weakening is visible:
exact / suffix / ambiguous (real citation, imprecise shorthand — not a failure) / sibling
repo (as-is or with the sibling's own name stripped from the token) / FAIL. A failure lists
every resolution attempted, so a 'not found' claim names what was tried. Per-root tallies
are the positive observable: '0 failures' alone cannot tell a working checker from a blind
one. Evidence trees (audits/, documentation/tests/) are excluded from the suffix index — a
copy of a file is not the file. An absent sibling is never a failure; an unreadable parent
says so and continues.
Result: 13/13 resolve, all four roots exit 0. felhom.eu 60 exact + 1 suffix; controller 126
exact + 6 suffix + 1 cross-repo; agent 88 + 1 + 1; catalog 17 exact + 3 cross-repo.
New scripts/test_reuse_refs_check.py: 13 fixture tests, one per resolution row plus the kill
condition. Red-proof: making resolve() return 'exact' for an unresolvable token turns 4 of
them red.
The Setup tab said 'host-install 1.19.0' while the served script was 1.22.0, and had
been wrong since 2026-07-14. Deriving the number honestly is not possible: the Option-1
command downloads felhom-host-install.sh from the website at RUN TIME and the website
git-syncs main every 30s (R-110), so no build-time value in the hub can be true. R-94(a)
offered derive-or-delete; deleted, which removes the drift class instead of automating it.
- configs.go: hostInstallVersion const, pageData.ScriptVersion field and its assignment
all removed; a NOTE in their place records why there is no constant here.
- customer_unified.html: the sentence now says the command always fetches the current
installer, and renders no version.
- hostinstall_gates.py gate 1: the third assertion INVERTS — it used to require the hub
const to equal SCRIPT_VERSION, it now asserts the hub carries no host-install version
literal at all, matched in six code shapes across every .go/.html under hub/ (comments
are deliberately not stripped: a // inside a URL literal would blind the scan).
- render_test.go: the assertion 'html contains hostInstallVersion' compared the constant
to itself and passed at ANY value — demonstrated green with the const at 9.9.9 while the
script was 1.22.0. Deleted, not replaced: there is no longer a version to assert.
- felhom-host-install.sh: COMMENT ONLY (SCRIPT_VERSION untouched) — it claimed the gate
keeps the hub copy equal, an invariant that no longer exists.
Red-proofs: restoring the const fails the rewritten gate 1 (3 shapes hit); the old
render_test assertion passes at 9.9.9.
Live: https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso
sha256 f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6, 1705322496 bytes.
PART 5 PASSED ON BOTH MENU ENTRIES, four observables each:
Graphical spikegfx.felhom.eu pairing code J7N-2DA
TerminalUI spikesix.felhom.eu pairing code ZY5-YY4
Both: manual install, own disk, own password, real completion signal, and the journal's
'not bound yet — polling every 30s ... normal waiting state, not an error'. Spike 4 had REASONED the
graphical path follows from shared Install.pm; it is now measured.
PART 6: G1-G10 + G13 all PASS against the uploaded file. G4's single hit is
felhom-bootstrap.sh:480's substring TEST ('$envtext' != *FELHOM_RETRIEVAL_PASSPHRASE=*), not a
value — my own regex matched the glob's asterisk.
PART 7: uploaded via rclone in a container configured ENTIRELY by environment variables, so no
credential file was ever written. Round trip verified from the public URL — not the local file.
Bucket stays private: unauthenticated GET to the S3 endpoint 400, custom domain has no index (404).
CORRECTED BEFORE UPLOAD: the generated manifest described a single automated entry with a 5s timeout
and listed Graphical/Terminal UI as 'menu-removed'. Generator fixed, sidecar regenerated, and the ISO
verified byte-identical before and after — the published file IS the file Part 5 validated.
Hub-side cleared: appliances 16, 17, 18 discarded (303 each); zero rows remain. The endpoint is
/appliances/<id>/discard, POST only (server.go:345) — not /delete.
Teardown: VMs purged, spike5 storage removed, demo-hp back to 6.6G, drill-r50 and 9201 untouched.
Still open and named: OPEN-ITEMS/ROADMAP dispositions for R-128/R-154/R-155 are not written; the
.deb is not byte-reproducible (G7 sub-clause); before-network stub unreached; Secure Boot and real
hardware not exercised.
The 1.26.1 manifest — the file a tester reads to know what they have, and which is published
alongside the ISO — carried four statements that were false for a release build:
boot-menu 'single entry Felhom telepítés, default, 5s' -> it has TWO, timeout 15
menu-entries '1 (... timeout 5s)' -> 2
menu-removed 'Graphical, Terminal UI, ...' -> those are exactly what it SHIPS
kernel-line '... proxmox-start-auto-installer' -> the release menu deliberately has none
secret-bearing 'no (embeds the customer retrieval passphrase...)' -> self-contradictory
All four came from branding/pairing notes that predate --release and were emitted unconditionally.
A public artifact whose own manifest misdescribes it is the false-claim class this arc exists to
correct, so it is fixed before publication rather than after.
FIX for the Part-5 failure. felhom-bootstrap.sh writes the appliance token (:431), the pairing code
(:435) and .bootstrap-done into /etc/felhom/. The old stub-first-boot.sh created it explicitly
('install -d -m 0755 /etc/felhom /usr/local/sbin'); packaging dropped the env FILE correctly and the
DIRECTORY with it. Measured consequence on a real interactive install: the box registered at the hub,
could not persist its token, and polled 'HTTP 401 — still retrying' forever with no claim code.
- build-deb.sh now ships ./etc/felhom/ (0755, empty) and ASSERTS it, plus ./usr/local/sbin/ and
./lib/systemd/system/, as G13. RED-PROOFED: removing the install -d makes the build exit 3 with
'is not in the package (G13)', and restoring it goes green.
- The gate gains G13 with the reasoning: G7/G8/G9 all passed on the broken package. G9 proves the
payload is the right payload and says NOTHING about what the payload depends on.
ISO_VERSION -> 1.26.1.
The release build copied the .deb to $WORK/felhom.deb before handing it to the repack, so the ISO
carried '/proxmox/packages/felhom.deb' — the version invisible from the image, and not matching the
release gate's 'exactly one felhom-*.deb' check (G7). Caught by running G5's enumeration against the
built artifact rather than trusting the build log.
Neither changes an artifact, but both are read by an operator deciding whether a build is sound:
- the closing banner printed 'root-pw : <iso>.rootpw.txt ... the console credential for this build'
unconditionally. In --release mode no password is minted and no such file is written (verified:
the release build emits only .iso, .sha256 and .manifest.txt). It now says so.
- the repack's menu-surgery line hardcoded '1 entry, 0 submenus' and printed it after a gate that
had just accepted TWO. It now reports the counts it actually asserted.
The repack read both correctly; the caller never set them, so a --release build reached the
narrowed R-155 guard still in 'single' mode and was refused (rc=10). Caught by the build's true
exit code. Also copies grub-release.cfg.tmpl into the brand dir and fixes the branding log line,
which claimed 'single-entry menu' unconditionally.
The mode validation still required one of --bootstrap-env / --pairing, so --release died at
'one of --bootstrap-env (direct) or --pairing (generic) is required' before reaching its own
validated branch. Caught by the build's true exit code (rc=1), not by a pipe.
Design inputs: SPIKE-universal-iso-{1,2,3,4}-2026-07-31.md. Every choice below is a measurement.
NEW: scripts/iso/pkg/ — the felhom-bootstrap .deb, built from committed source.
Two files only (script + unit), NOT three: felhom-bootstrap.sh:91 reads /etc/felhom/bootstrap.env
only 'if [[ -r ]]', and its defaults at :95-96 are EXACTLY what the pairing env set
(build-felhom-iso.sh:257-258) — so shipping it would add a 0600 file to a public package to express
values the script already defaults to. NO dependencies: the binaries it calls run at FIRST BOOT,
not at postinst time, so SPIKE 4's open 'dpkg --configure -a' ordering question does not arise.
The postinst is structurally incapable of failing (no 'set -e', every statement guarded, ends
'exit 0'); build-deb.sh self-asserts G8/G9 and REFUSES to emit a package that violates them.
iso-repack.sh — two changes, both narrowing rather than deleting:
- R-155 guard: now applies to FELHOM_MENU=single ONLY. It protected the single-entry mode's promise
(one button labelled 'install' must not drop into a disk-picker); a release image carries no
auto-installer-mode.toml BY DESIGN (gate G1), so refusing it would be the guard firing on the
shape it describes rather than the one it prevents.
- the menu collapse now has a release mode: two INTERACTIVE entries, Graphical default, timeout 15.
Entry-count and banned-token gates are per-mode; the six-token list is UNCHANGED for single mode.
- .deb injection into /proxmox/packages/, with a skip-list collision check (a colliding name would
be dropped silently — the inert-payload class) and a post-remaster assertion that it landed in
final.iso, not merely in the extract tree.
build-felhom-iso.sh — --release: no profile, no root hash, no answer.toml, no prepare-iso at all.
Skipping prepare-iso is what removes the Automated entry by construction, since the stock grub.cfg
emits it only inside 'if [ -f auto-installer-mode.toml ]'.
R-128 RULING — FIXED, by correcting the claim rather than inventing an assertion for it. The comment
said ISO_VERSION 'aligns with SCRIPT_VERSION'; nothing evaluated it and the two had drifted. The
coupling does not exist: the ISO is frozen, felhom-host-install.sh is fetched at run time from main
(R-94/R-110), so an assertion would invent a constraint. Comment corrected, ISO_VERSION -> 1.26.0.
Release gate G6 AMENDED before the build, with its reasoning recorded in the runbook: the six-token
ban existed to keep users away from the manual installer, which the ruling makes the product.
'proxtui' (the TUI installer we ship) and 'nomodeset' (its graphics fallback) are dropped for
release images; proxdebug/Rescue Boot/memtest/fwsetup stay banned in both modes.
Findings only — no script, profile or build file changed; no release ISO built, nothing published.
documentation/audits/SPIKE-universal-iso-4-2026-07-31.md
MEASURED, with a control, and the negative control is in the SAME box. One ISO (15 GRUB entries),
a trivial probe .deb injected into /proxmox/packages/, two qm-created VMs on demo-hp (400
interactive / 401 automated control) on a scratch dir storage at the /mnt/nvme-1tb mount ROOT.
Interactive (Terminal UI) install:
- package installed (ii felhom-spike4-probe 0.0.1)
- postinst RAN (marker + content intact)
- it enabled a systemd unit, and that unit FIRED ON FIRST BOOT (uptime 7.98s, pid1=systemd)
- while on the same machine proxmox-first-boot is NOT installed and /var/lib/proxmox-first-boot
does not exist — Spike 3's negative reproduced, not assumed.
Postinst environment (identical both paths): pid1=unconfigured.sh, NO running systemd, but
'systemctl enable' SUCCEEDS; /proc+/sys mounted; network+DNS happened to be up (inherited from the
installer's DHCP — must NOT be relied on). Constraints: never systemctl start/daemon-reload, never
require network, never fail, do the real work in the unit at first boot.
Repack preserves it, but a naive 'xorriso -boot_image any replay' fails with 'Overlapping MBR
partition entries' — iso-repack.sh:270-292 already documents that exact failure and its fix.
R-153 RETRACTED into R-94 leg (b): OPEN-ITEMS.md:15 carries it verbatim at READY (XS), and R-29
says explicitly 'do not mint a new ID for a new instance'. Spike 3's further claim that the drift
leaves the generator 'three minor versions stale' was FALSE and is corrected — R-94 retracts that
exact reading; the served script is always main, so 1.22.0 is what every install already gets.
No new R-rows opened.
Findings only — no script, profile or build file changed; no release ISO built, nothing published.
documentation/audits/SPIKE-universal-iso-3-2026-07-31.md
MEASURED with a control from the SAME image (one ISO, 15 GRUB entries):
- Automated entry -> hook fires: ttyS0 marker, marker file,
/var/lib/proxmox-first-boot/proxmox-first-boot (0700), activation symlink, unit active.
- Terminal-UI entry, normal manual install -> ALL absent, and the proxmox-first-boot PACKAGE is
not installed at all. A whole-filesystem grep for the marker returns nothing.
Mechanism cited: Config.pm:118 defaults first_boot.enabled=0 and set_first_boot_opt is never
called in the Perl tree; Install.pm:746 returns early without it; Install.pm:1360 skips the
package. proxinstall (graphical) has ZERO occurrences of first-boot. [first-boot] is an
automated-installer feature, unavailable on every interactive path by construction. R-154.
A delivery mechanism DOES exist and is UNTESTED: Install.pm:1343-1372 unpacks every .deb in the
ISO's /proxmox/packages/ into the target on every path (fixed skip-list), then dpkg --configure -a
runs postinsts (:1378) — how PVE ships first-boot itself. Read from source, not measured.
Q5: the public image should carry NO answer.toml at all — that removes the baked root hash, the
disk profile and the whole Spike 1-2 problem space, and makes it a one-line release gate. But
iso-repack.sh:100-106 refuses an ISO without auto-installer-mode.toml. R-155.
Incidental R-153: hub hostInstallVersion=1.19.0 vs SCRIPT_VERSION=1.22.0; hostinstall_gates.py
detects it and exits 1 — the gate works, nothing runs it.
Q3 (real stub at before-network) was NOT reached and is recorded as not reached.
Findings only — no script, profile or build file changed; no ISO built, nothing published.
documentation/audits/SPIKE-universal-iso-2-2026-07-31.md
Both Tier 0 boxes went offline mid-session (provider cable fault; four routes tried, no Tier 2
fallback used) and returned. All three scenarios then ran to completion on real PVE, each signalled
by reboot-mode='power-off' rather than a disk hash.
- A LOCKED ROOT CLOSES THE PVE WEB INTERFACE. Measured at the exact endpoint the UI uses
(POST /api2/json/access/ticket, root@pam) WITH A WORKING CONTROL: known-password install returns
HTTP 200 + ticket; locked install returns 401 for every password and none can exist.
passwd -S root = L, shadow = literal-asterisk, PVE uses the stock PAM stack.
- GRUB recovery mode is also closed ('the root account is locked') — but init=/bin/bash still gives
an unauthenticated root@(none):/#. A locked box is recoverable, operator-only, at the console.
The installed GRUB has NO password, so locking root is not a physical-security measure. R-152.
- before-network MEASURED (A/B, same image): the hook RUNS (marker, uptime 6.58s) with entropy 256,
writable /etc, all binaries and openssl_rand_len=32, while ip_global is EMPTY and
listen_22_8006 = 0. fully-up is the converse: sshd+pveproxy active, 3 listening. Zero window.
- R-148: answer.toml.tmpl:27 justifies fully-up with a pvesh/pct dependency the stub does not have
(grep rc=1) — it blocked the ordering now measured as the fix.
- R-149 three ordering values; R-150 Condition-guarded hooks skip silently; R-151 demo-felhom built
from an uncommitted profile.
Three probes failed and are recorded as failed: a container probe that ran as uid 0, a GRUB probe
that missed the 1-second menu timeout, and a kernel-line edit one line off (caught by a pre-typing
verification screendump). The interim 'Layer 1 teardown INCOMPLETE' is corrected — the fixture had
never landed, because the staging mkdir was in the SSH call that timed out.
Findings only — no script, profile or build file changed; no ISO built, nothing published.
documentation/audits/SPIKE-universal-iso-2-2026-07-31.md
BOTH Tier 0 boxes went offline mid-session (remote site, 12:28 CEST; four routes tried, our
tailscale pod healthy). Q1/Q2/Q3 each keep a part needing a nested VM: those are BLOCKED, not
answered. DooPlex was NOT used as a fallback — Tier 2, and this task did not authorise it.
Established without them:
- STRUCTURAL: ordering='before-network' maps to proxmox-first-boot-network-pre.service
(Before=network-pre.target, Type=oneshot) — it completes before ANY interface is configured,
so a rotation there has a zero-length window BY CONSTRUCTION, not by being fast.
- R-148: the stub does not need 'fully-up'. stub-first-boot.sh has no pvesh/pct/pveum/qm call
(grep rc=1); that usage is in felhom-bootstrap.sh under its own After=network-online unit.
answer.toml.tmpl:27 justifies the current ordering with a dependency that does not exist.
- R-149: the ordering enum has THREE values (before-network, network-online, fully-up), not two.
- MECHANISM (container, not PVE): locked root closes sulogin — 'the root account is locked' for
both '*' and '!', with a working control. So 'discard' and 'lock' are the SAME outcome for
recovery, making the escrow decision binary.
- R-150: all four proxmox-first-boot-* units are Condition-guarded; a failed condition is a SKIP,
so a hook that never ran looks identical to one that succeeded.
- R-151: demo-felhom was installed from an UNCOMMITTED profile — a Tier 0 reference box is not
reproducible from main.
- Q4: four gates in iso-repack.sh enforce the single-entry menu; default/timeout already settable.
The first mechanism probe was invalid (uid 0 bypassed pam_unix; sulogin had no tty) and a teardown
error (shredding the control plaintext) are both recorded as failures, not massaged.
demo-hp teardown is INCOMPLETE and named as such; the command is recorded, not claimed done.
Findings only — no script, profile or build file changed; no ISO built, nothing published.
documentation/audits/SPIKE-universal-iso-2026-07-31.md
- R-139 (HIGH): a disk filter matching >1 device does NOT fail safe. Observed in a nested VM —
the installer silently picked one of two matching disks and wiped it; validate-answer accepts
such an answer. The 'filter did not match any devices' guard covers the ZERO-match case only.
- No udev property distinguishes an internal system disk from external media. Measured on
demo-felhom with its 1TB external attached: ID_BUS='ata' for BOTH, lsblk RM=0 for both, and
device-info exposes no removability property. demo-hp's NVMe carries no ID_BUS/ID_TYPE at all.
- R-141 (HIGH): the answer schema makes a root credential mandatory, but root-password-hashed='*'
validates AND installs to completion. [first-boot].ordering accepts 'before-network', the only
ordering that closes the exposure window structurally.
- Q3: prepare-iso leaves grub.cfg byte-identical to stock (15 entries, automated AND interactive)
— a two-entry menu is purely a Felhom grub.cfg.tmpl change.
- R-129 resolved: demo-hp's key is the operator's own, added post-install; demo-felhom's IS baked
by an uncommitted profile.
The reachable-before-rotation measurement FAILED twice and is recorded as failed, not inferred.
Opens R-139..R-147; restates R-128.
Every box installed before this got local_backup_target "local" -- the vzdump
target on the SAME physical device as the guest, so a drive failure took the
guest and its only local backup together. E-1 fixed two machines by hand; this
fixes the installer.
Case A: an eligible secondary drive is already mounted -> create felhom-backup on
that drive's own mountpoint via the felhom-backup-target-apply wrapper (create +
grant) and point the primary tier at it.
Case B: system drive only -> the target stays on the system drive and this is
RECORDED AS DEGRADED, not as normal. The install still succeeds: a single-drive
appliance is a valid product, it just cannot survive drive loss.
Phase 0 inverts the emphasis: the installer has NO drive-enrollment step, so on a
fresh appliance Case A almost never fires. The common case is Case B with the
drive arriving later through the wizard (Part 3). Case A covers the reinstall
shape where an agent-generated .mount unit already brings the drive up by fs-UUID.
Eligibility suggests and refuses the absurd, never decides by transport: the
reference backup drive is an external USB HDD and BOTH demo boxes report
removable=0, so a transport rule disqualifies the reference drive and a removable
rule finds no candidate at all.
Scenario F: an already-configured box is never corrected -- an early return plus
setdefault, both load-bearing.
Proofs (installer-logic-tested against extracted functions with stubbed
pvesm/wrapper; NOT install-tested, no reinstall was performed):
A -> create + grant, resolved felhom-backup
B -> DEGRADED warnings, resolved local, rc=0 (install not failed)
F -> skipped, 0 wrapper calls
F red-proof (guard removed) -> 2 wrapper calls, i.e. it would have "corrected"
a correct box
A failed restore-test's scratch guest never joins the felhom pool, so the pool-scoped
grant cannot reach it and teardown 403s. Ten path-scoped /vms/<id> grants reach exactly
the scratch band and nothing else. Removal path + verify step extended.
Slice D.1 — host-install 1.20.0: a FRESH box defaults to local-daily +
offsite-weekly (felhom-pbs, 604800s, keep_last=2). setdefault semantics proven
both ways: fresh gets the tier, an UPGRADE preserves the existing backup block
verbatim — so an in-place upgrade can never silently start writing to an
offsite datastore. Existing boxes are migrated explicitly.
Slice E:
- 07-backup-architecture.md: honest status header per CONTEXT ruling S-2, with
an explicit STALE-outside-the-PBS-tier verdict (the controller tiers were last
verified 41 controller versions ago). The PBS row claimed 'PBS on DooPlex'
(the retired spike store) with no cadence; it now names felhom-pbs ->
felhom-offsite on ep0 over wg-felhom, weekly, keep_last=2. NOT marked
ratified — that is Viktor's review of the section 10 list. Discharges R-83.
- 06-offsite-connectivity.md: the target-split remaining-work note collapsed
(shipped), and records HOW S4.1's tier-aware timeout silently regressed — the
mechanism was never removed, its INPUT changed when local_backup_target was
retargeted to 'local'. Also notes S4.1 already diagnosed the teardown 403 as a
phantom (a timeout consequence, not an ACL gap).
- capability map: new row for recurring offsite backups actually LANDING, as
distinct from the existing row proving ACTIVATION. IMPLEMENTED, not
PROVEN-LIVE — the restore round-trip has not completed under the fixed code.
- ROADMAP: R-82 SHIPPED with its remaining gate named, R-83 DISCHARGED, R-84
left open.
- CONTEXT + REPORT: the arc, including the mid-arc correction I had to make.
Virgin-ISO nested drill closed the train: dead-NIC install baked the
fallback (incl. the dead default gateway), the R-59 screen painted
(capture committed beside the spike doc), the cable move healed +
registered at the hub in 23s unaided, and the build's rootpw file
matched the installed box's shadow hash. R-59 SHIPPED with the recorded
deviation (first-boot gate; installer-initrd abort out of scope by
operator ack). R-60 SHIPPED (spike + drill cited; F-P9 route-flush fix
included). R-61 slice 1 SHIPPED. New R-62 row (hub delete-dialog
cosmetics, XS). Capability map: new PROVEN-LIVE row (nested != metal,
said so). Cleanup verified: felhom-pve interfaces byte-identical,
bridge/VMs/ISO removed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
Live drill finding (nested leg, 2026-07-22): the installer's no-DHCP
fallback bakes not just the 192.168.100.2 static address but a DEFAULT
ROUTE via 192.168.100.1, and dhclient-script never replaces an existing
default route — so the sweep's hub probe rode the dead gateway and
failed even though the candidate's lease had landed, and the gate
oscillated (lease -> probe fail -> byte-identical restore -> screen).
The candidate probe now flushes vmbr0's addresses+routes after the
re-point, letting the lease install its own; the restore path and each
next candidate re-apply configured state via ifreload. Validated live
by hot-patching the drill box: heal in 33s from restart — self-heal
log, inet-dhcp persist, .felhom-bak, hub registration + pairing banner.
Full virgin-ISO drill follows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
network_gate() runs before mode dispatch. Hub reachable -> return with
ZERO new behavior (asserted). Unreachable -> diagnose (physical-NIC
table with raise-first carrier read; installer 192.168.100.2-fallback
signature named when present), then — ONLY while no install state file
exists, doubly confined by the unit's ConditionPathExists=!done-flag —
sweep: per candidate re-point vmbr0's bridge-ports (atomic tmp+mv,
always derived from the pristine copy) -> ifreload -> bounded dhclient
(20s) -> hub probe (any HTTP status = reachable; the hub answers 302).
First success persists winner + DHCP addressing (original kept as
interfaces.felhom-bak) and proceeds; total failure restores the config
BYTE-IDENTICALLY, paints a calm Hungarian console screen (spec copy,
print_pairing_banner pattern) and retries every 60s with the unit held
in activating — waiting is not failing (v1.21.0). Trigger is always
hub-unreachable, never the fallback signature (spike F-P6: the
truth-changed-after-install variant has no signature). Design inputs:
SPIKE-firstboot-nic-sweep-2026-07-22 F-P1..F-P8.
Harness: +4 scenarios (G1 zero-behavior in D incl. zero-sleeps +
fixture-untouched, G2 sweep-success/persist/proceed, G3 sweep-fail
byte-identical + screen content, G4 state-file-present = never sweep),
gate seams (FELHOM_NET_SYS/INTERFACES_FILE/CONSOLE_DEV), ip/ifreload/
dhclient fakes, hub-probe oracle incl. follow:<nic> mode. Red-proofs
run and restored: (A) restore-on-failure dropped -> G3 byte-identical
FAILs; (B) state gate removed -> G4 zero-call FAILs; (C) early return
removed -> all five G1 invariants FAIL.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
The old pairing/delivery scenarios expected a non-zero exit on an
unbound 204 poll — the one-poll-per-invocation design v1.21.0 (R-33)
deliberately removed; against the current script they would hang on a
real sleep. Now: a PATH-faked sleep counts the waits and flips the poll
to 200 after 3 cycles, so one scenario proves the whole v1.21.0 shape
in a single invocation (register -> in-script 204 waits -> delivery ->
host-install -> done-flag, exit 0), plus a 410 crash-window scenario
(still exits non-zero on purpose). Runaway guard: fake sleep kills the
loop after 25 calls. Assistant image gains python3 (the bootstrap's
JSON parsing needs it; PVE ships it on the real box) — the harness runs
in that image.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
The ISO build now writes the minted throwaway root plaintext to a 0600
sibling file (<iso>.rootpw.txt: password + ISO name + build date) — the
single record of truth. Never stdout/logs/manifest; the manifest carries
only a pointer line. OUT_ISO naming hoisted above the mint (the sibling
is named after the ISO); the mint itself and ROOT_PLAIN's lifetime are
unchanged beyond the one file write. FELHOM_ISO_KEEP_WORK=1 debug escape
added for the harness. New test/rootpw-emission.sh: dry-run emission,
0600, plaintext<->answer-hash cross-check (openssl -6 -salt), no
plaintext on stdout, manifest-heredoc guard. Red-proof run: plaintext
injected into the manifest heredoc -> harness FAILs -> restored.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
R-59 no-DHCP install must hard-abort (it baked 192.168.100.2 static and
completed - a box that can never call home). R-60 first-boot NIC sweep
self-heal. R-61 the baked root password must be knowable; a fixed well-known
password is explicitly rejected.
Positive evidence same-session: R-21 slice C PROVEN on a SECOND, virgin board
(HP t740) - and the shim loader booted with Secure Boot ENABLED, retiring the
assumption that Felhom installs need SB off. Fresh-box floor lift
0.153.0 -> 0.156.0 during day-0 cited on the publish-train row.
ISO README gains the t740 five-NIC trap: the 4-port igb card gets no lease,
the onboard r8169 port does.
R-58 records the operator ruling (2026-07-21) with the argument verbatim: the
installer should list available storage devices, excluding the install media,
and let one be selected. Third ISO mode alongside unattended-serial and
match-nothing-safety; unattended stays the appliance/factory mode. Slice 1 is
the abort-screen candidate table, same enumeration code, and it collapses the
two-boot dance on its own. Matters most for BYO/reinstall, where the serial is
unknown and a wrong guess is destructive.
ISO README gains the HP section: shim proven on this board by the safety boot,
the uncommitted-armed-profile pattern, verify-from-inside-the-ISO, and a
pointer to the prior-LVM abort that is the one likely failure on a
second-hand disk.
PROMPT-TEMPLATE: standard 'For the operator' plain-language section, mandatory
for M+ tasks and anything with a STOP.
ROADMAP rulings (operator, 2026-07-21): R-25b full-teardown cascade with three
acks + typed name (M-sized, spec to follow, no longer blocks R-3); R-11 channel
= direct Messenger, doc is the architect's; R-42 option (a), sidecars follow the
app; R-17 delete the archive - spike-lite found NO tooling verb targets it, so
it is an operator console action; R-4 complete (freemail.hu verified).
R-55 + R-41 slice 1 marked shipped; new R-56 (app difficulty classification -
the constructive half of the glance ruling).
scripts/build-hub.sh v1.23.0: the hub build script was outside any repo. Adopted
verbatim + versioned; the build-dir path is now a symlink to it.
felhom-testing skill: the ~1/5 recovery-code 'known flake' is retired - it was a
real defect the test was correctly detecting.
Mirrors profiles/n100.profile with one deliberate difference: the loader
default stays at the stock signed shim chain. mkimage exists to work around
the N100's AMI firmware GRUB relocation defect, not as a Felhom requirement,
and it costs Secure Boot. HP business firmware should take shim - so try it
first and carry mkimage as the fallback.
Ships the match-nothing disk filter, so built as-is it IS the hp-safety
pre-flight ISO. Arming instructions point at an uncommitted working profile,
per the n100-demo precedent.
R-38 flips to SHIPPED with the live-canary evidence. Two new items registered
from tonight's work: R-45 (unified async-job feedback — five one-off progress
surfaces now exist and agree on nothing; the two lessons any framework must
encode are "probe the terminal state, never infer it" and "a source reporting
nothing is normal, not broken") and R-46 (verification copies are now visible
but still cannot be browsed and never expire — the invisible-result gap and the
disk-lifecycle gap, designed together because they are the same object).
Pre-invite checklist gains the golden note: build-golden v2.1.0 makes goldens
>= 0.147.x carry all four infra images; no rebuild for this alone.
REPORT-<topic>.md per the parallel-session rule — another session is writing in
this clone tonight, so the shared REPORT.md is left alone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
The canary boot showed the entry sitting noticeably left of the centered logo.
gfxmenu's boot_menu draws items left-aligned inside the box with a ~29px inset
and offers no item alignment, so centering means placing the box: left 25%->40%,
width 50%->35%. Measured off the canary screenshot, not guessed; the comment
records the measurement so a future label change gets re-measured instead of
re-guessed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
The v1.22.0 gate asserts the rendered grub.cfg has no live reference to
proxtui/proxdebug/nomodeset/Rescue Boot/memtest/fwsetup. It grepped the whole
file, so the template's own header — which documents exactly which stock entries
were dropped, and names them — tripped it. Caught on the first canary build:
fail-closed, no ISO produced, which is the behavior we want from a safety gate
that is wrong.
Strip comments before matching. A comment naming a removed entry is the point;
a directive using one is the bug.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Two jobs, one repack pass.
BRANDING. Every ISO now carries a Felhom boot screen built from the website's
og-image_2.png at repack time (ImageMagick in the assistant container), so the
boot card has ONE source and not a second pre-rendered copy in the repo to
drift. The card is scaled onto a 1024x768 gfxterm canvas, top-centered, and the
card's own subtle background grid is continued across the letterbox fill
PHASE-LOCKED to where the card's grid lands — the fill is seamless rather than a
square of grid floating in flat navy. Menu positioning needs a gfxmenu theme
(plain background_image cannot move the menu off the wordmark), so the stock
pvetheme is replaced by felhomtheme, which puts the menu in the lower third the
layout deliberately leaves empty.
SAFETY — the half that matters. The stock PVE menu offers Graphical, Terminal
UI and serial installers plus an Advanced Options submenu (nomodeset x2, three
debug variants, Rescue Boot, memtest, UEFI settings). Every one of them reaches
the MANUAL installer, whose first question is which disk to wipe. A customer, or
their helpful nephew, must not be able to get there from a boot menu. They are
not hidden and not password-gated: they are NOT EMITTED. What ships is one
entry, 'Felhom telepítés', default, 5s.
Boot behavior is unchanged. The kernel/append and initrd lines are lifted
VERBATIM from the ISO's own 'Install Proxmox VE (Automated)' entry rather than
frozen into a copy here, so a PVE bump tracks automatically; the build fails if
they cannot be found, if the append line has lost proxmox-start-auto-installer,
or if auto-installer-mode.toml is absent (which would mean the one Felhom-
labelled entry boots a manual installer). The rendered menu is then gated for
exactly 1 entry, 0 submenus, and zero references to proxtui/proxdebug/nomodeset/
Rescue Boot/memtest/fwsetup — and re-verified by reading the menu back OUT of
the finished ISO, not merely out of the extract tree.
mkimage-surgery.sh -> iso-repack.sh: branding and the slice-B loader swap need
the same extract -> modify -> re-master cycle, so they share one pass instead of
re-mastering twice. The mkimage recipe is untouched. The embedded module list is
still derived from the STOCK grub.cfg (snapshotted before branding rewrites it),
plus gfxmenu's bitmap/bitmap_scale/trig renderer deps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Waiting to be bound is the NORMAL state of a freshly installed box, and it must
not be reported as failure. The PAIRING poll loop used to BE systemd's
Restart=on-failure/RestartSec=30 — one poll per invocation, exiting non-zero
until the bind landed — so every 30s systemd printed "Failed to start Felhom
host bootstrap" on the physical console the CUSTOMER is watching. The
2026-07-18 N100 rehearsal measured 52 FAILED lines in ~11 minutes while nothing
was wrong (VALIDATION-n100-rehearsal-2026-07-18.md F6).
felhom-bootstrap.sh: run_pairing() is now a while-loop that sleeps
POLL_INTERVAL (30s — the hub-side rate is unchanged) between polls, so the unit
sits in `activating`. Registration split into register_appliance(), which
returns non-zero for a transient problem (no network yet, no identity, no
token) and is retried by the loop instead of taking the unit down. Cadence
constants: POLL_INTERVAL=30, BANNER_EVERY=10 (5 min), HEARTBEAT_EVERY=20
(10 min).
Quiet without going dark: a 204 is logged once on entry (worded so nobody reads
it as an error) and then only on the 10-minute heartbeat with elapsed minutes;
404 and unexpected codes degrade the same way. 410 STILL exits non-zero on
purpose — delivery consumed but no local env is a real crash window, and a
clean systemd restart is the right response.
Console banner: every 5 min instead of every cycle, single accented spelling
instead of the parositasra/párosításra double, and the reassurance the
rehearsal showed was missing ("Ez a képernyő magától frissül — nincs teendő a
doboznál").
felhom-bootstrap.service: TimeoutStartSec=infinity. This is load-bearing, not
cosmetic — a Type=oneshot ExecStart is killed at DefaultTimeoutStartSec (90s),
so without it systemd would kill the new in-script wait after 90 seconds and
Restart=on-failure would silently reinstate the exact spam this removes, after
appearing to work for the first three polls. Restart=/RestartSec= are kept
deliberately: they still cover the DIRECT path, a failed host-install, and 410.
Verified behaviourally, not assumed: driven in a throwaway Debian container
against a stub hub answering 204 five times then delivering — logged the wait
once plus one heartbeat, never exited between polls, then consumed the
delivery, wrote the 0600 env, fell through to the direct install in the same
invocation and exited 0. The old design produced five unit invocations and five
"Failed to start" console lines for that same sequence.
Hub endpoints, payloads, polling rate and one-shot delivery semantics are all
unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Let a customer bind their own freshly-installed appliance without the
operator: operator "Send self-bind link" mints a 7-day tokenized
capability link, emailed (Hungarian, sibling sender) to the customer, who
opens a public /bind/<token> page and proves two factors — the console
pairing code shown on the box screen + their retrieval passphrase — and
the hub stages the bind via the same BindAppliance (provenance
customer_selfbind). The box's ~30s appliance poll delivers.
Viktor's three rulings verbatim: console pairing code (no appliance list
ever rendered), operator-sent tokenized link, 5-attempt lockout ->
"call support". Wrong code == wrong passphrase (one generic failure, no
oracle, both factors compared unconditionally); expiry falls back to
operator-bind unchanged.
THE TRAP: one public prefix /bind/, exempt from auth+CSRF at both /login
gate sites via a single isPublicBindPath predicate (tight trailing-slash
match; ServeMux ..-cleans; handler rejects '/' in token). 9 tests
(Scenarios A-F + F1/F2); 4 red-proofs verified red-then-green (lockout,
oracle, widened-prefix, single-active). GC verdict: no appliance GC ->
the 7-day TTL stands alone. Controller/agent untouched; R-27b deferred.
Green: full hub build/vet/test (17 ok) + bash -n + hub confirm gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qDiBqKKQ5vPB5fXBqu7Kp
Makes PBS DR storage visible like the restic pool box (v0.64.0), differentiated. Scoping
correction: restic = subaccounts on the shared Hetzner Storage Box (Hetzner API); PBS DR =
the felhom-offsite PBS datastore on the ep0 endpoint VM (NO Hetzner API). Option A
(Viktor-ruled): a read-only `usage` op on the felhom-tenantsync ep0 forced command (twin of
fingerprint), polled by a new hub checker on the 15-min throttle. READ-ONLY throughout.
Phase-0 (gate PASSED): on ep0 (PBS 4.2.3), df -B1 --output=size,used,avail <datastore path>
yields bytes (39990112256/7627939840/... ~19%), read-only, existing sudo context, no admin token.
- scripts/felhom-tenantsync.sh -> v1.2.0: read-only `usage` short-circuit (df on the datastore
path), no customer_id, no admin token, NO mutation. + a bash harness proving zero mutation.
- tenantsync.Client.Usage() + BoxUsage; unknown-op -> typed ErrUsageUnsupported (graceful).
- monitor.PBSDRBoxChecker: OffsiteBoxChecker clone over a usageReader seam; 15-min throttle,
cached PBSBoxSnapshot, escalation-only pbsdr_box_fill on the "pbsdr-box" scope (operator only,
no SaveEvent), recovery re-arm. Fill only. THREE states: ok / unavailable (ep0 <=v1.1.0,
neutral no-alert) / degraded (exec failed, keep last).
- config: Alerting.PBSDRBoxFill{Warn,Crit}Percent (80/90); built with the tenantsync client,
60s sweep, SetPBSDRBox. Hub deploy INDEPENDENT of the ep0 update (graceful degradation).
- web: /offsite splits into Restic + PBS DR hash tabs (endpoint cards under PBS DR); PBS panel;
the single dashboard tile becomes two gauges (RESTIC pct.ratio, PBS DR pct / n/a).
- runbook offsite-endpoint.md 10: v1.2.0 update steps (no sudoers/authorized_keys change).
Tests: 10 Go + the harness; 3 red-proofs (usage mutation, escalation-only, unavailable-drives-band)
confirmed red then restored. go build/vet/test + bash -n + hub confirm gate all pass.
A generic ISO carries NO customer secret. The box registers itself at the hub
as an unclaimed appliance; the operator binds it to a customer; the hub delivers
the customer-id + retrieval passphrase ONCE; day-0 completes via the slice-A path.
Hub (v0.62.0):
- store/appliance.go: appliance_registrations keyed by (uuid, mac_set) — MAC set
is the tiebreaker (duplicate SMBIOS UUIDs); token stored as sha256 only.
Idempotent register (sticky-discard), atomic one-shot delivery, bind/discard.
- api/appliance.go: POST /appliance/register (the one unauth endpoint, per-IP
rate-limited, 256-bit token); GET /appliance/poll (404 no-oracle / 204 unbound
/ 200 deliver-once / 410 delivered). Passphrase read live, never logged.
- web/appliances.go: Hosts-page "Unclaimed appliances" section + BIND (customer
picker, host count display-only) + DISCARD; SSH host-key fingerprints; events.
- Red-proofs: one-shot delivery + register idempotency (both proven red);
404-no-oracle, sticky-discard, bind staging, render. Green + confirm gate.
Scripts (v1.19.0):
- felhom-bootstrap.sh: ONE unit, TWO modes. Direct (env has customer/passphrase)
= slice-A path, byte-identical, only branched around. Pairing (generic) =
register + poll (RestartSec=30 is the poll timer); on delivery write the env
0600 and fall through to direct. Secrets + token shredded on success.
- build-felhom-iso.sh --pairing: generic secret-free ISO, -generic filename,
manifest mode=pairing. profiles/generic.profile (new).
- test/bootstrap-modes.sh: Scenario D (direct = zero appliance calls) + pairing
register/poll + delivery handoff — all green in a debian container.
Closes N100 F1 (HIGH): cheap AMI (AN3PLUS 0.01-class) UEFI firmware can't
relocate the ISO's stock signed GRUB from USB (relocation 0x0). The run's live
grub-mkimage workaround is now a first-class pipeline mode.
- build-felhom-iso.sh: --loader shim|mkimage (default shim, byte-for-byte
unchanged; profile-settable FELHOM_LOADER; --loader wins). Loud banner +
manifest loader:/grub-mkimage: fields + -mkimage filename suffix.
- mkimage-surgery.sh (new): post-prepare-iso, in the assistant container. Builds
a monolithic grub-mkimage loader from the ISO's own GRUB (module set from its
grub.cfg; embedded search --fs-uuid -> configfile the real menu). Swaps it into
the ISO9660 tree (real lowercase path) + the efi.img ESP; xorriso re-master
preserves BIOS-hybrid + UEFI + GPT-ESP, drops only Apple HFS+/APM. Recipe from
the N100 run evidence, not re-derived.
- Dockerfile.assistant: grub-common + grub-efi-amd64-bin + mtools + dosfstools.
profiles/n100.profile (new, mkimage + SB-off note).
- Validated on nested VM 311 (RUNBOOK-B legs): leg1 shim boots+installs under
OVMF SB-enforcing + SeaBIOS; leg2 mkimage boots+installs under SB-off; leg3
(red-proof) mkimage under SB-enforcing FAILS Access Denied (unsigned -> SB must
be OFF); leg4 surgery byte-identical payload. bash -n + shellcheck clean.
Physical N100 closure folds into the rehearsal (n100-safety match-nothing ISO
built + sha-recorded, unbooted). PXE stays a deferred R-21 note.
admin
Claude Code