New OPERATOR-ONLY event type recovery_unit_capture_failed (controller
v0.191.0, R-158): in allowedEventTypes AND notify.operatorOnlyEvents.
Deliberately not a reuse of backup_failed, which carries customer copy and
sits in the controller's DefaultEnabledEvents — reusing it would email the
customer in Hungarian about a failure they cannot act on. R-158's own
proposal said backup_failed; D-c overrides it.
disk_warning/disk_critical lose their generic customerMessages entries.
Both were allowlisted, copy'd, default-enabled and checkbox'd with NO
producer anywhere; controller v0.191.0 becomes that producer and sends a
DYNAMIC Hungarian message naming the drive and its free space.
FormatCustomerEmail prefers the entry over the message, so keeping a static
entry would discard the label and the byte figures — the same reason
offbox_enlarge_blocked and disk_health_degraded have none. The deletion is
pinned by a test.
New notify.IsOperatorOnly so the api package can pin BOTH registers of a new
event type in ONE test; allowlisted-but-not-operator-only is invisible when
they are checked separately, and it is the defect v0.78.0 shipped. The
register itself stays unexported.
REUSE.md's "new event type" extension point rewritten: it told readers to
always add a customerMessages entry, which is wrong for operator-only types
and harmful for dynamic-message ones.
Tests 574 -> 579. Red-proof: removing the operatorOnlyEvents entry shows the
customer being emailed; the skipped/operator_only row is asserted as a
positive observable.
R-172's root cause was not tuning — the WAL/busy_timeout pragmas had never been
applied, because the DSN used mattn/go-sqlite3 syntax against modernc.org/sqlite,
which ignores unknown parameters without an error. Recorded that way so nobody
re-reads it as "SQLite was slow".
R-173 NEW: while establishing who copies hub.db for the WAL change, found
pvc/hub-data labelled recurring-job-group.longhorn.io/default: disabled, with
backup-daily and backup-weekly the only recurring jobs and both on the default
group — so the hub database has no volume-level backup, and it holds every box's
break-glass root password plus the escrow custody records. Filed, not fixed:
whether the exclusion is deliberate is an operator question.
The session report is REPORT-r172-hub-wal.md, not REPORT.md, per the
parallel-session rule — REPORT.md belongs to the controller session that ran
immediately before this one.
It also records, plainly, that a 60-concurrent load test I ran OOM-killed the hub
pod three times against a 256Mi limit. Not the WAL change, and not a test I
should have run against a Tier-2 box; the unit tests already proved the property.
store.New opened the DB with `?_journal_mode=WAL&_busy_timeout=5000`, which is
mattn/go-sqlite3 syntax. The driver is modernc.org/sqlite, whose applyQueryParams
reads only _pragma/_time_format/_time_integer_format/_txlock/_inttotime and
IGNORES anything else WITHOUT AN ERROR. So the hub ran in rollback-journal mode
with busy_timeout=0 for its entire life while its own source said otherwise.
Surfaced as a false HOST STALE banner: in rollback-journal mode a reader excludes
a writer, so rendering an operator page blocks a host report; the hub 500s, the
agent waits its full 15-minute interval without retrying, and staleness fires at
30 minutes — two collisions is a false alarm plus an operator email. 13 collisions
in one pod lifetime; the alarm fired twice on 2026-08-02 for a host that was up
two days and reconciling throughout.
The observable that proved it: a 128 MB /data/hub.db with no -wal/-shm beside it
while the DB was open.
Fix: ?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate.
_txlock=immediate is not optional — database/sql's Begin() is DEFERRED, so a
read-then-write tx must upgrade its lock and a failed upgrade is
SQLITE_BUSY_SNAPSHOT, which busy_timeout does NOT retry; this store has 10+
db.Begin() sites and they are all write paths.
Every test asserts what the DATABASE reports, never the DSN string — a string
test would have passed for the whole life of the bug. Red-proof: restoring the
shipped DSN reproduces journal_mode="delete", the missing -wal, and the live
"database is locked (5) (SQLITE_BUSY)".
Operational consequence handled: a WAL DB cannot be copied by taking hub.db
alone — a bare `cat` opens cleanly and silently omits the newest writes. The
break-glass retrieval in operations/nodes.md used exactly that; it and the
recovery-inventory note are now WAL-aware.
The hub's /data/hub.db (128 MB) is in rollback-journal mode, not WAL, so a UI
render can block a report write; the hub returns 500 on SQLITE_BUSY without
retrying, and the agent waits its full 15-minute interval rather than retrying.
Staleness fires at 30 minutes, so two consecutive collisions produce a false
host_stale and an operator email for a healthy host. Observed twice on
2026-08-02 while the agent was up 2 days and reconciling throughout.
Pre-existing: 13 collisions in one pod lifetime, first ~3h before that day's
controller work, though a burst of restarts amplifies it.
Controller v0.190.0. Docs only here; no hub change, no hub version bump.
- audits/DIAG-bootrecon-drive-absent-2026-08-02.md — NEW. The Part 0 diagnosis,
including the run that produced a FALSE NEGATIVE and the mechanism behind it
(the agent re-binds an unmounted drive within ~60s, so the drive gate's startup
reconcile restarted the apps one second before the sweep looked). Records that
the write hazard was blocked only by an ACCIDENTAL filesystem permission that no
code owns and no test pins.
- architecture/02 §0a — the boot-recovery contract (S-1): both gates read desired
state; the sweep observes a SETTLED fleet and each sample must refresh first;
nothing is started without asking, fail-safe. Plus the durable warning:
Manager.StartStack has no gate of its own.
- 00-capability-map — the boot-recovery row, with the repeat count cited per N.5
(6 of 6 hard resets) rather than a bare PROVEN-LIVE.
- OPEN-ITEMS / ROADMAP — R-157 CLOSED (both mechanisms), R-170 CLOSED, R-171 NEW
and closed the same session, marked a regression from v0.189.0.
- STATUS.md — the power-cut line moved from "What's broken" to "What works right
now" with its repeat count; one dated bullet in the change log.
- CONTEXT.md S-13 — the lessons worth carrying: "it didn't happen this time" is
not a disproof; widening a window makes previously-unreachable overlaps
reachable; and a settle detector is only as good as the freshness of what it
samples — the fix's own defect, found live rather than by review.
Controller v0.189.0 implements operator decision D-b. Docs only here; no hub
change and no hub version bump.
- architecture/02-controller-module-map.md §0a — NEW, and it is the S-1 contract:
desired (app.yaml) / in-flight (own marker file) / observed (not persisted),
with the rule that ties them — never derive one from another. Absent desired
state means UNKNOWN, never "running". One file, one writer. D-b's binding
safety rule quoted verbatim.
- 00-capability-map.md — the boot-recovery row now rests on a recorded signal,
with the three live flows from 9201. The interrupted-operation half is marked
IMPLEMENTED, not PROVEN-LIVE: nobody killed the controller mid-backup on metal.
- OPEN-ITEMS/ROADMAP — R-166 SHIPPED with both blocking facts and their answers;
R-157 mechanism B CLOSED and A restated as the whole item; R-170 NEW (the
drive-backed boot gate still infers a Stop from a container count).
- STATUS.md — the "an app can stay switched off and nothing says so" line
rewritten to what is actually left: timing.
- CLAUDE.md — end-of-session checklist gains: confirm your own last push's CI run
went green, BY RUN ID. The failure email is a push signal; this is the pull check.
- CONTEXT.md S-12 — the rulings, and the two lessons worth carrying: a test that
constructs the thing it should prove the caller constructs is hollow (its
red-proof will say so), and a field-by-field struct rebuild in a save path is a
defect on sight.
SPIKE-ci-runner-2026-08-02.md: all six probes with method, measurement and ruling; none
STOPped. P2 (stock image has git but no python3) and P6 (a runner that loses its state
re-registers and orphans the old record) changed the design; P5 (a failed run signals
NOTHING) is why the alarm exists at all.
R-168 SHIPPED with its evidence. R-29 CLOSED — on the demonstrated alarm and not on a green
run, as required: the class it opened is answered at both ends, the hook refusing locally and
CI catching a --no-verify bypass and emailing. R-161 noted: its automatic half now exists for
the STATIC gate, while its original scope, the runtime gate, is deliberately still not
automatic and should stay that way.
NEW R-169 (grep established R-168 was the highest in use): CI can only report, because there
is no gate in the road. Making it blocking needs branch protection plus a PR workflow, both
of which change how the operator works — so it is theirs to decide, and the row states the
cost honestly rather than recommending it.
CONTEXT gains S-8 (CI detects, does not block, and why that is structural), S-9 (a detector
that tells no one is not finished, plus the curl and Cloudflare-1010 traps), S-10 (the runner
is unprivileged because DooPlex is Tier 2), S-11 (CI reproduces the sibling layout).
CLAUDE.md gains the rule earned by red-proofing: a go test -run pattern that matches no test
prints ok and exits 0, and an instrument that can silently drop results is not a measurement.
The deliberate hostInstallVersion const is removed. It existed only to produce a real red
run (#3-#6) and the demonstrated alarm; R-94's deletion stands.
Resend sits behind Cloudflare, which blocks the default 'Python-urllib/3.x' agent with its
own 403 (error 1010). That failure looks exactly like an auth failure and is not one, so
the reason is recorded next to the header. Verified from the runner image with a
deliberately invalid payload: with the agent set, Resend answers 422 missing-field, i.e.
the request now reaches the API.
The first version died on 'curl: command not found' — the runner image carries python3 and
git and nothing else on purpose. Reaching for a bigger image to send one HTTP request would
have been the wrong trade, so the step uses urllib. Verified from the image itself that
HTTPS to api.resend.com resolves and the certificate verifies.
The step also fails LOUDLY on an empty key or a non-2xx from Resend: a silent alarm is
worse than no alarm, because it reads as coverage.
P5 measured: a failed run produces NO mail, NO notification row and NO log line from Gitea.
A red tick in a web UI nobody watches is exactly the defect R-29 filed, rebuilt one layer
up, so the run alarms itself on the project's existing transactional path (Resend, the same
one the hub uses) and prints the provider's accepted id, making 'it was sent' an observable
rather than an assumption.
The key is a user-level Gitea Actions secret created out-of-band; it is in no committed
file. The recipient is the operator address the hub already uses and is not a secret.
This push is deliberately made while main is still carrying the Scenario B breakage, so the
resulting run fails and demonstrates the alarm end to end.
Replaces the Part 0 probe workflow, whose four measurements are recorded in
documentation/audits/SPIKE-ci-runner-2026-08-02.md.
Reports, does not refuse: pushes go straight to main with no pull request, so there is no
merge for a status check to stand at. The refusing half is .githooks/pre-push, which is
per-clone and --no-verify-able; this half notices when that was skipped.
No uses: step anywhere — JavaScript actions need a node runtime the host-mode runner does
not have. Probe P3 measured that a shallow git fetch of the exact pushed SHA from the
in-cluster Gitea service is sufficient, and that it equals the pushed commit.
The alarm step is deliberately absent until probe P5 measures whether Gitea already mails
on a failed run.
TEMPORARY. Deleted before the session ends. Measures whether a registered runner picks up
a job at all, whether python3 and git are visible to the JOB (not merely present in the
image), whether the source can be obtained with no JavaScript action, and that docker is
NOT reachable from a job.
REPORT overwritten per the standing rule; every red-proof, the core.hooksPath probe's four
measured outcomes, Scenario C's refusal-and-bypass, the hub deployment and the live Setup-tab
read are recorded there, plus three observations and two deliberate deviations from the spec
(a comment-only edit to felhom-host-install.sh, and __pycache__ in .gitignore).
STATUS: the 'check that needs a person to remember it' line is rewritten — the checks now run
themselves before every push, with both honest limits stated in plain words; and one entry
records the thirteen-check cleanup and the deleted installer version number.
hub/CHANGELOG v0.87.0 + scripts/CHANGELOG gate-enforcement entry. CONTEXT gains S-6 (the
hub renders no host-install version and the gate pins its absence) and S-7 (gates run from
one entry point per repo; reuse_refs_check was fixed rather than the REUSE.md convention,
with both rejected alternatives recorded).
OPEN-ITEMS: R-94 CLOSED all three legs, leg (a) by DELETION with its reason; R-29 leg (a)
CLOSED and leg (b) HALF-SHIPPED with the census result written into the row (13 gates; every
gate a CLAUDE.md names was green, two of the four unnamed were red); R-161 gains its
successor pointer. NEW R-168 (grep established R-167 was the highest in use): Gitea Actions
runner — measured 2026-08-02 as Gitea 1.26.2, Actions enabled on all four repos, 0 runners,
0 workflow runs, 0 branch protections, and the consequence that trunk-based direct-to-main
pushes leave no merge for a status check to gate, so CI here can detect but not block.
BLOCKED on a spike over host-mode vs privileged DinD on DooPlex and whether the workflow can
avoid JavaScript actions.
ROADMAP: R-94 collapsed to its one-liner, R-29 updated, R-168 added.
A census of all thirteen gate scripts across the four felhom repos on 2026-08-02 found one
clean correlation: every check a CLAUDE.md tells a person to run was passing, and two of the
four nobody is told to run were failing — one since 14 July. Neither failure was harmful in
effect (checked line by line); nothing would have said so if they had been. The fix is not
more gates, it is one place to run them from.
repo_gates.py runs site + hostinstall + hub-confirm + manifest-bearer + reuse-refs, streams
each gate's own output, and exits worst-wins non-zero. A missing gate script is a FAILURE and
prints the path tried — fail-closed, because a runner that quietly skips a gate is the
inert-seam failure this project has shipped four times. It copies catalog_gates.py (R-161),
NOT site_gates.py, which is a gate and not a runner.
.githooks/pre-push runs it with --fast and refuses the push. Honest limits are written into
the hook itself: per-clone (core.hooksPath is local config), and --no-verify bypasses it on
purpose. Any manual run WARNS when the clone is unarmed. Measured on git 2.47.3: a relative
core.hooksPath resolves correctly and the hook's cwd is the repo root from any subdirectory.
test_repo_gates.py is a SEAM test — it asserts each member gate's own distinctive stdout, not
the runner's summary line, which an inert runner prints while calling nothing. Red-proofed:
replacing run_gate's body with 'return 0' still prints 'all felhom.eu gates OK' and exits 0,
and turns the seam test red.
RED on all four repos with 13 findings, and a hand audit of all 13 on 2026-08-02 found
ZERO genuine drift: twelve were package shorthand whose file sits a couple of directories
deeper, and one (wgsync/reconciler.go, cited by the controller) lives in the hub. REUSE.md
cites by package shorthand and across repos on purpose; the tool was what was wrong.
Resolution order, first hit wins, every non-exact hit PRINTED so a weakening is visible:
exact / suffix / ambiguous (real citation, imprecise shorthand — not a failure) / sibling
repo (as-is or with the sibling's own name stripped from the token) / FAIL. A failure lists
every resolution attempted, so a 'not found' claim names what was tried. Per-root tallies
are the positive observable: '0 failures' alone cannot tell a working checker from a blind
one. Evidence trees (audits/, documentation/tests/) are excluded from the suffix index — a
copy of a file is not the file. An absent sibling is never a failure; an unreadable parent
says so and continues.
Result: 13/13 resolve, all four roots exit 0. felhom.eu 60 exact + 1 suffix; controller 126
exact + 6 suffix + 1 cross-repo; agent 88 + 1 + 1; catalog 17 exact + 3 cross-repo.
New scripts/test_reuse_refs_check.py: 13 fixture tests, one per resolution row plus the kill
condition. Red-proof: making resolve() return 'exact' for an unresolvable token turns 4 of
them red.
The Setup tab said 'host-install 1.19.0' while the served script was 1.22.0, and had
been wrong since 2026-07-14. Deriving the number honestly is not possible: the Option-1
command downloads felhom-host-install.sh from the website at RUN TIME and the website
git-syncs main every 30s (R-110), so no build-time value in the hub can be true. R-94(a)
offered derive-or-delete; deleted, which removes the drift class instead of automating it.
- configs.go: hostInstallVersion const, pageData.ScriptVersion field and its assignment
all removed; a NOTE in their place records why there is no constant here.
- customer_unified.html: the sentence now says the command always fetches the current
installer, and renders no version.
- hostinstall_gates.py gate 1: the third assertion INVERTS — it used to require the hub
const to equal SCRIPT_VERSION, it now asserts the hub carries no host-install version
literal at all, matched in six code shapes across every .go/.html under hub/ (comments
are deliberately not stripped: a // inside a URL literal would blind the scan).
- render_test.go: the assertion 'html contains hostInstallVersion' compared the constant
to itself and passed at ANY value — demonstrated green with the const at 9.9.9 while the
script was 1.22.0. Deleted, not replaced: there is no longer a version to assert.
- felhom-host-install.sh: COMMENT ONLY (SCRIPT_VERSION untouched) — it claimed the gate
keeps the hub copy equal, an invariant that no longer exists.
Red-proofs: restoring the const fails the rewritten gate 1 (3 shapes hit); the old
render_test assertion passes at 9.9.9.
Documentation only — no code, no box, no build.
STATUS.md (repo root, 652 words / 67 lines): what works · what's broken ·
what we're working on · waiting on you · changed since. A VIEW of
OPEN-ITEMS.md, holding nothing of its own; not CONTEXT.md, and both files
now say why they stay separate. No R-n is the subject of a sentence —
identifiers are bracketed pointers only.
CONTEXT.md S-5 records the four operator decisions taken 2026-08-02
(D-a … D-d), none of them implemented:
D-a merge mp1 into mp0 rather than resize it — before any external
install, and D-c ships in the same step → R-165
D-b desired/observed app state in its own store, with the state-store
safety rule verbatim → R-166 (BLOCKED)
D-c customer fill warning + operator backup-failure alert → R-167
D-d only DooPlex and Peti's box are protected → target-selection.md
R-163 RE-FRAMED, not closed: the sizing question is withdrawn rather than
answered; the row survives as the record of the constraint until R-165
lands. R-156's papra referral RESOLVED — deployed nowhere, so the template
fix strands nothing; the docker ps evidence is recorded with its
provenance and its scope limit.
target-selection.md: two protected machines, everything else disposable.
ep0 is no longer Tier 2 but is not scratch (it holds the only off-premises
copy of real customer data) — flagged for explicit operator confirmation.
The demo-box backup-target fence drops from prohibition to stated cost,
because D-d spends that reference anyway.
CLAUDE.md gains an End-of-session checklist carrying the STATUS.md
maintenance rule and "a finding goes in OPEN-ITEMS.md first".
The operator ruled on R-161 and the runner shipped in app-catalog-felhom.eu
(fd7747d), so the row moves from BLOCKED-needs-a-ruling to REDUCED SCOPE - open.
Both obvious enforcement points were rejected for measured reasons, and the row
now records them rather than leaving the rejection implicit. Controller-side at
template load: rejected because such a check can only read the file, and a static
audit of all 53 templates reports the catalog clean INCLUDING papra - it would
pass on the exact defect it exists to catch, the property being decidable only at
runtime. CI: rejected for now, neither repo has any and there are no users yet.
Shipped instead: scripts/catalog_gates.py, one entry point over all three gates,
non-zero exit on any failure, mandated in the catalog's CLAUDE.md the way
site_gates.py is. The rationale is recorded because it is the transferable part -
of this project's gates, the only ones that ever get run are those with a single
entry point named in a CLAUDE.md; site_gates.py is run and R-29's three orphans
are named nowhere and have stopped nothing.
What stays open is only the automatic half, which is sufficient while ONE person
touches templates - revisit when a second does.
Re-ranked accordingly: R-161 drops from 2nd to 7th, and R-156 is promoted to 2nd,
since R-161 was ranked high precisely because nothing ran the gate and that is no
longer true. The de-ranking is recorded inline with its reason, matching how R-94's
de-ranking is recorded, so a later reader sees a decision rather than drift.
Nine rows into OPEN-ITEMS.md and ROADMAP.md, matching each file's column shape.
R-156 and R-157 had lived only in audit documents - the identical "minted in a
spike doc and never carried across" failure the register already records for
R-153/R-154/R-155, caught by the catalog sweep's own section 8.0 while it was
happening. R-158 was minted by a second session the same day for an unrelated
finding, which is why the sweep's proposals were renumbered R-159..R-162 at filing
time. All nine IDs verified free in BOTH backlog files before use.
Part 0 settled the question the sizing item depended on, by reading:
mp1 is RETENTION, not staging, and neither of the two framings was right. A unit
is the KEPT copy on the app's OWN drive (backup.go:245-255); for an app with no
HDD_PATH the namespace falls back to the system SSD - "the SSD-only system-data
fallback" (appbackup/paths.go:26-27). There is no post-copy deletion: the only
prune is F5 residue-on-old-drives when an app MOVES (backup.go:1053-1112). So mp1
retains the units of driveless apps only - not every app, but not transient
either. Confirmed against the spike: sys_drive held exactly the four driveless
apps and not calibre-web, which had a drive and was still backed up.
A unit is volume tars + DB dumps only, never mp8 userdata
(recovery_unit.go:20-25), so a 1 TB photo library can never overflow one. And mp1
gates the WHOLE chain, not just Tier 1: Tier-2 mirrors the unit "(always)" from
RecoveryUnitPath (tier2.go:302,368) and Tier-3 carries it, so a unit that cannot
be written leaves both with nothing to copy.
Part 2 fired on both triggers - retention, and the fallback undocumented - so
07-backup-architecture.md gains section 7.5. Section 6.1 said a unit lives "on the
app's own drive", which is true and was the whole story only for drive-resident
apps; the no-drive case was undocumented, as was the sizing constraint. 7.5
records the mp0-50G-vs-mp1-20G mismatch, the measured ratios (DB app up to ~2x,
21.1GB -> 40.2GB; file-only 1.00x), and the bound this puts on D5's Lane-1
independence: restorable from the drive alone only while the unit still fits -
about 19 GB file-only, about 10 GB DB-backed. No number proposed; the ratio is the
operator's ruling (R-163).
R-159/R-160 marked SHIPPED only after verifying the template changes are in
app-catalog origin/main, and R-156's gate likewise (check-volume-persistence.py
present). papra is NOT fixed - referred - so R-156 stays open on that one app.
Ranked, with one line of reasoning each: R-157 first (an app can stay down
indefinitely with mechanism B silent on every channel), then R-161 (the gate
exists and nothing runs it, which is why R-156's class recurs - R-29's record is
three orphaned gates and one enforced), R-156, R-163, R-158, R-164, R-162.
Evidence-survival check FIRST: HEAD == origin/main == 7efb7a5, and every artefact
confirmed present in origin/main with git cat-file, including all 27 evidence
files. Nothing died with the rig.
1. VM 311 destroyed by exact VMID. qm list now shows only drill-r50; 311.conf does
not exist; images/311 gone. 126 GB returned - nvme 133G used / 758G avail ->
6.7G / 884G. A first attempt stopped the VM but its destroy never ran, killed by
an unescaped paren in my own echo; caught by checking qm list rather than the
exit code.
2. c10-scratch removed by name. storage.cfg 5 -> 4 entries. felhom-backup shares
the same path and is intact, config unchanged.
3. PBS on DooPlex (Tier 2). Identity confirmed first - the target carried the
comment this campaign set at creation. Datastore felhom-c10, user c10@pbs,
token !box, both ACLs and the on-disk chunkstore all gone; felhom-spike,
felhom@pbs, felhom@pbs!n100 and /mnt/5_hdd/backup/demo-felhom all intact. Only
the campaign's entries disappeared. "user delete" is not a PBS subcommand - it
printed usage and the user survived rc=0; caught by re-reading the user list.
4. Restic subaccount 281530 deleted after asserting home==felhom-campaign10 and
the description names Campaign 10 - the script aborts otherwise. 4 -> 3
subaccounts; demo-felhom, peti-felhom and demo-hp all present.
5. Hub customer c10-soak. The ONLINE gate REFUSED with HTTP 409 for 18 consecutive
attempts, exactly as anticipated (customer_delete.go:98-102, :146). Predicted
deletable at last-report 09:38:19 + 30 min = 10:08:19; measured 303 at 10:08:23.
Verified positively: 0 customer rows, 0 host/appliance rows, GET
/customers/c10-soak -> 404, and the hub logged the DELETE cascade COMPLETE with
residue purged. No /appliances/discard was needed - the appliance record was
bound and went with the cascade. drill-r50 verified INTACT (200, VM 300 present).
All 12 credential files shredded and the directory removed.
Register rows NOT written: app-catalog-felhom.eu has an uncommitted working tree
including scripts/check-volume-persistence.py, so the parallel catalog session is
still open and owns the register. R-156/R-157/R-158 rows and rankings remain owed,
itemised in the doc - and that session's volume-persistence checker looks like
R-156's fix shape, so the two should be reconciled before filing.
Q1 - what the customer sees when a backup refuses for lack of space. The failure
IS customer-visible: /backups renders "Adatmentés sikertelen" with a cross mark.
It is absent from the dashboard, the launcher, the app detail page, and - the one
worth fixing - from /backups/apps, the per-app page where you would naturally ask
whether a given app is backed up.
Point 5 measured across three runs: it retries, stays failed while constrained
(marker persists, unit mtime unchanged at 07:30:50), and clears on recovery with a
fresh unit at 07:37:38. /backups/apps reading "Utolsó: 3 perce" tracks the unit's
REAL mtime, not the failed run, so it is honest about the age of the last good
unit rather than claiming a fresh one. Explicitly NOT the R-156 family.
So R-158 is a NOTIFICATION GAP, not a silent-failure defect, and ranks BELOW
R-157 - whose mechanism B leaves a deployed app not running while deadapp reports
"0 currently down", silent on every channel.
Q2 - ValidateDump was right and no bad dumps are shipping. The live DB genuinely
had zero accounts (only _prisma_migrations 129, cc_proof 82, instance_settings 1).
An empty table proves nothing, so an account was SEEDED as the task required: the
warning then stopped entirely and the dump provably contained the rows (c10acct 1,
c10user 2; 102766 -> 103029 bytes).
But that kills C2's proposed ordering. A fresh appliance legitimately has zero
accounts, so gating on "accounts has rows" would block the backups of every new
customer until someone registers. The validator's fact is right; its inference
("may predate the customer's data") is wrong - there was no data to predate. The
chain is therefore longer: a sound predicate first (compare the dump against the
LIVE db, per-table counts, not an absolute expectation), then warn->gate, then the
tar-drop. Until then the DB volume tar stays load-bearing - not because dumps are
bad, but because nothing can yet prove one is good.
No new R-n; register grepped. Nothing fixed. Part 2 (teardown) follows.
Three headline answers.
1. The ceiling is REAL and on mp1 (/mnt/sys_drive), but its shape is a MISMATCH
rather than a single number. A1: docker's data-root is a SEPARATE 50G volume
(mp0) and every app volume resolves there, so app DBs are NOT on sys_drive -
build-golden.sh:68's "like the Docker-data" reading is correct. A2: the
recovery units ARE on sys_drive, which the golden ships at 20G. So a box
permits 50 GB of live app data while capping local backup at 20 GB, and
crossing that line is invisible until a backup fails.
A3 rules out the lab-default explanation: --sysdata-grow defaults to 0
(main.go:178) and is not computed from the drive. demo-hp's REAL guest 9201
runs a bare --config ExecStart and shows mp0 50G / mp1 20G; agent.json has no
sizing keys at all. A4, measured not read: restore extracts IN PLACE on the
docker volume - sys_drive avail was 799.2M before and after a restore run under
constraint - so the constrained mount is written only during backup.
2. Overflow behaves WELL. With sys_drive ballasted to 799 MB, backup refused
per-app ("No space left on device"), other apps continued, status reported
success=false, and the "last good dump preserved" claim VERIFIED byte-for-byte:
size and md5 unchanged, tar valid end-to-end, no .tmp residue. Restoring that
preserved unit under the same constraint returned correct data and claimed
success honestly. Explicitly NOT the R-156 family.
3. But it is SILENT - R-158, filed. Zero events reached the hub.
NotifyBackupFailed exists and the hub allowlists backup_failed, but the only
production caller is the off-box/NAS leg (main.go:659); the backup manager has
tier2/offbox/offbox-enlarge notify seams and none for the local recovery-unit
capture. This is R-97's shipped defect exactly one tier over, and the fifth
instance of "seam built but never wired" - a pattern the codebase names in its
own R-97 wiring test.
Sizing rule corrected: unit ~= volume-tar bytes + logical dump bytes, not a
constant 1.90x. Measured C1: file-only apps are 1.00x (homebox 2305->2305 MB, no
db-dumps dir at all), and the SAME DB app with an empty DB is also 1.00x. So a 20G
sys_drive holds ~19 GB file-only or ~10 GB DB-backed. That is the bound on D5's
Lane-1 independence.
C2: both representations are used for a reason stated in code (F17 - the dump is
authoritative and WINS over the tar; R-47 - replayed with only the DB service up).
The dump is single-database pg_dump --no-owner, so a fresh initdb plus the dump is
logically sufficient and the tar is a PHYSICAL FALLBACK. Dropping it would halve
DB-app units and also close the D5/R-127(b) password trap (restored PGDATA makes
postgres skip initdb and ignore POSTGRES_PASSWORD) - but only after ValidateDump
is promoted from a warning to a gate, since it currently WARNS on a dump whose
accounts table has no rows. In its present form the tar is load-bearing.
No production code, no template change. Teardown still owed and itemised.
The S figures (66 MB -> 42.0s, two passes agreeing to 0.6s) had a spread tight
enough to prove fixed work dominates, which is exactly why they said nothing about
M. Second point taken 327x larger, same app, same method: clock from restore
request to the app serving the correct discriminator.
rallly's postgres volume grown 66 MB -> 21.1 GB (200k rows, STORAGE EXTERNAL so
TOAST cannot compress it into a fake number). Two reps:
rep 1 backup 406.4s unit 41149 MB RTO 624.5s discriminator correct
rep 2 backup 387.2s unit 41133 MB RTO 591.8s discriminator correct
327x the data cost 14.5x the time - strongly sub-linear:
RTO ~= 40s + 26.9 s/GB backup ~= 29s + 17.4 s/GB
10 GB -> 5.2 min 20 GB -> 9.6 min (measured 10.1) 100 GB -> 46 min
The fixed ~40s dominates below ~1.5 GB, which IS the S band and explains its tight
clustering.
The more consequential result is capacity. A DB-backed app's recovery unit is
1.90x its data (volume tar PLUS SQL dump): 21.1 GB produced a 40.2 GB unit. The
default appliance ships /mnt/sys_drive at 20 GB, so the largest app that can hold
a local Tier-1/2 recovery unit on a default box is about 10 GB - and that fills the
volume. The M band does not fit on a default box at all; this test only reached
21 GB because sys_drive was first grown 20G -> 70G with the same operation the
product performs via SysDataGrowGB. A tier-sizing decision, not a defect, but it
is invisible until an app crosses it.
Caveats stated in the doc: two points define a line but do not test linearity; the
1.90x is DB-app-specific and a file-only app should be nearer 1.0x (inferred, not
measured); synthetic incompressible data; one app, one box.
Phase B completed in three passes: run 1 (27 cycles, 6 atom families, 0
violations), run 2a (10 cycles, stopped deliberately - two violations were harness
defects), run 2b (39 cycles, 12 of the brief's ~13 atom families). 1461 invariant
checks. Depth reached 39 consecutive cycles, past the brief's "drift at the
thirty-eighth", with c34-c39 clean on every invariant.
I7 headline: 66 restores across both passes, 66 correct discriminators - never
stale, never empty. I2/I3/I4/I5/I6/I10/I11 zero violations in either pass.
I1-under-load 5/5: the target pulled WHILE a backup ran still produced
backup_target_absent and a clean recovery. R-117's Q7 case holds - a filesystem
aborted in place surfaces and the gate stops the app on the dead namespace.
RTO Tier-1 rallly 66MB: run 1 median 42.0s, run 2b median 41.4s over 38 restores -
two independent passes agreeing to 0.6s. S band's lower end only; nothing
extrapolates to M or L. RPO not measured.
Monotonic growth, 9457 samples of 19 metrics over 13.5h: NO leak. Controller and
agent RSS flat, fds flat, no orphaned volumes/images/containers despite dozens of
redeploys, kills, reboots and hard resets. Only curve with real slope is the agent
journal at ~20MB/h, bounded by journald.
Findings: R-156 (papra's data neither persisted nor backed up, reports healthy)
and R-157 (bootrecon's start-once sweep, two mechanisms - the zero-container one
is silent on every channel). Four suspicions investigated and DISPROVED, each
recorded with what settled it.
Both halves of the disposition were run and neither survived as a finding.
The I1/I1-pair violations cluster at cycles 31-33 and nowhere else across 39
cycles; c34-c39 are clean, so it recovered with no intervention. Final tally I1
37 PASS / 2 VIOLATION, I1-pair 36 PASS / 3 VIOLATION. On the quiesced box one slow
detach with 4 minutes either side produced a perfect pair. And the alarming
false-healthy (mentes bound=False while degraded=false) does not survive
quiescence - I had been reading the two halves at different instants of a detach.
No R-n.
Separately cleared: the hub's SQLITE_BUSY event drops. 7 in 24h including one for
the real customer demo-felhom, and the hub does return 500 with notification
dispatch only after a successful save - so a lost event would be a lost alarm. But
the controller retries 3 times and ZERO events exhausted their attempts; the
07:04:39 drop landed at 07:04:42. Nothing lost. Only cosmetic note: the ERROR line
reads like data loss and is not.
The 4th hard-reset failure had a different signature, verified not assumed: all of
rallly healthy, papra missing entirely with state=stopped deployed=True
containers=0. Zero containers is exactly what bootrecon deliberately never touches,
because the UI's Stop is compose down which removes containers - but a hard reset
landing during a compose operation produces the identical state. The signature the
safety rule depends on cannot distinguish the two.
Worse: in that state the deadapp check reported 0 currently down while a deployed
app was not running. No app_start_failed, no banner. That is the workspace's own
false-invariant #4 (F-CRIT-1, StateStopped assumed deliberate) recurring through a
hard reset rather than quiesce. NOT filed as new - CLAUDE.md already records it -
but confirmed live on 0.188.0 via a new path.
papra returned after ~15 min, later than the harness's 10-min window, so this
instance was slow rather than permanent and the doc says so. What restarted it is
not established.
Mechanism A (Exited, missed by the unsettled snapshot) alarms; mechanism B (zero
containers) is invisible on every channel. A settle-condition fix closes A only.
Three I1/I1-pair violations in ~5 minutes, all "expected event absent". Recorded as
an OPEN observation, NOT a finding: the system was mid-abuse when it was seen, and
a verdict taken on a system being hammered is worth little.
Established: it is not hub-side suppression and not a truncated log. The hub pod
has 0 restarts over 43h and the controller's own log matches it line for line, so
the events were never emitted. It is specific to the backup_target_* pair - the
generic storage_disconnected/reconnected pair for the other drive kept firing
normally throughout the same window.
Also sampled, and the more serious half if it survives quiescence: mentes reads
bound_under_parent=False while the backup-target state simultaneously reports
degraded=false. Those cannot both be right - a false healthy on the backup target
is I5/I6's failure mode.
NOT established: whether the pair recovers once cycling stops (the harness detaches
every ~2 min; a customer does not), whether the 02:25:37 controller restart is
implicated, and whether the degraded=false sample was transient.
Disposition written into the doc: after the run ends, quiesce with both drives
attached, then do ONE slow detach/reattach and see whether the pair fires. That
distinguishes "does not survive rapid cycling" from "the target alarm has silently
stopped working", which would be severe.
The first write-up said R-157 reproduced "at the same cycle in both runs -
deterministic, not a coincidence". Wrong. The cycle numbers matched only because
the runner's RNG is seeded so both runs drew the same permutation. The failure
itself is a coin flip: run 2b's four hard resets went PASS(c2), FAIL(c10),
PASS(c18), FAIL(c26); run 2a went PASS(c2), FAIL(c10). Three failures in six.
The correction matters because it changes what kind of bug this is, and it
strengthens rather than weakens the root cause: intermittency is exactly what a
race against container-state settling predicts, whereas a wrong predicate would
fail every time.
Signature is identical on all three occurrences: rallly Exited 255 with
rallly-postgres healthy, bootrecon reporting "no boot-orphaned apps" about 5s
after controller start, and the container count still churning after the sweep
(third occurrence 01:05: refresh 8, bootrecon 01:05:13, then 8 -> 7 -> 8).
Reproduced twice, two independent runs, same cycle (the runner's RNG is seeded so
both drew the same permutation - deterministic, not coincidence).
A hard reset mid-backup brought everything back except the app half of the
DB-backed stack: rallly left Exited 255, oom=false, restarts=0, its own log ending
"Ready" - it died healthy - while rallly-postgres returned healthy.
20:28:13 Status refresh: 8 containers across 55 stacks <-- docker ps -a shows NINE
20:28:18 [bootrecon] Boot reconciliation: no boot-orphaned apps
20:28:25 Status refresh: 7 ... 8 containers <-- still churning AFTER the sweep
20:39:14 [deadapp] 20 scans, 5 deployed evaluated, 1 currently down
The predicate is sound: once settled the controller reports rallly state=degraded
containers=2, and IsDownState includes StateDegraded, so len>0 && IsDownState
holds. The SNAPSHOT was wrong. bootrecon fires as a goroutine ~5s after start
while docker is still restoring containers, and is start-once by design, so it
never re-checks.
Consequence: the app stays down indefinitely. Detection is perfect and recovery
never happens - R-52's original shape, an alarm with no recovery. Not fixed.
Distinguished from this campaign's two earlier HARNESS defects: both drives bound,
every other app returned incl. the drive-backed one, only the app half of a
two-container stack missing while its DB is healthy, and it surfaced through the
fixed check written for exactly this.
The previous commit message was truncated by an unescaped paren in the shell, so
the fix detail and the product observations were lost from the record. This adds
them as evidence, where they belong.
Covers: the cc_proof table showing no C010-A row at all (the seed never landed);
both harness defects; why an ambiguous I7 justified stopping a 10-cycle run; the
red-proofed controls; and two transient product observations recorded but NOT
filed as findings - the health probe naming the DB container on the app's port for
about 70s during recovery, and a ValidateDump WARN on a dump taken while the app
was down.
Run 2a hit its first two violations at cycle 10 and BOTH trace to my harness, not
the product. Recorded in full because a check that fails for the wrong reason is
as corrosive as one that passes for the wrong reason.
HARD-RESET VM returned=True canaries_intact=False
I7 want=C10-C010-A-194530 got=C10-C009-A-192929 restore_ok=True
Root cause, evidenced: the cc_proof table's highest row is C10-C009-A — there is
NO C010-A row at all, so the seed never landed. The hard-reset atom ran earlier in
the same cycle and left rallly Exited(255); atom_restore_verify called seed() and
never checked its return value, so an unwritten generation became a fake stale
I1-I11 are CORRECTNESS invariants: they answer 'is the system telling the truth
this cycle'. All 586 of them passed in run 1 while nothing at all watched whether
disk usage, snapshot count, log volume, fd count or RSS climbs. Accumulation is
exactly what depth was for, and it was missing from the invariant list.
Adds c10growth.py (Campaign 2's controller_rss.tsv precedent, widened to 19
metrics) sampling every 90s as a SEPARATE process, so the in-flight run 2 did not
have to be restarted. Attributes every sample to a cycle by reading the runner's
status.txt, and records NA rather than dying when the box is down during a
hard-reset or reboot atom.
c10growth_report.py turns it into Campaign 2's table shape (start/end/min/max/
slope-per-cycle) and splits verdicts by class: growth in RSS/fd/volumes/images/
restarts is a LEAK; growth in backup storage or the qcow2 is expected
accumulation, reported with a projection to cycle 45.
Caught a bug in the sampler itself on the first analysis: MENTES_USED_MB appeared
to jump 623 -> 5667 MB, which is exactly ROOT_USED_MB — when a drive is detached,
/mnt/<name> reverts to a plain directory on root and df silently reports the ROOT
filesystem. The same class of error as the agent's exactMount check, in the
measurement code. Gated on mountpoint and red-proofed both ways: a real mount
returns a number, a non-mount returns NA.
The case R-117's spike called the worse half — a drive dying with no detach/return
cycle, which before agent v0.117.0 emitted nothing on any channel indefinitely.
Box runs 0.119.0. Aborted ext4 in place (abort,emergency_ro; device still present):
bound_under_parent went false, storage_disconnected fired, the storage page named
the stopped app, and calibre-web (whose library binds that drive) was STOPPED
rather than restarted onto the dead namespace. Recovery needed a full device close,
not a remount — exactly as the fix intends (BindAborted => quiet no-op).
Runner extended with the 7 atom families run 1 skipped: abort-fs-in-place,
kill-agent-mid-backup, hard-reset-VM-mid-write, reboot-VM, concurrent
backup+restore, concurrent backup+detach, fill-drive-near-full. Also fixes a run-1
flaw recorded in the audit: reboot was appended AFTER the shuffle so it never
interleaved with a detach; heavy atoms are now permuted in with the rest.
Run-1 evidence preserved as *-run1.* (cycle numbering restarts per run).
Ran the soak on the Phase A rig. Ended on its own deadline — no watchdog halt,
no atom exception, no I11 breach.
I1 28+28 pairs, I2 28+28 pairs, I3 56, I4 56, I5/I6 28 each, I7 28, I10 135,
I11 28. Zero violations. The row counts are themselves the no-silent-skip check:
I3/I4 twice per cycle (both drives), I10 = 5 secret-class fields x 27, REBOOT on
cycles 7/14/21 only.
I7 is the headline: 28 restores, 28 correct discriminators — never stale, never
empty. RTO (Tier 1, rallly, 66 MB): min 38.8s, median 42.0s, p90 42.5s, max
44.3s. That is the S band's lower end ONLY; the 5.5s spread over 28 runs says
fixed work dominates, so nothing extrapolates to M or L. RPO not measured.
Every atom and invariant was proven BY HAND before automation — the runner
asserts nothing that was not first observed live.
Caught a Phase A gap before starting: no app had HDD_PATH, so all data sat on the
system disk and I3 could never have fired. Deployed calibre-web onto adatok
first; otherwise the run would have produced 27 green cycles that tested nothing
cross-drive.
Investigated and DISPROVED a suspected defect (audit 5.2): /api/disks reports
state=attached for a physically absent drive, and intermediary.go:230 really does
compute presence from State=="attached". It is inert — planDriveGates only gates
paths under /mnt/felhom-drives/ and uses BoundUnderParent there, which was
correctly false. The gate fired; the storage page showed "Meghajtó leválasztva".
No R-n minted.
Honest gaps: 6 of ~12 atom families ran. Not run — Tier 3 (structurally
un-isolatable), abort-fs-in-place, kill-agent-mid-backup, hard-reset-mid-write,
reboot-VM, both concurrency atoms, fill-drive-near-full. I8 not checked, I9 not
automated (cited from the tester-gate run, not re-claimed). kill_controller is
NOT mid-backup and reboot_guest never interleaved with a detach. 27 cycles does
not answer the brief's question about drift at the thirty-eighth.
Teardown still OWED, including hub customer c10-soak (disposition: DELETE).
Phase A passed every gate on a fresh box built from the PUBLISHED ISO 1.26.1:
install, claim, two drives enrolled through the real endpoints with the backup
target healthy, four apps spanning both sides of D5's secret split, and a
working discriminator across all four.
Isolation gate: both denials captured, each with a positive control. The PBS
control FAILED first — four clean-looking 403s were worthless because the token
was denied on its own datastore too (PBS token privilege separation). Fixed and
re-run; the denials stand.
R-156 (new, register grepped): papra's data is neither persisted nor backed up,
and it reports healthy. The template mounts papra_data:/app/data; the app writes
/app/app-data/db/db.sqlite. Volume empty and root-owned against a -rootless
image, real DB in the container writable layer, healthcheck only probes the HTTP
port. Its Tier-1/2 backup is real, verifiable and contains nothing. Not fixed.
Tier 3 could not be isolated so it was not run: offsite hard-requires the DR
tier (configs.go:1300) and the DR tier only provisions on ep0 (per-endpoint
allocation deferred, hub/README.md:260). Both are recorded deliberate positions,
so no R-n minted. The campaign touched neither ep0 nor the Storage Box.
Phase B did not start. Phase A was budgeted at ~1h and took ~5.5h (1.26.1 is a
public release image with no auto-install path, so the install was a blind
screendump+sendkey walk). That left the runner — which judges eleven invariants
and fires destructive atoms unattended — to be written at 04:00 with ~3h of
night left. Stopped on the brief's own fence: a rig producing false negatives is
worse than no rig. The rig is built and idle; teardown is OWED and itemised,
including hub customer c10-soak (disposition: DELETE).
PART 1 — the release gate.
G7 required the packaged .deb to sha256-match the one built from committed source. That is
unsatisfiable BY CONSTRUCTION: dpkg-deb stamps the build time into every archive, so two builds of
byte-identical source differ. It was already failing when the 1.26.1 release ran it. A criterion
nobody can satisfy gets waived once and read as advisory ever after — which is how R-29's shelf of
never-run gates was built. Sub-clause dropped, reason recorded in G7's own note the way G6's
amendment was, so a future reader can restore it if SOURCE_DATE_EPOCH ever makes it meaningful.
RULING ASKED FOR — is payload integrity covered by G9 alone? NO, and G9 is widened rather than a new
criterion invented. The package ships TWO payload files (build-deb.sh:54-55); G9 checked only the
script. The systemd UNIT was covered by nothing: G7 covered the container, G8 covers the postinst
behaviourally, G13 covers directory presence. The unit is not incidental — its After=, its
ConditionPathExists= and its Restart= decide WHEN AND WHETHER day-0 runs at all, so a drifted unit
would have shipped silently. Same shape as the /etc/felhom miss that G13 exists to prevent: a check
that proved the thing present and said nothing about what it depended on. The check passes today.
G13 moved to sit after G12 — it was minted late and left between G10 and G11.
PART 2 — register dispositions. BASELINE DISCREPANCY, reported rather than worked around: only R-128
had a row. R-154 and R-155 had NO row in either file — minted in a spike document and never carried
across, which is R-123's class, not the drift the task described. Rows created, closed, with the
reasoning, because in all three cases the reasoning is the durable part:
R-128 closed by CORRECTING a false claim, not by making the assertion real — the coupling does not
exist and asserting it would invent a constraint. Flagged so nobody 'restores' it.
R-154 closed with the measurement and where it now lives in pushed source.
R-155 NARROWED, not deleted — unchanged for FELHOM_MENU=single, inapplicable to release. Flagged so
the guard is not later removed wholesale on the strength of 'R-155 closed it'.
Documentation only: no code, no build, no ISO, no upload, no box touched.
Live: https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso
sha256 f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6, 1705322496 bytes.
PART 5 PASSED ON BOTH MENU ENTRIES, four observables each:
Graphical spikegfx.felhom.eu pairing code J7N-2DA
TerminalUI spikesix.felhom.eu pairing code ZY5-YY4
Both: manual install, own disk, own password, real completion signal, and the journal's
'not bound yet — polling every 30s ... normal waiting state, not an error'. Spike 4 had REASONED the
graphical path follows from shared Install.pm; it is now measured.
PART 6: G1-G10 + G13 all PASS against the uploaded file. G4's single hit is
felhom-bootstrap.sh:480's substring TEST ('$envtext' != *FELHOM_RETRIEVAL_PASSPHRASE=*), not a
value — my own regex matched the glob's asterisk.
PART 7: uploaded via rclone in a container configured ENTIRELY by environment variables, so no
credential file was ever written. Round trip verified from the public URL — not the local file.
Bucket stays private: unauthenticated GET to the S3 endpoint 400, custom domain has no index (404).
CORRECTED BEFORE UPLOAD: the generated manifest described a single automated entry with a 5s timeout
and listed Graphical/Terminal UI as 'menu-removed'. Generator fixed, sidecar regenerated, and the ISO
verified byte-identical before and after — the published file IS the file Part 5 validated.
Hub-side cleared: appliances 16, 17, 18 discarded (303 each); zero rows remain. The endpoint is
/appliances/<id>/discard, POST only (server.go:345) — not /delete.
Teardown: VMs purged, spike5 storage removed, demo-hp back to 6.6G, drill-r50 and 9201 untouched.
Still open and named: OPEN-ITEMS/ROADMAP dispositions for R-128/R-154/R-155 are not written; the
.deb is not byte-reproducible (G7 sub-clause); before-network stub unreached; Secure Boot and real
hardware not exercised.
The 1.26.1 manifest — the file a tester reads to know what they have, and which is published
alongside the ISO — carried four statements that were false for a release build:
boot-menu 'single entry Felhom telepítés, default, 5s' -> it has TWO, timeout 15
menu-entries '1 (... timeout 5s)' -> 2
menu-removed 'Graphical, Terminal UI, ...' -> those are exactly what it SHIPS
kernel-line '... proxmox-start-auto-installer' -> the release menu deliberately has none
secret-bearing 'no (embeds the customer retrieval passphrase...)' -> self-contradictory
All four came from branding/pairing notes that predate --release and were emitted unconditionally.
A public artifact whose own manifest misdescribes it is the false-claim class this arc exists to
correct, so it is fixed before publication rather than after.
iso 1.26.1, sha256 f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6.
Terminal-UI interactive install, host spikesix.felhom.eu — all four observables PASS:
1 package installed ii felhom-bootstrap 1.26.1
2 unit enabled enabled
3 unit fired first boot activating; 'registering unclaimed appliance at the hub'
4 box wants a claim code /etc/felhom/appliance-pairing-code = ZY5-YY4, token 64B mode 600,
'not bound yet — polling every 30s ... normal waiting state, not an error'
That is the product working end-to-end from a public image on a manual install.
G13 added and RED-PROOFED (removing install -d -> exit 3 with the G13 message; restoring -> green).
The first red-proof attempt was INVALID — a copied script failed on a missing control file, i.e.
non-zero for the wrong reason — and was redone in place.
NOT PUBLISHED: Part 5 requires BOTH entries. The Graphical entry reached the Target-Harddisk screen
but was not driven to completion (Enter lands in the Country field; monitor mouse_move does not move
the guest cursor), so Part 5 is not fully passed and Part 7 did not run.
TWO HUB ITEMS OUTSTANDING and NOT disposed of: unclaimed appliances 16 and 17. GET/POST on
/appliances, /appliances/17 and /appliances/17/delete all 404; the rows appear only inside /hosts,
which offers 'Bind & deliver' and no delete affordance. Stated at the top of the report too, because
R-131 is four orphaned objects that recorded commands never cleared.
Also recorded: 'qm set --scsi0 ... --boot' silently yields net0;ide2, and ide2-first sends a finished
install back into the installer — both made a COMPLETED install look like a stuck one.
FIX for the Part-5 failure. felhom-bootstrap.sh writes the appliance token (:431), the pairing code
(:435) and .bootstrap-done into /etc/felhom/. The old stub-first-boot.sh created it explicitly
('install -d -m 0755 /etc/felhom /usr/local/sbin'); packaging dropped the env FILE correctly and the
DIRECTORY with it. Measured consequence on a real interactive install: the box registered at the hub,
could not persist its token, and polled 'HTTP 401 — still retrying' forever with no claim code.
- build-deb.sh now ships ./etc/felhom/ (0755, empty) and ASSERTS it, plus ./usr/local/sbin/ and
./lib/systemd/system/, as G13. RED-PROOFED: removing the install -d makes the build exit 3 with
'is not in the package (G13)', and restoring it goes green.
- The gate gains G13 with the reasoning: G7/G8/G9 all passed on the broken package. G9 proves the
payload is the right payload and says NOTHING about what the payload depends on.
ISO_VERSION -> 1.26.1.
The Terminal-UI interactive install ran to completion from the release image and gave 3 of 4
required observables:
1 package installed PASS ii felhom-bootstrap 1.26.0
2 unit enabled PASS wants-symlink present; postinst enabled it from the chroot
3 unit FIRED first boot PASS journal shows PAIRING mode, registering unclaimed appliance
4 box wants a claim code FAIL
/etc/felhom/ does not exist on the installed system, so felhom-bootstrap.sh cannot write the
appliance token (:431) or the pairing code (:435), and the hub poll then 401s forever. The box can
never finish pairing and the customer never sees a claim code.
ROOT CAUSE, mine: stub-first-boot.sh opened with 'install -d -m 0755 /etc/felhom /usr/local/sbin'.
This task correctly dropped the env FILE from the package and dropped the DIRECTORY with it.
felhom-bootstrap.sh uses /etc/felhom for its runtime state (token, pairing code, .bootstrap-done).
WHY THE GATE MISSED IT: G9 proves the packaged script is byte-identical to HEAD, and it is. I
verified the payload files and never the directory the payload writes into — a check that proves
the thing present and not the thing it depends on. Added as G13.
The fix is one line and is deliberately NOT applied: a failing Part 5 stops the task, and proving a
fix needs both installs re-run.
Also recorded: 'qm set --scsi0 ... --boot order=scsi0;ide2' silently yields boot: order=net0;ide2,
so a COMPLETED install looked like a machine sitting in the installer. Set --boot separately.
Nothing uploaded; R2 credentials never read. Teardown complete: VMs purged, spike5 storage removed,
demo-hp back to 6.6G, drill-r50 and 9201 untouched. Hub-side: no appliance object was created
(searched /, /hosts, /configs for the hostname — zero hits), so R-131 gains no row.
Publication is gated on Part 5 (two interactive installs proving delivery end-to-end). Neither was
carried to completion, so nothing was uploaded. Per the task: stopping is the good outcome.
BUILT: felhom-installer-1.26.0-pve9.2-1.iso
sha256 24977bafd24d73262745fc1b3040939469c9b23b87ead927a8af86de73044a90, 1705322496 bytes.
GATE (Part 6) against that exact file: G1-G10 PASS, G11/G12 not run (nothing published).
G5 by ENUMERATION vs the stock PVE ISO: exactly four added paths — three felhomtheme files and
/proxmox/packages/felhom-bootstrap_1.26.0_all.deb. G9: the packaged felhom-bootstrap.sh is
byte-identical to repo HEAD. No .rootpw.txt is emitted at all, which is G2's own evidence.
PROVEN in Part 5 before stopping: the image boots to the branded TWO-ENTRY release menu and the
Terminal-UI entry reaches the stock PVE installer. NOT proven: package installed, unit enabled,
unit fired, box asking for a claim code — on either entry. The Graphical entry was never driven.
Reporting partial observables would be the LastRun-class error this arc has corrected three times.
R2 credentials were never read, never used, never echoed; no rclone/aws config was created.
Teardown complete: VMs purged, scratch storage spike5 removed, demo-hp back to 6.6G, drill-r50 and
9201 untouched, hub-side nothing created (verified by fetching the customer list).
admin