64 Commits

Author SHA1 Message Date
admin 51871a7ea6 installer 1.25.0: the off-site tier stops asking to prune (R-191)
gates / gates (push) Successful in 8s
Every weekly off-site run uploaded successfully and then failed the job on a prune
the box's token is deliberately refused — R-89 moved off-site pruning server-side
to ep0 and box tokens stay write-only. The 2026-07-26 'two weeks' ruling was not
reversed; where it is enforced moved, and keep_last: 2 did not follow. Now 0,
which the agent's existing guard already reads as 'never prune from the box'.

Verified read-only on ep0 before changing it: both namespaces have a prune job at
03:30 keep-last 2 that has run every day since 2026-07-27 — 18 tasks, all OK, the
newest keeping exactly two. Without that check this would have traded a weekly
false alarm for unbounded growth.

A gate asserts the offsite tier carries no client-side prune. The local tier is
untouched.
2026-08-04 09:46:09 +02:00
admin 33f0ab2f33 R-193: demo-hp offsite restored on the box; escrow correction recorded
gates / gates (push) Successful in 7s
Re-issue through the designed endpoint: fresh password staged 07:11:51, settle-gate
GO, consumed 15s later at 07:12:06, offsite configured at 07:12:09 against the SAME
sub-account (275124) and repo path. Both boxes now escrowed.

CORRECTION to this session's own reading: the escrow did NOT recover by itself.
escrow_state went pending -> escrowed 15s after the apply and I inferred automation;
the operator had run the ceremony. It needed a human on both boxes — demo-hp at
07:16:02, demo-felhom at 07:20:28 (its escrow had been pending since its own rebuild
the day before). A 15-second state change is not evidence of automation, which is the
same class as reading an absent log line as success.

Snapshot continuity is NOT established: both boxes report snapshot_count 0, but the
run-history keys are ABSENT rather than zeroed — a controller that has never run an
offbox backup, not an empty repo. The 04:15 run decides it.
2026-08-04 09:23:08 +02:00
admin 2584dfb938 R-193: a guest rebuild silently drops the offsite tier; R-192 cause established
gates / gates (push) Successful in 8s
Operator confirms no hub-side offsite config change, so the regression was not an
action. Evidence: demo-hp's controller went 0.187.0 -> 0.192.0 at 06:12:18 with a
new config hash and the agent re-keyed its leaf three minutes earlier — a guest
rebuild. The last pre-rebuild report shows the tier fully healthy: escrowed, last
success 02:16:39Z, 15 snapshots, 40.9 MB. No offsite object in the 108 reports
since.

Mechanism: the restic credential is delivered once. demo-hp consumed its secret on
2026-07-23; the rebuilt controller has no copy and no way to request another.
demo-felhom survived the SAME rebuild only because its secret was still unconsumed
— it consumed it four seconds after its config hash changed and was reporting
offsite again 76 seconds later. That difference was luck, not design.

Also sharpens R-192: the self-heal's guard refuses when any report since the
consume carried an offbox target, but that query reads the OLDEST 500 reports —
all of which predate the rebuild. Healthy history before a rebuild is not evidence
the credential still works, which is why the automation that exists for this case
declined to act.
2026-08-04 09:05:53 +02:00
admin 185661062a STATUS: back to one screen
gates / gates (push) Successful in 7s
2026-08-04 08:58:04 +02:00
admin 967ddf2f44 R-192: offsite_delivery_stuck states the opposite of what it measured
gates / gates (push) Successful in 7s
demo-hp has been emailing daily since 2026-08-03 06:12 UTC that '500 reports
since carry no offbox target — the credential is likely burned'. Measured against
the hub's own data: all 500 of those reports DO carry one. The credential was
consumed 2026-07-23 and applied successfully; the controller reported an offsite
object continuously until 2026-08-03 05:59:21, then it vanished at 06:12:19 and
has been absent for 108 reports. A regressed apply, not a burn.

Two defects. maybeEmitStuck builds its message from ReportsSinceConsume while
hardcoding 'carry no offbox target' and never consults OffsiteReportsSinceConsume,
so the recommended action targets a failure that did not occur. And maybeHeal
refuses SILENTLY on exactly that field ('regressed-apply shape -> operator's
call', a bare return), so there is a daily email with the wrong story, no heal,
and nothing saying why the heal declined.

The underlying condition is real: that box currently has no off-site copy of its
customer app data, since 08:12 CEST on 2026-08-03. What removed it is not
established.
2026-08-04 08:57:53 +02:00
admin d692c46db0 RUNBOOK: the host tier is restore-PROVEN unattended on both demo boxes
gates / gates (push) Successful in 7s
Four SCHEDULED runs, none triggered by hand: demo-felhom host 83.8s / offsite
540.4s; demo-hp host 109.3s / offsite 300.1s. Each restored into a scratch guest,
booted, verified and destroyed itself; zero 990000 guests or volumes afterwards
and both local-lvm figures returned to their pre-run values.

Both boxes had BOTH tiers due at once, so R-86's ordering was observed live for
the first time: never-proven sorted first, each box took its HOST tier, deferred
the offsite one, and picked it up on the next evaluation six hours later. The
host-tier proofs reached the hub through R-189's merge — demo-felhom's report
carries two tiers, and the local one can only have come from disk.

The capability map's optimistic half is cashed, with its scope stated: these two
boxes, not the fleet.

Surfaced and filed rather than fixed:
- R-190: a storage ACL that demonstrably worked at 04:44 was gone by 09:24, with
  a reinstall, any logged pveum activity and any cluster-log entry ruled out.
- R-191: every weekly offsite backup uploads successfully and then fails the job
  on a prune the box is deliberately not allowed to do (R-89 moved it
  server-side; both boxes still arm keep_last=2).

Two corrections to yesterday's record: the R-185 drift DID surface as 403s on the
write path (six, with the hub raising whole_guest_backup_failed at the first), and
my earlier "no restore_test_* events" was produced by grepping a 404 page.
2026-08-04 08:13:43 +02:00
admin b93ee06abc R-190 filed; two corrections to yesterday's R-185 record
gates / gates (push) Successful in 8s
CORRECTION 1 — the runbook annotation and the R-185 row both said the drift did
not surface as a 403 because writes go through a root path. That is WRONG.
demo-felhom's local-api backup jobs 403'd six times between 09:24 and 17:34 CEST
on exactly that storage and privilege, and the hub raised whole_guest_backup_failed
at the first with edge-triggering suppressing the rest. The impact was not only an
unreadable tier: the agent's own whole-guest backups to it were failing.

CORRECTION 2 — on this box the grant was LOST, not never issued. A vzdump by the
agent's token to that storage completed OK at 04:44:50 the same morning; the first
403 is 09:24:56. Ruled out by measurement: a host reinstall (uptime 12 days), any
pveum/ACL/user.cfg activity in syslog 04:00-10:00, any ACL entry in the cluster
log. Correlated but not established: guest 9201 was reprovisioned nine minutes
before the first failure.

R-190 files the unexplained disappearance, and notes that the new store-grant
probe detects the STATE but says nothing about the TRANSITION.
2026-08-03 20:11:09 +02:00
admin e3187c86d5 docs: R-185 closed — the silence as well as the grant
gates / gates (push) Successful in 8s
- OPEN-ITEMS: R-185 closed with the measurement, the corrected root cause (the
  installer's Scenario-F reuse arm, not PVE_STORAGES), and the live sequence.
  Records that demo-hp carried the same drift and was fixed too.
- capability map: the whole-guest row's HOST-tier half was OPTIMISTIC and now
  says so — that tier was not merely unproven, it was unprovable on both demo
  boxes, and every live proof cited was on the offsite tier.
- vzdump-target-move runbook: its item 5 predicted this; annotated (not
  rewritten) with what actually happened — the create arm did grant, the reuse
  arm did not, and it surfaced as a silent unreadable tier rather than the 403
  the item expected, because vzdump writes through a root path.
- CONTEXT: S-21 (an empty listing cannot distinguish forbidden from newborn; the
  measured trap that an ungranted path answers with INHERITED privileges) and
  S-22 (the Scenario-F arm must finish the job).
- STATUS: rewritten for the operator, back to one screen.
2026-08-03 19:02:25 +02:00
admin 311dc06c13 manifests: /scripts/ syncs installer-v1.23.0 -> installer-v1.24.0 (R-185)
gates / gates (push) Successful in 8s
Both refs — the git-sync sidecar and its init container. A fresh pod must not
serve a different installer from a running one.
2026-08-03 18:59:05 +02:00
admin 688470c945 installer 1.24.0: a PRE-EXISTING backup target is granted too (R-185)
gates / gates (push) Successful in 7s
configure_backup_target has two arms and only one granted. Case A creates the
storage and grants in the same breath; the Scenario-F arm ('the target already
exists') returned without granting. A box whose felhom-backup pre-dated the
install therefore pointed local_backup_target at a storage its own token could
not read — measured on BOTH demo boxes: {"data":[]} through the token while root
lists three archives. That tier was never restore-tested and nothing said so,
because an empty listing is also what a brand-new tier returns.

The reuse arm now ensures the ACL through the same guarded wrapper. Scenario F is
unviolated: the storage DEFINITION is untouched, and pveum acl modify is
idempotent. BACKUP_TARGET_ID is deliberately NOT added to PVE_STORAGES — that
list is granted a step before the target is resolved, and --acl-storages entries
are preflight-checked for existence; the comment now says so.

A gate asserts it: every arm that resolves the target must also grant on it.
Red-proved by reverting the arm.
2026-08-03 18:58:49 +02:00
admin 7a5694341d REPORT: point at the agent report for the later 2026-08-03 session (docs-only here)
gates / gates (push) Successful in 9s
2026-08-03 16:49:17 +02:00
admin 81d4438630 docs: R-189, R-188 and R-186 closed (agent v0.122.0); R-185 stays open
gates / gates (push) Successful in 8s
- OPEN-ITEMS: all three rows closed with their measurements. R-189 records the
  migration consequence seen live (a pre-R-189 record has no tier and is not
  reportable, so upgrading does not retroactively make an old proof visible).
  R-188 records both directions demonstrated and the fixture cleaned up. R-186
  carries the before/after sha values including the CGO_ENABLED discrepancy.
  The priority list states explicitly that R-185 is the one left open from that
  group and that this session did not touch it.
- capability map: the restore-proof row now says the evidence path itself had a
  gap — the proof reached the hub only because no restart intervened — and what
  closed it.
- CONTEXT: S-19 (a proof is durable and reportable, a failure is neither, and
  why) and S-20 (the release ordering, what each step protects, and how a binary
  is verified).
- STATUS: rewritten for the operator, back to one screen.
2026-08-03 16:46:49 +02:00
admin c9a3e48b21 R-189: a passing restore-test can be invisible to the hub — observed live
gates / gates (push) Successful in 8s
The agent reports restore_tests[] from an IN-MEMORY store ('lost on restart; the
cadence re-populates'), and the hub report interval is 900s. A restart inside
that window drops the result. That used to self-heal within 24h because the next
cadence re-tested the tier; under per-archive due-ness the agent will NOT re-test
a proven archive, so the hub can stay ignorant until the next archive generation
— a week on the offsite tier.

Observed today: the 15:25:14 PASS on felhom-pbs (a real 14.5 GB offsite restore)
reached no host-report at all, because the agent was restarted 2m43s later for
the v0.121.1 deploy.

The persisted proof already exists — RestoreTestState.Snapshot() even carries the
comment 'for the host-report gauge' and has NO production caller: a seam built and
never wired, and an invariant asserted in a comment with no test, in one method.
2026-08-03 15:35:17 +02:00
admin 0779c5db85 REUSE: the restore-proven window is per tier, and observation may only widen it (R-86)
gates / gates (push) Successful in 7s
2026-08-03 15:33:19 +02:00
admin 0f8f084817 R-188: a correct agent release emails a CI failure ~50% of the time
gates / gates (push) Successful in 8s
on: [push] fires the gates workflow on the TAG push too, and release-agent.sh
pushes the tag before publishing (deliberately). CI can therefore run the
published-versions gate inside the window where the tag exists and the package
does not, and correctly reports 'every released agent version must be
INSTALLABLE' for a release that completes seconds later.

Measured across two releases in one session: v0.121.0 -> runs 12/13 (success,
failure) on the same sha; v0.121.1 -> runs 17/18 (failure, success) on the same
sha; and one pair both green — a race, not a rule.

It matters because R-168 made CI email on failure so a red gate cannot be missed.
A signal that cries wolf on every second correct release is how that mail becomes
something you archive unread.
2026-08-03 15:29:59 +02:00
admin c102832892 docs: R-86 closed and proven live; ep0 recorded as protected; R-185/186/187 filed
gates / gates (push) Successful in 8s
- OPEN-ITEMS: R-86 CLOSED with the trap in its own wording recorded (the literal
  reading is never true on a daily tier); R-87 re-ranked UP because R-86 built
  most of what it waited for; R-185 (the agent cannot list demo-felhom's host
  backup tier — a missing storage ACL, pre-existing), R-186 (a released binary's
  sha is not reproducible from its tag), R-187 (R-115's publish leg had never
  actually run) filed. R-184 was the highest ID in use.
- ROADMAP: R-86 collapsed, keeping the reasoning and correcting the shape the row
  itself proposed — which would have been the never-fires version.
- 07-backup-architecture: new contract section — restore-testing is per ARCHIVE
  GENERATION, with the trap and what did not change (S-1).
- 00-capability-map: the unattended restore-proof row upgraded to PROVEN-LIVE on
  the 635 s due-triggered offsite run, with the restart and teardown evidence.
- CONTEXT: S-17 (the rule, the trap, the config key, the hub's derivation) and
  S-18 (ep0 is Tier 2 — extends D-d's protected list to three machines).
  Numbered 17/18 because S-14 and S-15 were already duplicated in the file.
- STATUS: rewritten for the operator, trimmed back to one screen.
2026-08-03 15:28:32 +02:00
admin ff2655cf19 manifests: hub 0.91.0 -> 0.91.1 (R-86: observation may only widen a tier's window)
gates / gates (push) Successful in 7s
2026-08-03 15:18:00 +02:00
admin 046df303b6 hub v0.91.1 — observation may only WIDEN a tier's window, never tighten it (R-86)
gates / gates (push) Successful in 7s
Found by checking v0.91.0 against the live box, not by review. demo-felhom's two
retained PBS snapshots sit 8h54m apart (one is a healing artefact), so the
mean-gap estimator reads a WEEKLY tier as nine-hourly: x4 = 36h, the 7-day floor
lifts it to 168h, and a weekly tier proved weekly reaches ~8.25d of proof age.
The false alarm this task exists to prevent would have returned within a week, on
the box it had just shipped to.

restoreProvenWindow now takes max(observed, declared). A gap SHORTER than the
declared rhythm is routine and means nothing (a retry, a manual run, a heal, a
catch-up); a gap LONGER than it is real information. Cost stated: a tier running
faster than its declared rhythm gets a slower stale signal — the right direction
for a signal that means 'unverified', since 'broken now' is a different event.
2026-08-03 15:16:59 +02:00
admin 687fedd8ee manifests: hub 0.90.1 -> 0.91.0 (R-86 Part 2, per-tier restore-proven window)
gates / gates (push) Successful in 7s
2026-08-03 15:07:52 +02:00
admin 323f45a5ef hub v0.91.0 — the staleness window learns each tier's own rhythm (R-86 Part 2)
gates / gates (push) Successful in 7s
Ships WITH agent v0.121.0, not after it. The agent now proves a tier once per
ARCHIVE GENERATION, so a weekly tier is proved weekly — in perfect health. The
flat 7-day restoreProvenStaleAfter derived its number from the 24h cadence R-86
removes, and a healthy weekly tier's proof age reaches EXACTLY 168h just before
its next proof: it sat ON the line, so any ordinary delay tipped it into a
nightly alarm about a working system.

restoreProvenWindow(tier, observed, ok):
- the tier's own archive interval, OBSERVED from reports the hub already holds
  (pbs_snapshots + successful backups attributed by TARGET TYPE, slice A.4)
- x4 generations = the same tolerance the flat constant expressed
- floored at 7d (never tighter than before), capped at 12d (strictly inside the
  2-week offsite retention)
- falls back to the DECLARED rhythm (26h host / 8d offsite — the thresholds the
  backup-freshness checker already uses) when history is too short to observe
  one; falling back to the FLOOR would recreate the false alarm on a fresh box

Kept: absence is UNKNOWN until the anchored window passes; the signal stays
edge-triggered; failed and stale remain distinct events. Every reason string now
states the window it was judged against (R-100's corollary).

Also backfills the missing v0.90.1 CHANGELOG entry (deployed since f21e7ca), and
records the operator's 2026-08-03 ruling that ep0 is Tier 2 / protected.
2026-08-03 15:03:35 +02:00
admin e34b614e5b docs: R-182 closed, R-90 closed on measurement, R-86 unblocked, ep0 record corrected
gates / gates (push) Successful in 7s
R-182 CLOSED (controller v0.194.0 + hub v0.90.0/.1), proven live on demo-hp. The
hub's notification_log for the run reads: two per-app failures RECORDED, one
digest SENT naming both, and the customer channel SKIPPED with operator_only.
Against the measured previous behaviour — two failures, one email naming one
app, one leaving no trace anywhere.

Scenario D proved itself on an event I had not planned: disk_critical alarmed on
two filesystems, the second was collapsed by the cooldown, and that collapse is
now visible WITH ITS KEY. Yesterday it would have left nothing at all.

A gap the spec did not anticipate is recorded with its fix: the per-app event
also fires from the periodic sweep, outside any run, so making it record-only
would have created a NEW silence. The sweep emits a digest too, with no run_id,
so it stays under the ordinary hourly cooldown.

ep0: MEASURED on the box — 7757 MB (8 GB), 4 vCPU, and the 4 GiB swapfile
SURVIVED the resize and is active (checked, because a resize is a stop/start).
The 40 GB local disk is UNCHANGED, so no disk figure was touched anywhere.

Five documents corrected — three of which the task's list did not name, found by
searching. Two audit/evidence documents ANNOTATED, body untouched: they record
what was true when written and that is their value.

R-90 CLOSED. R-86 unblocked and re-ranked, stated honestly: 8 GB is comfortable,
not unbounded — the original OOM was a 14.46 GB restore — so the restore-test
cadence should still be paced, just not by fear of the endpoint.

target-selection.md's "D-d did not name ep0 either way" is deliberately left
standing. It is the operator's question, not CC's.

STATUS.md 127 -> 83 lines, items rather than sentences.
2026-08-03 14:00:54 +02:00
admin f21e7caed1 hub v0.90.1 — the digest's per-app lines stop repeating the filesystem figures (R-182)
gates / gates (push) Successful in 7s
Found by reading the first REAL digest, not by design. Every app row ended with
the same usage clause the mail already prints once on its own Filesystem line.
On a two-app box that is untidy; down a list of a dozen it is the same forty
characters twelve times, pushing the part that DIFFERS off a phone screen at
07:00 — the only moment this mail has to work.

The reserve's refusal message is authored for a single-app alert where naming
the filesystem is right, so the message is unchanged; the digest trims the
duplicate when rendering. trimRepeatedUsage removes ONLY an exact
"— <target path>:" suffix, so an unrelated reason is untouched and a reason that
is nothing but the usage clause is left alone rather than emptied.

Also updates TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer, which
required the OPERATOR to be emailed a per-app capture failure. That was correct
when the event was the only signal and is wrong now that it is the record and
the digest is the notification. Its customer-safety claim is unchanged and is
why the test still exists; the operator assertion is inverted with the reasoning
written in place, and R-158's guarantee is shown to have MOVED, not weakened.
2026-08-03 13:54:02 +02:00
admin dd40f85bb8 hub v0.90.0 — a dropped notification leaves a trace, and the backup digest arrives (R-182)
gates / gates (push) Successful in 7s
processOperator's cooldown no longer returns bare. It dropped the event BEFORE
LogNotification, so a suppressed operator alert and an event that never happened
were indistinguishable — from the operator's side and from the hub's own records.
Measured 2026-08-03: nine recovery_unit_capture_failed events arrived, two were
mailed, seven left no row anywhere. That is why the defect took a day to get the
right way round: there was nothing to read.

A suppressed operator event now writes a `suppressed` row carrying the message
and the key that suppressed it. This applies to EVERY operator event, not only
the one that exposed it. It does NOT change the cooldown's duration or semantics.

backup_run_failures: the per-run digest. In allowedEventTypes AND in
operatorOnlyEvents — allowlisting alone does not make an event operator-only,
and FormatCustomerEmail falls back to the raw English message rather than
blocking. A test demonstrates a customer with the type enabled receiving nothing.

recordOnlyEvents: a third routing class — stored and recorded, never mailed.
recovery_unit_capture_failed moves here: it is the record, the digest is the
notification. A register rather than downgrading severity to info, which would
relabel a genuine failure as informational everywhere it is queried.

cooldownRunSuffix: a sibling of cooldownTierSuffix, not a branch inside it, so
tier keeps byte-identical semantics and R-97a's tests are untouched. It makes
the cooldown effectively inert for the digest, which is the intent — a digest is
already rate-limited by construction; the refresh sweep sends no run_id and so
stays under the ordinary hourly cooldown.

The email renders as a list, not a JSON blob. An absent space reading renders as
unavailable, never as zeros.
2026-08-03 13:46:48 +02:00
admin 7dc1744eec Merge branch 'main' of https://gitea.dooplex.hu/admin/felhom.eu
gates / gates (push) Successful in 7s
2026-08-03 13:14:17 +02:00
admin a5cd480280 reports deleted 2026-08-03 13:14:15 +02:00
admin b0b269b28d docs: R-110 + R-115 closed, R-182 re-scoped by measurement, R-183/R-184 filed
gates / gates (push) Successful in 7s
R-182's direction REVERSED by Part 0's measurement. Filed yesterday as "the
reserve re-alerts on every status refresh" — too many alerts, seen at the
sending end. Measured at the receiving end: 9 events received today, 2 operator
emails sent. When two apps are refused in the same second the operator is told
about ONE; the other is dropped before LogNotification, so it leaves no row on
any channel and cannot be audited. The operator cooldown key is
customerID:eventType(+tier) and the capture-failed event carries `app` but no
`tier`, so the key has no app identifier. Same failure mode as R-97a, in a
second event type that never opted into the narrow fix. Nothing changed —
Part 0 was investigation only.

Correction owed: yesterday's report said "one recovery_unit_capture_failed per
app, HTTP 200". True of what the CONTROLLER pushed; a reader would take it as
"the operator was told about each app", which is false.

R-110 CLOSED (installer v1.23.0). Both channels moved. The spec's mechanism for
channel 2 rested on a factual error — the run-time fetches are sixteen, not
nine, and come from felhom-agent, not this repo — so no tag here could cover
them; pinned to the agent version being installed instead, on the operator's
ruling. Channel 3 needed no change: the URL never carried a ref, so no hub
change and no hub bump.

R-115 CLOSED. release-agent.sh builds, tags, publishes and verifies by an
independent download; check-published-versions.py refuses a tag with no package;
CI now runs the full gate set so it actually runs.

R-183 NEW+CLOSED: a fresh install fetched the vouched agent binary and its
sixteen config files from two different refs, and nothing compared them.

R-184 NEW: nothing stops the hub vouching a version that was never released.
The R-115 gate cannot see it — measured, the hub manifest and Gitea's package
listing are both 401 anonymously.

capability map: new PROVEN-LIVE row for the published installer channel.
STATUS.md 138 -> 127 lines.
2026-08-03 12:44:08 +02:00
admin e79a20bbed test(R-110): remove the Scenario A marker — proof complete
gates / gates (push) Successful in 7s
Scenario A: pushed to main without moving the tag; the served bytes stayed
byte-identical (sha 2f859555…) and the marker never appeared, while the website
tree advanced to the new commit — both halves of the split in one observation.

Scenario B: moving the tag published it (~40s, sha ea2b4aa9…, marker present);
moving it back rolled it back to exactly the pre-publish sha with the marker
gone. The website returned 200 throughout.
2026-08-03 12:27:39 +02:00
admin 6a82719426 test(R-110): Scenario A marker pushed to main WITHOUT moving the tag
gates / gates (push) Successful in 8s
A comment only. It exists to be LOOKED FOR at the served URL: if it appears
there, /scripts/ is still tracking main and the publish channel is still
floating. It must not appear until installer-v1.23.0 is moved.
2026-08-03 12:24:22 +02:00
admin bee6848458 installer v1.23.0 — publishing becomes an act, not a side-effect (R-110, R-183)
gates / gates (push) Successful in 8s
Two channels moved off main in the same change, because either one left behind
makes the other cosmetic.

Channel 1 — the served script. webpage.yaml git-synced /scripts/ from
--branch=main every 30s and nginx served that tree, so pushing this file WAS
publishing it: within half a minute it was what every new machine downloaded and
ran as root, with no staging and no rollback but another push. The sync is now
SPLIT: the website keeps tracking main at the same cadence (a copy edit must
never need a release) and /scripts/ tracks the tag installer-v<SCRIPT_VERSION>.

PROVEN before the manifest was touched: git-sync v4.4.0 follows a tag AND
notices a MOVED one — measured on a throwaway sync against this repo,
"update required ... local:<old> remote:<new>" -> "updated successfully",
within one period. The moved-tag half is what the publish model rests on.

Channel 2 — the sixteen files fetched at run time. fetch_raw pulled from
$AGENT_REPO/raw/branch/main; it now pulls raw/tag/v$ART_AGENT_VER. That is a
correctness fix, not only a channel one (R-183): a fresh install fetched the
vouched agent BINARY while taking its unit file, sudoers and guarded wrappers
from whatever main held. Two refs, one install, nothing compared them. Their
correct ref was never SCRIPT_VERSION — they do not live in this repo.

No fallback to a branch: a vouched version whose tag is missing fails loudly
rather than quietly serving main.

Channel 3 — the URL — needed no change, recorded rather than left silent:
https://felhom.eu/scripts/felhom-host-install.sh never carried a ref, so both
producers follow the tag with no edit. No hub change, no hub version bump.

Gate 6 in hostinstall_gates.py pins all three structurally with no network, so
it stays in --fast and runs in CI. It deliberately does NOT assert "a tag exists
for the current SCRIPT_VERSION": that would go red on the very push that bumps
the version, before publishing — and publishing being separate is the ruling.
2026-08-03 12:08:37 +02:00
admin 8360f940bf docs: seventh row in the shipped-guarantees table (R-181), versioned workspace CLAUDE.md
gates / gates (push) Successful in 7s
Syncs documentation/runbooks/workspace-CLAUDE.md with the workspace root file.
Row 7: the B2 refusal claimed the previous unit was untouched; nothing-deleted
held, untouched was measured false.

Also records WHY it survived review: it passed a full green suite AND three of
its own red-proofs, because every one of them asserted the mechanism inside
captureAllRecoveryUnits and none asserted the consequence across the whole
backup run. The test that would have caught it is the one the fix ships —
fingerprint the tree before and after, and compare.
2026-08-03 11:36:47 +02:00
admin fb652024ea docs: R-181 closed, R-156 closed, R-110 + R-115 rulings recorded, R-182 filed
gates / gates (push) Successful in 7s
R-181 CLOSED (controller v0.193.0 + v0.193.1) and proven live on demo-hp for
BOTH reserve terms. The reserve is now a per-app, per-run ADMISSION decision
taken before the app's first write and covering all three write legs, and it
gained a size term. The refusal's wording was not weakened; the behaviour moved
so it became true, verified by sha256 tree fingerprint.

R-156 CLOSED — papra's template mounts the app's own data root. Precondition
re-measured rather than inherited (both boxes were wiped today).

Part 4, documentation only, nothing built:
- R-110 WAITING-ON-OPERATOR -> READY. Ruling: option (b), the installer's
  publish channel moves to a TAG. Recorded with the condition that decides
  whether it works at all — it must cover BOTH the /scripts/ git-sync and the
  nine files the installer fetches from raw/branch/main.
- R-115 WAITING-ON-OPERATOR -> READY. Ruling: mechanism (b), a build-side gate
  refusing to deploy or vouch an unpublished version. The third instance (agent
  v0.120.0) would have silently downgraded both demo boxes while succeeding.

R-182 NEW: the periodic status refresh has no admission scope, so a refused app
re-alerts on every poll (measured: a second alert pair 13s after the run's).
Pre-existing in v0.192.0; deliberately not fixed in the R-181 task.

capability map: the local-backup row moves to PROVEN-LIVE in BOTH halves.
ROADMAP: R-165 collapses to CLOSED; R-181 collapsed into it.
07-backup-architecture.md: the reserve's contract stated as what the code
provides (S-1 — an architectural contract changed in the same session).
STATUS.md trimmed 150 -> 111 lines, "What's broken" no longer holds shipped
work, and the stale "After:" line (pointing at work that shipped on 2 August)
is fixed.
2026-08-03 11:36:16 +02:00
admin 06cbf8df29 skill(build-deploy): the installer ISO — the one artifact the skill promised and omitted
gates / gates (push) Successful in 7s
The skill's own description claimed 'ANY Felhom artifact' and 'publish', and had no ISO section —
a description asserting coverage that did not exist. Description corrected and a section added.

POINTERS, NOT COPIES. The 13-criterion release gate stays in
documentation/runbooks/iso-release-gate.md and the measurements stay in the four spike audits;
duplicating them into a skill guarantees drift (the R-94/R-128 class). What the skill adds is the
ROUTING that was missing: nothing told anyone the gate exists, which is R-29's exact shape.

Records the two modes (--release public vs --pairing appliance) because picking the wrong one ships
the wrong product, the build and publish commands (rclone env-only, so no credential file is ever
written), and the round-trip verification.

The traps it carries existed only in commit messages until now, and each cost a wrong diagnosis:
  - 'qm set --scsi0 ... --boot order=' in ONE call silently yields boot: order=net0;ide2
  - after install the CD must be detached, or a COMPLETED install looks exactly like a stuck one
  - verify focus by screendump before every Enter (GTK Enter lands in fields, not Next)
  - proof installs register appliances; the verb is POST /appliances/<id>/discard, not /delete
  - scratch storage at the /mnt/nvme-1tb ROOT (a subdirectory reads disconnected forever)

Also flags that the ISO gate is NOT wired into repo_gates.py, so nothing reminds you to run it.

Docs only. python3 scripts/repo_gates.py --fast: all gates OK (rc=0).
2026-08-03 10:54:33 +02:00
admin 6b5d64c1fa REPORT: CI run ids and conclusions for both commits (46, 47 — both success)
gates / gates (push) Successful in 8s
2026-08-03 09:35:16 +02:00
admin aa62449694 R-178 CLOSED: both demo boxes reinstalled from the merged golden and proven
gates / gates (push) Successful in 8s
Two boxes, two DIFFERENT supply paths, so the session proved the disk shape and
the delivery route rather than one of them twice.

demo-hp (layout proof, --golden <local volid>): mp0 at /var/lib/felhom,
backup=1, 70G, no mp1; /var/lib/docker and /mnt/sys_drive both real mounts of
its subdirectories via fstab; one df figure and one device id (64519) on all
three paths; reboots 3/3 with the binds surviving each.

demo-felhom (pipeline proof, --force-gitea-golden): fetch_verify succeeding
against the vouched manifest for BOTH artifacts -- 'verified sha256
54e2a4c431daf580... matches the hub manifest' for the golden, a7763d31... for
the agent. 250G single volume, grep -c '^mp1:' = 0, reboots 3/3.

Journey proven on both, endpoint-level: claim -> deploy -> back up -> restore,
with a planted marker returning byte-identical on each box. Ceiling measured
gone: 65 GiB and 233 GiB available to a recovery unit, against 19 and 45.

R-165 -> IMPLEMENTED, not PROVEN-LIVE, on the operator's ruling. B2, which that
row records as the bulkhead's replacement, fired live for the first time and
does refuse per app, delete nothing and alert -- but it is checked only in
captureAllRecoveryUnits while runVolumeDumps writes the bulk unguarded, and its
'the previous unit is untouched' claim was measured false (182,272 B dump
replaced by 2,147,666,432 B under a manifest still dated 06:34:26). -> R-181.

New: R-179 (uninstall leaves NAS network-storage units), R-180 (--archive-storage
not cross-checked against the ACL grant; 403 at step 8/8 after root@pam is
rotated), R-181. Third instance of R-115 recorded (agent 0.120.0 unpublished).

No code written, no version bumps -- this was a runbook.
2026-08-03 09:34:15 +02:00
admin bdd1a9d130 REPORT: CI run ids and conclusions for all six commits (all green)
gates / gates (push) Successful in 7s
2026-08-03 07:16:44 +02:00
admin 14d8c00781 docs: R-165 merge built and proven at the bake; R-163 + R-175 closed, R-178 filed
gates / gates (push) Successful in 8s
07-backup-architecture.md gains §7.5.1 (S-1: the contract changed in the same
session): the ceiling §7.5 describes no longer exists for a box built from
golden >= 0.192.0, the bulkhead's replacement is recorded, and R-175 is FIXED
here rather than left standing — the bound is restated as a function of mp1
and scoped to split-layout boxes, naming all three real shapes.

Capability map: new row as IMPLEMENTED, deliberately NOT proven-live, with
the missing leg named — no box has been reinstalled from the golden, and
"the golden baked" is not "a box built from it works".

R-163 CLOSED: the ceiling it recorded stops existing. R-176(a) answered by
P1; (b) WITHDRAWN, since every node is reinstalled rather than migrated.
R-178 filed for the reinstalls, which were not done this session.

CONTEXT S-13 (the variant chosen on measurement; pruning rejected with its
reason) and S-14 (prove first, then vouch — the golden is published but
deliberately unvouched, because vouching is what makes a fresh install pick
up a layout no box has been proven from).

STATUS: plain-language section; both operator questions now answered, so the
waiting-on-you item is cleared. Two older entries trimmed so the page did
not grow.
2026-08-03 07:16:15 +02:00
admin e3525e62ac host-install: one data volume, derived from the disk (R-165)
gates / gates (push) Successful in 7s
felhom-agent v0.120.0 merges the two data volumes into one, and step_grows
computed two numbers while the install call passed both — so this had to
change with the agent or every install would have provisioned a half-sized
box. The 80/20 split is summed (226 = 184+42), so a standard appliance keeps
exactly the 250 G it had, no longer split by a wall.

The size still comes from the physical disk: step_grows already read the
thin pool's free space, and the merge only collapsed its two outputs into
one. --sysdata-grow is deprecated but still honoured, because the agent
folds a hand-passed value in rather than dropping it.
2026-08-03 06:43:58 +02:00
admin 7406ac7bbf audits: R-165 Phase 0 — P1 and P2 measured, nothing changed
gates / gates (push) Successful in 7s
P1 PASS: a pre-merge archive (mp0+mp1, confirmed from its own vzdump log)
restore-tests clean on demo-hp with mount_parity ok in 84s. mountParity was
not touched. Limit stated: run with the current agent because the merged one
does not exist until after the STOP; the comparison is archive-vs-its-own-
restore and never consults the host layout, so it carries provided Part 2
honours its constraint not to touch the restore path. Re-run after Part 2.

P2: all three probed variants are mechanically clean — both paths writable,
ONE df figure, dockerd 3/3 reboots, /mnt propagation, and a container's
statfs(/) reporting the merged volume. The task's flagged ordering risk for
V-b did not materialise. They are separated by SCOPING instead:

  V-a  container sees /mnt = 8.0K  but customer data sits inside Docker's
       data-root, so clearing /var/lib/docker destroys every local unit
  V-b  container sees Docker's ENTIRE data-root under /mnt (17.9M on an
       empty box), making the bootstrap's own scoping comment false
  V-c  neutral mount at /var/lib/felhom, both paths binds — breaks neither

V-c was probed because the measurements showed each named variant violates
a different documented invariant. It is offered as a measured option for the
operator, NOT adopted.

Teardown all three layers: 9401/9402/9403 destroyed, 5.19 GB returned, and
the hub registers verified unchanged (5 customers, 4 hosts).
2026-08-03 06:34:26 +02:00
admin 1806dfa8e9 REPORT: CI run ids and conclusions for all six commits (all green)
gates / gates (push) Successful in 7s
2026-08-02 23:56:45 +02:00
admin 41dbecb264 docs: R-167 + R-158 CLOSED, R-165 SPIKED, R-174..R-177 filed
gates / gates (push) Successful in 8s
R-167/R-158 shipped and proven live (controller v0.191.x, hub v0.89.0):
two new capability-map rows PROVEN-LIVE with live citations, and
07-backup-architecture.md §7.5's closing claim "nothing warns when an app
crosses the line" is now false and rewritten (S-1: an architectural
contract changed in the same session). §7.5 also gains the caveat that its
size bound is ONE BOX'S, not the fleet's.

Part 3 SPIKE (audits/SPIKE-r165-mp1-merge-2026-08-02.md): M1-M5 measured,
NO layout touched. Three findings the merge session must not re-derive:
"the layout" is not one thing (200G/50G vs 50G/20G vs 16G/8G); mp1 is a
BULKHEAD and not only a ceiling, so after the merge an overflow reaches
/var/lib/docker; the golden fails closed on the split in four places.
D-a's condition (1) is currently SATISFIED — no external box is in the
hub's register, and both demo boxes are Tier 0 and reinstallable.
Recommendation given, choice NOT made — it ends at the operator's ruling.

CONTEXT.md S-11 (D-c's routing, and why R-158's own backup_failed proposal
was overruled) and S-12 (the monitoring landed BEFORE the merge).
STATUS.md gains the plain-language section and the merge decision, with two
older entries trimmed so the page did not grow.

New rows R-174 (closed same session), R-175, R-176, R-177; each ID grepped
free before minting.
2026-08-02 23:56:16 +02:00
admin 6d359a5360 manifests: hub 0.88.0 -> 0.89.0 (R-167/R-158 event routing)
gates / gates (push) Successful in 7s
2026-08-02 23:21:17 +02:00
admin 179dd79882 hub v0.89.0 — the two halves of decision D-c (R-167, R-158)
gates / gates (push) Successful in 7s
New OPERATOR-ONLY event type recovery_unit_capture_failed (controller
v0.191.0, R-158): in allowedEventTypes AND notify.operatorOnlyEvents.
Deliberately not a reuse of backup_failed, which carries customer copy and
sits in the controller's DefaultEnabledEvents — reusing it would email the
customer in Hungarian about a failure they cannot act on. R-158's own
proposal said backup_failed; D-c overrides it.

disk_warning/disk_critical lose their generic customerMessages entries.
Both were allowlisted, copy'd, default-enabled and checkbox'd with NO
producer anywhere; controller v0.191.0 becomes that producer and sends a
DYNAMIC Hungarian message naming the drive and its free space.
FormatCustomerEmail prefers the entry over the message, so keeping a static
entry would discard the label and the byte figures — the same reason
offbox_enlarge_blocked and disk_health_degraded have none. The deletion is
pinned by a test.

New notify.IsOperatorOnly so the api package can pin BOTH registers of a new
event type in ONE test; allowlisted-but-not-operator-only is invisible when
they are checked separately, and it is the defect v0.78.0 shipped. The
register itself stays unexported.

REUSE.md's "new event type" extension point rewritten: it told readers to
always add a customerMessages entry, which is wrong for operator-only types
and harmful for dynamic-message ones.

Tests 574 -> 579. Red-proof: removing the operatorOnlyEvents entry shows the
customer being emailed; the skipped/operator_only row is asserted as a
positive observable.
2026-08-02 23:19:13 +02:00
admin 8ef92a3fa7 docs: R-172 CLOSED (hub v0.88.0), R-173 filed, session report
gates / gates (push) Successful in 7s
R-172's root cause was not tuning — the WAL/busy_timeout pragmas had never been
applied, because the DSN used mattn/go-sqlite3 syntax against modernc.org/sqlite,
which ignores unknown parameters without an error. Recorded that way so nobody
re-reads it as "SQLite was slow".

R-173 NEW: while establishing who copies hub.db for the WAL change, found
pvc/hub-data labelled recurring-job-group.longhorn.io/default: disabled, with
backup-daily and backup-weekly the only recurring jobs and both on the default
group — so the hub database has no volume-level backup, and it holds every box's
break-glass root password plus the escrow custody records. Filed, not fixed:
whether the exclusion is deliberate is an operator question.

The session report is REPORT-r172-hub-wal.md, not REPORT.md, per the
parallel-session rule — REPORT.md belongs to the controller session that ran
immediately before this one.

It also records, plainly, that a 60-concurrent load test I ran OOM-killed the hub
pod three times against a 256Mi limit. Not the WAL change, and not a test I
should have run against a Tier-2 box; the unit tests already proved the property.
2026-08-02 21:22:14 +02:00
admin d5774d3189 manifests: hub 0.87.0 -> 0.88.0 (R-172 WAL fix)
gates / gates (push) Successful in 8s
2026-08-02 21:08:17 +02:00
admin 0fc54e0122 hub v0.88.0 — the WAL that never was (R-172)
gates / gates (push) Successful in 7s
store.New opened the DB with `?_journal_mode=WAL&_busy_timeout=5000`, which is
mattn/go-sqlite3 syntax. The driver is modernc.org/sqlite, whose applyQueryParams
reads only _pragma/_time_format/_time_integer_format/_txlock/_inttotime and
IGNORES anything else WITHOUT AN ERROR. So the hub ran in rollback-journal mode
with busy_timeout=0 for its entire life while its own source said otherwise.

Surfaced as a false HOST STALE banner: in rollback-journal mode a reader excludes
a writer, so rendering an operator page blocks a host report; the hub 500s, the
agent waits its full 15-minute interval without retrying, and staleness fires at
30 minutes — two collisions is a false alarm plus an operator email. 13 collisions
in one pod lifetime; the alarm fired twice on 2026-08-02 for a host that was up
two days and reconciling throughout.

The observable that proved it: a 128 MB /data/hub.db with no -wal/-shm beside it
while the DB was open.

Fix: ?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate.
_txlock=immediate is not optional — database/sql's Begin() is DEFERRED, so a
read-then-write tx must upgrade its lock and a failed upgrade is
SQLITE_BUSY_SNAPSHOT, which busy_timeout does NOT retry; this store has 10+
db.Begin() sites and they are all write paths.

Every test asserts what the DATABASE reports, never the DSN string — a string
test would have passed for the whole life of the bug. Red-proof: restoring the
shipped DSN reproduces journal_mode="delete", the missing -wal, and the live
"database is locked (5) (SQLITE_BUSY)".

Operational consequence handled: a WAL DB cannot be copied by taking hub.db
alone — a bare `cat` opens cleanly and silently omits the newest writes. The
break-glass retrieval in operations/nodes.md used exactly that; it and the
recovery-inventory note are now WAL-aware.
2026-08-02 21:06:29 +02:00
admin 2c35c4204a OPEN-ITEMS: R-172 — false host_stale when SQLite refuses two consecutive host reports
gates / gates (push) Successful in 6s
The hub's /data/hub.db (128 MB) is in rollback-journal mode, not WAL, so a UI
render can block a report write; the hub returns 500 on SQLITE_BUSY without
retrying, and the agent waits its full 15-minute interval rather than retrying.
Staleness fires at 30 minutes, so two consecutive collisions produce a false
host_stale and an operator email for a healthy host. Observed twice on
2026-08-02 while the agent was up 2 days and reconciling throughout.

Pre-existing: 13 collisions in one pod lifetime, first ~3h before that day's
controller work, though a burst of restarts amplifies it.
2026-08-02 20:48:04 +02:00
admin ad28699761 docs: R-157 A / R-170 / R-171 closed — boot recovery finished
gates / gates (push) Successful in 7s
Controller v0.190.0. Docs only here; no hub change, no hub version bump.

- audits/DIAG-bootrecon-drive-absent-2026-08-02.md — NEW. The Part 0 diagnosis,
  including the run that produced a FALSE NEGATIVE and the mechanism behind it
  (the agent re-binds an unmounted drive within ~60s, so the drive gate's startup
  reconcile restarted the apps one second before the sweep looked). Records that
  the write hazard was blocked only by an ACCIDENTAL filesystem permission that no
  code owns and no test pins.
- architecture/02 §0a — the boot-recovery contract (S-1): both gates read desired
  state; the sweep observes a SETTLED fleet and each sample must refresh first;
  nothing is started without asking, fail-safe. Plus the durable warning:
  Manager.StartStack has no gate of its own.
- 00-capability-map — the boot-recovery row, with the repeat count cited per N.5
  (6 of 6 hard resets) rather than a bare PROVEN-LIVE.
- OPEN-ITEMS / ROADMAP — R-157 CLOSED (both mechanisms), R-170 CLOSED, R-171 NEW
  and closed the same session, marked a regression from v0.189.0.
- STATUS.md — the power-cut line moved from "What's broken" to "What works right
  now" with its repeat count; one dated bullet in the change log.
- CONTEXT.md S-13 — the lessons worth carrying: "it didn't happen this time" is
  not a disproof; widening a window makes previously-unreachable overlaps
  reachable; and a settle detector is only as good as the freshness of what it
  samples — the fix's own defect, found live rather than by review.
2026-08-02 20:38:21 +02:00
admin 5c97fbc397 docs: R-166 SHIPPED — the desired/in-flight/observed split (D-b)
gates / gates (push) Successful in 8s
Controller v0.189.0 implements operator decision D-b. Docs only here; no hub
change and no hub version bump.

- architecture/02-controller-module-map.md §0a — NEW, and it is the S-1 contract:
  desired (app.yaml) / in-flight (own marker file) / observed (not persisted),
  with the rule that ties them — never derive one from another. Absent desired
  state means UNKNOWN, never "running". One file, one writer. D-b's binding
  safety rule quoted verbatim.
- 00-capability-map.md — the boot-recovery row now rests on a recorded signal,
  with the three live flows from 9201. The interrupted-operation half is marked
  IMPLEMENTED, not PROVEN-LIVE: nobody killed the controller mid-backup on metal.
- OPEN-ITEMS/ROADMAP — R-166 SHIPPED with both blocking facts and their answers;
  R-157 mechanism B CLOSED and A restated as the whole item; R-170 NEW (the
  drive-backed boot gate still infers a Stop from a container count).
- STATUS.md — the "an app can stay switched off and nothing says so" line
  rewritten to what is actually left: timing.
- CLAUDE.md — end-of-session checklist gains: confirm your own last push's CI run
  went green, BY RUN ID. The failure email is a push signal; this is the pull check.
- CONTEXT.md S-12 — the rulings, and the two lessons worth carrying: a test that
  constructs the thing it should prove the caller constructs is hollow (its
  red-proof will say so), and a field-by-field struct rebuild in a save path is a
  defect on sight.
2026-08-02 18:58:27 +02:00
admin c04ea4f6c2 REPORT: record that Gitea soft-deletes runners, and the query trap that hid it
gates / gates (push) Successful in 7s
2026-08-02 16:38:35 +02:00
admin ca3c8f6784 REPORT: record how the session token was revoked, and what was tried first
gates / gates (push) Successful in 7s
2026-08-02 16:36:59 +02:00
admin c718aad1bc docs: R-168 SHIPPED, R-29 CLOSED on the demonstrated alarm, R-169 minted
gates / gates (push) Successful in 7s
SPIKE-ci-runner-2026-08-02.md: all six probes with method, measurement and ruling; none
STOPped. P2 (stock image has git but no python3) and P6 (a runner that loses its state
re-registers and orphans the old record) changed the design; P5 (a failed run signals
NOTHING) is why the alarm exists at all.

R-168 SHIPPED with its evidence. R-29 CLOSED — on the demonstrated alarm and not on a green
run, as required: the class it opened is answered at both ends, the hook refusing locally and
CI catching a --no-verify bypass and emailing. R-161 noted: its automatic half now exists for
the STATIC gate, while its original scope, the runtime gate, is deliberately still not
automatic and should stay that way.

NEW R-169 (grep established R-168 was the highest in use): CI can only report, because there
is no gate in the road. Making it blocking needs branch protection plus a PR workflow, both
of which change how the operator works — so it is theirs to decide, and the row states the
cost honestly rather than recommending it.

CONTEXT gains S-8 (CI detects, does not block, and why that is structural), S-9 (a detector
that tells no one is not finished, plus the curl and Cloudflare-1010 traps), S-10 (the runner
is unprivileged because DooPlex is Tier 2), S-11 (CI reproduces the sibling layout).

CLAUDE.md gains the rule earned by red-proofing: a go test -run pattern that matches no test
prints ok and exits 0, and an instrument that can silently drop results is not a measurement.
2026-08-02 16:35:34 +02:00
admin 4cc123809c revert the Scenario B breakage — main is green again
gates / gates (push) Successful in 7s
The deliberate hostInstallVersion const is removed. It existed only to produce a real red
run (#3-#6) and the demonstrated alarm; R-94's deletion stands.
2026-08-02 16:26:30 +02:00
admin 9530de722d ci: the alarm needs a User-Agent — Cloudflare 403s python-urllib
gates / gates (push) Failing after 8s
Resend sits behind Cloudflare, which blocks the default 'Python-urllib/3.x' agent with its
own 403 (error 1010). That failure looks exactly like an auth failure and is not one, so
the reason is recorded next to the header. Verified from the runner image with a
deliberately invalid payload: with the agent set, Resend answers 422 missing-field, i.e.
the request now reaches the API.
2026-08-02 16:25:58 +02:00
admin f7dbc335ac ci: send the failure alarm with python3/urllib, not curl
gates / gates (push) Failing after 8s
The first version died on 'curl: command not found' — the runner image carries python3 and
git and nothing else on purpose. Reaching for a bigger image to send one HTTP request would
have been the wrong trade, so the step uses urllib. Verified from the image itself that
HTTPS to api.resend.com resolves and the certificate verifies.

The step also fails LOUDLY on an empty key or a non-2xx from Resend: a silent alarm is
worse than no alarm, because it reads as coverage.
2026-08-02 16:25:17 +02:00
admin dd13f632c8 ci: a failed run sends its own alarm (R-168, probe P5)
gates / gates (push) Failing after 7s
P5 measured: a failed run produces NO mail, NO notification row and NO log line from Gitea.
A red tick in a web UI nobody watches is exactly the defect R-29 filed, rebuilt one layer
up, so the run alarms itself on the project's existing transactional path (Resend, the same
one the hub uses) and prints the provider's accepted id, making 'it was sent' an observable
rather than an assumption.

The key is a user-level Gitea Actions secret created out-of-band; it is in no committed
file. The recipient is the operator address the hub already uses and is not a secret.

This push is deliberately made while main is still carrying the Scenario B breakage, so the
resulting run fails and demonstrates the alarm end to end.
2026-08-02 16:23:56 +02:00
admin 3252d51104 SCENARIO B: deliberately break the hostinstall gate (reverted immediately)
gates / gates (push) Failing after 7s
Pushed with --no-verify ON PURPOSE: this simulates exactly the bypass that CI exists to
catch. The local pre-push hook would have refused this commit.
2026-08-02 16:21:35 +02:00
admin 666a34da88 ci: run the gate entry point on every push (R-168)
gates / gates (push) Successful in 7s
Replaces the Part 0 probe workflow, whose four measurements are recorded in
documentation/audits/SPIKE-ci-runner-2026-08-02.md.

Reports, does not refuse: pushes go straight to main with no pull request, so there is no
merge for a status check to stand at. The refusing half is .githooks/pre-push, which is
per-clone and --no-verify-able; this half notices when that was skipped.

No uses: step anywhere — JavaScript actions need a node runtime the host-mode runner does
not have. Probe P3 measured that a shallow git fetch of the exact pushed SHA from the
in-cluster Gitea service is sufficient, and that it equals the pushed commit.

The alarm step is deliberately absent until probe P5 measures whether Gitea already mails
on a failed run.
2026-08-02 16:20:43 +02:00
admin bbd6231909 probe: temporary CI workflow for R-168 Part 0 (P1/P2/P3 + Scenario E)
probe / probe (push) Successful in 48s
TEMPORARY. Deleted before the session ends. Measures whether a registered runner picks up
a job at all, whether python3 and git are visible to the JOB (not merely present in the
image), whether the source can be obtained with no JavaScript action, and that docker is
NOT reachable from a job.
2026-08-02 16:16:59 +02:00
admin af2d103880 REPORT + STATUS: gate enforcement session, hub v0.87.0 live
REPORT overwritten per the standing rule; every red-proof, the core.hooksPath probe's four
measured outcomes, Scenario C's refusal-and-bypass, the hub deployment and the live Setup-tab
read are recorded there, plus three observations and two deliberate deviations from the spec
(a comment-only edit to felhom-host-install.sh, and __pycache__ in .gitignore).

STATUS: the 'check that needs a person to remember it' line is rewritten — the checks now run
themselves before every push, with both honest limits stated in plain words; and one entry
records the thirteen-check cleanup and the deleted installer version number.
2026-08-02 15:37:19 +02:00
admin 8d9b78c153 manifests: hub 0.86.0 -> 0.87.0 (R-94, the Setup tab renders no version) 2026-08-02 15:29:51 +02:00
admin 4707be755c docs: R-94 closed, R-29 leg (a) closed + leg (b) half, R-168 minted
hub/CHANGELOG v0.87.0 + scripts/CHANGELOG gate-enforcement entry. CONTEXT gains S-6 (the
hub renders no host-install version and the gate pins its absence) and S-7 (gates run from
one entry point per repo; reuse_refs_check was fixed rather than the REUSE.md convention,
with both rejected alternatives recorded).

OPEN-ITEMS: R-94 CLOSED all three legs, leg (a) by DELETION with its reason; R-29 leg (a)
CLOSED and leg (b) HALF-SHIPPED with the census result written into the row (13 gates; every
gate a CLAUDE.md names was green, two of the four unnamed were red); R-161 gains its
successor pointer. NEW R-168 (grep established R-167 was the highest in use): Gitea Actions
runner — measured 2026-08-02 as Gitea 1.26.2, Actions enabled on all four repos, 0 runners,
0 workflow runs, 0 branch protections, and the consequence that trunk-based direct-to-main
pushes leave no merge for a status check to gate, so CI here can detect but not block.
BLOCKED on a spike over host-mode vs privileged DinD on DooPlex and whether the workflow can
avoid JavaScript actions.

ROADMAP: R-94 collapsed to its one-liner, R-29 updated, R-168 added.
2026-08-02 15:28:31 +02:00
admin 9bd1a54d71 gates: one entry point (scripts/repo_gates.py) + pre-push hook
A census of all thirteen gate scripts across the four felhom repos on 2026-08-02 found one
clean correlation: every check a CLAUDE.md tells a person to run was passing, and two of the
four nobody is told to run were failing — one since 14 July. Neither failure was harmful in
effect (checked line by line); nothing would have said so if they had been. The fix is not
more gates, it is one place to run them from.

repo_gates.py runs site + hostinstall + hub-confirm + manifest-bearer + reuse-refs, streams
each gate's own output, and exits worst-wins non-zero. A missing gate script is a FAILURE and
prints the path tried — fail-closed, because a runner that quietly skips a gate is the
inert-seam failure this project has shipped four times. It copies catalog_gates.py (R-161),
NOT site_gates.py, which is a gate and not a runner.

.githooks/pre-push runs it with --fast and refuses the push. Honest limits are written into
the hook itself: per-clone (core.hooksPath is local config), and --no-verify bypasses it on
purpose. Any manual run WARNS when the clone is unarmed. Measured on git 2.47.3: a relative
core.hooksPath resolves correctly and the hook's cwd is the repo root from any subdirectory.

test_repo_gates.py is a SEAM test — it asserts each member gate's own distinctive stdout, not
the runner's summary line, which an inert runner prints while calling nothing. Red-proofed:
replacing run_gate's body with 'return 0' still prints 'all felhom.eu gates OK' and exits 0,
and turns the seam test red.
2026-08-02 15:22:44 +02:00
admin 2137094799 scripts: reuse_refs_check resolves package shorthand and sibling repos
RED on all four repos with 13 findings, and a hand audit of all 13 on 2026-08-02 found
ZERO genuine drift: twelve were package shorthand whose file sits a couple of directories
deeper, and one (wgsync/reconciler.go, cited by the controller) lives in the hub. REUSE.md
cites by package shorthand and across repos on purpose; the tool was what was wrong.

Resolution order, first hit wins, every non-exact hit PRINTED so a weakening is visible:
exact / suffix / ambiguous (real citation, imprecise shorthand — not a failure) / sibling
repo (as-is or with the sibling's own name stripped from the token) / FAIL. A failure lists
every resolution attempted, so a 'not found' claim names what was tried. Per-root tallies
are the positive observable: '0 failures' alone cannot tell a working checker from a blind
one. Evidence trees (audits/, documentation/tests/) are excluded from the suffix index — a
copy of a file is not the file. An absent sibling is never a failure; an unreadable parent
says so and continues.

Result: 13/13 resolve, all four roots exit 0. felhom.eu 60 exact + 1 suffix; controller 126
exact + 6 suffix + 1 cross-repo; agent 88 + 1 + 1; catalog 17 exact + 3 cross-repo.

New scripts/test_reuse_refs_check.py: 13 fixture tests, one per resolution row plus the kill
condition. Red-proof: making resolve() return 'exact' for an unresolvable token turns 4 of
them red.
2026-08-02 15:16:11 +02:00
admin d319ae573e hub: delete the host-install version label (R-94) + invert hostinstall gate 1
The Setup tab said 'host-install 1.19.0' while the served script was 1.22.0, and had
been wrong since 2026-07-14. Deriving the number honestly is not possible: the Option-1
command downloads felhom-host-install.sh from the website at RUN TIME and the website
git-syncs main every 30s (R-110), so no build-time value in the hub can be true. R-94(a)
offered derive-or-delete; deleted, which removes the drift class instead of automating it.

- configs.go: hostInstallVersion const, pageData.ScriptVersion field and its assignment
  all removed; a NOTE in their place records why there is no constant here.
- customer_unified.html: the sentence now says the command always fetches the current
  installer, and renders no version.
- hostinstall_gates.py gate 1: the third assertion INVERTS — it used to require the hub
  const to equal SCRIPT_VERSION, it now asserts the hub carries no host-install version
  literal at all, matched in six code shapes across every .go/.html under hub/ (comments
  are deliberately not stripped: a // inside a URL literal would blind the scan).
- render_test.go: the assertion 'html contains hostInstallVersion' compared the constant
  to itself and passed at ANY value — demonstrated green with the const at 9.9.9 while the
  script was 1.22.0. Deleted, not replaced: there is no longer a version to assert.
- felhom-host-install.sh: COMMENT ONLY (SCRIPT_VERSION untouched) — it claimed the gate
  keeps the hub copy equal, an invariant that no longer exists.

Red-proofs: restoring the const fails the rewritten gate 1 (3 shapes hit); the old
render_test assertion passes at 9.9.9.
2026-08-02 15:16:01 +02:00
75 changed files with 5019 additions and 2908 deletions
+97
View File
@@ -0,0 +1,97 @@
# gates — re-run this repo's gate entry point on every push, on a machine that does not care who
# pushed or what they typed.
#
# *** THIS REPORTS. IT CANNOT REFUSE. ***
#
# felhom repos push straight to `main` with no pull request, so there is no merge for a status
# check to stand at. The refusing half is `.githooks/pre-push`, which is local to a clone and which
# `git push --no-verify` skips; this half is what notices when that happened. Neither half is the
# whole thing, and both are named in documentation/backlog/OPEN-ITEMS.md R-168.
#
# NO `uses:` STEP ANYWHERE, deliberately: JavaScript actions need a node runtime in the runner, and
# the runner is a host-mode container with python3 and git and nothing else (see
# homelab-manifests/gitea-system/act-runner.yaml for why it is not privileged). Probe P3 measured
# that a plain `git fetch` of the pushed SHA from the in-cluster Gitea service is enough.
#
# A failing run must reach a person — a detector nobody hears is the defect R-29 filed, rebuilt one
# layer up. That is the last step, and it runs ONLY on failure.
name: gates
on: [push]
jobs:
gates:
runs-on: felhom-gates
steps:
- name: Fetch the pushed commit
run: |
# Shallow, and pinned to the exact SHA that was pushed — not to the branch tip, which can
# move under us if two pushes race. Probe P3 proved the two are equal when done this way.
git init -q .
git remote add origin http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git
git fetch -q --depth 1 origin "$GITHUB_SHA"
git checkout -q FETCH_HEAD
echo "checked out $(git rev-parse HEAD)"
- name: Run the gate entry point
# The ONLY thing CI runs. No go build, no go test, no linting, no deploy — those are either
# already reliably run by a person or none of CI's business. The exit code IS the result:
# no `|| true`, no pipe that could swallow it.
run: python3 scripts/repo_gates.py --fast
- name: Alarm on failure
# THE POINT OF THE WHOLE THING. Probe P5 measured that a failed run produces NO mail, NO
# notification row and NO log line from Gitea itself — a red tick in a web UI nobody watches
# is exactly the shape R-29 filed against. So the run sends its own alarm, on the project's
# existing transactional path (Resend, the same one the hub uses), and prints the provider's
# accepted id so "a message left the machine" is an observable, not an assumption.
#
# Pure python3 and urllib, NOT curl: the runner image carries python3 and git and nothing
# else on purpose, and the first version of this step died on `curl: command not found`.
# Reaching for a bigger image to send one HTTP request would have been the wrong trade.
if: failure()
env:
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
run: |
python3 - <<'PY'
import json, os, sys, urllib.request, urllib.error
key = os.environ.get("RESEND_API_KEY", "")
if not key:
sys.exit("ALARM FAILED: RESEND_API_KEY is empty — the alarm cannot be sent, and a "
"silent alarm is worse than none. Set the user-level Actions secret.")
repo = os.environ.get("GITHUB_REPOSITORY", "?")
sha = os.environ.get("GITHUB_SHA", "?")
run = os.environ.get("GITHUB_RUN_NUMBER", "?")
srv = os.environ.get("GITHUB_SERVER_URL", "https://gitea.dooplex.hu")
body = json.dumps({
"from": "Felhom CI <monitoring@felhom.eu>",
"to": ["admin@felhom.eu"],
"subject": "[felhom CI] gates FAILED in %s" % repo,
"text": (
"The gate entry point exited non-zero.\n\n"
"Repository : %s\n"
"Commit : %s\n"
"Run : %s/%s/actions/runs/%s\n\n"
"The failing gate names itself in the run log.\n\n"
"If the local pre-push hook was GREEN for this commit, then CI and the hook\n"
"disagree - that is a finding about the gates themselves, not about CI, and it\n"
"outranks whatever the push was for.\n"
) % (repo, sha, srv, repo, run),
}).encode()
req = urllib.request.Request(
"https://api.resend.com/emails", data=body, method="POST",
headers={"Authorization": "Bearer %s" % key,
"Content-Type": "application/json",
# Cloudflare fronts api.resend.com and BLOCKS the default
# "Python-urllib/3.x" agent with its own 403 (error 1010) — which looks
# exactly like an auth failure and is not one. Measured 2026-08-02.
"User-Agent": "felhom-ci/1.0"})
try:
with urllib.request.urlopen(req, timeout=30) as r:
print("RESEND-ACCEPTED id=%s" % json.load(r)["id"])
except urllib.error.HTTPError as e:
sys.exit("ALARM FAILED: Resend returned HTTP %s: %s" % (e.code, e.read().decode()[:300]))
PY
+47
View File
@@ -0,0 +1,47 @@
#!/bin/sh
# pre-push — refuse a push that carries a broken gate. (2026-08-02, R-29 leg (b) first half.)
#
# Runs this repo's ONE gate entry point in --fast mode: only checks that touch no network and no
# container runtime, so a push stays a push and never pulls images or starts containers. The slow
# gates stay deliberate periodic runs; a hook that takes minutes gets bypassed within a week and
# the bypass becomes the habit.
#
# BOTH LINES BELOW ARE DELIBERATE. An absent log line is not evidence a hook ran — a silent pass is
# equally consistent with "gates green" and "hook never fired", so a passing push says so out loud.
#
# HONEST LIMITS, stated so this is not mistaken for enforcement it cannot provide:
# * per-clone — core.hooksPath is local config and a clone does not carry it. Arm a clone once:
# git config core.hooksPath .githooks
# Any manual entry-point run WARNS when the clone is unarmed.
# * skippable — `git push --no-verify` bypasses this entirely. That is on purpose: an escape
# hatch that cannot be reached is one that gets removed the first time it is
# inconvenient. USING IT MUST BE STATED IN THE SESSION REPORT.
# The half that is neither per-clone nor skippable is CI — felhom.eu OPEN-ITEMS.md R-168.
#
# Measured 2026-08-02 (git 2.47.3): a relative core.hooksPath resolves correctly and the hook's cwd
# is the repo root whether `git push` is issued from the root or from any subdirectory. The
# explicit rev-parse below does not depend on that.
set -u
root=$(git rev-parse --show-toplevel 2>/dev/null) || {
echo "pre-push: FAIL - cannot resolve the repo root (git rev-parse --show-toplevel)." >&2
exit 1
}
cd "$root" || exit 1
if ! command -v python3 >/dev/null 2>&1; then
echo "pre-push: FAIL - python3 not found, so the gates CANNOT run. This is a failure, never a" >&2
echo " pass by default. Install python3, or push with --no-verify and say so." >&2
exit 1
fi
echo "pre-push [felhom.eu]: running scripts/repo_gates.py --fast ..."
python3 "scripts/repo_gates.py" --fast
rc=$?
if [ "$rc" -ne 0 ]; then
echo "pre-push [felhom.eu]: PUSH REFUSED - gates exited $rc. Fix the finding above, or bypass with" >&2
echo " 'git push --no-verify' and state that you did in the session report." >&2
else
echo "pre-push [felhom.eu]: gates OK - push proceeding."
fi
exit $rc
+4
View File
@@ -32,3 +32,7 @@ Thumbs.db
# Temporary files
*.tmp
*.bak
# Python bytecode from the gate scripts + their fixture tests
__pycache__/
*.pyc
+53 -3
View File
@@ -56,6 +56,13 @@ UI. Package map, helpers, seams, extension points: **`REUSE.md`** (e.g. new even
(`{{if .Flag}}` around a button/form/script) ships with a render test per branch of the gate —
handler tests that POST directly prove nothing about reachability. The v0.70.0 ghost-delete was
fully implemented server-side and fully dead UI because the button sat inside the wrong gate.
- **A `go test -run` pattern that matches no test prints `ok` and exits 0.** Found 2026-08-02 while
red-proofing: `-run TestCustomerUnified` matched nothing in the target file and reported
`ok … 0.062s`, which was read as a passing red-proof. **A red-proof that uses `-run` must first
prove the filter matched something** (`-v` and look for `=== RUN`). This is the "an absent line is
not evidence" rule aimed at the one place a false green costs most — the proof itself. The same
class bit twice that day: a `| tail -5` inside a census query silently dropped rows and looked
exactly like a real finding. **An instrument that can drop results silently is not a measurement.**
- **A health check issues no block I/O.** A probe that touches a wedged device enters uninterruptible
sleep, survives `SIGKILL`, and cannot be recovered until the device returns or the host reboots — so
`systemctl restart` hangs too. A timeout protects the caller's control flow and nothing else: the
@@ -115,6 +122,11 @@ pushes; **you (Claude Code) implement**. A file being open in the editor is NOT
plain language, and is deliberately **not** `CONTEXT.md` — do not consolidate the two.
- **A finding goes in `OPEN-ITEMS.md` first**, never only in a report, an audit or `STATUS.md`. Four
items in this project were minted in a spike doc and lost (R-153/154/155, R-156/157).
- **Confirm your own last push's CI run went green, by run ID.** CI emails on failure, which is a
PUSH signal — this is the PULL check that catches a lost, filtered or unread mail. Quote the run
id and its conclusion in the session report, e.g.
`curl -s "https://gitea.dooplex.hu/api/v1/repos/admin/<repo>/actions/tasks?limit=3"` → match the
`head_sha` to your commit. An unchecked green is an assumption, not an observation.
## Tech stack (Hub)
@@ -174,11 +186,49 @@ Steps: commit+push code → `cd /mnt/5_hdd/felhom.eu/build/felhom-hub && ./build
(local) → bump `manifests/hub.yaml` tag + push → ArgoCD hard-refresh + sync (kubectl-patch method in
the skill, now local `sudo kubectl`) → verify Synced/Healthy + rollout + image + startup log.
## Gates — ONE entry point
**Run `python3 scripts/repo_gates.py` after ANY change in this repo.** It is the one entry point
and runs every gate — `site_gates.py`, `hostinstall_gates.py`, `hub_confirm_gate.py`,
`manifest_bearer_gate.py` and `reuse_refs_check.py` on this root — streaming each gate's own output
and exiting non-zero if any fails. `--fast` selects only the gates that touch no network and no
container runtime; today that is all of them. A missing gate script is a FAILURE, never a skip.
**Why a runner and not five invocations** (2026-08-02, R-29): a census of all thirteen gates across
the four repos found that every check a `CLAUDE.md` names was passing, and two of the four nobody
is told to run were failing — one since 14 July. The single-entry-point shape is the only one that
demonstrably gets run here; `app-catalog-felhom.eu/scripts/catalog_gates.py` is the canonical
version of it (R-161) and `repo_gates.py` copies it. `site_gates.py` is a *gate*, not a runner —
do not model new work on it.
**The pre-push hook.** `.githooks/pre-push` runs `repo_gates.py --fast` and refuses the push if it
fails. It is **per-clone** and switched on once with `git config core.hooksPath .githooks` — a
clone does not carry it, and any manual `repo_gates.py` run WARNS when this clone is unarmed.
`git push --no-verify` bypasses it deliberately; **say so in the session report when you use it**.
Both facts are why continuous integration is still owed (`OPEN-ITEMS.md` R-168) — this hook is
local and skippable, and only CI is neither.
## Build & deploy — Website / Manifests
- **Website** auto-deploys via git-sync; just push to `main` (live in 12 min). **Run
`python3 scripts/site_gates.py` after ANY website change**; new pages go into its `PAGES` list.
Emergency edits: https://files.felhom.eu. All `website/` HTML is **UTF-8 with BOM** — preserve it.
- **Website** auto-deploys via git-sync; just push to `main` (live in 12 min). Website changes go
through `repo_gates.py` above (it runs `site_gates.py`); new pages go into that gate's `PAGES`
list. Emergency edits: https://files.felhom.eu. All `website/` HTML is **UTF-8 with BOM** — preserve it.
- **THE INSTALLER DOES NOT (R-110, 2026-08-03).** `manifests/webpage.yaml` runs **two** git-syncs:
the website from `main` as above, and `/scripts/` from the tag **`installer-v<SCRIPT_VERSION>`**.
Pushing `scripts/felhom-host-install.sh` therefore changes nothing that any machine downloads —
which it used to, within thirty seconds, for the one artifact that runs as **root on a virgin box**.
- **To publish:** cut `installer-v<new SCRIPT_VERSION>`, bump the `--ref` in `webpage.yaml`
(both the sidecar and the init container), commit, and sync. `hostinstall_gates.py` gate 6
fails if the manifest stops naming an `installer-v…` tag or if the website stops tracking `main`.
- **To roll back:** move the tag back to the previous commit and wait ~30 s. **No ArgoCD sync and
no deploy** — git-sync picks up a moved tag on its next period, measured live on 2026-08-03 in
both directions. That is the emergency lever; fix forward with a new version afterwards.
- **Do NOT pin the website to the tag.** The sparse-checkout used to cover `/website/` and
`/scripts/` in one sync, and pinning that would turn every copy edit into a release.
- The **URL never carries a ref** (`https://felhom.eu/scripts/felhom-host-install.sh`), so
`felhom-bootstrap.sh` and the hub's day-0 command follow the tag with no edit — do not add one.
- The installer's own sixteen run-time fetches are pinned separately, to `raw/tag/v$ART_AGENT_VER`
in the **agent** repo (R-183) — they are the agent's configs, not this repo's.
- **Manifests** are GitOps via the `felhom` app — commit to `main`, then deliberate sync.
## Key patterns
+479
View File
@@ -17,6 +17,392 @@
## Standing rulings
**S-23 — the host (on-box) whole-guest tier is restore-PROVEN, unattended, on both demo boxes
(2026-08-04). Scope: those two boxes, not the fleet.**
Four SCHEDULED runs overnight, none triggered by hand: demo-felhom host **83.8 s** / offsite
**540.4 s**; demo-hp host **109.3 s** / offsite **300.1 s**. Every one restored into a scratch guest,
booted, verified and destroyed itself.
*What this closes.* Until yesterday every live restore-proof this project held was on the OFFSITE
tier. The on-box tier — the one an ordinary recovery uses — had never been proven on either box, and
not because it failed: the agent could not read the storage it lives on (R-185), so it never saw an
archive there to test.
*What was observed for the first time.* Both boxes had BOTH tiers due simultaneously. Never-proven
sorts first, so each took its host tier, deferred the offsite one, and picked that up on the next
evaluation six hours later — R-86's ordering and the one-heavy-operation gate, working together,
unsupervised. The host-tier proof then reached the hub through R-189's merge, which is that path
carrying a host-tier entry for the first time.
*The asymmetry worth remembering:* a host-tier restore is **83109 s**; an offsite one is
**300540 s**. The tier that matters for an ordinary recovery is also the cheapest to prove.
**S-21 — an empty listing cannot distinguish FORBIDDEN from NEWBORN, so the box asks the permission
question directly (2026-08-03, R-185; agent v0.123.0 + installer 1.24.0).**
*The defect.* On both demo boxes the agent's token had `FelhomAgentStore` on `local`, `local-lvm` and
`felhom-pbs` and **not** on `felhom-backup` — the storage the same installer configured as
`local_backup_target`. The content API answered `{"data":[]}` through the token while root listed
three archives. `pickForThisRun` skipped the tier as *"no settled archive yet"*, which is exactly
what a brand-new tier reports, so the host tier was never restore-testable and nothing said so.
*The rule.* The permission question has a definite answer where the listing does not. `Permissions`
reads `/access/permissions?path=/storage/<target>` **as the agent's own token** — asking as root
answers a different question and always says yes — and one `capability.Status` per configured tier
reports it. The probed set comes from `BackupTiers()`, never a fixed list: a hardcoded probe list is
the defect reproduced inside the fix.
*The measured trap, because the obvious reading is wrong.* An ungranted path answers **neither empty
nor 403**: it carries the privileges inherited from the box-wide `/` grant
(`Sys.Audit, SDN.Use, Datastore.Audit`). Testing for path-presence, or for `Datastore.Audit`, reports
a blinded storage HEALTHY. The probe tests **`Datastore.AllocateSpace`**, and re-measuring is required
before that constant is ever changed.
*Criticality, weighed once.* Critical, because the hub alerts only on critical and a non-critical
entry would ride the report and alert nobody. **Except** the `local` fallback target, which
host-install's own comment calls the DEGRADED configuration: still probed, still reported, but it
does not page — turning an ordinary documented setup into an alert is how a signal becomes something
an operator archives unread. It never consults content, so it cannot alarm on a newborn tier by
construction, and it never reports ok when it could not ask.
**S-22 — the installer's Scenario-F arm must finish the job, not just leave the definition alone
(2026-08-03, R-185).** `configure_backup_target` has two arms. Case A creates the storage and grants
in the same breath. The reuse arm — *"the target already exists"* — returned **without granting**, and
that, not `PVE_STORAGES`, is where the drift came from: a box whose target pre-dated the install
(the vzdump-target-move runbook, or a reinstall) pointed `local_backup_target` at a storage its token
could not read. The reuse arm now ensures the ACL through the same guarded wrapper. **Scenario F is
unviolated** — the storage DEFINITION is untouched, and granting the role the agent is supposed to
have on the target this script is about to write into `agent.json` is finishing the job, not
retargeting the box. `$BACKUP_TARGET_ID` stays OUT of `PVE_STORAGES`: that list is granted a step
before the target is resolved, and `--acl-storages` entries are preflight-checked for existence.
A gate asserts every arm that resolves the target also grants on it.
**S-19 — a restore-test PROOF is durable and reportable; a FAILURE is neither, and that asymmetry is
the design (2026-08-03, R-189; agent v0.122.0).**
*The rule.* Only successful restore-tests are written to `RestoreTestState`, and that state is what
the host report carries after a restart. Failures live only in the in-memory `backup.Store`.
*Why, in one line each.* A **success suppresses future work** — under R-86's per-archive due-check a
proven archive is never re-tested, so a lost proof leaves the box quietly less tested than it
believes, for a whole archive generation (a week on the offsite tier). A **failure causes future
work** — a failing tier stays due and is retried at the next evaluation, so a lost failure heals
itself within one interval, while a *persisted* failure would outlive the fault it describes.
*What the report does with the two.* The collector merges them: **one entry per tier, newest by
`TestedAt` wins**. A fresh failure therefore beats a stored success (the failure is the news and
exists nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier can
never appear twice — the hub would read that as two tests.
*It refuses to lie.* A persisted record missing the archive **or** the tier produces **no entry**: an
unproven tier reading as proven would be worse than the defect this closes. Run mechanics (scratch
VMID, duration) are not re-invented — an absent duration is not a claim, a fabricated one would be.
**Migration consequence, seen live:** a pre-R-189 record has no tier, so upgrading does not
retroactively make an old proof visible to the hub; the tier's next real proof fills it in.
**S-20 — the release order is build → tag LOCALLY → publish → push tag, and every step protects
something (2026-08-03, R-188 + R-186).**
The tag is created before the publish, so the build and the tag describe the same commit. It is
**pushed** after, because the push is what wakes CI (`on: [push]`) and a tag visible before its
package made `check-published-versions.py` correctly fail a *correct* release — measured on roughly
every second release, and R-168 mails those failures to the operator.
The invariant the old order protected is **asserted directly instead**: the gate now also refuses a
**published version with no tag**, as a bounded probe (frontier + patch gaps) that prints its own
coverage, because the package listing api is 401 without a token and absence cannot be enumerated.
A half-done release is loud: publish-then-failed-push dies naming the recovery command, and a failed
publish deletes the local-only tag so a retry is clean.
**A released binary is independently verifiable**`-trimpath -buildvcs=false` means the same source
yields the same bytes with or without the tag; the verification command lives in
`felhom-agent/CLAUDE.md`. Both build paths (`release-agent.sh` and `publish-agent.sh`'s fallback) use
identical flags: they differed by `CGO_ENABLED=0` and produced binaries 74 KB apart for one version.
**S-17 — restore-testing is PER ARCHIVE GENERATION, and the hub's staleness window follows each
tier's own rhythm (2026-08-03, R-86; agent v0.121.0 + hub v0.91.0).**
*The rule.* Let **A** be the newest archive on a tier that has settled for at least the settle lag
(24 h). The tier is **DUE** when A exists and **A has not already been proven**. The daemon-start
ticker survives only as the **evaluation interval** (6 h). A daily tier is proved daily on yesterday's
archive; a weekly tier weekly on its own; a tier with no archive is UNKNOWN, never a fault.
*The trap, written down so it is not reintroduced.* The literal reading of R-86 — *"due when the
newest archive is ≥24 h old"* — is **never true on a daily tier**, because a new archive resets the
newest-archive age to zero long before it reaches the lag. It would have switched restore-testing off
for the tier that matters most, silently. Red-proved at 0 runs over 5 simulated days
(`felhom-agent/internal/backup/restoretest_due_test.go`).
*What the state holds now.* `RestoreTestState` records **which archive** was proven, not just when a
tier passed — a timestamp cannot answer *"have we proven THIS archive"*. A pre-R-86 file keeps its
time (rotation ordering survives a deploy) and yields no proven archive, so each tier is due exactly
once after the upgrade.
*The old config key.* `backup.restore_test_cadence_seconds` is DEPRECATED. **Negative still disables**
verbatim; a positive value now seeds the **settle lag** only, and the daemon WARNs once at start-up
naming `restore_test_eval_interval_seconds` (default 6 h) and `restore_test_settle_seconds`
(default 24 h). It is deliberately NOT carried into the evaluation interval.
*The hub half is not optional.* `restoreProvenStaleAfter` was a flat 7 days **derived from the cadence
R-86 removes**, and a healthy weekly tier's proof age reaches EXACTLY 168 h just before its next
proof — it sat ON the line. `restoreProvenWindow(tier, observed, ok)` now takes the tier's own
observed archive interval × 4 generations, floored at 7 days, capped at 12 days (strictly inside the
2-week offsite retention), falling back to the tier's **declared** rhythm (`backupStaleAfter` 26 h /
`offsiteBackupStaleAfter` 8 d — the backup-freshness checker's own thresholds) when history is too
short to observe one. Shipping Part 1 alone would have produced a nightly false alarm.
**S-18 — `ep0` is Tier 2, PROTECTED (operator ruling, 2026-08-03).** D-d named two protected machines
and did not name ep0 either way; `runbooks/target-selection.md` carried the question in writing for
two days. The ruling **extends D-d's protected list to three machines**: DooPlex, Peti's cluster,
**ep0**. It is a classification, not a new set of prohibitions — destroying datastores, prune jobs,
tunnel config or nftables rules was already forbidden by what it would destroy, and the ordinary
off-site READ a restore-test performs remains permitted.
**S-13 — the `mp1` merge landed, and the variant was chosen on measurement (2026-08-03, R-165 / D-a).**
The appliance's two data volumes are one. **Variant V-c**: the volume mounts at the NEUTRAL path
`/var/lib/felhom`, and both `/var/lib/docker` and `/mnt/sys_drive` are binds of subdirectories of it.
**Three shapes were built and rebooted before choosing** (`audits/SPIKE-r165-phase0-2026-08-03.md`) —
all three boot, reboot 3/3, give ONE `df` figure and keep a container's `statfs("/")` on the merged
volume, so **the ordering risk that motivated the probe was not what mattered.** They differ only in
which documented guarantee they break: volume-at-`/var/lib/docker` puts customer backups inside
Docker's data-root, so the ordinary "clear `/var/lib/docker`" reflex destroys every local unit;
volume-at-`/mnt/sys_drive` puts Docker's entire data-root under `/mnt`, which the controller container
mounts wholesale — **measured: it then sees `/mnt/sys_drive/docker`**, falsifying the bootstrap's own
comment that `/mnt` holds only Felhom's namespace mounts. **V-c breaks neither**, for one extra path.
**B2 is the bulkhead replacement, and "or prune the oldest" is REJECTED with its reason**, because the
question will be asked again: nothing on that filesystem is generational — a unit is ONE fixed path per
app (`backups/primary/<app>`) refreshed in place, and a DB dump is `<stack>-<dbtype>.sql`, also fixed —
so pruning could only mean deleting a **different** app's only local recovery unit.
`pruneStalePrimaryDirs` is an ORPHAN sweep with no notion of age and must never be repurposed.
**No migration exists, and that is a ruling not an omission:** every node is REINSTALLED. Both demo
boxes are Tier 0; the colleague's box carries none of our customer data and is clean-installed shortly.
So R-176's in-place migration rehearsal is **withdrawn**, not deferred.
**S-14 — prove first, then vouch (2026-08-03) — SPENT, and the ordering did not survive contact.** The
rule was: golden **0.192.0** stays UNVOUCHED until a box has been proven from it, because vouching is
what makes a fresh install pick a golden up. **In the event the golden was vouched at 07:23:26 CEST on
2026-08-03, before any box was reinstalled** (hub log `Artifact manifest set: agent=0.119.0
golden=0.192.0`), so the ordering was already spent when R-178's session opened; the operator elected
to accept it rather than revert the manifest. **Both boxes were then reinstalled and proven** (R-178,
`REPORT.md`), so the end state is the intended one and no unproven layout was ever in front of a real
install — but the rule protected nothing, because nothing enforced it. **The lesson is R-115's, one
layer up:** an ordering that lives only in a `CONTEXT.md` sentence and a runbook's §7 is a reminder,
and reminders do not hold. If prove-then-vouch is to be a rule it needs the shape R-120's gate has —
a refusal at `handleSetArtifacts`, the sole path to `SetArtifactManifest`, which runs without anyone
choosing to run it.
**S-15 — the merged layout is proven live, by two different supply paths (2026-08-03, R-178).** Both
demo boxes were wiped and reinstalled from golden 0.192.0 and taken through claim → deploy → back up →
**restore**. **demo-hp** was installed with `--golden <local volid>` (the layout proof) and
**demo-felhom** by the normal manifest route with `--force-gitea-golden` (the pipeline proof —
`verified sha256 54e2a4c431daf580… matches the hub manifest`), deliberately different so the session
proved the disk shape *and* the delivery route rather than one of them twice. Live shape on both:
`mp0` at `/var/lib/felhom`, `backup=1`, **no `mp1`**; `/var/lib/docker` and `/mnt/sys_drive` both real
mounts of its subdirectories via `/etc/fstab`; ONE `df` figure and one device id on all three paths;
3/3 reboots each with the binds surviving every time. B2 was **not** proven on that pass → **R-181**:
the floor guarded `captureAllRecoveryUnits` and not `runVolumeDumps`, the leg that fills the volume,
and its refusal's "the previous unit is untouched" was measured false. **R-181 CLOSED the same day
(controller v0.193.0 + v0.193.1), so R-165 is now PROVEN-LIVE in both halves** — see S-14.
**S-14 — the reserve is a per-app, per-run ADMISSION decision, not a capture check (2026-08-03, R-181;
controller v0.193.0 + v0.193.1).** B2 as first shipped was consulted in exactly one place —
`captureAllRecoveryUnits`, a few KB — while `RunDBDumps`' database leg and `runVolumeDumps` wrote the
bulk into the same `backups/primary/<app>` tree, first and unguarded. The reserve was therefore
consumed by the very write it exists to bound, and the refusal then claimed *"the previous unit is
untouched"* about a tree the earlier leg had already rewritten (182,272 B → 2,147,666,432 B under a
manifest that had not moved). **Sixth entry in `CLAUDE.md`'s table of shipped guarantees the code did
not provide, and the fourth of those found on live hardware rather than by review.**
- **`internal/backup/admission.go``admitApp` is now THE gate**, and every per-app write leg calls
it. One verdict per app per run covers all three; they share one per-app root, which is what makes
that honest.
- **Decided lazily at the app's first write, never once at run start** (app A's dump can put app B
under the reserve), **never re-decided between an app's own legs** (that is the split it closes),
and **reset per run**.
- **Ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act — a refusal decided
inside it has already bounced the app. **After** the volume-less check, which has no write to gate.
- **Size term added:** *would THIS app's write cross the reserve?*, estimated from the app's previous
`.sql` + `.tar`. **No history → headroom-only**, deliberately — otherwise the first backup is the
one that can never happen.
- **A container-based `du` was MEASURED and rejected**, not waved away: median **~355 ms/volume** over
66 runs on demo-hp, on volumes holding tens of KB (container start-up, not the walk). Decisive on
top: `docker run` needs the writable layer, so the instrument can fail under exactly the pressure
the reserve handles.
- **The wording was NOT weakened; the behaviour moved so it became true**, and it is checked by
sha256 tree fingerprint, never by reading the log line — the log line is what lied.
- **v0.193.1**, found by the proof run itself: a 178 KB estimate printed as `0.00 GiB`, which reads as
*no estimate available*. Rendering moved to `humanizeBytes`; arithmetic still in GiB.
- **New finding, deliberately not fixed here → R-182**: `GetFullStatus`'s periodic capture sweep has
no run scope, so a refused app re-alerts on every status refresh (measured: a second identical alert
pair 13 s after the run's). Pre-existing in v0.192.0; R-181 changed neither caller.
**S-15 — publishing is an act, not a side-effect of pushing (2026-08-03, R-110 + R-115 + R-183).**
Two rulings, one shape: something became live because someone pushed, not because anyone decided.
- **The installer.** `/scripts/` now git-syncs the tag `installer-v<SCRIPT_VERSION>`; the **website
keeps tracking `main`** in a second sync, because pinning both would make every copy edit a
release. Publish = cut the next tag + bump the manifest `--ref` + sync. **Roll back = move the tag
back**, which takes ~30 s and needs no ArgoCD sync at all — git-sync v4.4.0 follows a moved tag,
and that half was measured before the manifest was touched because the whole model rests on it.
- **The sixteen run-time fetches were NOT what the spec described** — sixteen, not nine, and from
`felhom-agent`, not this repo — so no tag here could cover them. They are pinned to
`raw/tag/v$ART_AGENT_VER` instead, which is strictly better: the agent's configs now come from the
same ref as the agent binary being installed. That closed a real skew (**R-183**), not just a
channel.
- **The URL needed no change**, and that is worth knowing rather than re-deriving: it never carried
a ref, so both producers follow the tag automatically — and no hub change means no hub bump.
- **The agent.** `scripts/release-agent.sh` is THE release path: build → tag → publish → **verify by
an independent download**. It does not vouch. `check-published-versions.py` refuses a `v<semver>`
tag with no downloadable package, and **CI now runs the full gate set** rather than `--fast`,
without which that gate would have been registered and never run.
- **The gate's invariant is not the one specified, and P-C is why:** the hub manifest and Gitea's
package listing are both **401** anonymously; the package download and the tags api are not. So CI
can ask *is this installable* but not *what is vouched*. The residue is **R-184**.
- **Neither gate asserts "the newest version is published."** That would go red on the very push
that bumps a version, before publishing — and a gate that fails on the normal path is one people
learn to ignore.
**S-16 — a backup run NOTIFIES ONCE and RECORDS ALWAYS, and those are different things
(2026-08-03, R-182; controller v0.194.0 + hub v0.90.0/.1).** Measured: nine per-app capture failures
reached the hub, two were mailed, seven were dropped by a cooldown whose key carries no app
identifier — *before* `LogNotification`, so they left no row anywhere.
- **The record:** `recovery_unit_capture_failed`, per app, unconditionally, now routed
**record-only** by the hub (`recordOnlyEvents`) — stored and logged every time, never competing
for an e-mail slot.
- **The notification:** `backup_run_failures`, ONE per run, listing every failed app with its leg
and reason plus the counts and free space. **A clean run emits nothing.**
- **A suppressed operator event now leaves a `suppressed` row** carrying the key that suppressed it —
for every operator type. *"We chose not to e-mail you"* and *"nothing happened"* must never look
identical; that is the whole finding, stated as a rule.
- **The periodic sweep gets a digest too, with NO `run_id`**, so it stays under the ordinary hourly
cooldown. Without it the sweep's failures would be recorded and never notified — a new silence
created while closing one. A real run's digest carries a unique `run_id` precisely so the cooldown
can never collapse a manual run into the nightly one.
- **Why the silence is safe:** the hub's deadline check raises `expected_backup_missed` from report
freshness, independently of any mail the box sends (`monitor/deadline.go:396,417`). **If that check
is ever weakened, this design loses its footing.**
- **Not taken, and why:** putting `app` in the cooldown key. It fixes the swallowing by producing one
mail per failing app — a dozen on a full disk.
**ep0 was rescaled by the operator to a CX33 (2026-08-03): 4 vCPU, 8 GB RAM, measured on the box, and
the 4 GiB swapfile survived. The 40 GB local disk is UNCHANGED** — a CPU/RAM resize only, so no disk
figure in any runbook needed correcting. That closed **R-90** and unblocked **R-86**.
**S-11 — D-c's routing, and why R-158's own proposal was overruled (2026-08-02, R-167 SHIPPED).**
Decision D-c splits two signals by AUDIENCE, and the split is the ruling: **a fill warning is the
CUSTOMER's** (they can free space, delete files, add a drive) and **a per-app backup capture failure
is the OPERATOR's** (they can do none of those things about it). R-158 proposed emitting the existing
`backup_failed` for the capture failure. **That was rejected and D-c wins**, because `backup_failed`
carries a `customerMessages` entry AND sits in `settings.DefaultEnabledEvents` — so reusing it emails
the customer, in Hungarian, that their backup failed, about something they cannot act on. It is
exactly the mistake R-97a avoided by minting `whole_guest_backup_failed`, and the reasoning is written
into `hub/internal/api/handler.go`'s allowlist. New type: `recovery_unit_capture_failed`, in
`allowedEventTypes` **and** `notify.operatorOnlyEvents` — **the second register is what makes it
operator-only; the first does not**, and v0.78.0 claimed otherwise and shipped the defect.
**The customer half reused the pair that already existed rather than minting a seventh type.**
`disk_warning`/`disk_critical` were allowlisted, carried Hungarian copy, sat in `DefaultEnabledEvents`
and had a UI checkbox — and **nothing in any repo emitted them**. A complete customer pipeline with no
producer: the **sixth** *built-but-never-wired* instance in this project. `internal/fillwatch` is now
that producer. Their generic `customerMessages` entries were **deleted**, because
`FormatCustomerEmail` PREFERS the entry over the message and a static template would discard the drive
label and the free-space figures — the same reason `offbox_enlarge_blocked` and `disk_health_degraded`
have none. `notify.IsOperatorOnly` was added so ONE test pins both registers; checked separately, an
allowlisted-but-not-operator-only type is invisible.
**S-12 — the monitoring landed BEFORE the merge, not with it (2026-08-02).**
D-a's condition (2) says R-167 ships in the same step as the `mp1``mp0` merge and never after,
because the merge removes a wall that currently fails safely. **This session landed it FIRST**, which
is strictly better and costs nothing: the warnings went in and were proven on real hardware while the
wall is still standing, so the merge session inherits a proven signal instead of an untested one.
**No disk layout was touched.** R-165's measurement is `audits/SPIKE-r165-mp1-merge-2026-08-02.md`,
which STOPS at a question for the operator (which merge shape; what replaces the bulkhead). Its two
load-bearing findings for anyone picking that up: **"the layout" is not one thing** (demo-felhom
`200G/50G`, demo-hp `50G/20G`, golden `16G/8G` — so §7.5's bound is one box's, → R-175), and **`mp1`
is also a BULKHEAD**, not only a ceiling — today an overflow cannot reach `/var/lib/docker`, and after
the merge it can.
**S-8 — CI detects; it does not block, and that is structural (2026-08-02, R-168).**
A Gitea Actions runner in `gitea-system` re-runs every repo's gate entry point on every push,
independent of who pushed and of what they typed. It **cannot refuse a push**: every felhom repo
pushes straight to `main` with no pull request, so there is no merge for a status check to stand at.
That is not a gap in the runner — there is no gate in the road. The refusing half is
`.githooks/pre-push` (per-clone, `--no-verify`-able); this half is what notices when that hook was
skipped or was never armed. Making CI blocking needs branch protection plus a PR workflow, which
changes how the operator works and is **their** call → R-169. Do not "fix" this by adding branch
protection.
**S-9 — a detector that tells no one is not finished (2026-08-02, R-168 probe P5).**
Probe P5 measured that a failed run produces **no mail, no notification row and no log line** from
Gitea. So the workflow sends its own alarm on the project's existing Resend path and **prints the
provider's accepted id**, which makes "a message left the machine" an observable rather than an
assumption. The acceptance test for this work was never a green pipeline — it was a red run with a
message in hand (`RESEND-ACCEPTED id=…`). Two traps found while building it, both worth keeping:
the runner image has **no `curl`** (deliberately — python3 and git only, so use `urllib`), and
`api.resend.com` sits behind **Cloudflare, which 403s the default `Python-urllib` User-Agent with
error 1010** — a failure that looks exactly like an auth failure and is not one.
**S-10 — the runner is unprivileged, and the reason is the host (2026-08-02).**
The usual `act_runner` recipe pairs it with a `docker:dind` sidecar and `privileged: true`. Rejected:
DooPlex is **Tier 2** and *is* the recovery chain — Gitea, the hub, the registry, PBS and
k3s + Longhorn all live on it and it cannot be rebuilt from anything else. Host execution mode costs
nothing here because every CI job is one Python command. Consequence to remember: **in host mode the
job sees exactly the runner image's tools**, which is why `python3` had to be baked in (probe P2:
stock `act_runner` carries git but not python3). If a future job genuinely needs Docker, that is a
conversation, not a patch.
**S-11 — CI reproduces the workspace's sibling layout, because two entry points depend on it
(2026-08-02).** `controller_gates.py` and `agent_gates.py` invoke the shared `reuse_refs_check.py`
that lives in the `felhom.eu` clone next door and is deliberately never copied, and both repos'
`REUSE.md` files cite a path that lives in the hub. Their workflows therefore clone `felhom.eu` as a
sibling; without it the gate fails **closed** — correctly, but for the wrong reason. Verified that CI
and the local hook then agree exactly (controller 126 exact / 6 suffix / 1 cross-repo).
**S-6 — the hub renders no host-install version, and the gate pins its absence (2026-08-02, R-94).**
The Setup tab's *"host-install 1.19.0"* label is **deleted, not derived**. Deriving it is not
achievable honestly: the Option-1 command downloads `felhom-host-install.sh` from the website **at
run time**, and the website git-syncs `main` every 30 seconds (R-110) — so the hub cannot know which
version a given box will run, at build time or at render time, and any literal there is a guess
wearing a version number's authority. The real one drifted to 1.19.0-vs-1.22.0 and stayed wrong for
nineteen days. `hostInstallVersion`, `pageData.ScriptVersion` and the rendered label are gone; a NOTE
sits where the const was so it is not helpfully re-added, and `scripts/hostinstall_gates.py` gate 1
**inverted** — it now asserts the hub carries no host-install version literal in any of six code
shapes across every `.go`/`.html` under `hub/`. Corollary that generalises past this row: the
tautological `render_test.go` assertion (`html contains hostInstallVersion`, where the same constant
put it there) **passed at `9.9.9`** — an assertion that compares a value to itself tests the
plumbing, never the claim.
**S-7 — gates run from ONE entry point per repo, and `reuse_refs_check` was fixed rather than the
convention it polices (2026-08-02, R-29).** Two rulings from the same census.
*Where gates run.* Thirteen gate scripts exist across the four repos. Measured 2026-08-02: **every
check a `CLAUDE.md` tells a person to run was passing, and two of the four nobody is told to run were
failing** — one since 14 July. The correlation was exact, so the fix is not more gates but one place
to run them from: `scripts/repo_gates.py`, `felhom-controller/controller/scripts/controller_gates.py`,
`felhom-agent/scripts/agent_gates.py`, `app-catalog-felhom.eu/scripts/catalog_gates.py` — each
mandated in its `CLAUDE.md`, each wired to `.githooks/pre-push` with `--fast`. The canonical shape is
`catalog_gates.py` (R-161), **not** `site_gates.py`, which is a gate (eight assertions in one file)
and not a runner; copying it produces another monolith nobody invokes. **A missing gate script is a
FAILURE with the path printed, never a skip.** The hook's limits are real and are written into the
hook: per-clone (`core.hooksPath` is local config) and `--no-verify`-able on purpose. The
unbypassable half is CI → **R-168**.
*Why the checker moved and the docs did not.* `reuse_refs_check.py` was RED on all four repos with
13 findings, of which a hand audit found **zero** genuine drift — twelve were package shorthand
(`appbackup/userdata.go``controller/internal/appbackup/userdata.go`) and one, `wgsync/reconciler.go`,
is cited by the controller and lives in the hub. `REUSE.md` cites by package shorthand and across
repos deliberately; that convention is the useful one. **Rejected, so they are not revisited:**
rewriting all four `REUSE.md` files to full paths (makes the docs worse to serve the tool), and
deleting the checker (drift across four repos is a live risk). The checker now resolves
exact → suffix → ambiguous → sibling repo → FAIL, **prints every non-exact hit and a per-rule tally**
(because "0 failures" alone cannot tell a working checker from a blind one), and lists every
resolution attempted on a failure. It stays in **one** place and is invoked across the workspace —
never copied, which would recreate the drift it detects.
**S-1 — N.5 gains a third leg: architecture docs are same-session coupled (2026-07-26, R-81).**
Any task that changes an **architectural contract** — tiers, targets, cadences, trust boundaries —
updates the owning `documentation/architecture/*.md` in the **same session**, under exactly the same
@@ -87,6 +473,99 @@ Five decisions were deliberately **left open for the operator** and are recorded
stated**) · Hetzner as a single failure domain · and `local` vzdump sharing a physical device with
the guest it backs up. Gaps minted the same session: **R-102 … R-108**.
**S-13 — boot recovery finished, and the lesson is about the DIAGNOSIS ORDER (controller v0.190.0,
2026-08-02, R-157 A · R-170 · R-171).**
**The session's most valuable half hour was spent NOT writing code.** A hole was reasoned out of the
v0.189.0 diff — replacing the container-count term with recorded intent should make a
drive-gate-stopped app read as a boot orphan — and the task's own rule was to CONFIRM it on hardware
before writing a fix. **The first attempt to confirm it produced a false negative**, and reporting
that as a disproof would have been wrong: unmounting only the parent bind is healed by the agent
within ~60 s, so the drive gate's startup reconcile restarted the apps **one second before** the
sweep looked. `no boot-orphaned apps` in that log is a race that went the safe way, not a mechanism.
Holding the drive genuinely absent reproduced it immediately. **"It didn't happen this time" is not a
disproof — name the mechanism or run it again.**
**The confirmation also changed the severity, in both directions.** The write hazard did NOT
materialise: compose failed `mkdir …/userdata: permission denied`, because the unbound mountpoint is
host-root-owned and the guest is unprivileged. **That protection is accidental** — no code chose it,
no test pins it, it is one `chown` or one privileged guest away from gone, and its removal would be
invisible until data landed on the wrong disk. Meanwhile the harm that DID occur was real on every
box and was not in the hypothesis: two wasted attempts and a **false dead-app alarm for an app the
drive gate is deliberately holding**. Diagnosing first is what produced both facts.
**The fix was already in the codebase, on another path.** The API's `startGatedByMissingDrive`
refuses a customer's start on an absent drive with a Hungarian message. The sweep bypassed it by
calling `Manager.StartStack` directly. **`StartStack` has no gate of its own** — that is the durable
fact worth carrying: every caller that is not the customer must decide for itself whether the app may
run, and there are now fourteen of them.
**Widening a window makes previously-unreachable overlaps reachable, and that is a design input, not
an afterthought.** The old T+5 s sweep never met a quiesce or an in-flight app-data operation; a
50 s window can. All three holders answer one seam rather than three, because they differ only in
the reason string.
**A test rejected my first constant, and the comment now says so.** `settle + budget + one retry`
must fit inside `deadAppBootGrace` or a successful recovery stops being silent; 60 s gave 95 s
against a 90 s grace. The budget is 50 s **because a test said so**, and the code records that rather
than presenting the number as taste. Widening the grace to fit was rejected outright: it hides a late
recovery instead of reporting one.
**AND THE FIX HAD ITS OWN DEFECT, FOUND BY LIVE VALIDATION AND NOT BY REVIEW.** The window sampled
`GetStacks()` — the Manager's in-memory map, refreshed by the scheduler every 10 s — every 5 s. Two
identical samples could therefore mean *the cache did not update*, not that the fleet had settled. It
surfaced as a container removed ~5 s before the window closed still being in the sampled fleet, with
the sweep logging `no boot-orphaned apps` for an app that had none. **Generalise it: a settle
detector is only as good as the freshness of what it samples — if the source is cached, refresh it or
you are watching the cache settle, not the system.**
**Live: 6/6 hard resets on the shipped build** (every app back; a customer-stopped app down in all
six), window settle times 10/40/10/10/15/15 s — routinely 28× the old fixed 5 s. The sharpest
evidence is a same-app before/after on one box: missed at 18:08:35, recovered at 18:18:50.
**S-12 — D-b is BUILT (controller v0.189.0, 2026-08-02, R-166).** The desired/in-flight/observed
split now exists; the S-1 contract lives in `architecture/02-controller-module-map.md` §0a.
**Both facts D-b said to establish first were established at source, and both changed the shape.**
(a) *Does the crash-safe journal in the backup code already cover the in-flight case?* The pattern
DID already exist — twice (`quiesce` marker+`Recover`, `migrate` journal+`RecoverMigration`) — and
covered **none** of the app-data path: `DumpAppVolumesSafe` stopped and restarted an app with no
marker, no journal and **not even a `defer`**. So the answer was neither "it exists, wire it" nor
"build it": copy the proven shape into its own file. It was the fifth time the question was worth
asking and the first time the answer was "the pattern, not the coverage". (b) *Is the SQLite store
reachable?* Reachable and **deliberately not used**`metrics.db` is optional by design (the
controller runs with it absent), and operational state must not live in a store built to be dropped.
**The ruling that carried the design: absent means UNKNOWN, never "running".** Every `app.yaml` on
every box predates the field, so absent is what the whole fleet reads on upgrade; reading it as
running would have started every deliberately-stopped app on the first boot after the upgrade —
fleet-wide, silently. Where intent is unknown the box keeps the OLD inference rather than inventing
an answer. That is also why the backfill is **running-only**: "zero containers ⇒ stopped" is the
defect itself, so an ambiguous app stays ambiguous until a customer presses a button.
**The other load-bearing ruling: `StartStack`/`StopStack` are NOT writers of intent.** A census found
14 callers, of which exactly 2 are the customer. Recording intent in the primitive would make a
nightly backup indistinguishable from the customer pressing Stop — the confusion being removed.
**Found on the way, and it would have silently eaten the feature: `SaveAppConfig` rebuilt `AppConfig`
field-by-field.** That is the R-100 shape, which v0.181.0 shipped two live instances of. The literal
named five fields, so the sixth would have been dropped on every save across nine call sites — a
customer's Stop erased by the next unrelated `app.yaml` write. Copy-and-overlay is safe by
construction; the failure mode is generic, so **treat any field-by-field struct rebuild in a save
path as a defect on sight.** Measured and documented rather than assumed: `app.yaml` does NOT
round-trip YAML keys the struct does not model.
**Closes R-157 mechanism B; mechanism A (the sweep observes ~5 s after start and never re-checks) is
untouched and is now the whole of R-157** — and B's fix makes A cost more, since the sweep now has
more it could legitimately recover. **New: R-170**`shouldRecreateOnBoot`
(`internal/web/intermediary.go:131`) still infers a Stop from `hasContainers`, i.e. the same defect
one gate over for drive-backed apps. Left deliberately: the task scoped `bootrecon`, and two boot
behaviour changes under one live validation is one too many.
**Live on 9201, three flows, each with a positive observable.** The interrupted-operation half is
**IMPLEMENTED, not PROVEN-LIVE** — unit-proven and red-proofed, but nobody killed the controller
mid-backup on real hardware; the capability map says so rather than rounding it up.
**S-5 — four operator decisions taken in discussion on 2026-08-02, recorded before anything is
built.** They existed only in conversation, which is the condition the standing rules were written
against. Labels are the ones used in the discussion (**D-a … D-d**) and are deliberately kept
-113
View File
@@ -1,113 +0,0 @@
# REPORT — Campaign 10, two-storage adversarial soak (2026-07-31)
Follows `REPORT-campaign7/8/9.md`. Root `REPORT.md` is another session's (hub v0.85.0) and was not
clobbered — same shared-clone reasoning as `REPORT-iso-release.md`.
**Full audit + evidence:** `documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md`,
`documentation/tests/campaign10-evidence-2026-07-31/`.
## The sentence that matters
**Phase A passed every gate. Phase B ran to 39 consecutive cycles with the full atom set — past the
"drift at the thirty-eighth" depth the brief asked for. 66 restores, 66 correct discriminators. No
resource leak over 13.5 h. Two findings: R-156 and R-157.**
## What was established
- **Venue** — VM 311 on demo-hp (Tier 0), 200 G system + 2 × 50 G data, scratch storage at the
`/mnt/nvme-1tb` **mount root** (a subdirectory would have emitted `storage_disconnected` for demo-hp
all night — the exact signal I1/I2 discriminate).
- **Baselines, all read fresh.** controller `main` 0.188.0, **golden 0.188.0 (not behind)**, agent
0.119.0 published+vouched, hub 0.86.0, ISO **1.26.1** (`f3cc86d5…`, round-trip verified live).
The brief's ISO assumption (v1.25.0) was ~90 minutes stale; its "no baked SSH key" claim is R-129.
- **Isolation gate — both denials captured, each with a positive control.** The PBS control **failed
first**: four clean-looking 403s were worthless because the token was denied on its own datastore
too (PBS token privilege separation). Fixed, re-run, denials stand.
- **A1** fresh install from the **published** ISO. 1.26.1 is a public release image — verified against
its bytes that it has no auto-install path — so it was driven blind via screendump + `sendkey`
through the Terminal UI. Caught the Hungarian-keymap trap before typing the root password, which
would otherwise have been mangled and locked the box out.
- **A2** claimed for real; discriminator flipped `dashboard not yet claimed``authentication required`.
- **A3** both drives enrolled through the **real** endpoint; `mentes` accepted as backup target via
the offer flow, ending `degraded:false / target:felhom-backup` — the I5/I6 healthy baseline.
Four apps healthy spanning both sides of D5's split (4 × `type: secret`, 1 × `type: password`).
- **A4** discriminators seed and read back across all four apps; rallly's over the path
`DATABASE_URL` actually names, not the trusted socket that produced D5's false pass.
## Phase B — the soak
Three passes: run 1 (27 cycles, 6 atom families, 0 violations), run 2a (10 cycles, **stopped
deliberately** — two violations were harness defects), run 2b (**39 cycles, 12 atom families**).
1 461 invariant checks in total.
- **I7 is the headline: 66 restores across both passes, 66 correct discriminators.** Never stale,
never empty. Run 2b added an `I7-SKIP` verdict so a check with unmet preconditions is recorded as
skipped rather than silently green — it fired once.
- **I2, I3, I4, I5, I6, I10, I11: zero violations in either pass**, including the abort-in-place
variants and 330 secret-class assertions covering both sides of D5's split.
- **`I1-under-load` 5/5**: the backup target pulled **while a backup was running** still produced
`backup_target_absent` and a clean recovery.
- **R-117's Q7 case holds** — a filesystem aborted *in place* (device still present) surfaces via
`bound_under_parent=false`, the gate stops the app on the dead namespace, and the storage page names
it. That is the case R-117's spike called "the worse half".
- **RTO, both bands measured.** S: 66 MB → **42.0 s** / **41.4 s** across two passes (66 restores).
M: 21.1 GB → **608 s** mean over 2 reps, both returning the correct discriminator. 327× the data
cost 14.5× the time, giving **RTO ≈ 40 s + 26.9 s/GB** (backup ≈ 29 s + 17.4 s/GB). The fixed ~40 s
dominates below ~1.5 GB — that is the S band, and why its numbers clustered so tightly.
- **Capacity ceiling, and the more consequential result:** a DB-backed app's recovery unit is **1.90×**
its data (volume tar + SQL dump). The default `/mnt/sys_drive` is **20 GB**, so on a default box the
largest locally-backupable app is **≈ 10 GB** — **the M band does not fit at all** without a
per-customer `SysDataGrowGB`. **RPO still not measured.**
- **No resource leak.** 9 457 samples of 19 metrics over 13.5 h: controller and agent RSS flat, fds
flat, and **no orphaned volumes, images or containers** despite dozens of redeploys, kills, reboots
and hard resets. The only curve with real slope is the **agent journal, 194 → 463 MB (~20 MB/h)**
bounded by journald, but a lot of logging.
- Every atom and invariant was **proven by hand before automation**; the runner asserts nothing that
was not first observed live.
- **A Phase A gap was caught before the run:** no app had `HDD_PATH`, so all data sat on the system
disk and I3 could never have fired. calibre-web was deployed onto `adatok` first — otherwise the
soak would have produced green cycles that tested nothing cross-drive.
- **Two violations were my harness, not the product**, and run 2a was stopped for them: a seed that
never landed became a fake "stale restore", and a real one would have looked identical. Fixed and
red-proofed before restarting.
## Findings
- **R-156 (new)****papra's data is neither persisted nor backed up, and it reports healthy.** The
template mounts `papra_data:/app/data`; the app writes `/app/app-data/db/db.sqlite`. The volume is
empty and root-owned (the image is `-rootless`, so the app cannot even write there), the real DB
sits in the container's writable layer, and the healthcheck only probes the HTTP port. Its
Tier-1/Tier-2 backup is real, verifiable, and contains nothing. Not fixed.
- **R-157 (new)****bootrecon's start-once sweep misses the boot orphan it exists to recover.** Two
mechanisms. **A:** the container is left `Exited`, the sweep runs ~5 s after controller start while
docker is still restoring, sees "no boot-orphaned apps", and never re-checks (3 occurrences,
intermittent ~50%). **B:** the interruption leaves the stack with **zero containers**, which is
exactly the signature bootrecon deliberately skips as a user's Stop — and in that state the deadapp
check reported **`0 currently down`** while a `deployed: true` app was not running, i.e. silent on
every channel. A settle-condition fix closes A and leaves B open. Not fixed.
- **Tier 3 could not be isolated, so it was not run.** Offsite hard-requires the DR tier
(`configs.go:1300`), and the DR tier only provisions on ep0 (per-endpoint allocation deferred,
`hub/README.md:260`). Both are recorded deliberate positions, so **no R-n minted**. The campaign
therefore touched neither ep0 nor the Storage Box — stronger isolation than asked for, obtained by
not running the tier. Cost: all Tier-3 atoms, I8, and the Tier-3 RTO/RPO rows.
## What did not run
**12 of the brief's ~13 atom families ran** (run 1 covered 6; run 2b added abort-fs-in-place,
kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency atoms and fill-drive).
Previously reported as 6 of 12 — that was run 1 only.
**Superseded detail:** Still not run: **Tier-3 backup/restore** (§3, structurally un-isolatable) and **I8**. **I9** was not
automated — cited from the tester-gate run on this same controller 0.188.0, not re-claimed.
`kill_controller` is still not literally "mid-backup"; the dedicated concurrent backup+detach atom
covers that case properly. The run-1 flaw where `reboot` never interleaved with a detach was fixed.
**Depth reached: 39 consecutive cycles**, past the brief's "thirty-eighth", with c34c39 clean on
every invariant. Beyond 39 is untested, not proven clean.
## Teardown — OWED, nothing removed
Still intact: the rig is reusable for the atoms that did not run. VM 311, `c10-scratch`, PBS datastore
`felhom-c10` + user/token, restic subaccount `u629488-sub4`, and **hub customer `c10-soak` (disposition:
DELETE)** are all outstanding, with commands in the audit §9. Named explicitly because R-131 is four
orphaned scratch customers left by exactly this omission.
-43
View File
@@ -1,43 +0,0 @@
# REPORT — CAMPAIGN 7 (felhom.eu side: docs only)
> Written as `REPORT-campaign7.md`, **not** the shared `REPORT.md`, per the convention this run
> added to `CLAUDE.md`: `REPORT.md` is overwritten, so a second concurrent session in this repo
> would clobber it. This session's implementation work was in `app-catalog-felhom.eu`; here it only
> touched documentation.
**Run:** 2026-07-18 evening → 2026-07-19 morning. **Class:** campaign (record-and-rank + a defined
allowed-fix set). **Implementation repo:** `app-catalog-felhom.eu` (see its `REPORT.md`).
## What changed in this repo
| file | change |
|---|---|
| `documentation/audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md` | **new** — method, uninstall-semantics map, trio detail, full 53-app matrix, ranked findings, coverage |
| `documentation/backlog/ROADMAP.md` | **+3 items** — R-40 (multi-hop major upgrade path), R-41 (no standing catalog deployability check), R-42 (sidecar-major ruling) |
| `CLAUDE.md` | REPORT.md parallel-session rule: the second session writes `REPORT-<topic>.md` |
No hub/agent/scripts/website code was touched (campaign scope: catalog + docs).
## Headline for this repo's readers
1. **Uninstall semantics map row PARTIAL → PROVEN** (campaign doc §2), with live evidence from all
three trio apps: remove requires stop first; named docker volumes are **always destroyed**
(including the app's database); HDD bind-mount data and `backups/primary/<app>` survive unless
explicitly ticked; images are kept; `app.yaml` goes, the template stays; the per-app **offsite
toggle survives** the uninstall while tier-2 config is cleared. The confirmation modal does warn
about the volumes, so there is **no consent gap**.
2. **A lying healthcheck takes an app OFF-LINE, it does not merely mislead.** Traefik will not route
to an `unhealthy` container, so a probe that cannot execute → permanent unhealthy → **404 to the
customer while the app serves 200 on its own port**. 7 of 53 apps were in that state.
3. **The pre-flight gate's own signal is missing:** the 0.145.0 → 0.146.0 floor-lift emitted no
`controller_updated` event, though the identical bootstrap path emitted one for 0.143.0 → 0.145.0
two hours earlier (§0, finding F1). The box did converge — golden, floor and runtime all agreed —
but the event trail under-reports version transitions.
## Open items owned outside this repo
- **plant-it / wanderer** — images do not resolve at all (neither the new tag nor the one the
catalog already ships). Upstream research needed; recorded as findings, not deletions.
- **gokapi** — pinned back to v1.9.6; v2 needs the seeded `config.json` regenerated. Security-
relevant, should not sit on a superseded line indefinitely.
- **glance** — never had a seeded `glance.yml`; proven pre-existing.
-62
View File
@@ -1,62 +0,0 @@
# REPORT — CAMPAIGN 8: the backup & restore subsystem (2026-07-27/28)
Adversarial, destructive, unattended run against `demo-felhom`, `demo-hp` and `ep0`.
Full report: `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
Evidence: `DooPlex:~/campaign8/evidence/` (103 files, 35 MB, written continuously by 11 collectors).
**No production code was changed.** Findings are recorded and ranked, never fixed inline, per the
campaign's own rules.
## Scope safety
`peti-felhom`, its namespace and `u629488-sub2` were never touched. Phase 0 established with five
documented probes that peti has **no data at all** in `felhom-offsite`, which is what made the
operator-approved 100% datastore-fill safe. The 13 GB rollback copy `/srv/pbs-felhom` on ep0 is
intact.
## Findings
| # | Finding | Severity | Class |
|---|---|---|---|
| F-CRIT-1 | An app that fails to restart after a quiesce **never alarms**, on any channel | **HIGH** | DEFECT |
| F-CRIT-2 | A failed offsite backup leaves a phantom snapshot that **resets tier freshness** (up to 7 days silent on real cadences) | **HIGH** | DEFECT |
| F-A1 | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker, pages the operator | MEDIUM | DEFECT (behaviour) / ARTIFACT (frequency) |
| F-HUB | The hub dropped an event under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT |
| F-LEAK | A **failed** restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); leaks are never reclaimed | MEDIUM | DEFECT (root-caused by fault 18) |
| F-REBOOT | A guest rebooted during its backup **does not come back** — shutdown completes, start never happens, no self-heal | MEDIUM | DEFECT |
| F-DIAG | Four distinct offsite failure causes collapse into two operator-visible strings | LOWMED | DEFECT |
| F-OBS | `deadapp-check` leaves no positive observable on a default (info-level) box | LOW | DEFECT |
| F-OPS | A manual `pct restore` inherits the source guest's binds (live data drive + another guest's credentials) | LOW | Operational |
Both HIGH findings are in the same place: **the system's ability to tell you a backup did not
happen.** Both cite the code and the comment that asserts the property the code does not provide.
## What is now proven that was not before
R-88 breaker arming **and its full ladder** (15m/30m/1h/2h/4h/4h-cap) · per-tier isolation under a
real one-tier-fails case · `whole_guest_backup_failed` end-to-end with correct tier attribution ·
R-97c operator-only routing verified against the hub DB (zero customer rows with `status='sent'`) ·
`whole_guest_backup_recovered` + the R-68 pairing gate firing live · **`age_state=absent`** ·
R-97b's suppression half · the crash-recovery unquiesce by an actual SIGKILL (1 s) ·
**R-87 — the first restic restore round-trip ever performed**, byte-verified (6/7 sha256 identical,
the 7th explained) · R-82 one-quiesce-two-tiers · single-flight on two independent paths.
## Restore round-trips
restic (R-87) · local vzdump → fresh CT · PBS offsite → fresh CT · corrupted snapshot → fails
cleanly. `mount_parity` exact on both whole-guest tiers, `unprivileged: 1` preserved.
## Fleet state
**Healthy. Nothing left broken.** All four compression knobs reverted and verified; every fault
unwound; no leaked scratch guests, nft rules, ballast files or clock skew; ep0 datastore clean with
zero `.bad` chunks. demo-felhom 15/15 containers healthy, demo-hp 8/8.
## Not tested (with reasons)
Fault 4 (restic transport — four injection approaches defeated by guest-bridged networking; **the
most valuable follow-up**, because F-CRIT-2 raises the same question for restic), fault 12 (host
reboot — reasoned skip), and the agent's own DR bring-up path. Faults 6 and 8 were inconclusive for
documented reasons. Faults 11 and 18 WERE run in the campaign's tail and both produced findings.
**Campaign-caused outage, stated plainly:** fault 11 took demo-hp guest 9201 down for ~9m47s
(the guest did not restart after a mid-backup reboot) until manually started. Fleet healthy after.
## Note on repo conventions
This run touched no `hub/`, `scripts/` or `website/` code, so none of the per-area CHANGELOGs has an
entry — there is nothing shipped to log. The deliverable is the audit document plus this report.
-85
View File
@@ -1,85 +0,0 @@
# REPORT — CAMPAIGN 9: the restore paths, proven (2026-07-28)
**Overwritten** per the standing rule. **No production code shipped** — this was a proof campaign,
and findings are recorded, never fixed inline. Full write-up:
`documentation/audits/CAMPAIGN-9-restore-proof-2026-07-28.md`.
Evidence: `DooPlex:~/campaign9/evidence/` (69 files, 221 MB, 7 collectors, written continuously).
Fleet unchanged and healthy at close: hub v0.80.0, agent v0.110.0, controller v0.182.0 on both boxes.
**`peti-felhom` was never touched.** The ep0 rollback copy `/srv/pbs-felhom` (13 G) is intact.
## The headline — two never-proven restore paths are now proven
Driven through the **real endpoints the UI posts to**, over https through traefik with a real session
and CSRF token, on live hardware.
| proof | result |
|---|---|
| **A1** — Tier-2 restore of ordinary app data (`paperless-ngx`, demo-hp) | 6 deleted files back **byte-identical** (`sha256sum -c` all OK) |
| A1 — „A meglévő fájlok NEM módosulnak és NEM törlődnek" | 2 created files survived; 1 locally-edited file **not overwritten** (edit marker intact) |
| A1 — app stopped/restarted and healthy | stop→copy→start in 39 s, `paperless-webserver` healthy |
| A1 — data **usable by the app**, not just on disk | paperless resolved all 3 docs, checksums matched its own DB, and **served the restored bytes over its own HTTP API** at the exact pre-deletion sha256 |
| **A2** — Tier-1 recovery-unit restore is a **distinct** path | `POST /backup/restore``RestoreFromRecoveryUnit`; ran end-to-end in 18 s, 1 volume restored, app healthy |
| **A3** — restore after **total loss** (whole appdata dir `rm -rf`) | loss proven by doc download going **200 → 404**; restore returned **43/43 files byte-identical**, `documents_ok 16 of 16`, downloads back to 200 |
The honest boundary A1+A3 together establish: **existing files are untouched; destroyed files return
at their last-backup state.**
## Findings — 3 defects, ranked (none fixed)
| # | finding | severity |
|---|---|---|
| **C9-F1** | The Tier-2 restore button is offered for apps it can **never** restore (BookStack, Docmost). It takes a real app outage, restores 0 files, and reports „Nincs hiányzó fájl — minden fájl megvan a helyén." — while 156 MB of that app's data sits unread in the same copy | **HIGH** |
| **C9-F2** | An app in a **crash loop never alarms on any channel**. `StateRestarting` is in no down-set, so the dead-app heartbeat printed *"180 scans … 0 currently down"* while the app had been looping for 9 minutes | **HIGH** |
| **C9-F3** | An **interrupted offsite run** leaves an exclusive restic lock the existing self-heal cannot reach; the tier is dead until a human unlocks, and the operator is told *"unknown reason"* | **MEDIUM** |
Two things were deliberately **not** filed as defects: a recovery-unit poisoning that the catalog
sync self-healed within ~3 minutes (proven live — reporting it would have been reporting an
artifact), and a `snapshot_id` that looked ignored but is documented as logging-only and confirmed
so live.
## Mechanisms confirmed working, live
R-82's one-quiesce rule under mixed outcomes (2 tiers due, apps stopped **once**, per-target
breaker); R-88's breaker (edge-triggered, one WARN, one event, three silent DEBUG skips, **no app
thrash**); F-A1's contention deferral (409 → no breaker, no event, prompt restart — both sides of
the seam captured in the same second); **F-CRIT-2's size filter against a real 1-byte phantom** on
demo-hp, confirmed independently on ep0's filesystem; R-100's success anchor twice; **F-DIAG's
sanitiser on the exact bare-hostname case that defeated its first version** (nothing raw reaches the
hub event or the report); F-OBS's positive observable — which is precisely what made C9-F2 provable;
F-LEAK's fenced destroy (no leaked `990000` guests across ~10 restore-tests).
## Where it stopped, and what remains
Stopped at the **end of Phase B**, plus Phase D item 10, then full recovery. Phase C item 6 (host
reboot mid-backup) was deliberately not started — a large new fault class against boxes that are
remote until ~08-02, and starting it would have meant rushing it or leaving the fleet unknown.
**Approved but impossible:** Phase 0 cleared compressing the hub's `staleAfter` for R-100's
threshold test. It is **not a knob**`cmd/hub/main.go:552` passes `0`, selecting the compile-time
`defaultOffsiteStaleAfter = 48h`. Compressing it needed a hub code change, which the campaign
forbids. Reported rather than worked around. The no-code-change alternative (age the controller's
reported `last_success` past 48 h and let the hub judge at its real threshold) is the recommended
method next time.
**The honest residue — still not proven:** Tier-1 **content** recovery after real loss (A2 ran on an
intact app; A3 used Tier-2) — now the most valuable open item; host reboot mid-backup; three-way
concurrency with GC; Scenario C live; `offsite_stale` actually firing; F-HUB `SQLITE_BUSY`.
## Recovery
Every config reverted from `evidence/config-before/REVERT.md`, each verified with a **positive
observable**: agent cadences back to `0 / 302400 / 604800` on both hosts (`is-active` = active),
windows back to `02:30`, `pvesm` shows `felhom-pbs active` on both, 0 campaign iptables rules on
either host or guest, 0 scratch guests in the `990000` band, all stacks healthy on both boxes, and
the offsite tier not merely unblocked but **proven working again** (`ok`, 1m35s, 8 snapshots).
One benign residue: the in-memory R-88 breaker still holds a `felhom-pbs` failure count on each box.
Its `until` is long past so it blocks nothing; it clears on the next successful backup or any
controller restart (by design, not persisted). Clearing it would have cost another app outage for no
benefit.
**One operational lesson worth a runbook line:** a hand-run `docker compose up -d` in
`/opt/docker/stacks/<app>` starts a Felhom app **without its secrets** — they are injected by the
controller's `stackEnv` at start time, not stored in a `.env`. It turned a healthy docmost into a
crash loop during recovery. Manual recovery must go through `POST /api/stacks/<name>/restart`.
-71
View File
@@ -1,71 +0,0 @@
# REPORT — DIAGNOSE immich offsite restore (felhom.eu side: docs only)
> Written as `REPORT-diag-immich-restore.md`, **not** the shared `REPORT.md`, per the CLAUDE.md
> convention — `REPORT.md` is overwritten and currently holds the 2026-07-18 website refresh.
> No code shipped in this run; findings only.
**Date:** 2026-07-19 · **Box:** demo-felhom (felhom-pve guest 9201), controller 0.146.0, immich v3.0.3
> **CLOSED IN CODE 2026-07-19 — controller v0.148.0.** Findings 1 and 2 shipped as R-43 (offsite
> reconstitution: safety dump → stop → overwrite files → start → replay the snapshot's dump) and
> R-44 (every run dumps before it captures; manifests carry `offsite_run_id` + `dumps_at`). Deployed
> to demo 9201, healthy. **The §9 live acceptance has NOT run**, so no capability-map flip: the
> offsite row is PARTIAL, the customer-restore row stays MISSING, R-3 stays DRAFT. Implementation
> detail lives in `felhom-controller/REPORT.md`.
## What ran
A diagnosis of "immich offsite restore succeeds but photos do not reappear". No product code was
changed: no restore/backup logic, no labels, no flashes, no `restic prune`/`forget`, no snapshot or
escrow changes.
## Outcome
The restore did not fail. **It was never invoked on missing data, and could not have worked
if it had been.**
1. Viktor deleted the 11 photos in the immich UI to test offsite restore. A UI delete means
**trash**, not deletion — no file left the disk. Both „csak a hiányzó fájlok" runs merged
**0 files**, correctly, and flashed success. The test proved nothing.
2. A *valid* test would also have failed: **no offsite path loads a database.** All three buttons
are file-only. Files would return; the library would stay empty.
3. The shipped dump is additionally stale by design — from the 02:30 local run, never refreshed
before a manual push. Probed: **`asset: 0`, `user: 0`, `album: 0`**.
**Photos:** left in trash at Viktor's instruction (recovery not wanted). All 11 files verified
present on disk and all 11 rows intact, so an ordinary un-trash recovers them until immich's
30-day auto-purge.
**Answer to "can a customer trust same-day offsite?"** For a DB-indexed app: **no — files come
back, content does not.** The backup half is honest; the restore half cannot reconstitute the app.
## Decisive evidence
- `updatedAt` == `deletedAt` on all 11 asset rows ⇒ **no restore operation touched the DB.**
- Unit dump `immich-postgres.sql`, 51 954 452 B, mtime **02:30 CEST** ⇒ `asset: 0 / user: 0 /
album: 0`. The 52 MB is entirely immich's shipped geodata reference tables. It predates the admin
user (07:56:25) and the photos (07:57).
- **Latent hazard:** had a full restore loaded that dump, it would have written an empty DB over the
live one, destroying the trashed rows that were the only surviving recovery path.
## Files written
| File | Change |
|---|---|
| `documentation/audits/DIAG-immich-restore-2026-07-19.md` | new — timeline, evidence, source mechanism table, 7 findings |
| `documentation/backlog/ROADMAP.md` | **R-43** (P2-HIGH, offsite restore cannot restore a DB) + **R-44** (P2-HIGH, manual push ships unrefreshed dump) |
| `documentation/architecture/00-capability-map.md` | customer-restore row **stays MISSING**, gains the finding + a method note for R-3; offsite-restore row flagged *scope contested* |
## Needs Viktor
1. **Capability-map ruling (line 61).** The offsite-restore row claims PROVEN-LIVE citing
"immich end-to-end from offsite alone" (`CAMPAIGN-6D`). That phrase is contested by this
diagnosis. I flagged it rather than downgrading a proven row — did 6D's accept leg exercise the
DB half, or only the file half?
2. **Optional real red-proof.** Emptying the trash would create genuine data loss and convert the
dump-probe inference into a live proof. Offered, **not run** — irreversible, and the probe
already settles it.
3. **Unreconciled: the 704.6 MiB figure.** Measured 126 MB total on the library storage. If
704.6 MiB came off a controller Storage page, that gap is its own defect.
4. **Orphaned pre-v3 tree** `dccc13fe…` (~55 MB across upload/thumbs/encoded-video) stranded by the
immich 2→3 redeploy — worth a sweep policy for major redeploys.
-53
View File
@@ -1,53 +0,0 @@
# REPORT — Megosztás diagnosis (2026-07-20)
Topic-scoped report (parallel-session rule: shared `REPORT.md` untouched).
**Run:** RUNBOOK "Megosztás diagnosis — SMB unreachable from Mac + sharing-page reload loop".
Read-only diagnosis. **No code changes, no version bumps, no builds, no restarts.**
**Deliverable:** `documentation/audits/DIAG-sharing-2026-07-20.md`.
## Verdicts
* **Reload loop — ROOT-CAUSED (HIGH).** `sharingStatusHandler` (`sharing_handlers.go:246`, added in
`b5d78d1`, controller v0.147.0, 2026-07-19) coerces `idle``running` whenever the samba
container is alive. `sharing.html` L320326 treats `running` as a one-shot job-success edge and
calls `location.reload()` 1.2 s later. The first `tick()` fires synchronously on every page load,
so the page reloads forever. Unconditional for any customer with sharing enabled — the Megosztás
page is currently unusable. Proven live: 6 consecutive `/sharing/status` polls all returned
`{"phase":"running","running":true}`, and the controller log shows **no ensure job ran at all**,
so the phase is manufactured by that line rather than left over from a stuck job.
* **`smb://192.168.0.162` — ROOT-CAUSED.** `.162` is the Proxmox host and never was an SMB endpoint.
smbd runs in guest 9201 and binds `192.168.0.104:445`. `nc` from the host: `.104:445` **open**,
`.162:445` **refused**. Wrong target, stale Finder favourite.
* **`smb://FELHOM` — OPEN, narrowed.** NetBIOS resolution works on the wire
(`nmblookup -B 192.168.0.255 FELHOM``192.168.0.104 FELHOM<00>`), but the stack advertises **no
mDNS/Bonjour** (nothing on udp/5353; the R-6 spike selected `smbd + nmbd + wsdd` only) — the
mechanism macOS Finder prefers. Closing this needs one probe from the Mac (`smbutil lookup FELHOM`
/ `dns-sd -B _smb._tcp`), listed in the audit's Mac test matrix.
* **H2 (container down/crash-looping) — RULED OUT** at the first probe: `felhom-samba` Up 3 h, clean
logs, smbd/nmbd/wsdd all bound as `infra/samba.go` intends, live `smb.conf` matches the renderer
with no baked address literal, no PVE firewall in the path.
## Findings (full table + evidence in the audit doc)
S-1 HIGH reload loop (XS fix: latch a `sawInFlight` flag in the JS; red-proof required) ·
S-2 MED the UI never shows the connect address, only `\\FELHOM` — customers guess IPs ·
S-3 MED no mDNS advertisement (image slice, needs republish) ·
S-4 LOW ensure-job phase never resets — fold into the async-job-feedback roadmap item ·
S-5 INFO the guest's LAN IP is DHCP, so any displayed address must be read live, never cached.
Recommended packaging: S-1 as an immediate patch task (it bricks a shipped page), S-2 alongside it
if the guest IP is reachable from the sharing handler, S-3 as its own slice.
## Also noticed
The **remote site's LAN is `192.168.0.0/24` — the same prefix as the DooPlex home LAN** that the
Tailscale subnet router advertises (`192.168.0.180` shows `FAILED` in felhom-pve's neighbour table).
A successful `ping 192.168.0.162` therefore does not by itself prove the Mac is on the remote
segment; the Mac matrix starts by confirming which network it is on.
## Actions taken
None. Every command was a read, except a `POST /login` to obtain a session for the status polls.
No secrets are recorded in either document.
-85
View File
@@ -1,85 +0,0 @@
# REPORT — R-111 fixed, then E-2 proven on a fresh box (2026-07-29)
Two phases in one session. Full evidence: `documentation/audits/E2D-fresh-vm-2026-07-29.md`.
Root `REPORT.md` untouched.
## Phase 1 — R-111: the Day-0 channel now serves the current software
A Phase 0 gate earlier the same day stopped the E-2d run before any VM existed: a fresh box would
have installed **agent 0.96.0 + controller 0.161.0**, ~17 and ~24 releases behind `main`.
| | Before | Now |
|---|---|---|
| agent (Gitea generic) | 0.96.0 | **0.113.0**, sha `5f3247f7…`, round-trip verified |
| golden (Gitea generic) | 0.161.0 | **0.185.1**, sha `dba00f3e…`, embeds controller 0.185.1 |
| hub `min_agent` | 0.93.0 | **0.113.0** (what controller v0.185.0 declares) |
Bake clean on every marker: `Result=success`, overlay2, **all three mounts in the archive**, 0
FATAL/exclusions, HTTP 201, token-leak grep 0. GL-1 teardown: guest 9100 purged, secrets shredded,
drill disk restored to `virgin`. Agent + golden moved in **one** manifest POST so it never vouched a
new agent against an old golden. `min_agent` verified zero-impact first (all three enrolled hosts
already at 0.113.0). Global floor deliberately **not** raised — the golden now bakes 0.185.1.
Commit `3dff357`.
## Phase 2 — the E-2d run, full ISO/PAIRING route
Nested PVE VM on demo-hp, one disk, outside the `felhom` pool. Bind → running controller in
**3 m 35 s**. The install fetched exactly the artifacts published an hour earlier and restored
`vzdump-lxc-9100-2026_07_29-12_37_56` — the golden baked 20 minutes before. The publish train is
proven end to end on a real install.
| Claim | Verdict |
|---|---|
| **C1** host-install 1.22.0 completes a real install, rc=0 | ✅ **PROVEN** |
| **C2** Case B fires naturally | ✅ **PROVEN** — both DEGRADED lines verbatim, `local_backup_target=local`, install did not abort |
| **C3** degraded banner renders **to a customer** | ⚠️ **PARTIAL** — API byte-exact; **no UI consumer exists****R-112** |
| **C4** offer appears and moves the target | ⚠️ **PARTIAL** — decline path, `restart_required:true`, no self-restart, E-2a wrapper, healthy-renders-nothing all PROVEN at API level; offer equally invisible → **R-112** |
| **C5** `backup_target_absent` end to end | ❌ **FAILED** — zero events on any channel → **R-113** |
## The three findings
**R-112 (P1)** — E-2's banner and offer have **no UI consumer**. The endpoint returns byte-exact copy;
`grep 'backup-target'` across every `*.html`/`*.js`/`*.css`**0 hits**, and no page handler injects
the state. Decisive contrast: templates fetch **18** distinct `/api/storage/*` endpoints;
`backup-target` and `backup-target/assign` are the only two with zero references. v0.185.1 fixed the
router mount and stopped one layer short of the render. Fifth instance of seam-built-but-never-wired.
**R-113 (P1)** — the drive-absent gate **cannot fire on device loss**. `planDriveGates` reads presence
from `BoundUnderParent` = "is this path in the guest's mountinfo". The raw mount is a device-bound
systemd unit and dies; **the agent's own bind is not device-bound and outlives the device**, so the
gate sees "present" forever. Live: agent said `enrolled drive absent by UUID` every 20 s for 4½
minutes, controller logged **0** `[gate]` lines, hub got **zero** events — neither the specific nor the
generic one. Sixth instance of the class, one layer deeper: E-2b wired the seam to a condition that
cannot occur.
**R-114** — on target-drive loss the message says the backup is *"on the same disk as the system"*
(false) and offers **the drive that just vanished**. Invisible today only because of R-112 — so
**R-114 must be fixed before R-112 is wired.**
Also filed as a **second instance under R-110** (not a new ID): host-install fetches **nine** files
from `raw/branch/main` and the hub vouches a sha for **one**; E-2a's wrapper is installed 0755 to
`/usr/local/sbin`, root-fenced in sudoers, validated only by `bash -n`.
## Record
- `OPEN-ITEMS.md`**R-112/R-113/R-114 opened**; E-2d re-stated with results and left open for the
residue; E-2's "NOT yet live-proven" list resolved into proven / known-broken; R-94 fully unblocked;
R-110 extended. The drill-cleanup row was opened and then **closed the same session** once the
teardown completed, so it is not carried in the register.
- `ROADMAP.md` — R-112/R-113/R-114 under P1; R-111 marked SHIPPED.
- **`architecture/00-capability-map.md` not touched** — for two reasons: the customer-facing legs are
broken rather than proven, and the map has **no E-2 / backup-target rows at all** (worth noting
against the ROADMAP's coupling rule).
## Teardown
VM destroyed, scratch storage removed, **`pvesm status` after == before** (`local-lvm` 38.77 %,
byte-identical), guest 9201 and drill-r50 untouched. **Hub records removed — teardown complete.** The delete was correctly refused at four gates while the host still read ONLINE; once the destroyed host aged to DOWN (`delete-impact``deletable:true`) the documented cascade ran and completed: host deleted, PBS tenancy deprovisioned, claim reset, residue purged. Verified after: **0** `e2d` occurrences on the hosts page, fleet unchanged. The one purged `appliance_registrations=1` was this run's own appliance; the unrelated stale 2026-07-25 appliance (`206c8838…`) was not touched by the cascade — the operator removed it separately.
## One human step, and a premise correction
The runbook's §5.1a operator STOP (the bind) is **retired** — CC did it. But E-2d's premise that a
fresh install yields a CC-drivable claimable customer is **wrong**: the claim code is bcrypt-hashed and
email-only, and the gate covers everything except `/claim`, `/api/health`, `/static/`. One operator
relay of the emailed code was required — which also proved the claim flow end to end.
-92
View File
@@ -1,92 +0,0 @@
# REPORT — ep0 PBS datastore relocated onto the 100 GB volume (2026-07-27)
**Class:** supervised operational run (RUNBOOK execution). **No code changed. No version bump.**
Written as `REPORT-<topic>.md` per the parallel-session rule — the shared `REPORT.md` was not touched.
**Full record with all evidence:** `documentation/runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md`
---
## Outcome: DONE and verified
`felhom-offsite` now lives on a dedicated 100 GB Hetzner Cloud Volume instead of ep0's 40 GB root disk.
| | Before | After |
|---|---|---|
| Path | `/srv/pbs-felhom` (root disk) | **`/mnt/pbs-datastore`** (volume) |
| Datastore total | 37.2 GB | **98 GB** (hub gauge: 97.9 GB) |
| Used | 28.9 % | **13 %** (hub gauge: 12.6 GB, 13 %) |
| Headroom to the 80 % warn | 19 GB | **≈65 GB** |
| Additional customers before warn | ≈2 | **≈713** |
Datastore **name unchanged** — the PBS-DR descriptors, per-box storage ids, ACLs and namespace
layout that R-39/R-82 made self-healing are untouched.
**Window:** 06:58 → 07:19 UTC (PBS down 07:00 → 07:17). **Nothing was deleted.**
### Acceptance evidence
| Gate | Result |
|---|---|
| Copy integrity | 13,242,207,822 = 13,242,207,822 B · **9,748 = 9,748 chunks** · 7 = 7 snapshots · `backup:backup` · itemised dry-run **0 lines** |
| Snapshot counts per ns | `demo-felhom` 2=2, `demo-felhom-01` 3=3, `demo-hp` 2=2 |
| atime semantics | `rw,relatime,discard`**`relatime` present, `noatime` absent** (GC correctness) |
| Verify job | `TASK OK`, 3/3 groups, forced re-verification of every snapshot, **0 errors** |
| §6 mount guard | **refusal observed**`Job … failed with result 'dependency'`; mountpoint stayed empty |
| §8 restore round-trip | `source_tier: pbs`, `pass: true`, `mount_parity: ok`, clean teardown, 12m1s |
---
## Three findings the operator should act on
1. **`scratch` datastore is configured at a path that does not exist** (`/srv/pbs-scratch`).
Pre-existing, not caused here, but now logs `ENOENT` on every PBS start. This is the PRIME RISK
shape ("reports fine, is not there") already live in the config. **Decision needed:** remove the
stanza or create the directory.
2. **The runbook's §6 acceptance test proves the wrong proposition.** `RequiresMountsFor` is a
mount-first ordering guarantee, not a refusal — systemd silently *re-mounts* an unmounted volume
and PBS then starts safely. The test only bites when the device is genuinely unavailable, which
is how it was re-run and passed. **Amendment recommended in the runbook record.**
3. **§11 — storage box `u629193` is NOT simply unused.** No live backup path references it (no
datastore, no restic repo, no fstab, no `known_hosts` pin; R-17 already deleted `u629193-sub1`),
**but ep0 carries an enabled, currently-mounted sshfs unit** `mnt-pbs\x2dstoragebox.mount`
`/mnt/pbs-storagebox`, holding spike leftovers. Disable and remove that unit before deleting the
box, or ep0 logs a failed mount every boot. **The deletion is the operator's console click.**
## Deviations from the runbook as written
- **The volume arrived already formatted and mounted** by Hetzner at `/mnt/HC_Volume_106469259`
(§2 assumed neither). Operator ruled: reformat + repath. The 5 % reserve was reclaimed (`-m 0`).
- **§8 ran on demo-felhom, not demo-hp** — DooPlex holds no SSH key for demo-hp (the G1 gap). Same
tier, same relocated datastore, larger archive.
- **The window was contended** by a stale 10-minute restore-test cadence on demo-felhom: the config
had already been reverted to 3.5 days on disk, but the cadence is read once at daemon start and
`NRestarts=0`. Restarting the agent applied it (`cadence=84h0m0s`). The in-flight test was allowed
to finish rather than aborted.
## Process errors made during this run (recorded deliberately)
- `rsync -aHAX` **OOM-killed** ep0 (3.7 GB RAM, no swap). Cause: a PBS `.chunks/` tree pre-creates
all 65536 shard dirs → 75,341 inodes, and `-H` retains the whole inode map. `-H` was dropped only
after **proving** no hardlinks exist (`-links +1` → 0; max link count → 1); PBS references chunks
by digest, never by hardlink.
- `/usr/bin/time -v` is not installed on ep0 → exit 127, rsync never ran, and a `| grep … || true`
wrapper swallowed it while printing a success-looking line.
- `rsync --version | head -1` reported a working rsync 3.4.1 as "missing" — **the §12 pipe-into-head
trap, fourth recorded instance in this project.** Both fixed by capturing the command's own `$?`.
## Deferred
1. **Old copy retained** at `/srv/pbs-felhom` (13 GB, 9,748 chunks) as the rollback. Rollback is a
two-line `datastore.cfg` revert. Reclaim only after a new weekly offsite backup lands on the
volume, with explicit go-ahead.
2. **GC not run** — now unblocked by the round-trip, but left for a separate deliberate run. No GC
schedule is configured on this PBS at all.
3. ~~Hub PBS-DR capacity gauge not re-read.~~ **CLOSED — verified correct.** The hub operator UI
(Offsite → PBS DR) reports `felhom-offsite (ep0)` at **97.9 GB capacity, 12.6 GB used, 13 % full**,
agreeing with the on-box `df`. The gauge follows the datastore's configured path, so the move
needed no hub-side change and the suspected "wrong filesystem" bug does not exist.
4. **ep0 has no swap** (temporary 4 GB file removed; box left as found). Worth a small permanent
swapfile — outside this runbook's scope.
-63
View File
@@ -1,63 +0,0 @@
# REPORT — F-CRIT-1 + F-A1 fixed (controller v0.179.0, 2026-07-28)
Docs here. Implementation, all six red-proofs and the full live replay live in
`felhom-controller/REPORT.md`. The campaign that found both:
`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
## What changed
**F-CRIT-1** — an app that failed to restart after a quiesce never alarmed, for two independent
reasons, either of which alone kept it dead: `restartAll` returned nothing (the failure was logged
and dropped), and `classifyRunStates` whitelisted `StateStopped` on invariant I1 ("the user stopped
it") — which the quiesce loop had made false by stopping stacks the same `docker compose down` way.
A failed restart and a user stop are the *same* Docker state; the only difference is that the loop
tried and could not, now surfaced by `Loop.FailedRestarts()`.
**F-A1** — HTTP 409 is the agent's single-flight gate refusing while a restore-test holds it, not a
failure. It is now contention: the tier stays DUE, is dropped before anything stops, and unending
contention raises its own **BLOCKED** signal rather than going silent.
## Bounds, justified against measured reality
- `contentionRetryAfter` **15m** — longest restore-test observed on the fleet is 12m01s; the agent's
local restore-test wait is 10m. Caps app-stop churn at 4/hour instead of 12/hour.
- `contentionAlarmAfter` **3h** — the agent's own PBS restore-test task is capped at 120 minutes, so
contention outliving that is a *stuck* gate, not a busy one. 3h adds margin and is 15× the longest
contention actually observed.
## Verified live, with the hub DB as arbiter — not from logs
Same box, same day, same event type; the only difference is 409 versus a genuine error:
| injection | operator emails (demo-hp) |
|---|---|
| **409 contention** | 8 → **8** (none) |
| **real transport failure** | 8 → **9** |
And for F-CRIT-1: the failed restart alarmed **9 seconds** after grace expiry with the dashboard
banner naming the `(stopped)` state, while a **deliberate** user stop on the same box stayed silent
through **9** dead-app scans (the positive observable that the silence is suppression, not a dead
detector).
## The rule this arc earned
Added to **both** copies of `CLAUDE.md` (live + `documentation/runbooks/workspace-CLAUDE.md`):
**a comment asserting an invariant needs a test pinning it, or it is a wish.** Six instances in this
project have shipped guarantees the code did not provide — `EffectiveProtected`, `newestArchiveOn`,
the R-97a operator-only claim, `classifyRunStates`' I1, `inflight.go`'s defer claim, and
`quiesce.go`'s spurious-failure claim. Two were found only on live hardware, and one of those had a
green, red-proofed test suite over a production path broken two independent ways.
Corollary recorded with it: prefer a test that asserts the **consequence** (does the alarm fire?)
over one that asserts the **mechanism** (does suppression expire?). R-97b's Scenario F proved the
mechanism; the consequence was still broken.
## Docs touched
- `documentation/backlog/OPEN-ITEMS.md` — F-CRIT-1 and F-A1 → SHIPPED + PROVEN-LIVE.
- `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md` — both rows struck through, closing
section added. **All three of the campaign's alarm findings are now closed** (F-CRIT-1, F-CRIT-2,
F-A1).
- `documentation/runbooks/workspace-CLAUDE.md` — the invariant rule.
## Still open, highest first
**Fault 4** (restic transport interruption) — four injection approaches were defeated by
guest-bridged networking, and it is now the most valuable follow-up: F-CRIT-2 answered the phantom
question for PBS and left the identical question open for restic. Then **R-99** (prune never removes
phantoms) and **F-LEAK** (a failed restore-test cannot destroy its own scratch guest — observed
again during this work).
-40
View File
@@ -1,40 +0,0 @@
# REPORT — F-CRIT-2 fixed: a failed backup no longer looks like a fresh one (2026-07-28)
Scope: `felhom-agent` v0.105.0 → **v0.106.0**. Docs here. Implementation detail and the full live
re-test live in `felhom-agent/REPORT.md`; the campaign that found it is
`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
## What changed
`NewestArchiveTime` counted an aborted PBS upload (1 byte, manifest-less, and NEWEST) as a
successful backup, so the tier read fresh, went **not due**, and was never retried — seven days of
silence on the real 168h cadence, invisible to both the R-88 breaker (defers only *due* tiers) and
the hub deadline monitor (reads the same freshness). It now counts only *plausibly complete*
entries via a measured 1 MiB floor; undecidable ⇒ not counted.
**Size is the only tier-agnostic discriminator.** `verification` and `encrypted` are absent on every
local (dir) archive AND on a good PBS snapshot until `verify-new` catches up — gating on either
would have rejected 100% of local backups and produced fleet-wide backup thrash. That inverse risk
is a first-class test, red-proofed by making the filter reject everything.
## Verified live, not just in unit tests
Campaign fault 2 was replayed against the fixed agent on demo-hp — phantom created, rejected and
announced once; the tier correctly reported DUE and backed up (4,359,968,099 B landed); and the
inverse showed **no thrash**, with 91 scheduler ticks as the positive observable that the loop was
alive rather than dead.
## Settled along the way — no retention bug
Server-side prune does **not** count phantoms toward `keep-last`: a dry-run against three real
snapshots plus a phantom retained two real ones plus the phantom. The feared "two phantoms ⇒ zero
real backups" does not occur. Prune never removes them either, so they accumulate one per aborted
upload — filed as **R-99** (LOW, hygiene), not as a retention bug.
## Docs touched
- `documentation/backlog/OPEN-ITEMS.md` — F-CRIT-2 → SHIPPED+PROVEN-LIVE; **R-99** filed;
**F-CRIT-1** filed as READY-HIGHEST (Campaign 8's other HIGH finding, untouched here).
- `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md` — F-CRIT-2 row struck through and
a closing section added.
## Still open, highest first
**F-CRIT-1** — an app that fails to restart after a quiesce never alarms, for two independent
reasons. Then fault 4 (restic transport interruption), which this fix makes more pointed: the
phantom question is now answered for PBS and still open for restic.
-140
View File
@@ -1,140 +0,0 @@
# REPORT — F-REBOOT + F-LEAK + F-OBS, and two investigations (2026-07-28)
Scope in this repo: **`scripts/felhom-host-install.sh` v1.20.0 → v1.21.0** (which is where F-LEAK's
*actual* fix lives), plus the Campaign 8 audit doc and `OPEN-ITEMS.md`. Written as
`REPORT-freboot-fleak-fobs.md` so the shared `REPORT.md` is not clobbered.
Code companions: `felhom-agent` v0.106.0 → **v0.110.0**, `felhom-controller` v0.179.0 → **v0.180.0**.
**Correction to this repo's part of the story:** v1.21.0's band-scoped ACL is *not* the final F-LEAK
fix. It works, but only **once per slot** — PVE's destroy path calls
`AccessControl::remove_vm_access($vmid)` (`API2/LXC.pm:906`), which deletes every ACL at `/vms/<vmid>`
(`AccessControl.pm:1898`), so **the grant is consumed by the operation it authorises**. Found by counting
ACL rows after the first successful teardown (`/vms/990000` → 0 grants), not by reasoning about it. The
durable fix is agent **v0.110.0**'s band-scoped fenced destroy; v1.21.0 remains valuable because it makes
the common case need no privileged call, and it is now the *first* of two layers rather than the only one.
## Baselines (reconfirmed, not copied)
`felhom.eu d0cec9d`, `felhom-agent af1c21a`, `felhom-controller fb91c8d`, all clean. Agent `0.106.0`
and controller `0.179.0` live on both demo boxes.
---
## host-install v1.21.0 — F-LEAK, and why the fix landed *here* rather than in the agent
**The finding.** A restore-test whose restore **fails** leaves a scratch guest the agent cannot destroy
(`403 missing privilege VM.Allocate`), so a half-restored guest holds its disks until a human removes
it and the 10-slot scratch band shrinks silently.
**The cause is structural, not a missing privilege in the role.** `FelhomAgentGuest` is granted at
`/pool/felhom`, and **a guest joins that pool only when its restore completes**. A failed restore
therefore produces a guest that exists, is in no pool, and is out of the token's reach entirely.
**The first fix was wrong, and its own live replay is what proved it.** Agent v0.107.0 shipped a
teardown fallback that adopted the stranded guest into the pool and retried — reasoning from
`Pool.Allocate` on `/pool/felhom`. It fired exactly as designed and PVE refused it:
```
ERROR restore-test: pool adoption failed; left for Recover vmid=990000
err="proxmox: PUT /pools/felhom -> HTTP 500: permission denied at /vms/990000 (missing privilege ...)"
```
`PUT /pools/{pool}` **also** requires `VM.Allocate` on the VM being added. **Pool membership cannot
bootstrap its own authority.** Removed in agent v0.108.0 rather than left in place — a path that
provably cannot work is worse than none, because it reads as a fix.
**What shipped instead.** `apply_scoped_acl` now grants `FelhomAgentGuest` at each `/vms/<id>` in
`PVE_SCRATCH_VMID_MIN..PVE_SCRATCH_VMID_MAX` (990000990009 — the band the restore-test already picks
from), to **both** the user and the token, because the privsep-intersection rule applies here as
everywhere.
Two supporting changes, both load-bearing rather than tidy-up:
- **`remove_scoped_acl` deletes the band grants before the role delete.** PVE refuses to delete a role
still referenced by any ACL, so omitting this would have broken the uninstall — a failure that would
only surface on a decommission.
- **`step_verify` asserts the band grants.** A missing one is otherwise invisible until a restore-test
*fails*, which is precisely the case that leaked a guest in the first place.
### Why the grant is still not a widening — proven live, at the seam the defect lives in
A real PBS restore to `990000` **without `--pool`** reproduced the exact stranded state
(`990000 stopped`; `felhom pool members: [9201]`; `990000 in pool: False`). Then, with the agent's own
token, same guest, minutes apart:
| | `DELETE /nodes/<node>/lxc/990000` |
|---|---|
| **grant removed** (the original defect) | `403 Permission check failed (/vms/990000, VM.Allocate)` — guest still present |
| **grant restored** (the fix) | `200 UPID:...:vzdestroy:990000:felhom-agent@pve!agent` — guest gone |
And it still cannot reach anything else:
| target | result |
|---|---|
| `/vms/990010` (one past the band) | **403** `Permission check failed (/vms/990010, VM.Allocate)` |
| `/vms/100` | **403** same |
`990010` does not exist and PVE **still** answered 403 rather than "does not exist" — so PVE evaluates
**permission before existence**, which makes these genuine authorization refusals rather than artifacts
of a missing guest. Granting at `/vms` was considered and rejected: it would authorise destroying every
guest on the box, including a co-tenant's.
Applied on **both** demo boxes (demo-hp and demo-felhom) so the live fleet matches the installer.
**A careless step of mine, recorded rather than buried.** The probe loop also issued a live `DELETE`
against running guest **9201**. It was refused with `500 container is running` — but the *permission
check passed* (9201 is a pool member by design), so had the guest been stopped I would have destroyed
the live demo guest. The scratch-band probes were the safe ones; 9201 had no business in that list.
---
## Documentation changes
- **`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`** — F-REBOOT, F-LEAK and F-OBS
written up as FIXED with their live evidence, including F-LEAK's refuted first attempt (recorded
precisely *because* it looked right), plus a new **§6b** for the follow-up investigation.
- **`documentation/backlog/OPEN-ITEMS.md`** — three findings closed, **R-100** filed.
---
## R-100 — the investigation's finding, deliberately NOT fixed
**A restic offsite tier that fails every night never goes stale on the hub.** This is **F-CRIT-2's
defect class one layer up and on the other tier** — a *failed* run resetting the freshness clock — and
it was found by asking whether F-CRIT-2's shape existed anywhere else.
Both halves verified in the source, not inferred:
- **Controller:** `o.LastRun = time.Now()` is set **unconditionally** at
`controller/internal/backup/offbox.go:716`, *outside* the `runErr` branch. The failure is recorded
faithfully, but into a different field — `o.LastStatus = "error"` at `:725`.
- **Hub:** `isStale()` reads **only** `off.LastRun` (`hub/internal/monitor/offsite.go:120`, `:127`,
`:131`) and never consults `LastStatus`.
So a nightly restic run that fails every night keeps `LastRun` fresh, `isStale` is permanently false,
and the staleness alarm never fires — with no successful offsite backup having occurred at all.
**Scope of the silence, stated precisely rather than dramatically.** `LastStatus` *does* reach the hub —
it is parsed into the report struct and **only logged** (`offsite.go:270`); it drives no checker and no
notification. The controller's own guest UI surfaces `LastStatus="error"`, so the failure is visible to
someone who looks. What is missing is the **push**: the operator's fleet-wide alarm plane is silent,
which is the plane that matters for an unattended appliance.
Not fixed, per this task's investigation-only scope. Fix direction: gate staleness on the last
*successful* run rather than the last attempt — exactly what F-CRIT-2's `NewestArchiveTime` fix did for
the PBS tier.
---
## Verification
`bash -n scripts/felhom-host-install.sh` clean. The ACL behaviour was verified **live on real hardware**
rather than by dry-run, since the whole finding is about what PVE's authorizer actually does — and the
live run is what refuted my first design.
## Fleet state
Agent **0.110.0** (with the updated sudoers) and controller **0.180.0** on both demo boxes, all healthy.
Scratch-band ACLs at 20 rows on both — re-applied after the attempt-2 destroy consumed one. No leftover scratch guests. demo-hp's `restore_test_cadence_seconds` reverted **600 → 302400**
(a bounded change made for the replay).
`felhom.eu`: this repo has a **foreign uncommitted WIP file** (`documentation/PROMPT-TEMPLATE.md`) from
another session in the shared worktree. Left untouched; my commits staged explicit paths only, per the
never-`git add -A` rule.
@@ -1,119 +0,0 @@
# REPORT — ISO boot branding + single-entry GRUB menu (R-38) · website grid restored (2026-07-19)
> `REPORT-<topic>.md` per this repo's parallel-session rule: another session was writing in this
> clone tonight (CAMPAIGN 7 / `DIAG-immich-restore-2026-07-19.md`), so the shared `REPORT.md` is left
> untouched.
Parts 1 and 2 of the polish train. Parts 3 and 4 landed in `felhom-agent` and `felhom-controller`;
see their own `REPORT.md`.
---
## Part 1 — scripts v1.22.0: GRUB branding + single-entry safety (R-38)
### What shipped
Every ISO is now **repacked** after `prepare-iso`. `mkimage-surgery.sh`**`iso-repack.sh`**:
branding and the slice-B loader swap need the same extract → modify → re-master cycle, so they share
one pass instead of re-mastering twice. **The mkimage recipe is untouched.**
**The safety half — the one that matters.** The stock PVE menu offers *Graphical*, *Terminal UI*, a
serial variant, and an **Advanced Options** submenu holding two `nomodeset` entries, three debug
entries, *Rescue Boot*, memtest and *UEFI Firmware Settings*. Every one of those reaches the
**manual** installer, whose first question is which disk to wipe. They are **not emitted** — not
hidden, not password-gated. What ships is one entry, „Felhom telepítés", default, 5 s.
**Boot behavior is unchanged.** The `linux`/`initrd` lines are lifted **verbatim at repack time**
from the ISO's own *Install Proxmox VE (Automated)* entry rather than frozen into a copy in this
repo, so a PVE bump that moves the kernel path or edits the append line tracks automatically. The
build **fails** if they cannot be found, if the append line has lost `proxmox-start-auto-installer`,
or if `auto-installer-mode.toml` is absent — that last one because without it the single
Felhom-labelled entry would boot a *manual* installer, i.e. exactly what this change prevents.
**Gates, then a re-check against the shipped artifact.** The rendered menu is asserted to have
exactly 1 `menuentry`, 0 `submenu`s and no *live* reference to
`proxtui`/`proxdebug`/`nomodeset`/`Rescue Boot`/`memtest`/`fwsetup` (comments are stripped first —
the template's header names the dropped entries deliberately). Then the menu and theme background
are read back **out of `final.iso`**, not out of the extract tree.
**The boot card.** `grub/generate-grub-background.sh` letterboxes `website/assets/og-image_2.png`
onto a 1024×768 gfxterm canvas at repack time (ImageMagick added to the assistant image), so the boot
screen has **one source** and not a second pre-rendered PNG to drift. The card's own subtle grid
(measured: 4px lines of `#0D131A` on `#0D1117`, pitch 131px) is continued across the letterbox fill
**phase-locked** to where the card's grid lands, so the fill is seamless instead of a 500px square of
grid floating in flat navy. The generator refuses a source whose geometry no longer matches the
measured constants — a swapped asset would misplace every line, and that only shows up on a boot
screen nobody re-checks. Menu positioning needs a gfxmenu theme (plain `background_image` cannot move
the menu off the wordmark), so `grub/felhom-theme.txt` puts it in the lower third the layout leaves
empty, optically centered (measured off a canary screenshot; the comment records the measurement).
### Live validation — nested canary, UEFI/OVMF, PVE 9.2-1
Booted the built canary ISO under QEMU with OVMF and captured the framebuffer.
| Check | Result |
|---|---|
| GRUB renders the Felhom card | **PASS** — background + grid visible at 1024×768 |
| Exactly one entry, selected | **PASS** — „Felhom telepítés" only |
| Hungarian accents under gfxterm | **PASS** — „telepítés", „Indítás … másodperc múlva" render correctly |
| Countdown visible and counting | **PASS** — 5 → 0 |
| Auto-fires at 0 | **PASS** — serial shows ``Booting `Felhom telepítés'`` |
| Unattended install proceeds | **PASS** — „Fetching answers for automatic installation" → auto installer |
| Same abort as v1.21.0 | **PASS**`ERROR: Installation failed: filter did not match any device``Installation aborted`; no disk touched |
A first build **correctly failed closed**: the banned-entry gate matched the template's own
explanatory header. Fixed to strip comments before matching (a comment naming a removed entry is the
point; a directive using one is the bug), which is a gate behaving as designed.
### Artifacts (rebuilt on 180, `/mnt/5_hdd/felhom.eu/felhom-iso/out/`)
| ISO | sha256 | bytes |
|---|---|---|
| `felhom-pve-9.2-1-v1.22.0-n100-generic-mkimage.iso` (safety) | `ff6f06ba1dbfe10f27d703afc29516001000349147426b43c9a424a0ea28bdbf` | 1 704 482 816 |
| `felhom-pve-9.2-1-v1.22.0-n100-demo-generic-mkimage.iso` (real) | `494db0ddf859b6b152cad4d0e0d9e9cefd27255cde07e2b41aba3ac12a217888` | 1 704 482 816 |
| `felhom-pve-9.2-1-v1.22.0-nested-canary-generic.iso` (validation) | `83c61c0413c84e27b26a37bb5dfaed2fcd44fd25e3e571c7310142bd305f2f9d` | 1 705 338 880 |
Both shipping ISOs: `embedding 60 modules`, `El Torito boot images=2`, fs-uuid preserved, and the
post-re-master verification confirming 1 entry + theme background inside the finished image.
**Deliberately not done** (per the task): no squashfs/initrd rebranding — post-GRUB screens are still
Proxmox-branded; no disk-setup or answer-generation change; the pairing banner is untouched.
---
## Part 2 — website: the index grid background, restored
**Archaeology.** Not a deliberate removal. The grid lived as a fixed `body::before` in
`index.html`'s inline `<style>` block and was dropped in **`bed8675`** ("D3 Part 2: index + kapcsolat
on design system v2"), the commit that migrated the page onto the shared `assets/site.css`.
`dd54e4c`, which *created* `site.css`, has no `body::before` at all — it was a porting omission and
nothing took its place. `ccbb13a` (the other five pages) never had it. No asset was lost: the
mechanism was pure CSS (two stacked `linear-gradient`s), which is why nothing looked missing in the
worktree.
**Restoration, not redesign.** Same 50px cells, same 1px lines, same 3% opacity, same
`position:fixed` / `z-index:-1`. One deliberate difference: the accent is the v2 `--blue` `#0083D8`
instead of the retired legacy `#0088cc`, which `site_gates.py` bans. Scoped to `body.page-index`,
because index is the only page that ever had it. `site.css` cache-bust bumped `?v=1``?v=2` across
all seven pages (nginx caches 7 days); BOM preserved on every file.
**Live verification** (felhom.eu, after git-sync deploy):
- Desktop: grid renders behind the hero, at its original subtlety.
- **376px viewport** (via a same-origin iframe — the browser window would not resize in this
environment, so the narrow case was exercised for real rather than asserted): grid renders, mobile
layout unchanged, `scrollWidth === clientWidth` so **no horizontal overflow**.
- Computed style confirmed live: `linear-gradient(rgba(0,131,216,0.03) 1px, …)`, `50px 50px`,
`position: fixed`, `z-index: -1`, `pointer-events: none`.
- `python scripts/site_gates.py`**OK** (BOM, no legacy tokens, no `<style>` blocks, cache-busted).
---
## Docs
- `scripts/CHANGELOG.md` — v1.22.0 entry.
- `website/CHANGELOG.md` — grid restoration entry.
- `documentation/backlog/ROADMAP.md`**R-38 flipped to SHIPPED**; **R-45** (unified async-job
feedback) and **R-46** (verification-copy browse + expiry) added; pre-invite checklist gained the
"golden ≥ 0.147.x carries all four infra images" line.
- Capability map: **untouched** — no capability moved. These are UX and packaging.
-307
View File
@@ -1,307 +0,0 @@
# REPORT — the universal ISO: **PUBLISHED** (2026-07-31)
**Live:** `https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso`
**sha256:** `f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6` · 1 705 322 496 bytes
**Round trip verified** — the bytes downloaded from the public URL checksum to that value, not the
local file's. `.sha256` and manifest published beside it.
> Written as `REPORT-iso-release.md`, not root `REPORT.md`, per the task and the shared-clone rule.
## 0. Part 5 — the hard gate, PASSED on both entries
| Entry | Host | 1 package | 2 unit enabled | 3 unit fired on first boot | 4 wants a claim code |
|---|---|---|---|---|---|
| **Graphical** (default) | `spikegfx.felhom.eu` | `ii felhom-bootstrap 1.26.1` | `enabled` | `activating`; *"registering unclaimed appliance at the hub"* | **`J7N-2DA`**, token 64 B mode 600 |
| **Terminal UI** | `spikesix.felhom.eu` | `ii felhom-bootstrap 1.26.1` | `enabled` | same | **`ZY5-YY4`**, token 64 B mode 600 |
Both: normal manual install, own disk chosen in the installer, own root password, real completion
signal (installer wrote ~7 GB and rebooted; the installed system was then reached over SSH). Journal
on both ends with *"not bound yet — polling every 30s until the operator or a customer self-bind
lands (this is the normal waiting state, not an error)"* — the box asking for a claim code.
Spike 4 reasoned the graphical path would follow from shared `Install.pm`. **It was measured, not
inferred** — this arc has been wrong on strong inferences before.
## 1. Venue and baselines
| | |
|---|---|
| **Host** | `demo-hp` (t740), Tier 0 |
| **VMs** | **500 `spike5-gfx`**, **501 `spike5-tui`** — both created with `qm` so the run is visible in the web console |
| **Storage** | **`spike5`**, `dir` at **`/mnt/nvme-1tb` — the mount ROOT**, `content=images`. Root chosen deliberately: a storage at a *subdirectory* reads `disconnected` forever via the agent's `exactMount` check. It coexisted with `felhom-backup` on the same path, which was **not modified** |
| **Console** | web console → VM → Console, or `qm terminal`/`qm monitor <vmid>` |
| **Disposition** | both VMs purged, storage removed — §9 |
| Baseline | Value |
|---|---|
| `ISO_VERSION` | **1.26.0** (was 1.25.0) — `scripts/iso/build-felhom-iso.sh:51` |
| `SCRIPT_VERSION` | `1.22.0``scripts/felhom-host-install.sh:187` |
| `felhom-bootstrap.sh` @ HEAD | `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` |
| PVE base | `proxmox-ve_9.2-1.iso`, `4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` |
| controller on `main` | `0.188.0` (`4115e88`) |
| `felhom.eu` HEAD at build | clean, pushed, `== origin/main` |
## 2. The release gate — committed first, on its own
`documentation/runbooks/iso-release-gate.md`, commit **`e787391`**, written and pushed **before the
first build** so it could not be rationalised afterwards. Twelve criteria, each checkable against the
uploaded file rather than the build inputs, each carrying the spike measurement that justifies it.
**One criterion was amended before the build, with its reasoning recorded in the runbook.** G6 was
first written with the six-token ban `iso-repack.sh:160-164` enforces, on the rationale *"no live route
to a manual disk-picker"*. That rationale is obsolete for a public image — the ruling makes the manual
installer **the product**. `proxtui` (the Terminal-UI installer we deliberately ship) and `nomodeset`
(its graphics fallback) are dropped **for release images only**; `proxdebug`, `Rescue Boot`, `memtest`
and `fwsetup` stay banned in both modes, and the six-token list is **unchanged** for appliance images.
## 3. The stub package
`scripts/iso/pkg/` — source committed, built by `build-deb.sh`.
**Contents: exactly two files, deliberately not three.**
```
-rwxr-xr-x ./usr/local/sbin/felhom-bootstrap.sh
-rw-r--r-- ./lib/systemd/system/felhom-bootstrap.service
```
The old first-boot stub also wrote `/etc/felhom/bootstrap.env` (0600). This package does not:
`felhom-bootstrap.sh:91` reads it only `if [[ -r ]]`, and its defaults at `:95-96`
(`https://hub.felhom.eu`, `https://felhom.eu/scripts/felhom-host-install.sh`) are **exactly** what the
generic pairing env set (`build-felhom-iso.sh:257-258`). Shipping it would add a 0600 file to a public
package to express values the script already defaults to.
**Dependencies: none, and that is a finding.** `dpkg-deb -I` shows no `Depends` line. The payload is a
shell script and a unit file; the binaries the script calls (`curl`, `ip`, `dhclient`, `python3`,
`systemctl`) run at **first boot**, not at postinst time. **Spike 4's open `dpkg --configure -a`
ordering question therefore does not arise** — confirmed, not carried.
**How the postinst is structurally incapable of failing** — no `set -e`, every statement individually
guarded with `|| true` or an `if`, and an unconditional `exit 0`. `build-deb.sh` refuses to emit a
package that violates any of it.
**The guarantee was tested, not asserted.** Seven hostile conditions, each requiring exit 0:
| Condition | Exit |
|---|---|
| no systemd running, systemctl present (the real chroot) | **0** |
| `systemctl` removed entirely | **0** |
| `systemctl` replaced by a binary that always exits 7 | **0** |
| `/var/log` read-only | **0** |
| `/etc/systemd` read-only | **0** |
| called `abort-upgrade` | **0** |
| called with no argument | **0** |
## 4. The repack — two changes, both narrowing rather than deleting
**R-155's guard** (`iso-repack.sh:100-106`) **protected the single-entry mode's promise**: that menu
shows one item labelled "Felhom telepítés" which boots the *automated* installer, and without
`auto-installer-mode.toml` the same label would drop the user into a manual disk-picker — a button
promising an unattended install that silently does the opposite. That promise is real, so the guard is
**kept unchanged for `FELHOM_MENU=single`** and simply does not apply to `release`, where the absence
of that file is release-gate criterion G1 rather than a defect.
**The menu collapse** happens at `iso-repack.sh:144-148` (the stock `grub.cfg` is replaced by a
rendered template). A `release` template now renders **two interactive entries**; entry-count and
banned-token gates are per-mode; the post-remaster verification reads the count back out of
`final.iso`.
**Ruling — default entry and timeout.** Default is **the graphical interactive entry**; timeout **15 s**.
Reasoning: Spike 1 measured that no automated disk selection can be safe on unseen hardware (no
property distinguishes an internal disk from a customer's backup drive; a two-disk match silently wipes
one), so a public image whose default is unattended puts the unsafe path in front of anyone who boots
and walks away. And Spike 2 lost a probe to a **1-second** menu — a person reading two options needs
longer than a machine.
**The automated entry is absent, not broken.** Skipping `prepare-iso` means no
`auto-installer-mode.toml`, and the stock `grub.cfg` emits the Automated entry only inside
`if [ -f auto-installer-mode.toml ]`. There is no entry that could fail in front of a customer.
## 5. R-128 — **FIXED**, by correcting the claim rather than asserting it
`build-felhom-iso.sh:44` claimed `ISO_VERSION` "aligns with felhom-host-install `SCRIPT_VERSION`".
Nothing evaluated it and the two had drifted. **I did not turn it into a real assertion, because the
coupling it claimed does not exist:** the ISO is a frozen artifact, while `felhom-host-install.sh` is
fetched at run time from the website's git-sync of `main` (R-94/R-110), so whatever version an ISO
carries, the script a box runs is always current. An assertion would invent a constraint. The comment
now states the independence, and `ISO_VERSION` is `1.26.0`.
## 6. Part 5 — the defect, the fix, and where it now stands
### Round 1 (`1.26.0`) — the Terminal UI install FAILED on observable 4
Three of four passed: the package installed, the unit was enabled from inside the installer chroot,
and the unit **fired on first boot** and registered at the hub. The fourth failed:
```
felhom-bootstrap.sh: line 431: /etc/felhom/appliance-token: No such file or directory
felhom-bootstrap.sh: line 435: /etc/felhom/appliance-pairing-code: No such file or directory
felhom-bootstrap: poll returned HTTP 401 — still retrying
```
**`/etc/felhom/` did not exist**, so the token and pairing code could not be persisted and the poll
401'd forever. No claim code would ever appear.
**Root cause, mine.** `stub-first-boot.sh` opened with
`install -d -m 0755 /etc/felhom /usr/local/sbin`. §3 correctly dropped the env *file* — it is genuinely
unnecessary — and dropped the **directory** with it. `felhom-bootstrap.sh` uses `/etc/felhom/` for its
runtime state.
**Why the gate missed it.** G9 proves the packaged script is byte-identical to repo HEAD, and it was.
**I verified the payload files and never the directory the payload writes into** — a check that proves
the thing present and not the thing it depends on.
### The fix, and its red-proof
`build-deb.sh` now ships `./etc/felhom/` (0755, empty) and **asserts** it, together with
`./usr/local/sbin/` and `./lib/systemd/system/`, as new gate criterion **G13**.
**Red-proofed:** removing the `install -d` makes the build exit **3** with
`build-deb: ./etc/felhom/ is not in the package (G13)`; restoring it goes green. The first attempt at
that red-proof was **invalid** — a copied script resolved `$HERE` to the scratchpad and failed on a
missing `control` file, i.e. non-zero for the wrong reason — and was redone in place.
### Round 2 (`1.26.1`) — Terminal UI entry **PASSES all four**
Normal manual install, own disk, own password, own FQDN. Host `spikesix.felhom.eu`.
| # | Observable | Result |
|---|---|---|
| 1 | the `.deb` is installed | **PASS**`ii felhom-bootstrap 1.26.1 all` |
| 2 | the unit is enabled | **PASS**`enabled` |
| 3 | the unit fired on first boot | **PASS**`activating`; journal shows *"PAIRING mode (generic ISO, no baked customer/passphrase)"**"registering unclaimed appliance at the hub"**"registered — appliance token stored (0600)"* |
| 4 | **the box wants a claim code** | **PASS**`/etc/felhom/appliance-pairing-code` = **`ZY5-YY4`**; `appliance-token` present, 64 B, mode `600`; journal: *"not bound yet — polling every 30s until the operator or a customer self-bind lands (this is the normal waiting state, not an error)"* |
That is the product working end-to-end from a public image on a manual install: own disk, own
password, nothing baked, and the box asking for a claim code.
### The Graphical entry — **NOT COMPLETED**, and this is why nothing is published
It reached the installer from the same image (KVM dialog, EULA, and the **Target Harddisk** screen
showing `/dev/sda (20.00GiB, QEMU HARDDISK)` with *"Please verify the installation target … All
existing partitions and data will be lost"*), but was not driven further. `Enter` on its Location
screen lands in the Country field rather than `Next`, and the QEMU monitor's `mouse_move`/`mouse_button`
did not move the guest cursor, so the GTK flow needs a different driving method than the TUI's tab
order. **Part 5 requires both entries. It is not fully passed, so Part 7 did not run.**
The `.deb` path lives in `Install.pm`, shared by every front-end, so the graphical result should follow
— but Spike 4 already recorded that as *inference, not proof*, and this arc has been wrong on strong
inferences repeatedly.
### A fixture bug of mine, recorded twice because it cost two diagnoses
`qm set <vmid> --scsi0 … --boot order="scsi0;ide2"` silently produced `boot: order=net0;ide2` — PVE
processed `--boot` before `--scsi0` existed. Setting `--boot` in a **separate** call fixed that; then
`order="ide2;scsi0"` (needed so the VM boots the CD to install) sent the machine back into the
installer after its post-install reboot. **Detach the CD, or flip the order to `scsi0`, once the
install completes.** Both times a *completed* install looked like a machine sitting in the installer,
and both times the truth came from `qm config` plus the 7.0 GB disk rather than from the screen.
## 7. Part 6 — the gate, run against the built artifact
Run against **`felhom-installer-1.26.1-pve9.2-1.iso`**,
sha256 **`f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6`** — the image the
Terminal-UI install in §6 was performed from, and the one that would be uploaded.
| # | Criterion | Scanned for | Result |
|---|---|---|---|
| **G1** | no `answer.toml` / `auto-installer-mode.toml` | both names at ISO root | **PASS — 0** |
| **G2** | no root password or hash | `.rootpw.txt` companion; the answer file that would carry a hash | **PASS** — no `.rootpw.txt` emitted; no answer file exists to hold one |
| **G3** | no SSH key | `root-ssh-keys`, `ssh-rsa`, `ssh-ed25519` | **PASS** — no answer file; package carries only a script and a unit |
| **G4** | no customer identity | `FELHOM_CUSTOMER_ID`/`RETRIEVAL_PASSPHRASE` with values, claim code, api key, Bearer | **PASS** — only the empty initialisers at `felhom-bootstrap.sh:89` |
| **G5** | credential scan **by enumeration** vs the stock PVE ISO | full recursive file-list diff, both directions | **PASS** — exactly **four** added paths: the three `felhomtheme/` files and `/proxmox/packages/felhom-bootstrap_1.26.0_all.deb`; three removed (`pvetheme/`) |
| **G6** | menu present, both paths, human timeout | entry count, `set default`/`timeout`/`timeout_style`, banned tokens | **PASS** — 2 entries, `default=0` (graphical), `timeout=15`, `timeout_style` underscore |
| **G7** | one `felhom-*.deb`, version recorded | `/proxmox/packages/felhom-*` | **PASS** — exactly 1, `Package: felhom-bootstrap`, `Version: 1.26.0`, **no `Depends`** |
| **G8** | postinst cannot fail | live (comment-stripped) `systemctl start\|daemon-reload\|restart`, network commands, `set -e`, last line | **PASS — 0, 0, 0**, ends `exit 0` |
| **G9** | `felhom-bootstrap.sh` == repo HEAD | sha256 of the packaged file vs the repo file | **PASS** — both `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` |
| **G10** | build inputs committed | `git status --porcelain`, HEAD vs origin | **PASS** — clean and pushed at build time |
| **G11** | published checksum + round trip | — | **NOT RUN** — nothing was published |
| **G12** | bucket stays private | — | **NOT RUN** — the bucket was never touched |
| **G13** | *(new, from Part 5's failure)* every directory the payload writes into is in the package | `./etc/felhom/`, `./usr/local/sbin/`, `./lib/systemd/system/` in `dpkg-deb -c` | **PASS** — all three present in `felhom-bootstrap_1.26.1_all.deb`; asserted by `build-deb.sh` and red-proofed |
**A gate refinement found by running it.** G7 also asked that the ISO's copy of the `.deb` sha256-match
the package built from source. It does not, and cannot: `dpkg-deb` embeds build timestamps, so two
builds of identical source differ. **G9 — the payload's identity — is the meaningful check**, and it
passes. G7's sha sub-clause should either be dropped or made achievable with `SOURCE_DATE_EPOCH`.
## 8. Publication — done, and verified by round trip
Uploaded with `rclone` **in a container, configured entirely by environment variables**, so no
credential file was ever written to disk — the fence asks for config files to be kept out of repo
paths and removed at teardown; none was created to remove. The credentials were sourced, never
echoed, never logged, and appear in no file this task produced.
| Check | Result |
|---|---|
| objects in the bucket | the ISO (1 705 322 496 B), `.sha256` (103 B), `.manifest.txt` (2 492 B) |
| **round trip** | `curl https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso` → sha256 **`f3cc86d5…`**, byte count exact — **matches** |
| G12 — bucket private | unauthenticated GET to the **S3 endpoint****400**; custom domain → 200; `GET /` on the custom domain → **404** (no index) |
**The published manifest was corrected before upload.** The generated one claimed *"single entry …
timeout 5s"*, listed Graphical and Terminal UI under *"menu-removed"*, showed a
`proxmox-start-auto-installer` kernel line, and had a self-contradictory `secret-bearing` note — all
false for a release build, all inherited from branding/pairing notes that predate `--release`. The
generator is fixed and the sidecar regenerated. **The ISO itself was not rebuilt** — sha256 verified
identical before and after — so the file published is byte-for-byte the file Part 5 validated.
## 9. Teardown
**demo-hp:** VMs 500/501 `qm destroy --purge`; **scratch storage `spike5` removed**
(`storage.cfg` back to 4 entries, `grep -c spike5` = 0); `/mnt/nvme-1tb/images/` empty; usage
**6.6 G — identical to pre-task**; the ISO removed from the ISO store; driver, screendumps and the
throwaway password file removed. `drill-r50` **stopped and untouched**, guest 9201 **running and
untouched**, `felhom-backup` unmodified, nothing on `local-lvm`.
**demo-felhom:** not contacted.
**DooPlex:** scratchpad 84 K; build logs and the package build tree removed. `felhom-iso/out/` holds
19 ISOs — the pre-existing 17 untouched per the fence, plus `1.26.0` and `1.26.1`, both unpublished
and **neither with a `.rootpw.txt`**, which is G2's own evidence. Repo tree clean and pushed.
### Hub-side — **cleared**
Observable 4 works *by* the box registering itself, so each proof install created an unclaimed
appliance. All three were discarded: **16** and **17** (the 1.26.0 round), then **18** (the two
1.26.1 proofs). `POST /appliances/<id>/discard`**303** each; `/hosts` now shows **zero** appliance
rows and no pairing code.
The endpoint is `/discard`, **not** `/delete``hub/internal/web/server.go:345`, POST only. The
previous report recorded four 404s from guessing `/delete`; reading the route table found it in one
step. **R-131 gains no row.**
## 10. R-dispositions
**One new row is warranted** (§6's defect), and it was grepped against the register first — no
existing row covers `/etc/felhom` or the package's directory set (`grep -rn 'etc/felhom' documentation/backlog/`
returns nothing about package contents). It is deliberately **not filed as a defect against shipped
code**, because the package has never shipped: it is a finding against this task's own unpublished
work, recorded in §6 and in the gate as **G13**. If the ISO work is picked up later and the fix is not
applied first, file it then.
Otherwise, no new rows. Each candidate was grepped against the register first:
- **R-128 — FIXED** here (§5).
- **R-155 — RESOLVED** here (§4): the guard is narrowed, not deleted.
- **R-154** (`[first-boot]` is automated-only and nothing in the tree says so) — **addressed in code
rather than by a row**: `pkg/build-deb.sh`'s header and `grub-release.cfg.tmpl` both state it with
the measurements. The register row can close when the docs land.
- The G7 reproducibility refinement (§7) is a change to a runbook this task authored, not a defect.
## 11. What did not happen, and what is still open
- **Part 8 partially done.** The release-gate runbook (`e787391`), `day0-install.md` C.0 (ISO vs
manual, and when to use which) and `scripts/CHANGELOG.md` are written. **`OPEN-ITEMS.md` /
`ROADMAP.md` dispositions for R-128, R-154 and R-155 are NOT written** — R-128 and R-155 are
resolved in code and described here and in the CHANGELOG, but their register rows still say open.
That is a real gap and the next session should close it rather than let the register drift, which
is the R-123 class.
- **The `.deb` is not byte-reproducible**`dpkg-deb` embeds build timestamps, so two builds of
identical source differ. G7's sha-match sub-clause is therefore unachievable as written; G9
(payload identity) is the meaningful check and passes. Either drop the sub-clause or set
`SOURCE_DATE_EPOCH`.
- **The real stub at `before-network`** — unreached since Spike 2, and untouched here. It is now
narrower than it was: on the `.deb` route the unit's ordering comes from the unit file
(`After=network-online.target …`), not from `[first-boot].ordering`, so it governs operator-built
appliance images only.
- **Secure Boot** was not exercised. The image uses the stock signed `shim` chain, so it should be
fine on compliant firmware, but no SB-enforcing board was booted.
- **Only virtual hardware** was tested. Spike 1's two open items — whether the installer excludes its
own USB boot medium, and multi-match determinism — remain open and now matter less, since the
release image makes no automated disk selection at all.
-71
View File
@@ -1,71 +0,0 @@
# REPORT — PBS prune moved server-side, write proof closed (2026-07-27)
**Class:** supervised operational run. **No code, no version bump.** Topic-scoped per the
parallel-session rule; shared `REPORT.md` untouched.
**Full record:** `documentation/runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md`
---
## Outcome — all parts complete
| Part | Result |
|---|---|
| 1 — prune gate | **Config-gated.** `keep_last: 0` on the PBS tier, both boxes → `prune_pbs_allowed=false`. No code, **no grant** |
| 2 — prune jobs | 2 jobs, per live namespace, `keep-last 2`, daily **03:30 UTC / 05:30 CEST** |
| 3 — dry run → real | Gate passed; both `TASK OK`; demo-hp 3→2, demo-felhom untouched |
| 4 — write proof | **CLOSED — `TASK OK`, no job errors** |
| 5 — GC | Scheduled `sun 04:30 UTC / 06:30 CEST`. **NOT run** |
| 6 — `verify-new` | **Enabled** (operator ruling) |
| — legacy ns | `demo-felhom-01` deleted with its ACLs + token (operator ruling, confirmed twice) |
| 7 — roadmap | **R-89** + CONTEXT.md note |
## The fix, in one line
`allowPBSPrune := !t.Primary && t.KeepLast > 0` — so setting the PBS tier's `keep_last` to `0`
disables both the `--prune-backups` value and the gate, in one config edit, **while the tier stays
armed**. Verified: `backup tier armed target=felhom-pbs cadence=168h0m0s keep_last=0
prune_pbs_allowed=false`, no `tier REJECTED` line.
## The proof
```
07-27 08:25:47 UTC vzdump (felhom-pbs) -> job errors ← prune denied
07-27 09:37:29 UTC vzdump (felhom-pbs) -> OK ← after the change
```
New snapshot `ns/demo-hp/ct/9201/2026-07-27T09:37:29Z`, chunks 9,787 → **9,813**, 97.0 % reused,
45.80 s, **prune step absent entirely**. Driven via `POST /api/guest-backup/trigger``TriggerNow()`
— the UI's „Mentés most" path, not `--selftest`, not raw `vzdump`.
**Hub gauge evidence NOT satisfied** — a +32.8 MB delta is below its 0.1 GB display granularity, so it
still reads 12.6 GB / 13 %. Stated plainly rather than dressed up.
## The demo-felhom prediction — CLOSED
The claim was that demo-felhom's next weekly backup would make 3 snapshots and reproduce the prune
failure. Neutralised on both halves: the box no longer attempts prune, and `prune-demo-felhom` covers
the namespace server-side (verified live, `TASK OK`). **It will not reproduce.**
## Why it mattered more than the unpruned snapshots
demo-hp's PBS tier had reported failure on **every** backup since the tier was created on 07-26, while
the data landed correctly every time. A tier that cries wolf on every success makes a genuine failure
invisible — which is precisely what happened at 07:13 UTC, when a real outage produced an
indistinguishable result.
## Security property preserved
**No prune right was granted to any box.** Final ACLs are four entries, write-only
(`DatastoreBackup`), live namespaces only. A compromised box still cannot delete its own offsite
backups. `felhom-tenantsync.sh` was **not** edited — the ruling makes its current grant correct.
## Open
1. **R-89** — hub-owned retention policy (today's jobs are increment 1, not a stopgap).
2. **Does the restic key on `storage-box-pool-1` have DELETE rights?** Unanswered, carried in R-89,
and the more urgent half — if so, the daily app-data tier has the identical exposure and
append-only mode is the equivalent answer. Rule once for both tiers.
3. **GC has still never run.** First execution Sunday 04:30 UTC; worth watching, as nothing has ever
exercised it here.
4. Old 13 GB datastore copy still at `/srv/pbs-felhom` — rollback intact.
-194
View File
@@ -1,194 +0,0 @@
# REPORT — R-100: a failing offsite tier must go stale (2026-07-28)
Hub **v0.79.0 → v0.80.0**; companion `felhom-controller` **v0.180.0 → v0.181.0** (the producer, shipped
first). Written as `REPORT-r100.md` so the shared `REPORT.md` is not clobbered.
## Baselines (reconfirmed, not copied)
`felhom.eu 6369570`, `felhom-controller 4056fec`, `felhom-agent d5c7691` — all = origin/main. The only
dirt in `felhom.eu` was a **foreign** `documentation/PROMPT-TEMPLATE.md` (shared worktree, untouched).
Hub manifest **and** running pod both `0.79.0`; `staleAfter` = 48h; controller 0.180.0 and agent 0.110.0
live on both boxes.
---
## The premise was wrong, and it was mine
R-100 was filed yesterday claiming *"the operator's fleet-wide alarm plane is silent"*. Phase 0 refuted
that, twice:
1. **A failing offsite run does alarm.** `main.go:655` wires `SetOffboxNotify``NotifyBackupFailed`;
the notify cooldown is 6h against a 24h cadence, so a nightly failure alarms nightly. Live hub DB:
`backup_failed | operator | sent | 5`, latest 2026-07-27 17:42. The `isStale` doc comment —
*"a recent-but-failing run is NOT stale (backup_failed owns that signal)"* — was **accurate**.
2. **The orphaned-repo path I expected to be an indefinite hole is already covered.** The scheduled run
returns early at `offbox.go:606`, *before* the `LastRun` write at `:716`, so `LastRun` freezes and
`offsite_stale` fires normally.
I could find no failure mode that both advances `LastRun` and produces no operator signal.
**The real defect — defeated defence in depth.** `offsite_stale` is the hub-side, *pull-based* net that
exists to be independent of controller-*pushed* events. Anchoring it on `LastRun` made it depend on the
very thing it backs up: when the push is lost, the net cannot compensate, because the failing controller
keeps refreshing the field the net reads. **F-HUB — this campaign's own finding, the hub dropping an
event under `SQLITE_BUSY` with no retry** — is exactly that loss.
**Honest severity: MEDIUM**, not the top-ranked item. The fix is unchanged; the justification is not.
---
## Phase 0 answers
**P0.1 — a last-success timestamp did not exist.** `OffboxTarget` carried `LastRun`/`LastStatus`/
`LastError`/`LastDuration` only. Recording one is a new field, not a transmission of something known.
**P0.2 — `LastStatus` on the wire**, from 4000 live reports (not from source alone):
| value | count | paired with |
|---|---|---|
| `ok` | 2269 | `last_run` set |
| absent/null | 541 | `last_run` **empty** — never-ran |
| `error` | 27 | `last_run` set |
| **`running`** | 7 | a report captured **mid-run** |
Plus 1156 reports with no `offsite` object at all. **The legacy trap — status absent *with* a real
`last_run` — occurs 0 times**, because `LastStatus="running"` is written the moment a run starts. It is
still handled explicitly, but it is not a live shape. `running` being real is why the verdict ignores
status entirely.
**P0.3 — sweep**
| tier | `LastRun` written on failure? | read as success by a verdict? |
|---|---|---|
| **Offsite restic** | YES (`offbox.go:716`) | **YES — hub `isStale`.** The defect |
| **Tier 2 cross-drive** | YES (`recordTier2Failure`) | No hub verdict; UI only → **R-101, filed** |
| Tier 1 recovery units | **NO** — derived from an actual artifact | structurally immune |
| Shares offsite leg | YES | `sharing.html:180` shows the time only when status=="ok" — honest |
| DB dump | n/a — **event-based** (`db_dump_completed`/`db_dump_failed`) | immune by design |
`offsite.go` is the **only** hub verdict anchored on a `LastRun`-shaped field. The deadline checker
already uses distinct success/failure *events* — the pattern this converges on.
**P0.4 — the customer is NOT shown a failed offsite run as successful.** `backups_remote.html:34-36`
leads with the status (`✓ Rendben` / `✗ Hiba` / `Fut…`). Two narrower Tier-2 instances → **R-101**.
---
## The fix
**Controller v0.181.0 (producer, shipped first).** `OffboxTarget.LastSuccess`, carried on the report as
`last_success`. The rule is a pure function called unconditionally beside the `LastRun` write:
```go
func offboxAnchorAfterRun(prev, at string, runErr error) string {
if runErr != nil { return prev } // failures neither advance nor clear
return at
}
```
Both directions are separate bugs: a failure must not **advance** it (the original defect) and must not
**clear** it (one bad night making an established tier read as never-succeeded).
**Two silent-wipe sites found and closed** — the "seam built but never wired" shape, where the field
exists, the writer sets it, and an unrelated routine path zeroes it:
- `offboxConfigHandler` rebuilds the target from the form and copies runtime status field by field, so
an ordinary settings save would have erased the anchor;
- `ApplyOffsiteTarget` does the same on a hub re-apply.
Neither would have surfaced until the verdict changed, days later. **The first was proven live** — see
below.
**Hub v0.80.0.** Three deliberate branches:
- **never ran** — unchanged v0.73.0 anchored behaviour, still keyed on `last_run` on purpose: that field
answers "has anything ever happened here", and a box whose *first* run failed is a run, not a newborn.
- **legacy** (`last_run` set, no `last_success`) — degrades **explicitly** to the old behaviour, logged
**once** per customer. Absence-as-failure would alarm the whole un-upgraded fleet; absence-as-success
keeps the bug. Same degrade direction as R-88 Part 2's `age_state`.
- **anchored** — counts from `last_success`; `last_status` is deliberately not consulted, because
"error ⇒ stale" pages on every blip (the F-A1 noise path).
**The alarm text had to move with the verdict.** `emitStale` still said `last run 8h ago` while firing on
a six-day-old success — a true alarm that reads as false. `staleAge` now separates *"runs are happening
and failing — check the error, not the schedule"* from *"the offsite leg is silently not running"*.
---
## Red-proofs — all observed failing
| # | red-proof | observed failure |
|---|---|---|
| A | restore the `LastRun` anchor | `a tier that has not succeeded in 6 days reads as FRESH — that is R-100` |
| B | delete the never-ran branch | `a newborn box alarmed — this is the 2026-07-23 cry-wolf that v0.73.0 fixed` |
| C | collapse to `LastStatus == "error"` | `a single transient failure alarmed — 20h ... well inside the 48h threshold` |
| D | delete the legacy degrade | `a legacy controller alarmed — that is a fleet-wide alarm storm on an un-upgraded fleet` |
| + | drop the `runErr` guard (controller) | `a FAILED run advanced LastSuccess ... that is the R-100 defect in mirror image` |
| + | always return `prev` | `a successful run did not advance the anchor` |
| + | drop the wire field | `OffboxReportStatus dropped LastSuccess — the hub would degrade forever` |
| + | drop the handler preservation | `a settings save erased LastSuccess` |
**A hollow test of my own, caught by red-proofing it.** The first version of the controller test
re-implemented the rule in a local closure — mutating production code left it **green**. That is why
`offboxAnchorAfterRun` was extracted: the test now calls the real rule.
Fixtures are the **real** wire shapes from P0.2, not invented JSON.
`go build`/`go vet`/`go test` green in both repos (hub 17 pkgs, controller 27 pkgs), run separately
from every commit.
---
## §6 — LIVE, on demo-hp (disposable; `peti-felhom` never touched)
A genuine restic failure was induced by pointing the target at a **closed port** (23 → 2) — it creates
nothing, touches no data, and is exactly reversible.
```
success run → last_status=ok last_run=11:24:20Z last_success=11:24:20Z
INJECT port 23 → 2 ... and the settings save PRESERVED last_success = 11:24:20Z ← the wipe-site fix, live
failing run → last_status=error last_run=11:25:48Z last_success=11:24:20Z ← ANCHOR HELD
```
**As the hub received it:**
| box | status | `last_run` | `last_success` | anchor |
|---|---|---|---|---|
| **demo-hp** (induced failure) | `error` | 11:25:48Z | **11:24:20Z** | **HELD** |
| **demo-felhom** (healthy) | `ok` | 11:29:22Z | **11:29:22Z** | **advanced** |
Also observed live, unplanned: **Scenario E**. Both boxes were still on the old controller at hub
startup, and the degrade logged **exactly once per customer**
`[WARN] [offsite] demo-hp: controller sends no last_success — staleness degraded to the last-ATTEMPT
anchor`. Two lines, two customers, same second.
**No spurious alarms:** 0 `offsite_stale` events since deploy (correct — both tiers succeeded minutes
ago). `backup_failed` fired for demo-hp at 11:25:48 from the induced failure, confirming the
pre-existing channel is intact and re-confirming the Phase 0 correction.
**Config restored** and verified field by field: `host=u629488-sub3.your-storagebox.de port=23
user=u629488-sub3 repo=/home/felhom-repo enabled=True escrow=escrowed`.
### Proven live vs. proven by injected clock — stated plainly
- **Live:** the anchor does not advance on failure; it does on success; it survives a settings save;
`last_success` reaches the hub; the legacy degrade fires once per customer; no spurious alarms.
- **Unit, injected clock only:** the 48h **threshold** behaviour itself — Scenarios A/B/C/D turning on
elapsed time. A live threshold test would take days. **The threshold was NOT proven live.**
---
## Part 2 — the rule
**"Presence is not success"** added to `CLAUDE.md` and its versioned copy, with both instances
(F-CRIT-2's phantom ctime, R-100's `LastRun`) and the corollary R-100's fix produced: when a verdict
changes which field it counts from, the **alarm text must change with it**. `// R-100` notes sit at
`isStale` and at the controller write site, each naming the test that pins it.
## Filed, not fixed
- **R-101** — Tier-2 `LastRun` is also written on failure, and three customer surfaces render it without
a status (two degraded branches plus the restore-confirm dialog). No hub verdict reads it.
## NOT yet live-validated (carried forward)
- **The 48h staleness threshold itself** (see above) — and with it Scenario A end-to-end: no
`offsite_stale` event has yet been *observed firing* from a genuinely stale success anchor, because
that needs 48h of failure.
- **Fault 4** — restic transport interruption; four injection approaches defeated by guest-bridged
networking. (This task's closed-port injection sidesteps it rather than solving it.)
- **R-99** — prune never removes phantom snapshots.
- **R-101** — filed today, unvalidated.
- `contentionAlarmAfter` (3h) — injected clock only.
-181
View File
@@ -1,181 +0,0 @@
# REPORT — R-101 + F-DIAG + F-OPS (2026-07-28)
Controller **v0.181.0 → v0.182.0**; `felhom.eu` gains the manual-restore runbook (F-OPS) and the
OPEN-ITEMS rows. Written as `REPORT-r101.md` so the shared `REPORT.md` is not clobbered.
## Baselines (reconfirmed, not copied)
`felhom-controller 3db8bfb`, `felhom.eu 6b7d516`, `felhom-agent d5c7691` — all = origin/main; the only
dirt in `felhom.eu` was a **foreign** `PROMPT-TEMPLATE.md`. Controller **0.181.0** live on both boxes,
hub `felhom-hub:0.80.0` ready 1/1.
---
## Phase 0
**The render sites — three dishonest, two already honest.** The spec listed `backups_apps.html:216`
as a defect site; it is in fact the one branch that *already* pairs its timestamp with a status badge.
The real third site is the `Tier2DestInactive` branch.
| site | rendered | honest? |
|---|---|---|
| `:231` **restore confirm dialog** | `Legutóbbi másolat: {{.Tier2LastRun}}` — raw RFC3339, no status | **NO** — the one that matters |
| `:195` `Tier2DestDisconnected` | `Utolsó: …`, no status | **NO** |
| `:206` `Tier2DestInactive` | `Utolsó: …`, no status | **NO** |
| `:217` main configured branch | `Utolsó: …` **+ status badge** | already honest |
| `sharing.html:181` | rendered **only** when status=="ok" | already honest |
`Tier2LastStatus` was already set unconditionally at `handlers.go:1182`, so this was a wording/anchor
problem, not a plumbing one. The restore button was gated on `{{if .Tier2LastRun}}`, so **Scenario C
was live-reachable**: a tier that had attempted and never succeeded offered a restore and a timestamp.
**`cd.LastRun` is written on failure** — `recordTier2Failure` (`tier2.go:573-574`) writes it alongside
`LastStatus:"error"`. Identical shape to R-100.
**Legacy state is universal, not an edge case.** All 7 Tier-2 rows across both boxes had `last_run` and
no anchor. Scenario E was therefore the *initial state of every customer*, which is what made the
legacy marker non-optional.
---
## Part 1 — the strings shipped
| case | string |
|---|---|
| dialog, normal | `… Legutóbbi sikeres másolat: 2026-07-28 16:43.` |
| dialog, newest attempt failed | `… Legutóbbi sikeres másolat: 2026-07-28 16:40. Figyelem: a legutóbbi mentési kísérlet nem sikerült, ezért a visszaállított fájlok ennél régebbiek lehetnek.` |
| card | `Utolsó sikeres: 2 perce` |
| never succeeded | `Még nincs sikeres másolat` + `Még nincs sikeres másolat, amiből vissza lehetne állítani.` (restore removed) |
| **legacy row** | `Utolsó: …` / `Legutóbbi másolat: …`**today's wording, unchanged**, logged once per stack |
**Timestamp made human-readable** (agreed): new `fmtTimeStr` renders Budapest-local `2026-07-28 16:40`
instead of the raw UTC `2026-07-28T14:40:55Z` a customer was previously asked to reason about.
**`SuccessTracked` is what makes the legacy case possible at all.** Without it, "row predates the
anchor" and "row has an anchor and it is empty" are indistinguishable — both are `LastSuccess==""`
and every existing row would have rendered as never-succeeded on deploy. Legacy rows migrate on first
touch: a row whose last known state was `ok` adopts that time (truthful — under the old code that run
did succeed); a row whose last state was `error` seeds **nothing**, because the old data evidences no
success.
## Part 2 — the copy-site hazard, and it was in the path
The three `record*` helpers each built a **whole `CrossDriveBackup` literal**, with a helper re-applying
exactly two fields; everything else was zeroed on every status write. Adding `LastSuccess` to that shape
would have had `recordTier2Failure` **clear** it — the mirror image of the defect, firing on the *first*
failure rather than lying dormant.
Replaced with **`tier2Update`**, which copies the existing row and overlays the outcome: **compile-safe
by construction** — a new field carries over unless deliberately overwritten, so nothing is preserved by
a list that can fall out of date. Callers now clear explicitly what a run invalidates, reproducing the
old behaviour exactly.
**Sweep of other rebuild sites:** `SetTier2Preference` mutates in place (safe); `SetCrossDriveConfig(name, nil)`
in `api/router.go:774` is a deliberate delete. No others.
## Part 3 — F-DIAG
| class | signal it maps to | message head |
|---|---|---|
| `quota` | the pre-run soft-quota gate | `A távoli mentés nem fért el a tárhelykereten belül` |
| `orphaned` | `ErrOffboxOrphaned` sentinel | `A távoli tárhely egy korábbi, már nem elérhető kulccsal készült` |
| `no_repo` | restic "unable to open config file" | `A távoli tárhelyen nincs mentési adattár` |
| `no_units` | "produced no snapshots" | `Nem volt mit menteni: egyetlen kijelölt alkalmazásnak sem található mentése` |
| `transport` | refused/reset/timeout/authn/host-key | `A távoli tárhely nem érhető el (hálózat vagy bejelentkezés)` |
| **`unknown`** | anything else | `A távoli mentés ismeretlen okból nem sikerült` |
The `unknown` class is deliberate: a cause that cannot be told apart where the error is produced is
reported as unknown rather than folded into a neighbour.
**Secrets — and this caught a bug in my own first attempt.** The old message was
`"…: " + err.Error()`, carrying the repo reference `sftp:<user>@<host>:<path>` off the box. My first
sanitiser regex-matched `sftp:…` and `user@host` and *looked* complete; its own test caught it leaking
on `ssh: connect to host <host> port 23: Connection refused` — a bare hostname in neither shape. It now
redacts the target's **actual** host/user/repo-path literally, with the regex kept only as a backstop.
Guessing at what a secret looks like fails exactly where it matters.
## Part 4 — F-OPS
`documentation/runbooks/RUNBOOK-manual-guest-restore.md`. Grounded in the real bind shape read off live
guest 9201, not written from memory. Covers: which `mpN` are storage volumes (restored) versus **host
binds** (taken as-is on the target); the `mp9` trap — it embeds the **source** VMID, so restoring to a
different VMID can bind **another guest's bootstrap credentials**; strip-and-re-add before first boot;
the hookscript check; and a positive pre-start verification that asserts every bind path exists rather
than accepting "no error". Docs only, by design.
---
## Red-proofs — all observed failing
| # | red-proof | observed failure |
|---|---|---|
| A | dialog back on the attempt clock | `the dialog does not name the last SUCCESSFUL copy` |
| C | gate the restore on `LastRun` again | `a tier that has NEVER succeeded still offers a restore — the dialog would promise a copy that does not exist` |
| D | make the caution unconditional | `a HEALTHY tier shows the failed-attempt caution ("nem sikerült")` |
| F | clear the anchor on failure | `a FAILED run wiped the success anchor (round 1) — one bad night would read as 'no copy has ever succeeded'` |
| + | raw sanitiser | `the repo reference reached the message ("sftp:" leaked)` |
**F exercises the real `recordTier2Success``recordTier2Failure` sequence**, not a modelled copy — the
R-100 lesson. The Scenario A/C/D tests **render the production template tree** and assert on the string
the customer reads; a test asserting a template variable would prove nothing about wording, which is
the defect.
`go build`, `go vet ./...`, `go test ./...` — 27 packages, `rc=0`; `template_id_gate.py` and
`emoji_gate.py` both OK. Run separately from every commit.
---
## LIVE on demo-hp — the rendered dialog, which is the deliverable
**Legacy state** (before any run under v0.182.0) — today's wording, no fright:
```
Legutóbbi másolat: 2026-07-28 03:30
```
Failure induced genuinely: the Tier-2 destination directory was **moved aside** and replaced by a file,
so `mkdir …/recovery-unit` fails. (`chmod` does not work — the controller runs as root, which bypasses
permission bits; `chattr +i` is refused, the unprivileged container lacks `CAP_LINUX_IMMUTABLE`. Both
were tried and reported rather than glossed.) The real data was only ever moved, never deleted.
```
status = error
last_run = 2026-07-28T14:42:18Z ← ADVANCED
last_success = 2026-07-28T14:40:55Z ← HELD
last_error = mkdir …/paperless-ngx/recovery-unit: …
```
**The rendered dialog, failed state:**
```
Visszaállítja a hiányzó fájlokat a másodlagos másolatból? A meglévő fájlok NEM módosulnak és NEM
törlődnek. Az alkalmazás a művelet idejére leáll. Legutóbbi sikeres másolat: 2026-07-28 16:40.
Figyelem: a legutóbbi mentési kísérlet nem sikerült, ezért a visszaállított fájlok ennél régebbiek
lehetnek.
```
**The rendered dialog, healthy state** (after restoring the destination and a successful run) — no
caution, no tonal change:
```
Visszaállítja a hiányzó fájlokat a másodlagos másolatból? A meglévő fájlok NEM módosulnak és NEM
törlődnek. Az alkalmazás a művelet idejére leáll. Legutóbbi sikeres másolat: 2026-07-28 16:43.
```
Card lines: `Utolsó sikeres: 2 perce``Utolsó sikeres: most`.
**Everything restored:** destination is a directory again, 86 MB intact, mode 755, `.r101-aside` gone,
`status=ok`, `last_success=2026-07-28T14:43:23Z`.
**demo-felhom is the untouched control:** all 5 rows still `tracked=None` after the deploy, rendering
today's way, 15/15 containers up. Scenario E holding across a whole box nobody ran.
---
## NOT yet live-validated (carried forward)
- **F-DIAG's classes** — unit-proven only. No live offsite failure of each class was induced; the
`transport` class is the only one this arc exercised indirectly.
- **Scenario C live** — the never-succeeded rendering is unit-proven; no fleet row is in that state
(every row either migrated or has a real success), and manufacturing one would mean breaking a
customer app's only Tier-2 history.
- **The Tier-2 restore itself** was not executed — this arc changed what the dialog *says*, not what the
restore does.
- **R-100's 48h staleness threshold** — injected clock only.
- **Fault 4** (restic transport interruption), **R-99**, **F-HUB**, fault 12, the three-way concurrency
overlap — next campaign's material, untouched here.
-136
View File
@@ -1,136 +0,0 @@
# REPORT — R-106 + R-109 (+ R-122): closing the recipe-completeness set (2026-07-30)
Non-overwritten sibling per `CLAUDE.md:82-87` — the shared `REPORT.md` holds R-117 and is not touched.
Shipped: **agent v0.118.0 → v0.118.1** (`felhom-agent` `1c8a67e`, `6b5dade`) + **hub v0.83.0**
(`felhom.eu` `acfc2b7`). Neither half is useful alone.
**Read §3 first if you read nothing else:** v0.118.0's R-106 half shipped INERT and live validation is
what caught it — the recipe still said `"root"`, now with `namespace_state: resolved` beside it. Full
account in the audit §6, filed as **R-125**.
## Part 0 — the answers, before the fix
### 0.1 Which items are actually open, and R-105/R-106's registration
`OPEN-ITEMS.md` calls itself "the single source of truth for open work" (`:1`), with `ROADMAP.md` keeping
"the full history and reasoning" (`:3-4`).
| item | `ROADMAP.md` | `OPEN-ITEMS.md` | verdict |
|---|---|---|---|
| R-105 | row, `READY — 2026-07-28` (`:108`) | **absent** | **open but UNREGISTERED** |
| R-106 | row, `READY — 2026-07-28` (`:109`) | **absent** | **open but UNREGISTERED** |
| R-108 | row (`:111`) | row (`:50`) | registered |
| R-109 | row (`:112`) | row (`:61`) | registered |
So R-109's own cell — "third recipe-completeness defect beside R-105/R-106" — was the **only** place in the
register naming two open items. That is exactly the thread-loss the register exists to prevent, and it is
itself a finding (filed **R-123**). Both now have rows.
**The set this task closes is R-106 + R-109**, matching the arc's stated definition of done (`OPEN-ITEMS.md:14`).
**R-105 is NOT in it** and was not worked: it is M-sized and is about three *hub-held DR records* being `{}`
(`hosts.dr_record_json`, `host_escrow.directive_json`, and the `drives` third — already traced and populated
by the 2026-07-28 target move). Different fields, different owner, different size.
### 0.2 Where the recipe is generated — three producers, not two
| half | repo | function |
|---|---|---|
| host (guests/pbs/drives/pve_storage) | `felhom-agent` | `BuildDRRecipeHostHalf`, `internal/hub/dr_recipe.go:86` |
| app (customer/apps/offsite_restic) | `felhom-controller` | `controller/internal/report/dr_recipe.go` |
| **assembly + delivery** | `felhom.eu/hub` | `AssembleDRRecipe`, `internal/store/dr_recipe.go:104`; served by `handleDRRecipeDownload`, `internal/web/dr_recipe.go:14`, route `internal/web/server.go:439` |
R-109's "host-half" is therefore the **agent**, and the field must also pass the **hub's** allow-list — see §2.
### 0.3 What the namespace field actually contained — verified, and the brief was RIGHT
The eleven-session-old brief held up. Live, pre-fix, from the hub for **both** boxes:
```json
"pbs": { "repo_id": "felhom-pbs", "namespace": "root", "latest_snapshot_id": "9201" }
```
against `/etc/pve/storage.cfg` on the same boxes:
```
pbs: felhom-pbs
datastore felhom-offsite
namespace demo-felhom # demo-hp reads: namespace demo-hp
```
Traced to source: `Snapshot.Namespace` decodes `ns` (`internal/pbs/client.go:97`), which PBS does not echo
per item once the list is namespace-scoped via `?ns=` (`:118-120`) → always empty → `ToHub` normalises empty
to `"root"` (`internal/pbs/report.go:22-25`) → `latestPBSCoord` writes it in.
**The authority taken, and why:** storage.cfg's `namespace` on the pbs storage. It is the same field
`vzdump --storage <pbs>` makes PVE read, and the agent's own verify client is built from it
(`cmd/felhom-agent/main.go:1164`). Deriving the recipe from anything else is how it drifts again.
## 1. R-109's ambiguity is real, in the boxes' own pre-fix recipe
```json
"pve_storage": [
{ "name": "local-lvm", "type": "lvmthin", "content": "images,rootdir" },
{ "name": "felhom-backup", "type": "local-dir", "content": "backup" },
{ "name": "felhom-pbs", "type": "pbs", "content": "backup" },
{ "name": "local", "type": "local", "content": "backup,import,vztmpl,iso" }
]
```
No `backup_target` key anywhere. `felhom-backup` (live, `/mnt/hdd_1`) and `local` (`/var/lib/vz`, archives
frozen 2026-07-28) are both `content=backup` dir storages; `local` is also the *historically* correct answer,
which is what makes guessing it so easy.
## 2. R-122 — a fourth defect, found here, and it had already shipped
`AssembleDRRecipe`'s `hostHalfShape`/`appHalfShape` are **allow-lists** dressed as forward-compat. The
controller has emitted `offsite_restic` since fork-4 (`controller/internal/report/dr_recipe.go:39-41`, "so DR
knows WHERE to recover from"); `appHalfShape` never listed the key. Verified both ways:
- **stored**: `dr_recipe.app_half_json` carries it for all three real customers —
`peti-felhom`, `demo-felhom` (`u629488-sub1.your-storagebox.de:23/home/felhom-repo`), `demo-hp`.
- **delivered**: the downloaded recipe's top-level keys were
`recipe_version, customer, guests, pbs, drives, pve_storage, apps`**no `offsite_restic`**.
So a restorer reading the recipe had **no offsite location at all**, for the whole life of the feature, with
a green suite throughout — because the test fixture `drAppHalf` is hand-written and omits the field.
**Deviation from the task's §7.10 ("Findings — filed as R-n, none fixed"), stated rather than absorbed:**
I fixed it. Reasons — (a) Part 0 authorises working the real set if it differs; (b) it is the same
symptom the task is named for (the recipe is incomplete), and the worst instance, a whole section missing;
(c) it is in the *same two structs* R-109 forced me to edit, and leaving one of three known keys off a
drop-list I was already correcting would be indefensible. It is filed as R-122 with a SHIPPED disposition.
## 3. The before/after recipe — both boxes, quoted
```
demo-felhom BEFORE "namespace":"root" backup_target absent offsite_restic absent
AFTER "namespace":"demo-felhom" backup_target {resolved, felhom-backup, /mnt/hdd_1}
offsite_restic {u629488-sub1…}
demo-hp BEFORE "namespace":"root" backup_target absent offsite_restic absent
AFTER "namespace":"demo-hp" backup_target {resolved, felhom-backup, /mnt/nvme-1tb}
offsite_restic {u629488-sub3…}
```
The two boxes DISAGREEING is the point — nothing is hardcoded. And the ambiguity was not theoretical:
on both boxes `felhom-backup` holds an archive from **07-30 04:36** while `local` stops at
**07-28 17:5x**, frozen at the target-move date. The recipe now names the live one.
Full evidence, all seven red-proofs and the publish observables:
`documentation/audits/R106-R109-recipe-completeness-2026-07-30.md`.
## 4. Findings filed (none of them fixed except R-122, see §2)
| id | finding |
|---|---|
| **R-122** | `AssembleDRRecipe` allow-list dropped `offsite_restic` for the feature's whole life — **FIXED here**, hub v0.83.0 |
| **R-123** | R-105 and R-106 were `READY` in `ROADMAP.md` with no `OPEN-ITEMS.md` row — referenced only inside R-109's prose. Registered here |
| **R-125** | v0.118.0 shipped an inert R-106 because the "production path" test injected `fakeObserver` one layer below the break — **FIXED** in v0.118.1; filed for the doctrine point (name the seam you inject at) |
| **R-124** | The recipe spells PBS's root namespace `"root"`, but the PBS API spells it `""` and there is no namespace literally named `root` — a restorer pasting it into `pct restore --ns root` would fail. Pre-existing wire convention, deliberately unchanged; documented at `PBSRootNamespace` |
## 5. Not done, and why
- **R-105, R-108, D5** — out of scope by the task's §6. R-108 blocks D5; starting either would leave both half-done.
- **The backup machinery** — untouched. This corrects the record, not the doing.
- **`sess-f` (0.116.0) and `drill-r50` (0.113.0) were not upgraded** — neither was named as a venue, and `drill-r50` is fenced by the task's §6.
- **R-124 not fixed** — changing the wire's spelling of the root namespace mid-R-106 would shift the field's meaning during the fix meant to make it trustworthy.
-117
View File
@@ -1,117 +0,0 @@
# REPORT-r116-diag — the `/disks` payload captured, R-116's mechanism isolated (2026-07-30)
Read-only diagnosis run by CC on DooPlex. **No code written, nothing built, nothing published.**
Full evidence: `documentation/audits/DIAG-r116-disks-payload-2026-07-30.md`.
A `REPORT-*.md` sibling, not the shared `REPORT.md` (`CLAUDE.md` parallel-session rule).
## Outcome
**Both goals met.** The `/disks` read path is solved and written down verbatim, proven by a
present-drive control run *first*; and the absent-state payload was captured, which isolates the
mechanism.
**R-116 is theory #1 — "the registry-union row writes `false`" — the theory that was raised, declared
wrong, and retracted. The retraction was the error.**
In the absent state `/disks` returns **4 rows, not 3**. The drive appears twice and the two facts the
controller needs sit on different rows:
| row | source | `mount_path` | `guest_path` | `backup_target` |
|---|---|---|---|---|
| `felhom-backup` | Observe (`disks.go:196-284`) | `""` | `""` | **`true`** |
| `694034cc-…` (the UUID) | registry union (`disks.go:297-339`) | `/mnt/cel` | `/mnt/felhom-drives/cel` | **field ABSENT ⇒ `false`** |
So the row holding the flag contributes **no key** to `driveTargetByPath`, and the row that owns the key
says `false``isTarget[a.Path]` is `false` → generic `storage_disconnected`. On return the rows
re-merge into one carrying both facts → specific `backup_target_restored`. Applying
`intermediary.go:602-618` to the captured payloads gives PRESENT `True` / ABSENT `False` /
RETURNED `True` — **the live asymmetry reproduced from payload alone.**
The union row's `MountPath` survives the device because the union source is the systemd **`.mount` unit
file** (`registry_known.go:40-75` via `main.go:605``:764`), which never reads the mount table. The
dedup at `:298` therefore does not fire, because `seen` is keyed on the one field the absent state
empties (`:290-295`).
**Theory #2 (the basis of the shipped v0.115.0) is false on both halves**; **#3 is false too**
(`isTarget["/mnt/cel"]` is `false` as well). **v0.115.0 is provably inert** — its fallback calls
`StablePathForRaw("")`, which returns `""` (`intermediary.go:69-75`), so it assigns nothing.
## The read path (this cost two prior sessions — it should never cost again)
The token plaintext exists in exactly one place: `bootstrap.json` **on the Proxmox host**. The agent's
own store keeps SHA-256 hashes only (`tokenstore.go:26-32`), which is what defeated the earlier attempts.
```bash
ssh felhom-pve
B=/var/lib/felhom-agent/guests/9201/bootstrap/bootstrap.json
TOK=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['token'])")
EP=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['endpoint'])")
curl -sS -k -H "Authorization: Bearer $TOK" "https://$EP/disks" | python3 -m json.tool
```
Control run, live felhom-pve, drive present: **HTTP 200, 2483 bytes, 4 plausible rows** — so Part 5's
0-rows-on-a-present-drive failure mode is excluded.
## Where the absent state was staged
No new box. The existing DooPlex **nested-PVE drill fixture** (`drill/drill.qcow2`, snapshot `virgin`) —
my own host, zero production exposure, and it can hot-unplug a disk for a genuine device loss. Run with
the **byte-identical live agent binary** (`sha256 f48544ad…`, `--version` 0.115.0) and every
state-producing step through the real endpoints (`format``assign``guest-attach`
`backup/target`). Its present-state row matched felhom-pve's control run field-for-field before it was
trusted. Non-production aspects (root/direct privileged mode, stubbed hub, a hand-written bearer-token
record, no controller) are enumerated in the audit §4.
## Two new findings, filed not chased
- **R-117 (READY M) — outranks R-116.** After a detach/reattach the guest's bind is a **dead mount**:
host is healthy on the new device node, guest still names the old one, and `ls`/write through it
return **`EIO`** — while `/disks` reports `attached` + `bound_under_parent:true` + `backup_target:true`.
`planDriveGates` therefore takes the `Return` branch and **restarts the customer's apps onto a dead
namespace, reporting healthy, with no alarm on any channel.** R-113's conjunction cannot catch it:
one half is satisfied by the stale entry, the other by the new device, and neither compares them.
This is the "stale bind" seen and dismissed as cosmetic in three consecutive runs.
- **R-118 (READY XS).** An absent drive's union row reports the **root filesystem's** capacity as its own
(46 GiB / 9.2 % for a 4 GB drive) — `statfsCapacity` at `disks.go:335-338` statfs's a bare directory on
root. `observe.go:176-183` guards the Observe path against exactly this; the union path does not.
`durable_id` is still correct, so re-attach identity is safe — it is a false capacity, not a DR mis-id.
## Register
`documentation/backlog/OPEN-ITEMS.md` — R-116 updated with the mechanism and the fix constraints;
R-117 and R-118 added. The single register edit this session makes.
## Record correction
The brief's baseline `controller 0.185.1` is the version the **golden bakes**. **0.186.0** (R-114 +
R-112, 2026-07-29) is what **demo-felhom** runs — **demo-hp is still on 0.185.1**, so the fleet is
split, and R-114's `TargetAbsent` branch exists only on demo-felhom. Confirmed: agent **0.115.0**
(felhom-pve) / **0.113.0** (demo-hp), hub **0.81.0** (manifest pin and live pod image agree),
host-install **1.22.0**, `felhom.eu` HEAD `c3ce4c7`.
> **Correction, 2026-07-30.** As first written this section said 0.186.0 was what *both* demo boxes run.
> That was wrong — only felhom-pve's guest was sampled and the result generalised to the fleet. demo-hp
> re-checked directly → `0.185.1`. Fixed here and in the audit's baseline table.
## Teardown and fences
Drill guest destroyed, scratch storage removed, mount unit deleted, secrets `shred -u`'d, VM powered
off, **`drill.qcow2` restored to `virgin`** (the golden-bake fixture is exactly as found), scratch qcow2
and console dumps deleted. DooPlex `/mnt/5_hdd` at 24 %, unchanged.
Both demo boxes **read-only throughout** and re-verified after teardown: demo-hp `local-lvm` **38.83 %**
(identical before/after and to Part 5), `drill-r50` still stopped, felhom-pve `felhom-backup` still
active on `/dev/sdb`, both guests running, **v0.115.0 untouched**.
`sess-d-0452c4` now reads **STALE**, and the delete gate refuses only on ONLINE
(`hub/internal/web/customer_delete.go:220-228`) — so it **is** now deletable; the command is recorded in
the audit rather than executed (customer delete runs external teardown plus a DB purge). **`sess-c` is
also still present and was not recorded by the Session-C audit** — same terms.
## Not done, deliberately
No fresh controller gate-log / hub-event correlation: that observable was already captured live and
identically twice, and only the payload was missing. Staging a controller meant a hub customer, a
pairing, a golden fetch and a claim — the work that consumed the three prior sessions. The audit §8
correlates the payload to those existing measurements and labels that step as inference.
-96
View File
@@ -1,96 +0,0 @@
# REPORT — SPIKE R-117: a dead bind that reports healthy (2026-07-30)
Written as `REPORT-<topic>.md`, not `REPORT.md`, per this repo's parallel-session rule and the
established local pattern (`REPORT-r116-diag.md`, `REPORT-session-c.md`, …). The shared `REPORT.md`
was not touched.
**Class: Spike.** Deliverable is a findings document. **No production code was written; no `.go` file
in either repo was modified; nothing was built for deployment, published, or version-bumped.**
## Deliverables
| File | Action |
|---|---|
| `documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md` | **new** — the findings doc (Q1Q7, evidence, recipe, probe comparison, recommendation) |
| `documentation/backlog/OPEN-ITEMS.md` | **R-117 row rewritten** with the mechanism, the reproduction recipe and the fix constraint — the one register edit, per the brief |
**No CHANGELOG entry.** This repo's changelogs are per-area (`hub/`, `scripts/`, `website/`); a
documentation-only change belongs to none of them. Stated rather than silently skipped, per standing
rule 4.
## Baselines
agent `main` **v0.116.0** @ `d4eb259` · controller `main` **v0.186.0** @ `b331f18` ·
`felhom.eu` HEAD `29bcfeb` · hub **live 0.82.0** · golden **0.186.0** ·
demo-hp PVE 9.2.2 / kernel 7.0.2-6-pve, **live agent 0.113.0** (= manifest `MinAgent`; never used as
the source of a behavioural claim — every predicate result came from a probe built from `main`).
## Results
**All seven questions answered empirically.**
- **Q1 — reproduced 3/3**, two device classes, on a purpose-built scratch LXC (9301) on demo-hp.
**The device-node change is a consequence of the defect, not a precondition** — control test: with
the stale bind held the drive returns as `sdc` (8:32); released, the letter is reused (`sdb`, 8:16).
- **Q2 — two death states**: device removed ⇒ superopts gain `shutdown`, `EIO`(5) on read and write,
host and guest; device errors in place ⇒ `emergency_ro`, write `EROFS`(30), reads served from cache.
The raw host mount is genuinely healthy in both. **No cross-device mis-identification is possible**
on this path — the unit is fs-UUID-keyed.
- **Q3 — confirmed at source and live.** Both halves of the R-113 conjunction compare **field 5** of a
mountinfo line and **never read field 3 (`major:minor`)**, so neither can see that the bind and the
raw mount name different devices. Measured `BoundUnderParent = TRUE` over an `EIO` namespace.
- **Q4 — a pure-`/proc` check costs 0.160.45 ms**, cannot hang, spins up no disk, writes nothing.
**`statfs` and `getdents` both SUCCEED on a dead namespace** — probes built on either are hollow. The
hang case is below; it is the sharpest result in the run.
- **Q5 — the agent**, and not on balance: the controller runs inside the guest and cannot see the host
mount tables the check needs.
- **Q6 — recovery works in place, guest never restarted** (init PID identical). **The repair code
already exists and three call sites already invoke it**, including the controller's `Return` branch
*before* it restarts apps — all defeated by one early return.
- **Q7 — a bind can die in steady state, no cycle at all.** The gate produces no action and **nothing
is emitted on any channel.** A `Return`-branch fix cannot reach this half.
## Q4's hang case — measured, and it is the sharpest result
Against a `dmsetup suspend`ed device (I/O queues instead of returning `EIO`):
- **P1 and P2 completed in 364 µs / 206 µs.** They read `/proc`, so no block device is involved.
- **`statfs` and `getdents` completed and reported HEALTHY** — on a wedged device they do not even hang.
- **Every probe that touches the device blocked, including a buffered write with no `fsync`** — the
`O_CREAT` metadata path needs journal access (`wchan=do_get_write_access`). There is no cheap-and-safe
write probe.
- **The blocked process survived `SIGTERM` and `SIGKILL`** (`stat=D`, still alive 3m50s after `kill -9`)
and died only when the device was resumed. So **`systemctl restart felhom-agent` would hang**, leaving
the agent unrecoverable until the device returns or the host reboots. The thread count does not reveal
the leak (5→5, 5→6).
**A timeout protects the caller's control flow and nothing else.** This turns "prefer a cheap probe" into
a fence: **the fix must issue no block I/O.**
## Teardown — done, all three layers
Guest 9301 destroyed; `r117scratch` removed; both dm devices and both loop devices gone; `scsi_debug`
unloaded (`/dev/sd*` back to `sda1..3`); no `r117` mounts, `/mnt` and `/root` exactly as found; `local`
**37.02 %** against a session-start **37.00 %**. Fences re-verified *after* teardown: 9201 `running`,
`drill-r50` `stopped`, `local-lvm` **38.84 % byte-identical**, `felhom-backup` `content backup`
unchanged, live `/mnt/felhom-drives` intact with both submounts, agent service `active`. **Layer 3 is
genuinely empty** — 9301 had no network interface and ran no controller, so no hub-side record was ever
created.
**Ordering trap worth keeping:** a suspended dm device must be `dmsetup resume`d *before* any `umount`,
or the teardown itself blocks on the same uninterruptible sleep.
## Not measured
No load or duration testing of the recommended check — P1/P2 were single calls, not a sustained
reconcile loop on a many-drive box. Nothing suggests a problem (they are two `/proc` reads the code
already performs), but it was not measured.
## Findings filed, none fixed
R-117 (mechanism + recipe), **R-117a** steady-state death with no event (HIGH, larger than R-117 as
filed), **R-117b** `statfs`/`getdents` are hollow liveness probes, **R-117c** three untested comments
promising "live + usable in the guest", **R-117d** the self-heal that already exists is short-circuited
(HIGH), **R-117e** both demo boxes share one failure domain — no route survives the site losing internet,
including the WireGuard OOB path, **R-117f** an I/O liveness probe turns a wedged drive into an
unkillable agent (HIGH — disqualifies a whole probe class).
-203
View File
@@ -1,203 +0,0 @@
# REPORT — installer-channel record correction + R-29 filing (2026-07-29)
Two commits, documentation only. No code, no version bump, no CHANGELOG entry, no build, no deploy,
no box touched. Written as `REPORT-<topic>.md` per `CLAUDE.md:82-87` so root `REPORT.md` (the E-2
increment-1 report) is preserved.
| # | Commit | Baseline | Scope |
|---|--------|----------|-------|
| 1 | `d4c07873ca0c3d3e547373a9fafc0e472a6535e8` | `36d635a4cdc1`, unmoved | Retract a false R-94/E-2d finding; open R-110 |
| 2 | (this commit) | `d4c07873`, unmoved | File R-29 to the register; three record defects; this report |
---
## Commit 1 — `d4c07873`
### What was false
`36d635a4` recorded that `felhom-bootstrap.sh` fetches the installer **from the hub**, that the hub
therefore serves 1.19.0, and that a fresh ISO install runs the pre-E-2 installer. All three wrong.
The claim had propagated into two `OPEN-ITEMS.md` rows, the ranking rationale, and `ROADMAP.md:149`.
### Confirmation table — all PASS
| # | Claim | Read at | Result |
|---|-------|---------|--------|
| F1 | bootstrap fetches from the **website** | `scripts/iso/felhom-bootstrap.sh:96` | PASS — `INSTALL_URL="${FELHOM_INSTALL_URL:-https://felhom.eu/scripts/felhom-host-install.sh}"` |
| F2 | hub-rendered command points at the same URL | `customer_unified.html:563`, `:1262` | PASS — and **three** emission sites, not two: `:563` static, `:1262` JS error branch, **`:1267` JS success branch** |
| F3 | website serves `/scripts/` from a git-sync tree tracking `main` | `manifests/webpage.yaml` — nginx `:74-77`, sparse-checkout CM `:211-218`, git-sync `:272-281`, init `:299-307` | PASS — `--branch=main --period=30s --link=current`; sparse-checkout `/website/` + `/scripts/`; `location /scripts/ { root …/current; }`. No image build, no ArgoCD step |
| F4 | `hostInstallVersion` selects nothing | `configs.go:28`, `:487`; `render_test.go:219`; `customer_unified.html:494` | PASS — repo-wide grep returns exactly those 4 code sites; all other hits prose. Rendered as a text label |
| F5 | every generated flag is parsed by 1.22.0 | generator `customer_unified.html:1206-1239` vs parser `felhom-host-install.sh:1175-1212` | PASS — `--mode --cores --memory --vmid --node --acl-storages --operator-pubkey-file --preserve-state-from --skip-provision --dry-run --preflight-only --allow-new-leaf` (+ `--customer-id`); every one a parser case. **No functional gap** |
| F6 | installer is 1.22.0 | `scripts/felhom-host-install.sh:187` | PASS |
| F7 | the drift test is hollow | `render_test.go:219-221` | PASS — `strings.Contains(html, hostInstallVersion)` compares the constant to itself; passes at any value |
### Live command 1 — what the URL actually serves
```
$ curl -fsS https://felhom.eu/scripts/felhom-host-install.sh | grep -m1 '^SCRIPT_VERSION='
SCRIPT_VERSION="1.22.0" # the SINGLE version source (F-1): -h, the run banners, and the hub
```
### Live command 2 — the drift gate's real state
```
$ python3 scripts/hostinstall_gates.py; echo "exit=$?"
ok: SCRIPT_VERSION=1.22.0
ok: header has no version literal
FAIL: hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0 (F-1: bump both together)
ok: age is in the installed package set
… (six further ok lines) …
hostinstall gates: 1 FAILURE(S)
exit=1
```
### Phase 0 source read — PAIRING reaches the same installer invocation
Mode selection `felhom-bootstrap.sh:537-541`: a fresh VM with no baked customer-id calls
`run_pairing`. On HTTP 200 from `/api/v1/appliance/poll` the loop writes the hub-delivered
`FELHOM_CUSTOMER_ID` + `FELHOM_RETRIEVAL_PASSPHRASE` into the 0600 env, re-sources it, and calls
`run_direct` **in the same invocation** (`:495-499`). `run_direct` is the single site that fetches
`$INSTALL_URL` (`:322-330`), builds the args (`:334`) and invokes `bash "$SCRIPT_TMP" "${args[@]}"`
(`:343`). The customer it yields is the one the operator bound — claimable. **So the ISO leg is the
spine for E-2d**, not an obstacle to it.
### Rows changed
| ID | Before | After |
|---|---|---|
| R-94 (line 14) | `READY — deferred until E-2d`, blocked on E-2d, false ISO/hub framing | `READY (XS)`, blocked on nothing, retracted + re-scoped to three legs |
| R-94 (line 16) | duplicate row, `READY #2`, 1.19.0 vs 1.20.0 | **deleted** — merged |
| R-110 | did not exist | opened, `WAITING-ON-OPERATOR (S)` |
| E-2d | ISO implied as obstacle | Next-action appended; ISO is the spine |
| ranked list | 1 R-95 · 2 R-94 (high-consequence) · 3 R-86 · 4 R-87 | 1 R-95 · 2 R-94 **de-ranked** · 3 R-86 · 4 R-87 · 5 R-110 |
Files: `documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/ROADMAP.md`,
`documentation/runbooks/day0-install.md`.
---
## Commit 2 — R-29 filing + record hygiene
### The §1.1 ruling — is R-29 the right home for a non-design-v2 gate? **Yes. Proceeded.**
R-29's title says *"the design-v2 green gates"*, and `scripts/hostinstall_gates.py` is not one — it
comes from drill F-1 (2026-07-12) and postdates the item. Four things decide it anyway:
1. **R-29's own audit list already spans well beyond design-v2 subject matter.** It names
`docker_run_volume_path_gate` (docker mount safety), `offbox_rename_gate`, `app_row_dedup_gate`
and `manifest_bearer_gate` (secrets — `runbooks/secrets.md:76`). The title is a misnomer relative
to the item's own body.
2. **Part (b) — "the systemic half is the real item" in R-29's words — is stated with no
design-v2 restriction.** It is about the *enforcement mechanism*: "the gates run only when a human
remembers to run them… decide where they run (pre-push hook, `build.sh` step, or a CI job) and make
a red gate block the train." That is gate-agnostic and repo-wide.
3. **`hub_confirm_gate.py` is already on R-29's list and lives in the same `scripts/` directory** as
`hostinstall_gates.py`. Wiring one and not the other would be arbitrary.
4. **Identical failure shape, identical genre.** Both self-describe as mechanical grep-assertions
(`hostinstall_gates.py:2`, `hub_confirm_gate.py:1-8`); both exist, assert something true, and are
invoked by nothing.
R-29 has already absorbed one independent re-raise without minting an ID (2026-07-18 rehearsal note)
and says so explicitly. This is the third. No new ID minted.
### Orphan-search evidence — `hostinstall_gates.py` and `hub_confirm_gate.py`
Re-established at `d4c07873`. Pattern `hostinstall_gates\|hub_confirm_gate`.
| # | Scope | Command | Result |
|---|-------|---------|--------|
| S1 | `felhom.eu`, all file types | `grep -rn "$PAT" . --exclude-dir=.git` | **19 hits, zero invocations.** All are docstrings (`scripts/hostinstall_gates.py:5`, `scripts/hub_confirm_gate.py:7`), code comments (`hub/internal/web/configs.go:27`, `scripts/felhom-host-install.sh:189`) or prose (`REUSE.md:62`, `CONTEXT.md:540,564`, `hub/CHANGELOG.md:371,1292,1351,1385`, `scripts/CHANGELOG.md:483,524`, 3 files under `documentation/audits/`, `ROADMAP.md:149,158`, `OPEN-ITEMS.md:14`) |
| S2 | sibling repos | `grep -rln "$PAT" /mnt/5_hdd/felhom.eu/git --exclude-dir=.git` minus this repo | 3 files, all in `.claude-memory/` (`MEMORY.md`, `drtier-by-default-2026-07-12.md`, `polish-batch-2026-07-13.md`). Notes, not invokers |
| S3 | `~/.claude` | `grep -rln "$PAT" /home/kisfenyo/.claude`, and targeted on `settings*.json`, `skills/`, `hooks/` | **Zero hits in settings, skills or hooks** — where an invoker would live. Remaining hits are `file-history/` (Claude Code's own backups of files edited in past sessions) and `paste-cache/` (pasted task specs). Neither is an invocation site |
| S4 | git hooks | `ls -1 .git/hooks/ \| grep -v '\.sample$'` | **Empty — every hook is a `.sample` stub** |
| S5 | build files | `find . -type f \( -iname Makefile -o -iname '*.mk' -o -iname justfile -o -iname 'Taskfile*' \)` | Only `hub/Makefile`; `grep -n gate hub/Makefile`**zero occurrences** |
| S6 | CI | `find . -type d \( -name .github -o -name .gitea -o -name .woodpecker* -o -name .drone* -o -name .circleci \)` | **Empty — `felhom.eu` has no CI configuration at all** |
Of the four gates in `scripts/`, only `site_gates.py` is mandated (`CLAUDE.md:153`);
`manifest_bearer_gate.py` is named in `runbooks/secrets.md:76`.
### Rows changed
| ID | Before | After |
|---|---|---|
| **R-29** | **absent from `OPEN-ITEMS.md`** (`grep -c` → 0) while present at `ROADMAP.md:158` since before the 2026-07-27 rebuild | **opened, `READY (S for (a) / M for (b))`**, owner CC |
| R-94 | leg (b) stood alone | leg (b) carries `→ R-29` as its class. Row otherwise untouched |
| E-2d | cited `:322-341` for an invocation at `:343` | cited `:322-343`, with the fetch / args / call each pinned separately |
| R-95, R-86, R-87 | `READY #1`, `READY #3`, `READY #4` | `READY` — markers dropped (see below) |
Not added to the ranked list under *"Why the READY rows rank this way"*: that list is a top-N
rationale, not a complete ordering (R-99, R-102, R-103, R-104, R-108, R-109, R-89, R-92, R-93 and
E-2d are all READY and absent from it). Ranking R-29 was not asked for and would be a priority claim
this task has no basis to make.
### §2.2 — markers dropped, not renumbered
The `#1/#3/#4` markers duplicated ranked-list positions 1/3/4 for exactly those three rows; two
orderings of one set is what produced the hole when the `#2` row was merged in `d4c07873`. Removing
the duplicate leaves the ranked list as the single maintained ordering.
### The other two record defects
- `ROADMAP.md:147` — cited a non-existent **R-164**; it means controller **v0.164.0**'s
deliberate-stop filter. Corrected. (It had already cost one max-ID scan a false positive.)
- `CONTEXT.md:564` — asserted in the present tense that the single VERSION source is *"gated by
`scripts/hostinstall_gates.py`"*. The gate exists, asserts exactly that, is red, and runs nowhere.
Corrected to say the cross-check exists but is not enforced, tracked as R-94 leg (b) / R-29.
`hub/internal/web/configs.go:27` carries the same false claim in a code comment and was left alone
— R-94 leg (b) territory, needs a hub build.
Files: `documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/ROADMAP.md`,
`documentation/backlog/README.md`, `CONTEXT.md`, this file.
---
## Commit 3 — `de5a3e5` — the record-hygiene rider
Baseline `7383400a`, unmoved. Four XS items from `7383400a` §9; the three deferred observations stay
deferred.
| File | Change |
|---|---|
| `CONTEXT.md:540` | *"`scripts/hub_confirm_gate.py` enforces"* → asserts, but is not enforced (R-29). Third instance of the class after `:564` and `configs.go:27` |
| `REUSE.md:62` | Same claim, *"enforces zero"*. The **rule stays** — never native `confirm()`/`prompt()` is correct guidance and this is a reuse-reference row — only the enforcement claim changes |
| `OPEN-ITEMS.md:4` | Root `REPORT.md` = overwritten per-session; `REPORT-<topic>.md` = non-clobbering sibling (`CLAUDE.md:82-87`), 14 of them. Prohibition unchanged |
| `OPEN-ITEMS.md:55` | Heading scoped to *"the **TOP** READY rows"* + a half-sentence that it is deliberately not a full ordering. **No row added to the list** |
`hub/internal/web/configs.go:27` — the fourth instance — left alone (R-94 leg (b), needs a hub build).
### Part 3 NOT done — its stated evidence is false
The spec proposed recording a convention: *"`ROADMAP`'s P3 `Status` column only ever takes `idea` or a
`SHIPPED`/`PROVEN-LIVE` variant, and never takes `READY`, `BLOCKED` or `WAITING-ON-OPERATOR`"*, and
concluded from that vocabulary split that `ROADMAP` `Status` is **disposition** while `OPEN-ITEMS`
`State` is **live actionability**.
Refuted inside P3 itself (`ROADMAP.md`, lines 113172):
| Line | ID | P3 `Status` cell |
|---|---|---|
| 151 | R-90 | `BLOCKED on Hetzner CX33 availability (operator, 2026-07-27)` |
| 152 | R-91 | `WATCHING — gated on demo-felhom's first post-migration PBS backup` |
| 150 | R-110 | `idea — found 2026-07-29, **WAITING-ON-OPERATOR (a ruling, not a defect)**` |
| 137 | R-86 | `**NEXT — operator ruling 2026-07-27**` |
And P2's recovery-gaps sub-table (lines 99107) uses a bare `READY` for nine rows, including `E-2`.
Both halves fail: the column **does** take the `OPEN-ITEMS` vocabulary, and `BLOCKED`/`WATCHING`/
`NEXT` are live-actionability words, not dispositions. The real shape is a free-text status field
that has absorbed both vocabularies over time — usually disposition, sometimes not. Writing the
proposed convention down would install a **new false invariant** in the commit chain whose whole
purpose was removing them, so per the spec's own instruction (*"If you think this ruling is wrong, say
so and make no edit"*) `documentation/backlog/README.md` was not touched.
The narrow question that prompted it still resolves benignly: R-29 and R-94 reading `idea` in
`ROADMAP` and `READY` in `OPEN-ITEMS` is **not** a defect and needs no sync. What is unsupported is
generalising that into a rule.
---
## Standing note
**No gate was wired, fixed, run as a hook, or deleted by any of the three commits.** Filing R-29 is
not doing R-29; the enforcement decision (pre-push hook / `build.sh` step / CI) and the audit of the
remaining eight gates are R-29 part (b), M-sized, and its own task.
-94
View File
@@ -1,94 +0,0 @@
# REPORT — Session C: R-113, R-114, R-112 proven on a real box; C5 fails on a new defect (2026-07-29)
`RUNBOOK-session-c-2026-07-29.md`. Full evidence: `documentation/audits/SESSION-C-2026-07-29.md`.
Root `REPORT.md` untouched.
## Verdicts
| Claim | Fix | Verdict |
|---|---|---|
| **C4** — offer appears and moves the target | R-112 | ✅ **PASS** |
| **C3** — absent target tells the truth, offers nothing | R-114 | ✅ **PASS** |
| **C5**`backup_target_absent` + `backup_target_restored` | R-113 | ❌ **FAIL** — generic alarm, specific recovery → **R-116** |
**All three shipped fixes work.** R-113's gate fires in **4 seconds** (E-2d measured zero over 4½
minutes). R-114's message is correct. R-112's banner reaches the customer. C5 fails on a **fourth,
separate defect that was unreachable until R-113 made the gate fire at all.**
## What the drill box ran
Agent **0.114.0 from the Day-0 manifest** — the shipped binary, so C5 tested the real artifact and
closed R-115's observation 1 for R-113. Controller **0.186.0 hand-deployed** after install (§3.1
ruling (a)); the vouched golden bakes 0.185.1, so **C3/C4 prove the code, not the shipped golden**
that lag is filed against R-115, not a new ID.
## The three headline observables
**R-113** — detach at 18:43:50, gate at **18:43:54**, on exactly the shape that defeated it before:
```
raw /mnt/mentes : NOT mounted
bind /mnt/felhom-drives/mentes : /dev/sdb[/felhom-data] <- the stale bind SURVIVED
```
**R-114** — with the target absent: absent copy 1, **system-disk copy 0**, **offer block 0**. Both of
E-2d's falsehoods gone.
**R-112** — banner element 1 and the never-configured copy in the HTML; after the wizard, the offer
with `data-path="/mnt/felhom-drives/mentes"`. Healthy renders nothing, **proven positively**: idle
delta 0 `/backup/tiers` calls, page-load delta **+1**, single caller ⇒ the seam ran and chose silence.
**Decline path proven** (registration confers no role), `restart_required:true`, agent did **not**
self-restart, in-flight check recorded before I restarted it, wrapper created the storage at the
drive's own mountpoint.
## C5's failure
```
absent : Event pushed: storage_disconnected (error) <- GENERIC
return : Event pushed: backup_target_restored (info) <- SPECIFIC
```
`backup_target_absent` count **0** across the whole run. The alarm and its recovery cannot be matched
— precisely what `notifyDriveReturned`'s own comment forbids.
**Root cause (R-116):** `driveTargetByPath` builds `out[GuestPath] = d.BackupTarget`, but the drive is
**two `/disks` rows** and the flag and the guest path sit on different ones — the `felhom-backup`
storage row carries `BackupTarget: true` and gets a guest path only while classified user-data; the
registry union row carries the guest path and **never assigns `BackupTarget`**. Absent ⇒ they separate
⇒ generic. Return ⇒ they rejoin ⇒ specific. v0.184.1 fixed the *keying*, not this.
## Mirror + over-correction guard — PASS, with a caveat
Non-target drive detached ⇒ `storage_disconnected`, `backup_target_absent` count 0. **Over-correction
guard passes**: both drives present ⇒ 0 ABSENT lines, target stayed healthy — R-113's stricter presence
did not make a healthy drive read absent. **Caveat: the mirror passes trivially**, because the target
also produced the generic event; it confirms no over-correction but cannot confirm discrimination.
## Record
- `OPEN-ITEMS.md`**R-113, R-114, R-112 → SHIPPED + PROVEN-LIVE**; **R-116 opened** (READY (S), P1);
**E-2 and E-2d CLOSED as partially proven** with R-116 as the one named open leg, per the runbook's
§9 decided-in-advance rule.
- `ROADMAP.md` — R-116 under P1.
- **Capability map NOT touched** — it still has **no E-2 / backup-target rows at all**, so no row could
be moved to PROVEN-LIVE. Creating them is a design act, not a validation act. Third session running
that this has been noted.
## Teardown
VM destroyed, storage removed, **`pvesm status` after == before** (`local-lvm` 38.78 %), guest 9201 and
`drill-r50` untouched. **Customer ruling: DELETE**; attempted and correctly refused (`host … is
ONLINE`) — deletable once the destroyed host ages to DOWN (>1 h), command recorded in `OPEN-ITEMS.md`.
## What did not happen
`backup_target_absent` never fired, so its severity, Hungarian copy and hub routing remain unexercised
— R-116 blocks them. The offer was accepted via the endpoint the button POSTs, not a browser click (no
browser automation on DooPlex); the rendered control and its non-auto-submission were verified in HTML.
The stale bind still naming a dead device node after return was observed, not investigated.
## The arc
E-2 ends here. Its stated definition of done is **R-106 + R-109, R-108 and D5** — none of which this
detour touched. The detour was worth taking: it found six real defects (R-111 through R-116), four of
them customer-affecting, none of which any unit suite had caught.
-98
View File
@@ -1,98 +0,0 @@
# REPORT — tester gate: golden re-baked to 0.188.0, fresh-install proof PASSED (2026-07-31)
Written as `REPORT-<topic>.md` per `CLAUDE.md:82-87` so the shared `REPORT.md` (E-2 increment 1) is
not clobbered. Full record with every observable: `documentation/audits/tester-gate-golden-0.188.0-2026-07-31.md`.
## Outcome
**§7.2 — YES: a fresh install is safe to hand to an external tester.** ISO boot → claimable,
app-serving box in ~10 minutes unattended, and an app's data restored **from the drive with the
guest's `app.yaml` gone**, proven readable by the application over its own TCP path.
**Golden 0.186.0 → 0.188.0** baked, published, vouched. **No ISO rebuilt** — Part 0 proved none was
needed. No existing box changed; floor still v0.156.0, MinAgent still 0.113.0.
## Part 0 — the ISO does not need rebuilding
Verified against the ISO **on disk**, not from source. It bakes exactly three Felhom payloads
(`felhom-bootstrap.sh`, its unit, the secret-free pairing env) — full-base64 match, 1 hit each — and
**0** hits for `SCRIPT_VERSION="1.2`, `felhom-controller`, `vzdump-lxc-9100`. The installer is fetched
at run time (`felhom-bootstrap.sh:96`) and the live URL is byte-identical to repo HEAD
(sha `ab8b283e…`, v1.22.0, committed six days *after* the ISO). The golden arrives via the hub-vouched
artifact manifest (`felhom-host-install.sh:423-433`). The one genuinely baked, drift-capable thing is
`felhom-bootstrap.sh` itself — currently at repo HEAD.
**Proven live**, not just argued: the fresh box ran `felhom-host-install v1.22.0` and fetched golden
**v0.188.0**, sha-verified.
## Part 1 — bake / publish / vouch
Baked **0.188.0**, not the brief's 0.187.0: 0.187.0 lacks D5, and Part 2 step 6 *is* the D5 claim, so
that golden could not have passed the proof this task exists for. 0.188.0 satisfies R-120 anyway.
Stated rather than absorbed, per standing rule 4.
GOLDEN_VERSION=0.188.0
GOLDEN_SHA256=7353d8beb63641f87a848e45f8aa12e465647e1190ad164a65b32ad01fc3d299
Three observables: 404 pre-gate (with a 200 control on 0.186.0 so it is not vacuous), then an
**anonymous** download returning `http=200 bytes=649310288` and a matching sha; the manifest read back
showing `0.188.0` selected; and the consumer call `GET https://hub.felhom.eu/api/v1/artifacts/sess-g`
returning the pair. Plus a fourth: a real fresh box fetched and sha-verified it.
**R-120's gate evaluated and allowed.** Exercised both ways rather than inferred from silence —
vouching 0.185.1 first produced `flash=golden_behind_fleet`, the logged `artifact vouch REFUSED`, and
**no write** (the manifest still read 0.186.0); then 0.188.0 produced
`Artifact manifest set: agent=0.118.1 golden=0.188.0`.
## Part 2 — the clean-install proof, on demo-hp
All seven steps PASS. Venue was demo-hp (Tier 0, the designated drill host) using the scratch dir
storage at `/mnt/nvme-1tb` that `target-selection.md:38-40` names; `local-lvm`, `drill-r50` and both
9201s untouched.
Highlights: real day-0 pairing → bind → install; a **real** claim (the code is emailed-only, R-119 —
the operator relayed it), with the gate flipping `dashboard not yet claimed` → `authentication
required`; controller **0.188.0** confirmed *from the box*; **rallly** (postgres) + **homebox**
deployed through the real endpoints.
The D5 leg: recovery unit `portable-carried=2/2, withheld=0`; the carried `DB_PASSWORD` matched the
live one **by fingerprint** (`14c8f515…`, never printed); guest `app.yaml` moved aside; restore
returned `secrets recovered=2/2`. Step 7 read the data from **rallly's own network namespace** over
TCP to `rallly-postgres` — not the localhost trust socket that produced D5's false pass — and the same
path with a wrong password returned `FATAL: password authentication failed`, proving the credential
does real work. **The discriminator held: PRE-BACKUP row = 1, POST-BACKUP row = 0.**
## Part 3 — runbook integrity
`RUNBOOK-manual-build.md` told the reader to use a "RECORDED" qemu line that is itself labelled
*reconstructed*, and whose source says it *"was never saved"*. The real invocation is now captured
from this bake and recorded as canonical in **§4.0**, alongside the bake/publish/teardown steps, the
template-rot warning and where the R-120 gate actually lives. The old runbook's deviation entry is
marked SUPERSEDED with a forward pointer.
## Teardown — three layers
1. VM 310 destroyed with `--purge --destroy-unreferenced-disks 1`; `/mnt/nvme-1tb/images/` empty.
2. `cc-scratch` removed, `storage.cfg` back to its original four entries; `felhom-backup` available
**926 492 284 KiB before and after** — space returned exactly.
3. Hub: **`sess-g` and its host record DELETED, full cascade** — `customer DELETE cascade COMPLETE
for sess-g (journal #8) — full teardown`, residue purged including `appliance_registrations=1`.
Verified positively: `/configs` and `/hosts` both loaded (10160 / 9880 bytes) with **0** hits for
`sess-g` and 0 for the appliance UUID. The gate refused twice first (409 host ONLINE, then 400
missing acknowledgements) — the record cannot be deleted until the destroyed box ages out of
ONLINE, ~30 min. **`sess-f` deliberately NOT deleted** (R-131); its command is in the audit §7.1.
Secrets shredded in the guest and on the box.
## Findings — filed, none fixed
`R-128` ISO_VERSION/SCRIPT_VERSION comment is false · `R-129` demo-hp's "no baked SSH key" is stale
(key auth works) · `R-130` `HARD_MIN_LVM_GIB` warns and proceeds — a hard min that is not hard ·
`R-131` `sess-f` is a fourth orphaned scratch customer · **`R-132``curl -w '%{redirect_url}'`
printed the hub operator password into a session transcript; `HUB_PW` needs rotating.**
## Not done, deliberately
No ISO built; no defect fixed; no golden deployed to an existing box; no floor or MinAgent change;
offsite/PBS-DR legs not exercised (the task forbids pointing anything at production PBS or the real
restic offsite, so `sess-g` ran DR-tier off); Campaign 10, the demo-hp repartition and subdomain
onboarding untouched.
-97
View File
@@ -1,97 +0,0 @@
# REPORT — SPIKE 4: can a `.deb` in the ISO deliver the stub on an interactive install? (2026-07-31)
> Written as `REPORT-universal-iso-spike.md`, not `REPORT.md`: the shared file belongs to today's hub
> v0.85.0 session and the second session in a shared clone never touches it. Supersedes this file's
> Spike 13 contents.
**Class: Spike.** Findings only — no production file changed, no release ISO built, nothing published.
Evidence: `documentation/audits/SPIKE-universal-iso-4-2026-07-31.md`.
## The answer is yes, and it was measured with the negative control in the same box
One ISO, 15 GRUB entries, a trivial probe `.deb` injected into `/proxmox/packages/`. Two VMs on
demo-hp built with `qm` so the run was visible in the web console: **400 interactive**, **401
automated control**.
On the **interactive** install (`spikefour.felhom.eu`):
- the package is installed — `ii felhom-spike4-probe 0.0.1`
- its **postinst ran** — marker file present, content intact
- it **enabled a systemd unit**, and **that unit fired on first boot** (uptime 7.98 s, `pid1: systemd`)
- and on **that same machine**, `proxmox-first-boot` is not installed and `/var/lib/proxmox-first-boot`
does not exist — Spike 3's negative reproduced, not assumed
So the two delivery mechanisms are independent, and the one that survives the path we are actually
shipping is the `.deb`. **The product — insert the stick, install Proxmox normally choosing your own
disk and password, box sets itself up and waits for a claim code — is now measured rather than hoped
for.** With one honest caveat: what was measured is a trivial probe package, not Felhom's real stub.
Packaging the real stub and confirming pairing end-to-end is the last step before a spec (~60 min).
## What a postinst may and may not do
Identical on both paths: `pid1 = unconfigured.sh`, **no running systemd**, `/proc` and `/sys` mounted,
and **`systemctl enable` succeeds** (it wrote the symlink). Network and DNS *happened* to be up —
inherited from the installer's own DHCP.
Four constraints for the real postinst, so they get written against rather than discovered:
1. Never `systemctl start` or `daemon-reload` — there is no systemd running. `enable` is the only verb.
2. **Never require the network**, despite it being present here. A box installed with the cable out
gives a postinst no route, and a failing postinst breaks the customer's install.
3. Never fail — guard everything, `exit 0`.
4. Do the real work in the unit at first boot, where systemd, network and a booted kernel exist.
## Two smaller results
**The repack preserves the `.deb`, but not naively.** `xorriso … -boot_image any replay` fails with
*"Overlapping MBR partition entries"* — and `iso-repack.sh:270-292` already documents that exact
failure and its fix. Mirroring it produced a working image (19 El Torito entries; the `.deb` extracted
back out is byte-identical). So this is an insertion into an extract→modify→re-master cycle our repack
already performs, not a new build stage.
**Q3:** `iso-repack.sh:100-106` refuses an ISO without `auto-installer-mode.toml`. It is a guard, not
a structural requirement, and its reasoning is sound for the shape it was written for — already R-155,
cited exactly here. With no mode file the stock grub.cfg does not emit the Automated entry at all; with
a mode file but no answer, that entry aborts safely and loudly.
## A correction I owe you from last session
**R-153 is retracted.** The register grep this task mandated shows R-94 already carries it verbatim at
`OPEN-ITEMS.md:15`, status `READY (XS)`, with leg (b) being precisely "the gate fails today and is
invoked by nothing" — and R-29, the class, says in terms *"do not mint a new ID for a new instance."*
Spike 3 filed a duplicate.
**And the substantive half of that Spike 3 claim was wrong.** I wrote that the drift left the customer
page's install-command generator "targeting a flag surface three minor versions stale." R-94 explicitly
retracts exactly that reading: the constant selects no script — it renders as a text label, and the
command beneath it fetches the script the website git-syncs from `main`, so **1.22.0 is what every
install already gets**. It is a wrong number on your screen and nothing more. I overstated it.
## Still unknown
**The real stub has not been packaged** — that is the one thing between here and a build spec.
Also unproven: `dpkg --configure -a` ordering for a package with dependencies; an ISO that never went
through `prepare-iso` (blocked by R-155, which this spike was fenced from changing); and the Graphical
installer, where the result should hold *a fortiori* since the `.deb` path is in `Install.pm`, shared
by all front-ends — but that is inference, not measurement.
**Spike 3's Q3** — the real stub at `before-network`**this session did not touch it.** Note it is
now partly superseded: on the `.deb` route the unit's ordering comes from the unit file, not from
`[first-boot].ordering`.
## R-rows
**None opened.** Each candidate was grepped against the register first: the delivery result is a
positive finding, the postinst constraints belong in the build spec, and the repack guard is already
R-155. **R-153 retracted** into R-94 leg (b) / R-29.
## Teardown
All three layers plus the scratch storage, verified positively. demo-hp: VMs 400/401 purged, **storage
`spike4` removed** (`storage.cfg` back to 4, `grep -c spike4` = 0), `/mnt/nvme-1tb/images/` empty,
**disk usage 6.6 G — identical to pre-spike**, probe ISO and driver removed, 0 loop devices,
`drill-r50` stopped and untouched, 9201 running, `felhom-backup` unmodified, nothing on `local-lvm`.
DooPlex: workspace scratch **4.8 GB removed**, scratchpad **3.3 GB → 88 K**, both throwaway passwords
destroyed, 17 ISOs in `out/` untouched, no production file modified. **Hub-side: nothing created**
the VMs took LAN DHCP leases but never ran `felhom-host-install.sh` or contacted the hub; verified by
fetching and searching the customer list. Nothing published.
+143 -138
View File
@@ -1,168 +1,173 @@
# REPORT — `STATUS.md` created, and the 2026-08-02 operator decisions recorded (2026-08-02)
# REPORT — RUNBOOK: the first host-tier restore-test, on both boxes
**Overwritten** per the standing rule. The prior contents (hub v0.85.0 Network card + v0.86.0 Copy
without reveal, 2026-07-31) have their durable record in `hub/CHANGELOG.md` and
`documentation/audits/host-addresses-visible-2026-07-31.md`; nothing was lost by this overwrite.
**Date:** 2026-08-03 → 2026-08-04 · **Repos:** `felhom.eu` docs + registers only. **Nothing was built
and no version was bumped.** The only binary that moved is the already-published `v0.123.0`, onto the
box that did not have it (P1). Baselines re-read and matched: `felhom-agent` `72161f6cf010` /
`v0.123.0`; `felhom.eu` `e3187c86d58d` / hub `v0.91.1`, installer `1.24.0`. Constants re-confirmed at
source: `defaultRestoreTestEvalInterval` **6 h**, `defaultRestoreTestSettle` **24 h**.
**Class: documentation only.** No code, no template, no box, no build. Repo `felhom.eu` only —
`app-catalog-felhom.eu` was read for context and **not** modified. **No `CHANGELOG.md` entry exists
for this change and none is missing:** this repo has no root changelog, only per-area `hub/`,
`scripts/`, `website/` (`CLAUDE.md`), and this session touched none of those areas.
Baselines: `felhom.eu` @ `260a8f6`, `app-catalog-felhom.eu` @ `fd7747d`. Part 1 was derived by reading
the register's rows and both ranking sections, not from memory.
**Outcome: four scheduled runs, all passed, nothing triggered by hand.**
---
## 1. `STATUS.md` — the file
## 1. Preconditions
Root of `felhom.eu`, so it is the first thing visible. Sections in the specified order: what works ·
what's broken · what we're working on · waiting on you · changed since last update.
**Word count: 652 total, 581 excluding the header block** (`wc -w`; the header carries the
view-not-source, not-`CONTEXT.md` and maintenance rules, which the spec requires). **That is over the
~500 target and it is a deliberate miss, stated rather than hidden.** Five passes took it from 819 to
652. Getting under 500 needed either dropping a mandated item or dropping the off-site-credential line
— the register's **top-ranked** open item and the largest customer-data exposure on it. Cutting the
biggest data risk to save forty words is the wrong trade on a page whose job is to show the operator
what is at stake. It is 67 lines and fits a screen. **If the operator disagrees, the line to cut is
the R-95/R-87 one** and the page drops to ~545.
Content, in the operator's ranking: an app can stay off after a power cut, silently (R-157) · the
off-site copy can be erased by the box that wrote it, and has never been restored from (R-95, R-87) ·
three of fifty-three apps saved data where backups never looked (R-156) · 20 GB of backup space
against 50 GB of apps (R-163) · when that trips, one page says so and nothing alerts (R-158) · the
checker exists but a person has to remember it (R-161).
**Constraints honoured:** no `R-n` is the subject of any sentence — every identifier is a bracketed
pointer at the end of a line; no file paths, function names or version numbers appear; every broken
item is stated as what a customer or the operator would notice. Shipped, watching and
blocked-on-a-predicate rows (R-159, R-160, R-162, R-164) are absent by design.
**One deviation, flagged per standing rule 4.** "Waiting on you" is specified as *decisions only*, and
it carries one non-decision: **the hub password needs rotating** (R-132, owner Viktor). It is the only
thing on the register waiting on the operator with a live credential consequence, and omitting it from
the operator's own page to honour a section rule would be the letter over the point. It is labelled
*"a job, not a decision"* so the section's shape is not quietly eroded.
## 2. The decisions — where each one went
All four are in **`CONTEXT.md` as standing ruling S-5**, labelled **D-a … D-d** as in the discussion
and deliberately kept distinct from S-3's `D1…D6`. Open work is carried as backlog rows, per the
existing convention — no new home was created for either.
| Decision | Recorded | Work |
| # | demo-felhom | demo-hp |
|---|---|---|
| **D-a** — merge the backup partition away (not resize) | `CONTEXT.md` S-5 | **R-165** (new) |
| **D-b** — desired/observed app state, own store | `CONTEXT.md` S-5 | **R-166** (new, `BLOCKED`) |
| **D-c** — storage monitoring + backup alerts | `CONTEXT.md` S-5 | **R-167** (new) |
| **D-d** — only DooPlex and Peti's box are protected | `CONTEXT.md` S-5 | `runbooks/target-selection.md`, this session — no row; the decision *is* the change |
| **Maintenance rule** (Part 3) | `STATUS.md` header **and** `CLAUDE.md` § End-of-session checklist | — |
| **P1** agent | `v0.123.0` ✓ | **`0.120.0` — below the 0.121.0 floor, so it could not become due at all.** Remediated with the published `v0.123.0` (sha `74910135…`, deployed sha identical) |
| **P2** tiers | host `felhom-backup` + offsite `felhom-pbs` (weekly) | **the same — the runbook expected demo-hp to have no offsite tier, and it has one**, active with 2 snapshots |
| **P3** grant | `ok=70 total=70 degraded=0` | `ok=70 total=70 degraded=0` (once the probe existed) |
| **P4** storage | `/dev/sdb → /mnt/hdd_1`, ext4, on the N100 | `/mnt/nvme-1tb` on the t740 — **different hosts, different disks ⇒ INDEPENDENT ⇒ parallel is safe** |
| **P5** space | target 889 GB free; restore pool `local-lvm` 358 GB | target 925 GB free; restore pool `data` **53.9 G at 30.79 %** (~37 GB free) against a 2.35 GB archive — adequate, and measured *because* that pool is the over-subscribed one |
| **P6** candidate | `…2026_08_02-04_42_14.tar.zst` (08-02) | `…2026_08_02-04_49_29.tar.zst` (08-02) — both correctly the settled archive, not the day's |
| **P7** other heavy work | daily backup ~04:44, outside the window | daily backup ~04:49, outside the window |
Recorded verbatim inside D-b, because it is the decision's binding constraint: *losing the state store
must never cause an app to be deleted, restarted wrongly, or reported healthy when it is not — the
worst acceptable outcome is re-running a backup that already ran.* Its two "establish before speccing"
items are carried on R-166 as the reason that row is `BLOCKED` rather than `READY`.
## 2. The due verdicts before the run, quoted
**Deliverable 5 asks for "the five decisions".** Part 2 defines four (D-a … D-d); the fifth deliverable
line is the Part-3 maintenance rule, and it is in the table above. Nothing else in the task reads as a
fifth decision — flagged rather than invented.
```
demo-felhom tier=felhom-backup due=true archive="…2026_08_02-04_42_14.tar.zst"
reason: newest settled archive … has not been proven; nothing proven on this tier yet
tier=felhom-pbs due=true archive="…2026-07-28T04:49:43Z"
reason: … has not been proven (last proven archive was a different one)
**D-a's two conditions are recorded as conditions, not commentary:** it changes the disk layout so it
must land **before any external install**, and it removes a wall that currently fails safely so
**R-167 ships in the same step, never after**. R-165 restates both; R-167 names R-165 as the thing it
gates.
demo-hp tier=felhom-backup due=true archive="…2026_08_02-04_49_29.tar.zst"
tier=felhom-pbs due=true archive="…2026-07-28T19:19:45Z"
```
**None of D-a, D-b or D-c is implemented.** No controller, agent, installer or hub file was opened for
editing.
**Both boxes had BOTH tiers due**, which made §4's ordering question live rather than theoretical.
## 3. R-163 re-framed, and R-156's papra referral resolved
## 3. The runs — all four SCHEDULED, none triggered
**R-163 is re-framed, not closed** — as instructed. State went `WAITING-ON-OPERATOR — the ratio is a
tier-sizing ruling` → `RE-FRAMED 2026-08-02 — open, no longer waiting on a ratio`; "Blocked on" went
from `the operator's sizing decision` to a pointer at R-165; owner `operator``CC`. The cell now says
the sizing **question is withdrawn rather than answered**, that the row survives as the record of the
constraint until the merge lands, and that the original finding follows unchanged. The intake ranking
(item 4) was updated with it, and records that **R-165 inherits R-163's rank and is the highest-ranked
item that must land before any external install**.
| box | tier | due at | archive | result |
|---|---|---|---|---|
| demo-felhom | **host** | 00:55:21 | `…2026_08_02-04_42_14.tar.zst` | **passed, 83.8 s**, scratch torn down 00:56:45 |
| demo-felhom | offsite | 06:55:21 | `…2026-07-28T04:49:43Z` | **passed, 540.4 s**, torn down 07:04:21 |
| demo-hp | **host** | 02:05:39 | `…2026_08_02-04_49_29.tar.zst` | **passed, 109.3 s**, torn down 02:07:28 |
| demo-hp | offsite | 08:05:39 | `…2026-07-28T19:19:45Z` | **passed, 300.1 s**, torn down 08:10:39 |
**R-156's papra referral is resolved.** The referral existed because moving a mount relocates live data
out from under a running app; with papra deployed nowhere there is nothing to strand, so the cheaper
leg — the template mounts `/app/app-data` — is takeable without waiting on upstream.
**No box failed to fire, so Phase C was not entered and no `--selftest` was used as a proof.** The
only selftest invocations in this session were the read-only `restore-test-due` verdict prints in §2,
which start nothing.
**The provenance is recorded with the claim, because it decides the row.** The evidence is
`docker ps -a` on **demo-hp's guest 9201** returning empty, **supplied with the task**; this session
**did not re-measure** — it is documentation-only and every box was fenced. The recorded scope is
honest about its edge: it covers the one guest papra was convicted on in Campaign 10, and **no other
customer's guest was enumerated**, so the row instructs the task that edits the template to re-check
first. Next action on the row is the catalog edit plus `catalog_gates.py`, explicitly not done here.
**§4's question, answered live:** each box took its **host** tier first — never-proven sorts ahead of
proven, and ahead on the id tie-break — deferred the offsite one, and picked it up on the **following
evaluation six hours later**. One heavy operation at a time, per box, with nobody sequencing it. That
is R-86's oldest-proven ordering and the heavy-operation gate observed together for the first time.
## 4. `target-selection.md` per D-d
**The asymmetry worth keeping:** a host-tier restore is **83109 s**; an offsite one **300540 s**. The
tier an ordinary recovery uses is also the cheapest to prove.
The rule at the top is now D-d: **two protected machines, everything else disposable**, with the
correction stated as a correction — the earlier caution was costing sessions and pushing drills onto
DooPlex. The tier table's Tier 2 row is DooPlex + Peti's cluster "and, by D-d, nothing else".
## 4. What the runs left behind
**Two consequences the decision did not name, both handled visibly rather than silently:**
**Persisted state — v3, naming the archive, the tier and what was verified:**
- **`ep0` + the Hetzner Storage Boxes.** D-d's protected list has two machines and ep0 is not one, so
the page no longer calls it Tier 2. It is **not** thereby scratch: it holds the PBS-DR datastore and
the restic copy of a real customer's data — the only off-premises copy that exists. Read the narrow
way (not protected, but not wipeable), using the page's own *fences-name-acts* rule, and **flagged
in the page for the operator to confirm explicitly.**
- **The shared "do not re-point either backup target" fence** on the two demo boxes was **downgraded
from a prohibition to a stated cost**, because D-d makes both boxes freely reinstallable, which
spends that reference configuration just as thoroughly — keeping the fence would have left the page
self-contradicting. The reason survives: know you are spending the regression reference, and put the
box back.
```json
demo-felhom felhom-backup → {archive …2026_08_02-04_42_14.tar.zst, tier local, verified boot+running,
proven_at 2026-08-03T22:56:45Z}
felhom-pbs → {archive …2026-07-28T04:49:43Z, tier pbs, proven_at 2026-08-04T05:04:21Z}
demo-hp felhom-backup → {archive …2026_08_02-04_49_29.tar.zst, tier local, proven_at 2026-08-04T00:07:28Z}
felhom-pbs → {archive …2026-07-28T19:19:45Z, tier pbs, proven_at 2026-08-04T06:10:39Z}
```
Also corrected while in the file: the *fences-name-acts* example cited the fence this edit removed, and
demo-hp's access line asserted "no baked SSH key" — which **R-129** records as measured false on
2026-07-31. It now points at R-129 instead of sending the next session to the hub vault for a
credential it may not need.
**The hub received the host-tier proofs — R-189's path carrying one for the first time.** demo-felhom's
latest report holds **two** entries, one per tier; the `local` one can only have come from the
persisted state, because the in-memory store held only that morning's offsite run:
## 5. The maintenance rule
```
demo-felhom tier=local pass=True archive=felhom-backup:…2026_08_02-04_42_14.tar.zst tested_at=2026-08-03T22:56:45Z
tier=pbs pass=True archive=felhom-pbs:…2026-07-28T04:49:43Z tested_at=2026-08-04T05:04:21Z
demo-hp tier=local pass=True archive=felhom-backup:…2026_08_02-04_49_29.tar.zst tested_at=2026-08-04T00:07:28Z
```
In two places, as specified: the `STATUS.md` header block, and a new **`## End-of-session checklist`**
in `CLAUDE.md` — which also gathers the couplings that were previously scattered (CHANGELOG + REPORT,
REUSE, the capability map's own end-of-session line, S-1's architecture coupling) and closes with *a
finding goes in `OPEN-ITEMS.md` first, never only in a report, an audit or `STATUS.md`*.
**A subsequent evaluation runs nothing on a proven tier**, quoted from demo-felhom now:
`CONTEXT.md` gained a header block stating why it and `STATUS.md` are separate — same subjects,
different readers, and `STATUS.md` holds nothing of its own. `STATUS.md` says the same from its side.
```
tier=felhom-pbs due=false proven="…2026-07-28T04:49:43Z"
reason: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven
```
## 6. What could not be translated into plain language
…while its **host** tier reads `due=true` again — on the **08-03** archive, which has now settled 24 h.
That is not a defect: it is "proved daily, on its own archive", visible one day later.
Asked for explicitly, because an untranslatable row usually means the row itself is unclear.
## 5. Teardown — three layers, per box
- **R-29***"gates are enforced nowhere"*. The class is stateable ("we have checks nobody runs"), but
its instances are four differently-broken scripts across two repos with no shared consequence, so
every plain sentence either says nothing or misstates one instance. **R-161 is its readable
fragment**, which is why R-161 is on the page and R-29 is not.
- **R-123 / R-125** — process findings about how the register and how tests are written. Real, and they
belong on the register; there is no customer-visible symptom to lead with, so they have no honest
first sentence for this page. They are not "broken" in the operator's sense.
- **R-133 (the plaintext break-glass credential)** — translatable, and left off only for space. It is
the strongest candidate for the next update if something else closes.
- **R-115 vs R-110** — separate rows, one plain-language paragraph. Merged into a single "Waiting on
you" bullet carrying both pointers, because two adjacent bullets about publishing read as one item
the operator has already half-decided.
| layer | demo-felhom | demo-hp |
|---|---|---|
| the machine | `pct list`**0** entries for 990000 | **0** |
| the host | `lvs`**0** volumes for 990000; `local-lvm` **1.95 % → 1.95 %** | **0**; `local-lvm` 30.79 % → 40.86 % during the offsite run → **30.83 %** after |
| the hub | the `restore_tests[]` entries are **RETAINED DELIBERATELY** — they *are* the proof the staleness check reads, so deleting them would delete the result | same |
**A register defect found while reading, filed here because the fix is not mine to guess: `R-133` is
used TWICE** — `OPEN-ITEMS.md:80` (duplicate `domain` values accepted by the hub) and `:86` (the
plaintext break-glass credential). Two different findings, one ID, both `READY`. One needs renumbering,
and which one is the operator's call since both are cited from elsewhere (`CONTEXT.md` S-4 cites the
credential one).
Nothing else was created: no scratch customer, no fixture storage, no probe tag, no package version.
## 7. Files changed
## 6. What the run surfaced — three findings, two of them corrections to my own record
| File | Change |
|---|---|
| `STATUS.md` | **new** — the operator page |
| `CONTEXT.md` | S-5 (D-a … D-d); header note on the `STATUS.md` separation |
| `documentation/backlog/OPEN-ITEMS.md` | R-165/166/167 filed; R-163 re-framed (state, blocked-on, owner, ranking); R-156's referral resolved with its provenance |
| `documentation/runbooks/target-selection.md` | D-d rule; tier table; ep0; the demo-box fence; two stale lines |
| `CLAUDE.md` | new `## End-of-session checklist`, carrying the `STATUS.md` maintenance rule |
### R-190 (new) — a storage ACL that worked in the morning was gone by mid-morning
Nothing was built, deployed, published or touched on any host. Every claim about the register above is
a claim about pushed source in this repo at the commit below.
A `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50** on
2026-08-03; the first `403 … missing privilege Datastore.Allocate` on that path is **09:24:56**, and by
~14:50 `pveum acl list` held **no row at all** for it. Ruled out by measurement: a host reinstall
(uptime 12 days), any `pveum`/ACL/`user.cfg` activity in syslog 04:0010:00, any cluster-log ACL entry.
Correlated but not established: guest 9201 was reprovisioned at 09:1509:19, nine minutes before.
**A permission that can vanish silently makes every ACL-based guarantee on these hosts provisional**,
and v0.123.0's probe detects the *state* but says nothing about the *transition*.
### R-191 (new) — every weekly offsite backup reports FAILED although it worked
demo-felhom, 06:4906:53 today: the upload **succeeded** (223 s, 629 MiB of 1.874 GiB, 67.2 % reused
incrementally) and the job then failed on the prune —
`missing Datastore.Modify|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `TASK ERROR: job
errors`, and the hub raised `whole_guest_backup_failed`. **The token behaves exactly as R-89 designed**
(box tokens are write-only; ep0 prunes). What did not follow is the config: **both** boxes still arm
the offsite tier with `keep_last=2 prune_pbs_allowed=true`, so every weekly run asks for a prune that
must fail. The data is safe; the verdict and the weekly e-mail are wrong, which is the R-100 corollary
— an alarm whose text is true and whose trigger is not the thing you would act on. **Not fixed here**
(§6 rule 2), and the fix needs one check first: whether ep0's prune jobs actually cover these two
namespaces.
### Two corrections to yesterday's record
1. **The R-185 drift was NOT silent on the write path.** demo-felhom's local-api backup jobs 403'd
**six times** (09:24 → 17:34) on that storage and privilege, and the hub raised
`whole_guest_backup_failed` at the first with edge-triggering suppressing the rest. My annotation
said backups kept landing because writes go through a root path — wrong, and now corrected in the
runbook and on the row.
2. **My "no `restore_test_*` events at all" was an instrument error.** The hub has no `/events` route;
I grepped a **404 page**. Read from the events table: five such events exist (2026-07-27/28), none
since the R-86 work — and one **more** since, below.
**One further event, correctly raised and worth stating:** `restore_test_stale` for demo-felhom at
2026-08-03 22:33:42 UTC — **22 minutes before** the host-tier run. It was **true**: at that moment the
host tier had never been proven, and the hub said so on its own, without being asked. It has not
re-fired; the signal is edge-triggered, so the return to healthy is silent by design.
## 7. The capability map
The unattended restore-proof row now reads **PROVEN-LIVE for the host tier, unattended, on both demo
boxes**, with the four runs, their durations, the deferred-tier ordering, the hub-side proof and the
teardown cited — and it states its **scope explicitly**: `demo-felhom` and `demo-hp`. The tester's box
is untested and untouched, and one box proving something does not make it a fleet property.
## 8. Registers
- **R-185** — its consequence is now demonstrated; the row already closed yesterday, and carries the
correction in §6.
- **R-190**, **R-191** — filed. `grep` established R-190 and R-191 were free before minting (R-189 was
the highest in use).
- `ROADMAP.md` holds none of these rows, so nothing to collapse.
- `STATUS.md` rewritten for the operator and kept to one screen (85 lines); R-191 appears under
"What's broken" because it produces a weekly e-mail you would otherwise learn to ignore.
## 9. Observations — noticed, NOT acted on
- **demo-hp's `local-lvm` thin pool reached 40.86 %** during its offsite restore (from 30.79 %,
returning to 30.83 %). Comfortable, but that is the over-subscribed pool the target-selection notes
warn about, and the offsite archive is the larger of the two. A materially bigger guest would want
the restore pointed at `/mnt/nvme-1tb` instead.
- **demo-hp has an offsite tier**, contrary to the runbook's §2 premise and to the note that it "has
none". Nothing depends on that assumption now, but the operations notes still carry it.
- **The two boxes' daily archives are ~6.3 GB (demo-felhom) vs ~2.35 GB (demo-hp)** for the same guest
role — a 2.7× difference worth understanding before either is used to size anything.
- **Both waiters this session produced no output** despite the runs completing; the evidence was
gathered by direct query afterwards. A watcher that silently produces nothing is exactly the
instrument class this project distrusts — the conclusions here rest on the boxes' own journals and
the hub's database, not on the waiters.
+7 -2
View File
@@ -104,7 +104,7 @@
| `offsite.DeliveryStateFor` (+ `DeliveryStatus`) | hub/internal/offsite/delivery.go | `(st, customerID) (DeliveryStatus, error)` | THE R-70 offsite last-mile detector — one implementation for every consumer (customer card `deliveryViewFor`, `monitor.OffsiteDeliveryChecker` event + R-71c heal) | Precedence: `applied` (latest report has offsite) wins over every secret-row shape; applied+unconsumed-staged = applied + `StaleStagedSince` flag (demo-felhom's live specimen). Never add a sibling derivation — consumers read THIS. |
| `(*Store).GetOneTimeSecretInfo` / `LastEventAt` / `LatestReportOffsitePresence` / `CountReportsOffsiteSince` | hub/internal/store/store.go | `(customerID) (*OneTimeSecretInfo, error)` / `(customerID, eventType) (time.Time, error)` / … | Detector inputs + DURABLE event-cooldown source (events table survives restarts — prefer over in-memory maps for hub-emitted checker events) | `GetOneTimeSecretInfo` never selects the value column — keep it that way. `SetOneTimeSecretTimesForTest` is the back-dating seam (PBSDR pattern). |
| `monitor.OffsiteDeliveryChecker` + `OffsiteReissuer` | hub/internal/monitor/offsite_delivery.go | `NewOffsiteDeliveryChecker(st, reissuer, onEvent, logger)` | R-70 stuck event + R-71c self-heal on the shared 60 s ticker | THE R-39(a) GUARD lives in `maybeHeal`: re-reads the secret row at act time and refuses over an UNCONSUMED row — `SaveOneTimeSecret` clobbers by design (Re-issue depends on supersede); never "fix" the store, never bypass the guard. reissuer nil = heal disabled (no provisioner) — required, else a heal-event fires for a silent no-op. |
| `monitor.RestoreTestChecker` + `assessRestoreProven` | hub/internal/monitor/restoretest.go | `NewRestoreTestChecker(st, onEvent, logger)`; `.Check()` | R-85: turns a restore-test result into a SIGNAL — it was a `[WARN]` log line and nothing else, even for the tier already being tested | **TWO event types, never merged**: `restore_test_failed` (broken now, error) vs `restore_test_stale` (unverified — *not* known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. **Anchored on R-81** (`assessRestoreProven` reuses `backupAssessment`/`verdict*`): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: **no `customerMessages` entry** — do not add one without copy review. |
| `monitor.RestoreTestChecker` + `assessRestoreProven` | hub/internal/monitor/restoretest.go | `NewRestoreTestChecker(st, onEvent, logger)`; `.Check()` | R-85: turns a restore-test result into a SIGNAL — it was a `[WARN]` log line and nothing else, even for the tier already being tested | **TWO event types, never merged**: `restore_test_failed` (broken now, error) vs `restore_test_stale` (unverified — *not* known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. **Anchored on R-81** (`assessRestoreProven` reuses `backupAssessment`/`verdict*`): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: **no `customerMessages` entry** — do not add one without copy review. **R-86 (2026-08-03): the window is PER TIER, not one constant.** `restoreProvenWindow(tier, observed, ok)` = `clamp(4 × max(observed, declared), floor 7d, cap 12d)`, where `declared` is that tier's own backup-freshness threshold (`backupStaleAfter` 26 h / `offsiteBackupStaleAfter` 8 d — reuse those, never a second opinion) and `observed` comes from `observedArchiveIntervals` over the retained window. **Observation may only WIDEN**: a gap shorter than the declared rhythm is routine (a retry, a heal, a catch-up) and a live box proved it — demo-felhom's two PBS snapshots sit 8 h 54 m apart, which would read a WEEKLY tier as nine-hourly and re-create the false alarm. The cap keeps the window strictly inside offsite retention. `assessRestoreProven` takes the window as an argument and **every reason string names it** (R-100's corollary). |
| `(*Server).applyPBSDR` + `mergePBSDR`/`readPBSDR` | hub/internal/web/pbsdr.go | `(ctx, r, cfg) error` | The config form's DR-tier section → HOST desired_json `pbs_dr` descriptor + generation bump | Descriptor lives in the host desired_json, NOT ConfigJSON (buildConfigJSON drops foreign keys on re-save). v0.51.0: driven by `cfg.DRTier` (set from the form BEFORE applyOffsite/applyPBSDR); UNMET preconditions are honest waiting stages (save succeeds), REAL failures stay fail-closed; already-provisioned = success-no-op (red-proofed); disable keeps the ep0 tenancy. |
| `(*Server).pbsdrProvisionAtom` + `PBSDRAutoProvision` | hub/internal/web/pbsdr.go | `(ctx, customerID, host, storageID) (blocked string, err error)` / `(ctx, customerID)` | The shared fresh-provision cascade atom; the WG-registration hook target (api `SetWGRegisteredHook`, wired in hub/cmd/hub/main.go when tenantsync is on) | `blocked != ""` = waiting stage (never an error); the hook runs in a detached goroutine and must never fail registration. Scenario-A e2e test: TestPBSDR_AutoProvisionOnWGRegistration. |
| `cfg.DRTier` + offsite coupling | hub/internal/store/store.go (CustomerConfig), hub/internal/web/configs.go (applyOffsite guard) | bool | Per-customer DR-tier flag: new-customer default ON (handleConfigNewForm); offsite REFUSED without it (exact F-6 message) | One-time migration backfill initializes legacy rows from descriptor reality — never re-runs (opt-outs survive re-open; store test pins it). Form field `dr_tier` (formBool helper). |
@@ -189,7 +189,12 @@
## 5. Extension points (where new features plug in)
- **New event type**: add to `allowedEventTypes` (hub/internal/api/handler.go ~L1063) **and** `customerMessages` (hub/internal/notify/templates.go) **and** the customer-prefs default list if customer-notifiable. Missing the first = controller POST 400s (the known gotcha).
- **New event type — THREE registers, and which ones depend on the AUDIENCE.** Always: `allowedEventTypes` (hub/internal/api/handler.go) — missing it means the controller's POST 400s and the event vanishes (the known gotcha). Then decide the audience and stop guessing from the other registers:
- **Operator-only** → add to `notify.operatorOnlyEvents` (hub/internal/notify/dispatcher.go) and give it **no** `customerMessages` entry. **Allowlisting alone does NOT make a type operator-only**`FormatCustomerEmail` treats a missing `customerMessages` entry as a *fallback to the raw message*, not a block, and the only customer gate is configuration. v0.78.0 asserted the opposite in a comment and shipped the defect (R-97c). Examples: `whole_guest_backup_failed`, `recovery_unit_capture_failed`.
- **Customer-facing with a STATIC message** → add a `customerMessages` entry (hub/internal/notify/templates.go) and the controller's `settings.DefaultEnabledEvents` if it should be on by default.
- **Customer-facing with a DYNAMIC message** (the producer builds Hungarian text carrying names/numbers) → deliberately **no** `customerMessages` entry: `FormatCustomerEmail` PREFERS the entry over the message, so adding one silently discards the specifics. Examples: `offbox_enlarge_blocked`, `disk_health_degraded`, and since v0.89.0 `disk_warning`/`disk_critical`.
- Pin BOTH registers in ONE test (hub/internal/api/recovery_unit_event_test.go is the model) — fixing one and not the other is the realistic mistake, and `notify.IsOperatorOnly` exists so the api package can assert it.
- **A type in these registers with no PRODUCER is inert.** `disk_warning`/`disk_critical` were allowlisted, copy'd, default-enabled and checkbox'd from early on, and nothing in any repo emitted them until controller v0.191.0 — grep for an emitter before assuming a type works.
- **New monitor checker**: copy hub/internal/monitor/staleness.go (§2 pattern); wire in hub/cmd/hub/main.go with an `EventNotifyFunc`; severity must be warning/error/critical to notify.
- **New API route**: switch in `api.ServeHTTP` (handler.go ~L139); auth helper first line.
- **New web page/action**: switch in `web.ServeHTTP` (server.go ~L182) — non-GET gets CSRF automatically; template into hub/internal/web/templates/ (embedded FS, parsed in `web.New`); new helpers into the funcMap (server.go ~L67).
+57 -34
View File
@@ -1,6 +1,6 @@
# STATUS — what works, what's broken, what's next
**Updated 2026-08-02.**
**Updated 2026-08-03.**
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this
> page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**,
@@ -14,53 +14,76 @@ A blank machine boots the Felhom disc, installs itself unattended, and is claime
who sets their own password. They install apps from a catalogue of fifty-three, share files over the
home network, and open apps from a launcher or a shared link. Backups run on their own to three
places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can
restore files and app data from the drive alone. Proven end to end on real hardware.
restore files and app data from the drive alone. Apps come back after a power cut: hard-reset the demo
box six times, everything returned every time, and an app switched off deliberately stayed off.
Proven end to end on real hardware.
## What's broken
**After a power cut, an app can stay switched off — and nothing says so.** The machine looks for apps
that didn't come back, but looks too early and never again. In one case it decides the customer
switched it off deliberately, so it isn't even counted as down. *(R-157)*
- **Rebuilding a machine silently takes away its off-site app-data backup.** `demo-hp` was rebuilt on
3 August and came up without one, and stayed that way for a day. **Fixed on 4 August** — re-issued,
the machine picked the new password up in 15 seconds and reattached to the same repository, and you
escrowed the key. **The underlying fault is not fixed:** the off-site password is delivered exactly
once and a rebuilt machine cannot ask for another, so this will happen again on the next rebuild.
The other machine survived the same rebuild only because an unused password happened to be waiting
for it. *(R-193)*
- **The daily email about it tells you the wrong story**, and the automatic repair that exists for
this declines without saying why. The message says the password was never applied; it was, on
23 July, and worked for eleven days. *(R-192)*
- **The weekly off-site backup reports FAILED although it worked.** It uploads correctly and then
trips on a tidy-up step it is deliberately not allowed to perform, so the job ends in an error and
you get an email. The backup itself is safe and on the endpoint. Both demo machines do it; one
setting per machine fixes it. *(R-191)*
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
also delete it. A daily snapshot is armed as a stopgap.
*(R-95, R-87)*
**The off-site copy can be erased by the machine that made it** — the credential that writes it can
also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy.
*(R-95, R-87)*
## What shipped recently
**Three apps out of fifty-three kept their data where backups never looked.** They reported healthy;
the data would vanish on the next update. Two are fixed, the third is now clear to fix because it is
installed nowhere. *(R-156)*
- **The on-machine backup copy has now been proved to restore — by the machines themselves.** Both
demo machines restored their own on-machine backup into a throwaway machine overnight, booted it,
checked it and destroyed it, without being asked: 84 and 109 seconds each. Every restore proof we
had before this was of the *off-site* copy; the copy an ordinary recovery would actually use had
never been tested on either machine. Both also proved their off-site copy on the same night, one
after the other rather than at once, which is the machine deciding for itself what to do first.
*(closes the last open half of R-86/R-185)*
- **A backup copy the machine was never allowed to read — and could not tell you about**, on both
demo machines. The permission was one command; the silence was the real fault, and the machine now
checks whether it may read each copy it depends on and says so when it may not. *(R-185)*
- **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked
data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was
restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a
day, because the machine correctly refuses to re-prove an archive it has already proven. It is now
written to disk with the result and survives. This was caught happening, not predicted: a real
14.5 GB off-site restore passed and left no record at all. **(2)** Every release had about a
fifty-fifty chance of emailing you a failure for a release that worked; the version tag is now
published after the binary, and a new check catches the opposite mistake so nothing is traded away.
**(3)** A released binary can now be rebuilt by anyone and checked against the fingerprint you
approve — until today, rebuilding produced different bytes. *(R-189, R-188, R-186)*
**Local backups get 20 GB while apps get 50 GB.** An app that outgrows the smaller space stops being
backed up locally — and the off-site copy is made from the local one, so that stops too. Nothing is
lost: the last good copy is kept intact. *(R-163)*
**When that happens, only one page says so** — no email, no alert. The page that answers "is this app
backed up?" is the one that stays silent. *(R-158)*
**The check that catches this needs a person to remember it.** One command, run by hand; nothing
refuses a change that skipped it. *(R-161)*
## What we're working on
- **Now:** the last app whose data was never saved; today's decisions written down.
- **Next:** merging the small backup partition into the large one, with the drive-filling warning and
the backup-failure alert in the same step.
- **After:** rebuilding how the machine records whether an app is meant to be running.
- **Now:** nothing outstanding.
- **Next:** proving the off-site *app-data* copy can actually be restored — the one tier nothing
tests unattended. Most of the machinery it needed arrived with the restore-test change below.
*(R-87)*
- **After:** the off-site copy that the machine making it can still erase. *(R-95)*
## Waiting on you
- **How a new version reaches a machine.** Pushing the installer publishes it — half a minute later
every new machine downloads it, with no staging and no way back but another push. And publishing is
a step we remember rather than one the release performs, forgotten twice: a fix can be live here
and still not reach a new machine. Nothing is installing today, so this is the cheapest moment to
settle both. *(R-110, R-115)*
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
session log; nothing suggests anyone else saw it. *(R-132)*
- **One small question, not urgent.** The automatic check cannot see which version you have told
machines to install, only which ones exist. Closing that needs either a password given to the build
server or a check inside the hub itself. *(R-184)*
- **Nothing else.**
## Changed since last update
- **2026-08-02**Decided: the 20 GB backup partition goes away and shares space with app data. That
changes the disk layout, so it happens before any machine is installed outside the house.
- **2026-08-02** — Decided: only this machine and the tester's box are protected; every other box,
both demo boxes included, may be broken or reinstalled freely.
- **2026-08-02** — Two of the three apps that never saved their data are fixed; this page created.
- **2026-08-04**Both demo machines proved their on-machine backup restores, on their own,
overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the
weekly off-site backup reports failure after a successful upload. *(R-185, R-191)*
- **2026-08-03** — Fixed three ways the alarm system misreported itself: a proof of a working backup
that could vanish on a restart (seen happening), a release that emailed a failure for a release
that worked, and a released binary nobody could rebuild and check. *(R-189, R-188, R-186)*
File diff suppressed because one or more lines are too long
@@ -53,6 +53,114 @@ Risk tags: **clean** · **needs-rework** · **hazard** (entangles a delete-targe
---
## 0a. App state: desired / in-flight / observed — S-1 CONTRACT (2026-08-02, decision D-b, R-166)
> **This section is a live contract, not migration history** — the rest of this document is the
> v0.33 keep/port/delete inventory. Read this before touching `stacks/`, `backup/` or `bootrecon/`.
> Shipped in controller **v0.189.0**.
An app's state is **three different kinds of information**, and conflating them is what produced
R-157 mechanism B and F-CRIT-1. They are stored differently on purpose.
| Kind | Question it answers | Where it lives | Persisted? |
|---|---|---|---|
| **Desired** | *What did the customer ask for?* | `app.yaml``desired_state` | Yes, beside the app's other settings |
| **In-flight** | *Is an operation part-way through, and did it finish?* | its **own** marker file under `<data_dir>` | Yes, written before the operation and cleared after |
| **Observed** | *Is it running, unhealthy, restarting, is its drive gone?* | nowhere | **No — rebuilt by looking** |
**The rule that ties them together: never derive one from another.** The defect this replaced did
exactly that — it derived *desired* from *observed* (zero containers ⇒ "the customer stopped it"),
and zero containers is equally what a power cut mid-compose, an interrupted deploy and an interrupted
backup leave behind. Two real faults were therefore read as deliberate stops and stranded silently.
### Desired — `app.yaml`, `desired_state`
Tri-state: `""` (unknown) · `"running"` · `"stopped"`.
- **ONE OWNER: the customer's own action.** Writers are the `/api/stacks/{name}/{action}` switch,
`DeployStack`, `UpdateOptionalConfig`'s redeploy branch, and the `.fab` import. **`StartStack` and
`StopStack` are NOT writers** — a census found 14 callers of which only 2 are the customer; the
rest are quiesce, the backup volume dump, offbox reconstitution, app export/restore, the storage
gate, migration and the boot reconciler. Intent recorded in the primitive would make a nightly
backup indistinguishable from the customer pressing Stop.
- **Written BEFORE the act; a failed write REFUSES the act.**
- **Absent means UNKNOWN — never "running".** Every `app.yaml` predating v0.189.0 lacks the field,
so consumers must fall back to the pre-v0.189.0 behaviour rather than assume. A running-only
backfill converges the unambiguous cases; **`stopped` is never inferred, from any signal.**
### In-flight — a marker file, one per owner
Two exist and they are deliberately **separate files**: `quiesce-state.json` (the whole-guest backup
window, `internal/quiesce`) and `appstop-state.json` (app-data operations that stop an app —
`backup.AppStopGuard`, covering the volume dump, offbox reconstitution and `.fab` export). **One
file, one writer**: sharing would give one record two lifetimes, and one owner clearing the other's
note is a stranded app by a different route.
- Written **before** the stop; cleared **only** after a restart that succeeded; a **failed** restart
keeps the marker so the next startup retries.
- **A `defer` is not the mechanism.** A SIGKILL runs no deferred function — established on live
hardware by Campaign 8 fault 10, where what brought the stacks back was the marker read at startup.
- Recovery runs at startup and **completes before** the boot reconciler is launched, so an app the
marker explains is not also reported as an unexplained boot orphan.
### Observed — not persisted, by design
`aggregateState` walks **every container** of a stack and any unhealthy or mixed result wins, so a
partly-dead app cannot read as healthy (F-CRIT-1's shape). This requirement is met here and must not
be re-implemented downstream. Nothing about observed state is written to disk: a controller restart
re-observes it within one refresh, whereas persisting it risks carrying a stale verdict across the
very restart that fixed it (the same argument as `RestartingSince`).
### The binding safety rule (verbatim, from decision D-b)
> *Losing the state store must never cause an app to be deleted, restarted wrongly, or reported
> healthy when it is not — the worst acceptable outcome is re-running a backup that already ran.*
Applied: a lost or corrupt marker means the app is not auto-restarted **by that mechanism**, which is
the pre-v0.189.0 position, not a new hazard. A lost `app.yaml` already means the app is not deployed.
**Nothing here may make an absent file more dangerous than a present one.**
### Boot recovery reads desired, and asks before it acts (v0.190.0)
**Both boot gates now read desired state.** `bootrecon.isBootOrphan` (the R-52 sweep) and
`shouldRecreateOnBoot` (the drive-backed recreate gate) answer the same question — *did the customer
want this running?* — with the same three-way table, absent falling back to the pre-v0.190.0
container count in both. Their agreement is pinned from both sides against one fixture table, because
an import cycle prevents testing them together. R-170 closed the last gate that still guessed.
**The sweep observes a SETTLED fleet, not a single early sample.** It samples (name, state, container
count) every 5 s, calls the fleet settled after 3 identical samples, and sweeps **once**, at the end.
The window ends on settled or a 50 s budget, and the log says which. Two constraints bound it:
- `bootReconcileSettle + budget + one DefaultRetryDelay` must stay inside `deadAppBootGrace`, or a
successful recovery stops being silent. This is arithmetic, pinned by a test.
- **each sample must REFRESH first.** `GetStacks()` is the Manager's in-memory map, refreshed by the
scheduler on its own 10 s cadence; sampling it faster without refreshing lets "settled" mean "the
cache did not update". Found by live validation, not review.
A recovery completing after the grace emits a `LATE RECOVERY` warning naming the apps. The grace is
**never** widened to make a late recovery look silent.
**Nothing is started without asking whether it may be.** `bootrecon.StartGate` is the one question
the sweep asks per candidate, and it is **fail-safe: cannot determine ⇒ do not start.** Three holders
answer it, and the last two only became reachable once the window widened past T+5 s:
| Holder | Why starting would be wrong |
|---|---|
| the drive-absent gate | compose creates bind sources wherever the mountpoint points — the guest rootfs |
| a quiesce | a running app inside a snapshot meant to be clean-shutdown-consistent |
| an in-flight app-data operation | restarting an app under its own tar |
The rule is **not new** — the API's `startGatedByMissingDrive` already refused a customer's start on
an absent drive. The sweep bypassed it by calling `Manager.StartStack` directly, which is what R-171
closed. Held apps are reported separately from "still down": they are not a fault the sweep failed to
fix, and reporting them as one is a false alarm.
**Read this before adding a fourth caller of `Manager.StartStack`.** That method has no gate of its
own; every caller that is not the customer must decide for itself whether the app may run.
---
## 1. v0.33 module inventory (package → purpose, key deps)
| Package | Purpose | Key internal deps |
@@ -132,6 +132,41 @@ executed** (`CAMPAIGN-8…:522`), the host-loss plan **executes nothing by const
(`felhom-agent/internal/dr/plan.go:1-4`), and **no host has ever been rebuilt as its former self**
(INV Part D1).
### Lane 2's restore-test is scheduled PER ARCHIVE GENERATION (R-86, 2026-08-03)
**[CONTRACT, changed 2026-08-03 — agent v0.121.0 + hub v0.91.0.]** The scheduled restore-test used to
fire on an interval started at daemon start. It no longer does. The rule is:
> Let **A** be the newest archive on a tier that has settled for at least the settle lag (24 h).
> The tier is **DUE** when **A** exists and **A has not already been proven**.
So a tier is proved **once per archive**, on its own archive, and the proof follows the backup rather
than the process's uptime:
| tier rhythm | what is proved, and when |
|---|---|
| daily (host tier) | yesterday's archive, once a day |
| weekly (offsite tier) | last week's archive, once a week |
| newborn (no archive yet) | nothing — **UNKNOWN, never a fault** |
**The trap in the obvious formulation, recorded so it is not reintroduced:** *"due when the newest
archive is ≥ 24 h old"* is never true on a **daily** tier — a new archive resets the newest-archive
age to zero long before it reaches the lag — so the literal reading silently switches restore-testing
off for the tier that matters most.
What survives unchanged: the restore-test itself (restore → boot → verify → destroy the scratch), its
journal and crash recovery, the scratch VMID band, the one-heavy-operation gate, proof credit only on
success, and oldest-proven ordering, which is now the tie-break **between due tiers**. A ticker
remains, but only as the **evaluation interval** (6 h by default, chosen from a measured cost: one
due-check is 18 ms on a local dir storage and 392 ms on the PBS tier over the WAN).
**The hub's half is not optional.** `restoreProvenStaleAfter` was a flat 7 days derived from the very
cadence this replaced, and a weekly tier proved weekly reaches a proof age of **exactly** one interval
just before its next proof — 168 h against a 168 h window. It sat ON the line, so any ordinary delay
tipped a healthy tier into a nightly alarm. The window is now per tier, from that tier's observed
archive interval, floored at the old 7 days, capped at 12 days (strictly inside the two-week offsite
retention), and falling back to the tier's declared rhythm when history is too short to observe one.
### Why the split is right, stated once
**[DESIGN]** A customer can reason about "my photos are gone". A customer cannot reason about
@@ -529,6 +564,102 @@ portable secrets in the unit. **That independence is bounded by app size**, and
> Past that the unit cannot be captured, and the app falls back to Lane 2's operator-driven
> whole-guest route.
**AS OF 2026-08-02 SOMETHING NOW WARNS, AND THE ALERTING IS PART OF THIS CONTRACT (R-167 / R-158,
decision D-c; controller v0.191.x + hub v0.89.0).** The last sentence of this section used to end
"nothing warns when an app crosses the line". Two signals now exist and both are PROVEN-LIVE:
- **To the CUSTOMER, before anything fails**`internal/fillwatch` warns per FILESYSTEM (never per
app: one full disk holding ten apps would fire ten times) on **whichever trips first, used ≥ 85% or
free < 5 GiB**, critical at 95% / 2 GiB, clearing at 75% / 7 GiB. **Two terms, because a percentage
alone lies at both ends of the range this section itself documents:** 85% of a 20 G `mp1` leaves
3 G — less than one DB-backed app's unit — while 85% of a 4 TB drive leaves 600 G. It watches the
app-data volume, the system-data volume **and** every registered drive, which the previous
`health_degraded` signal did not. Edge-triggered against persisted state; the hub owns cooldown.
- **To the OPERATOR, when a capture actually fails**`recovery_unit_capture_failed`, per app, with
the target filesystem's used/free bytes at the moment of failure, so the *why* needs no login. It is
**operator-tier** (`notify.operatorOnlyEvents`) and deliberately not `backup_failed`: a customer can
take no action on a capture failure.
### 7.5.1 — THE CEILING THIS SECTION DESCRIBES HAS BEEN REMOVED (2026-08-03, R-165 / decision D-a)
**Everything above describes the SPLIT layout, which is now the legacy shape.** A golden built by
`build-golden.sh` **v3.0.0** ships **one** data volume; `mp1` does not exist. Both consumer paths are
binds of subdirectories of it (variant **V-c**):
```
mp0 -> /var/lib/felhom ├─ docker/ --bind--> /var/lib/docker
└─ sys_drive/ --bind--> /mnt/sys_drive
```
**So the size bound below no longer applies to a box built from that golden.** A driveless app's
recovery unit is limited by the box's actual free space, not by a partition set at build time. The
mismatch table above (`mp0` 50 G vs `mp1` 20 G) describes what a merged box no longer has.
**R-175, fixed here rather than left standing.** The bound below was stated as the fleet's and was
**one box's**: it is derived from `mp1 = 20 G`, which is demo-hp exactly and never was demo-felhom
(`mp0 200G / mp1 50G`, where the same arithmetic gives ≈ 49 GB / ≈ 24 GB), nor the golden (`16 G / 8 G`
before provision grew them). **Read it as a function of `mp1`, and only for a box still on the split
layout.** Measured: `audits/SPIKE-r165-mp1-merge-2026-08-02.md` M1.
**What replaced the partition's second job — the reserve.** `mp1` was also a BULKHEAD: an overflow was
refused per app with the last good unit byte-identical, and it **could not reach `/var/lib/docker`**,
because that was a different filesystem. On a merged box it can. Decision **B2**, shipped in controller
**v0.192.0**, is that bulkhead made deliberate — a two-term reserve (97% used or 1 GiB free) in
`internal/fillwatch`'s shape, sitting beyond its critical band so the customer is always warned first.
It **refuses per app and never deletes**: nothing on this filesystem is generational, so pruning could
only destroy a different app's only local copy.
**WHAT IS RECORDED, WHAT IS E-MAILED, AND HOW OFTEN (controller v0.194.0 + hub v0.90.x, R-182).**
The two are deliberately different mechanisms, because conflating them is how seven failures went
missing on 2026-08-03 without leaving a trace.
| | Record | Notification |
|---|---|---|
| what | `recovery_unit_capture_failed`, one per failed app | `backup_run_failures`, one per RUN |
| when | every time, unconditionally | at the end of a run, **only if something failed** |
| gated by | nothing — not cooldowns, preferences or delivery | the hub's operator cooldown |
| where it lands | the events table **and** `notification_log` (status `recorded`) | the operator's inbox |
- **A clean run e-mails nothing.** Silence means the run finished and found nothing wrong — and that
is only safe because the hub's daily deadline check raises `expected_backup_missed` from the box's
REPORT freshness, independent of any mail the box sends. That check is load-bearing for this
design; weakening it re-opens a silent-failure path.
- **A suppressed operator notification leaves a `suppressed` row** naming the key that suppressed it.
Deciding not to tell someone is itself an event worth recording.
- **Deliberate skips are not failures** and never appear in the digest — a disconnected or
decommissioned drive has its own alert, and a nightly digest about an unplugged drive is one the
operator stops reading.
- **Cadence:** a nightly run gives at most one mail a day. A manual run always reports, even within
the hour, because someone pressing the button is actively trying to get a backup. The periodic
capture sweep is capped by the ordinary hourly cooldown.
**THE CONTRACT, stated as what the code provides (controller v0.193.0, R-181).** The reserve is a
**per-app, per-run ADMISSION decision, not a capture check.** It is taken once for an app, immediately
before that app's FIRST write of the run, and it covers **all three write legs — the database dump, the
volume dump and the recovery-unit capture**. Those three write under one per-app root
(`backups/primary/<app>`), which is what makes one verdict able to cover them honestly.
- **What it guarantees.** A refused app has **nothing written for it in that run**, its previous unit
is **byte-identical**, it is **not stopped**, nothing anywhere is deleted, and the operator gets
**exactly one** alert naming the app, the term that bound and the disk figures.
- **Two terms, two questions.** *Headroom*: is the filesystem already below the reserve? *Size*: would
THIS app's write take it below? The size estimate is the app's previous `.sql` + `.tar` on disk;
with no history the decision degrades to headroom alone, deliberately — otherwise the first backup
is the one that can never happen.
- **Why it is decided lazily and not once per run.** Space changes during a run: app A's dump can put
app B under the reserve, so a verdict taken at run start reads a disk that no longer exists.
- **Why it is never re-decided between an app's own legs.** That is precisely the shape v0.192.0 had —
the two dump legs unguarded and only the capture refused — under which the reserve was consumed by
the very write it exists to bound, and the refusal's *"the previous unit is untouched"* was measured
false. Proven live on demo-hp 2026-08-03 (R-181), fixed the same day, and re-proven by filling the
box for each of the two terms.
- **It sits ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act — a refusal
decided inside it would already have bounced the app it is refusing to back up.
**Status caveat, deliberately explicit:** every box in the field that has not been reinstalled is still
on the split layout and everything above still describes them exactly. This subsection describes what a
box built from golden ≥ 0.192.0 gets. Both demo boxes were reinstalled from it on 2026-08-03 (R-178).
Two things are deliberately **not** recorded here. **The sizing ratio is the operator's ruling**
(**R-163**) — this section states the constraint, not a number. And **the same-device placement is
intended, not a defect**: a driveless app's unit sits on the same SSD as its volumes, but Tier-2's
@@ -724,7 +855,7 @@ does **not** hold as written. → **R-108**
| ~~**R-108**~~ | ~~Network storage can host an app's namespace~~ | **CLOSED 2026-07-30, controller v0.187.0 — D5 UNBLOCKED.** An app namespace may no longer be placed on network storage (5 surfaces guarded by one fail-closed predicate); the share-root bind is deliberately UNCHANGED because it is load-bearing and unscopable (§10.1). `audits/R108-network-app-namespace-2026-07-30.md` |
| **R-126** | A `.fab` bundle — plaintext secrets, optional password — can be exported ONTO a NAS: `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | split out of R-108, which closed without it. NOT a D5 precondition: an explicit customer-chosen export destination, not a browsing surface reaching a backup tree (§5, §7.3) |
| R-95 (open) | The restic offsite credential **can delete** — the box can `forget --prune` its own repo | the tier holding the customer's documents and photos is the one whose credential can destroy it (matrix row 10) |
| R-86 (open) | Restore-tests are interval-scheduled, not backup-aligned | a tier's proof cadence is unrelated to when its archives are written |
| ~~R-86~~ | ~~Restore-tests are interval-scheduled, not backup-aligned~~ | **CLOSED 2026-08-03 — agent v0.121.0 + hub v0.91.0.** Restore-testing is now **per archive generation**: a tier is due when its newest archive that has settled ~24 h has not been proven, so a daily tier is proved daily on its own archive and a weekly tier weekly on its own. The ticker survives only as the evaluation interval (6 h, chosen from a measured cost). The hub's staleness window moved with it — per tier, from that tier's observed archive rhythm — because a weekly tier proved weekly sat EXACTLY on the old flat 7-day line (§3, Lane 2's per-archive rule) |
| R-87 (open) | The restic tier is never restore-tested | matrix row 4's route has no unattended proof |
### 10.3 Divergences that are documented elsewhere and are not re-opened here
@@ -790,7 +921,7 @@ to now *implement* D5 remains an open scheduling decision, not a blocked one.
| whole-guest restore, local and PBS, exact mount parity | **PROVEN-LIVE** | CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C |
| corrupted PBS snapshot fails cleanly | **PROVEN-LIVE** | CAMPAIGN-8 fault 17 |
| the box cannot delete its own **PBS** snapshots | **PROVEN-LIVE** | CAMPAIGN-8, R-89 |
| unattended restore-test across tiers | **IMPLEMENTED** (rotation not observed across consecutive cadences) | `00-capability-map.md:41`; LIVE per-tier timestamps this session |
| unattended restore-test across tiers | **IMPLEMENTED**; **per-archive due-ness PROVEN-LIVE 2026-08-03** (agent v0.121.0) | `00-capability-map.md:41`; the due verdict + a real offsite run on demo-felhom (§3, Lane 2's per-archive rule) |
| guest-power watchdog | **PROVEN-LIVE** | agent v0.107.0, 120 s |
| quiesce crash recovery | **PROVEN-LIVE** | CAMPAIGN-8 fault 10, 1 s, by SIGKILL |
| break-glass | **PROVEN-LIVE** | `runbooks/break-glass.md` |
@@ -45,7 +45,10 @@ hub, guests and PBS are UTC — every timestamp below carries its zone.
the hub SQLite DB was taken with `kubectl exec … cat /data/hub.db > hub.db.copy` at **17:40:00 UTC**
(113,033,216 bytes) into the session scratchpad, and every hub-DB figure below comes from that copy.
It is a hot copy of a live database; row counts and metadata are consistent enough for an inventory
but are a snapshot of that instant, not a transactionally consistent dump. The copy is read with
but are a snapshot of that instant, not a transactionally consistent dump. **Do not reuse this
command as a recipe: since hub v0.88.0 the DB is in WAL mode (R-172), so `cat /data/hub.db` alone
yields a copy that opens cleanly and silently omits the newest writes — the `-wal` must be copied
beside it (`documentation/operations/nodes.md`).** The copy is read with
`mode=ro`. No hub table was written.
**Secret hygiene.** No secret value, key, token, password or key fingerprint is reproduced in this
@@ -0,0 +1,139 @@
# DIAG — does the R-166 boot sweep start an app whose data drive is absent?
**Date:** 2026-08-02 · **Box:** demo `felhom-pve` guest 9201 · **Controller:** v0.189.0
**Question raised by:** reading the v0.189.0 diff, **not** by an incident. Diagnosed before any fix
was written (task §9.2, "diagnose before theorising").
## Verdict
**CONFIRMED — the sweep starts it.** Observed directly, twice in the same run (both attempts).
**With a qualification that changes the severity but not the fix:** on this box the resulting
`docker compose up -d` **failed** and **nothing was written** to the wrong disk. The mechanism that
prevented the write is a **filesystem-permission accident that no code owns**, and the harm that DID
occur is a different one: a false dead-app alarm for an app the drive gate is deliberately holding.
## The reasoning under test
`bootrecon` imports `stacks` alone and has no storage awareness. Since v0.189.0 `isBootOrphan`
returns true for `desired_state: running` + zero containers. The drive-absent gate stops apps with
`compose down` (leaving exactly zero containers) and never touches `desired_state`, because it is not
the customer. `Manager.StartStack` has no drive gate. Therefore the sweep should start an app whose
drive is absent.
## Method
The drive is a real USB disk (`/dev/sdb`), bound under the stable parent at
`/mnt/felhom-drives/hdd_1`. The gate's absence signal is the agent's `BoundUnderParent`
(`planDriveGates`, `intermediary.go:226`), so the drive was made absent **by unmounting it**, not by
editing controller state — the state edit would have proven a different thing.
**Run 1 was contaminated and is reported because it produced a mechanism.** Unmounting only the
parent bind was not enough: **the agent re-binds it within ~60 s** while `/dev/sdb` is still mounted
at `/mnt/hdd_1`. In that run the drive gate's startup reconcile re-attached the drive and restarted
the apps at `17:18:47`, **one second before** `bootrecon` looked at `17:18:48` — which then logged
`no boot-orphaned apps`. That is a race that happened to go the safe way, **not** a disproof. Had it
been reported as one, the conclusion would have been wrong.
Run 2 therefore unmounted **both** `/mnt/felhom-drives/hdd_1` and `/mnt/hdd_1`, and held them
unmounted against the agent's healing for the duration.
## Preconditions, all verified before the observation
| # | Precondition | Observed |
|---|---|---|
| 1 | drive absent | `mount \| grep -c hdd_1`**0** |
| 2 | app has zero containers | `docker ps -a --filter name=calibre`**0** |
| 3 | intent still says running | `desired_state: running` |
| 4 | app still deployed | `deployed: true` |
The gate stopped the apps itself first, on its own signal:
```
17:22:07 [WARN] [gate] drive ABSENT /mnt/felhom-drives/hdd_1 — stopped+blocked 2 app(s): [calibre-web immich]
```
`immich` was set to `desired_state: stopped` beforehand to scope the blast radius to one app; the
gate is per-drive, so it stops every app on that drive regardless.
## The observation
```
17:22:29 [INFO] [stacks] desired-state backfill: 0 app(s) recorded as running, 0 left unrecorded
17:22:34 [INFO] [bootrecon] Boot reconciliation: 1 boot-orphaned app(s) found: [calibre-web] — up to 2 attempt(s)
17:22:34 [INFO] [stacks] Starting stack: calibre-web
17:22:35 [WARN] [bootrecon] attempt 1/2: start "calibre-web" failed after 0.4s: exit code 1
17:23:05 [WARN] [bootrecon] attempt 2/2: start "calibre-web" failed after 0.4s: exit code 1
17:23:05 [WARN] [bootrecon] gave up after 2 attempt(s): recovered=[] still down=[calibre-web] (the dead-app alarm now owns these)
```
**The sweep selected a drive-absent app and called `StartStack` on it.** That is the confirmation.
## What stopped the write, and why it must not be relied on
```
Error response from daemon: error while creating mount source path
'/mnt/felhom-drives/hdd_1/userdata/media/books':
mkdir /mnt/felhom-drives/hdd_1/userdata: permission denied
```
With the drive unbound, `/mnt/felhom-drives/hdd_1` is an empty directory on the host's `pve-root`,
and it is **host-root-owned**:
```
/mnt/felhom-drives uid=0 gid=0 mode=755
/mnt/felhom-drives/hdd_1 uid=0 gid=0 mode=755
```
Guest 9201 is an **unprivileged** LXC, so its container root is uid 100000 and cannot `mkdir` there.
Confirmed no write occurred: `find /mnt/felhom-drives/hdd_1/` returned the directory alone, and
`df -h /` was unchanged at `25G used / 28%` before and after.
**This protection is accidental.** Nothing in the controller chose it, no test pins it, and it rests
on two conditions that are not guaranteed and are not checked anywhere:
1. the guest is unprivileged (a privileged guest maps root→0 and the `mkdir` succeeds);
2. the stable-parent mountpoint directory is root-owned. When the drive **is** bound, that same path
shows `uid=100000 gid=100000` — i.e. guest-writable. Any code path or agent version that
pre-creates the mountpoint with guest ownership removes the protection silently.
It is one `chown` away from being gone, and its removal would be invisible until data landed on the
wrong disk. It is therefore **not** a reason to leave the sweep unguarded.
## The harm that DID occur
Independent of the write question, and real on every box:
- the sweep burns **both** attempts and 30 s of retry delay on an app that cannot start by design;
- it then hands the app to the **dead-app alarm** — `still down=[calibre-web] (the dead-app alarm now
owns these)` — producing a **false alarm about an app the drive gate is deliberately holding**,
which is exactly the noise class R-97/F-A1 exist to prevent;
- it leaves a `Created` container behind on each attempt.
Before v0.189.0 none of this happened: `isBootOrphan` required `len(Containers) > 0`, and a
gate-stopped app has zero. **This is a regression introduced by v0.189.0.**
## A second asymmetry found while restoring the box
The **API** start path already refuses this correctly. Restoring `immich` through the endpoint the UI
calls, while the path was still flagged disconnected, returned:
```
{"ok":false,"error":"A(z) /mnt/felhom-drives/hdd_1 tárhely jelenleg nem elérhető —
az alkalmazás nem indítható, amíg a meghajtó vissza nem csatlakozik."}
```
That is `startGatedByMissingDrive` (`internal/api/router.go`). So the controller already holds the
rule "do not start an app whose drive is missing" — it is enforced on the customer's path and
**bypassed by the sweep**, which calls `Manager.StartStack` directly. The fix is to give the sweep
the same question to ask, not to invent a new rule.
## Consequence for the task
**Part 3 applies and is implemented first**, before the sweep's window is widened — a wider window
makes both the false alarm and the (currently accident-blocked) write hazard wider. Fail-safe
direction per §8.4: **cannot determine drive liveness → do not start.** Not starting is recoverable
(the gate's `Return` branch restarts the app when the drive comes back, and the alarm reports it
meanwhile); starting on an absent drive is not recoverable by anything automatic.
Register row: **R-171**, marked as a regression from v0.189.0.
@@ -0,0 +1,220 @@
# SPIKE — a Gitea Actions runner for the felhom gate entry points (R-168, 2026-08-02)
**Verdict: the mechanism works and is SHIPPED.** All six probes answered; none produced a STOP.
The runner is unprivileged, host-mode, one registration for all four repos, and a failed run now
sends its own alarm — measured, not assumed.
**Context.** Session 1 (same day) gave every felhom repo one gate entry point and a
`.githooks/pre-push` that runs it and refuses a failing push. That hook is per-clone and
`git push --no-verify` skips it, so nothing independent of the person pushing ever saw whether the
gates passed. This spike built the independent half.
---
## Arrival state, re-confirmed live (all matched the anchor)
| Fact | Expected | Measured |
|---|---|---|
| Gitea version | 1.26.2 | **1.26.2** |
| Actions enabled | all five repos | **all five** (`repo_unit` type 10 present on each) |
| Runners registered | 0 | **0** (`action_runner` empty) |
| Workflow runs, ever | 0 | **0** (`action_run` empty) |
| Branch protections | 0 | **0** (`protected_branch` empty) |
| `.gitea/` directory | none | **none in any of the five** |
| ArgoCD `gitea` app | `path: gitea-system`, auto-sync off | **confirmed**, Synced at `420e819`, Healthy |
Repo visibility, which decided P3's design: the four product repos are **public**;
`homelab-manifests` is private.
> **A near-miss worth recording.** My first census query reported Actions enabled on only five
> *unrelated* repos, which looked like a baseline drift big enough to change the task. It was a
> `| tail -5` inside my own query helper truncating the result. The measurement was never wrong;
> the instrument was. Re-run without the pipe, all five target repos had the Actions unit. **A tool
> that silently drops rows is indistinguishable from a finding** — the same class as the
> `go test -run` filter that matches nothing and prints `ok`.
---
## P1 — does a registered runner pick up a job at all?
**Method.** Registered one runner at **owner scope** for `admin` (`repo_id=0`), label `felhom-gates`,
via a token minted with `gitea actions generate-runner-token --scope admin` and stored out-of-band as
`Secret/act-runner-registration`. Pushed a temporary `.gitea/workflows/probe.yml` with one
`run: echo` step to `felhom.eu`.
**Measured.** Run #1 appeared against the pushed commit, was claimed by `felhom-gates-runner`, and
finished **success**:
```
felhom-gates-runner(version:v0.6.1) received task 1 of job probe, be triggered by event: push
P1-OK runner picked up the job
🏁 Job succeeded
```
**Ruling: PASS.** No STOP.
---
## P2 — can the stock image run our checks without a container runtime?
**Method.** Inspected `gitea/act_runner:0.6.1` directly, then re-measured *inside a job* — because
what matters is what the **job** sees, not what the image contains, and in host mode those are the
same thing only if host mode is really in effect.
**Measured.** Stock image (Alpine Linux v3.23):
```
python3: sh: python3: not found ABSENT
git: git version 2.52.0 PRESENT
```
In-job, after building the minimal image:
```
P2 python3: Python 3.12.13
P2 git: git version 2.52.0
```
**Ruling: the known branch, not a failure.** Host mode works; only `python3` was missing. Built
`gitea.dooplex.hu/admin/felhom-act-runner:0.1.0` = stock (pinned) + `python3`, nothing else, from
`homelab-manifests/gitea-system/act-runner/Dockerfile`. Verified by deleting the local copy and
**re-pulling from the registry** rather than trusting the push's own output.
**Host mode was never in doubt, so the privileged/dind pattern was never reached for** (§5/§12 of the
task, and the reason is written into `act-runner.yaml`: DooPlex is Tier 2 and *is* the recovery
chain).
---
## P3 — can a workflow obtain the source without JavaScript actions?
**Method.** No `uses:` anywhere. A plain `run:` step clones from the **in-cluster** Gitea Service
(`http://gitea.gitea-system.svc.cluster.local:3000`) — no ingress, no TLS hop, no geo rule — and
checks out `$GITHUB_SHA`. The four product repos are public, so no credential is needed at all.
**Measured.**
```
P3 pushed sha = bbd62319096a1fb92f72218a28756e6fa101e87b
P3 checked-out sha = bbd62319096a1fb92f72218a28756e6fa101e87b
P3-OK checkout equals pushed commit
```
**Ruling: PASS.** The shipped workflows use the tighter form — `git init` + `git fetch --depth 1
origin $GITHUB_SHA` + `checkout FETCH_HEAD` — which is both shallow and pinned to the **exact pushed
commit** rather than the branch tip, so two racing pushes cannot make a run test the wrong tree. The
full clone in the probe took ~48 s for `felhom.eu`; the shallow fetch is materially faster.
---
## P4 — does one registration serve all four repos?
**Method.** The registration is owner-scoped (`owner_id=1`, `repo_id=0`). Pushed the real workflow to
all four repos and read which runner claimed each task.
**Measured.**
```
task repo runner_id runner
7 felhom.eu 2 felhom-gates-runner
8 felhom-controller 2 felhom-gates-runner
9 felhom-agent 2 felhom-gates-runner
10 app-catalog-felhom.eu 2 felhom-gates-runner
```
**Ruling: PASS.** One registration, four repos. No per-repo registration needed.
---
## P5 — does a failed run signal anything outside the UI? *(the probe that decided Part 4)*
**Method.** Broke a gate deliberately and pushed it **with `--no-verify`**, which is precisely the
bypass CI exists to catch. Run #3 concluded `failure`. Then looked for any outbound signal in the
ten minutes around it: Gitea pod logs filtered for mail/SMTP/notification activity, and the
`notification` table.
**Measured — nothing left the machine.**
```
gitea pod logs, mail/smtp/notif lines since the failure : (none)
notification rows created in the last 10 minutes : 0
```
Gitea's mailer is *configured and enabled* (`[mailer] ENABLED = true`, Gmail SMTP, FROM is the
operator's own address) and the sole user `admin` is active with
`email_notifications_preference = enabled` — so this is not a disabled-mailer artefact at the
config level.
**Honest limit on this measurement.** I did not independently prove that Gitea's SMTP path can
*deliver*, so "no mail" cannot be split with certainty between *"1.26.2 has no action-failure
notification"* and *"the mailer is broken"*. That distinction does not change the design: the alarm
is built on **Resend**, a different path entirely, and Scenario C proves that path end to end. It is
recorded here so nobody later reads this probe as a clean bill of health for Gitea's mailer.
**Ruling: Part 4 applies — build the alarm.** A red tick in a web UI nobody watches is exactly the
defect R-29 filed, rebuilt one layer up.
---
## P6 — does the runner need persistent state?
**Method.** Two measurements, not one, because "it survived a restart" and "the PVC is load-bearing"
are different claims.
- **P6a — restart with the PVC intact:** deleted the pod, let it come back.
- **P6b — restart with state lost:** removed `/data/.runner` (exactly what ephemeral storage would
do), deleted the pod, let it come back.
**Measured.**
| | runner list before | after | runner log |
|---|---|---|---|
| P6a | `1 felhom-gates-runner` | `1 felhom-gates-runner` | no re-registration; straight to `Starting runner daemon` |
| P6b | `1 felhom-gates-runner` | `1 …` **and** `2 …` | `Registering runner…``Runner registered successfully` |
**Ruling: the PVC is load-bearing.** Without persistence, every restart mints a new registration and
leaves the previous one behind as a permanently-offline record — the runner list would silently fill
with corpses. Shipped with a 5 Gi Longhorn PVC (largest repo checked out is ~125 MiB).
**The P6b casualty was cleaned up immediately**, not left for the teardown section:
`DELETE /api/v1/admin/actions/runners/1 → 204`, leaving exactly one runner.
---
## What the probes changed about the design
1. **A custom image** (P2) — stock + `python3` only, base pinned by tag.
2. **Shallow fetch of the exact SHA** rather than a clone of the branch tip (P3).
3. **A PVC** (P6), with the reason recorded in the manifest.
4. **A self-sent alarm** (P5) — the task's whole second half.
5. **A sibling clone for two of the four repos** — discovered while writing the workflows, not by a
probe. `controller_gates.py` and `agent_gates.py` invoke the shared `reuse_refs_check.py` that
lives in the `felhom.eu` clone next door and is deliberately never copied, and both repos'
`REUSE.md` files cite a path that lives in the hub. Without the sibling, CI would have failed
**closed** — correctly, but for the wrong reason. CI now reproduces the workspace's sibling
layout, and the resulting tallies match the local hook exactly (controller 126 exact / 6 suffix /
1 cross-repo; agent 88 / 1 / 1). **CI and the hook agree.**
## Two things that failed on the way, and why they are recorded
Both were caught because the step failed **loudly**; either would have shipped as a silent
non-alarm if the step had swallowed its exit code.
- **`curl: command not found`** — the first alarm used `curl`, which the deliberately minimal image
does not carry. Fixed by using `python3` + `urllib` rather than by growing the image: reaching for
a bigger base to send one HTTP request is the wrong trade, and every added tool becomes something
the next person assumes is load-bearing.
- **Cloudflare 403, error 1010**`api.resend.com` sits behind Cloudflare, which blocks the default
`Python-urllib/3.x` User-Agent. **This failure looks exactly like an auth failure and is not one**,
which is the reason it is written down: the next person to see a 403 from Resend should check the
agent before rotating a key. Verified the fix from inside the runner image with a deliberately
invalid payload — with a User-Agent set, Resend answers `422 missing_required_field`, i.e. the
request reaches the API rather than the CDN.
## Standing limit, stated so it is not mistaken for something it is not
**CI here DETECTS. It does not BLOCK.** Every felhom repo pushes straight to `main` with no pull
request, so there is no merge for a status check to gate. This is not a gap in the runner — there is
no gate in the road. The refusing half is the local pre-push hook; this half notices when that hook
was skipped. Making CI blocking requires branch protection and a pull-request workflow, which is a
change to how the operator works and is **their** decision, not this task's.
@@ -10,7 +10,7 @@
**Class:** SPIKE (empirical validation; no product code). **Repos:** felhom.eu (this doc only);
felhom-agent read-only for grounding (`internal/pbs/{client,pin}.go`, `configs/build-golden.sh`,
`internal/hub/cloudflared.go`, `internal/escrow/identity.go`).
**Probe ends:** `felhom-hetzner` = Hetzner CX23, Debian 13.4, public IPv4 `167.233.158.164`,
**Probe ends:** `felhom-hetzner` = Hetzner CX23 (**note added 2026-08-03: rescaled to a CX33, 8 GB RAM — this spike records the machine as probed and its body is deliberately unchanged**), Debian 13.4, public IPv4 `167.233.158.164`,
global IPv6 `2a01:4f8:...::/64` (throwaway — NOT the live jarrs.eu box) ⟷ **demo-felhom** =
the real PVE 9.2.2 host on the operator's home line (One Hungary fixed cable, Budapest), driven
over the existing LAN SSH path; the tunnel itself always dialed **out**.
@@ -0,0 +1,248 @@
# SPIKE — measuring the `mp1``mp0` merge (R-165, decision D-a)
**Date:** 2026-08-02 · **Author:** Claude Code · **Status:** MEASUREMENT ONLY — **no layout changed**
> **Decision D-a is already taken** (`CONTEXT.md` S-5): the dedicated backup partition is merged away
> rather than resized. This document measures **how**, never **whether**. It ends in a question for the
> operator and a **STOP** — the merge itself is next session's supervised work.
## 0. What was and was not touched
Nothing was created, resized, moved or deleted. No golden was rebuilt, no guest config edited, no
partition altered. Every figure below is a read of live state or of committed source. `ep0` and Peti's
box were not contacted at all (`runbooks/target-selection.md`, decision D-d).
The monitoring D-a requires (**R-167**) shipped **before** this measurement, in controller v0.191.x and
hub v0.89.0, and was proven on guest 9201. That ordering is the point: D-a's condition (2) says the
monitoring lands with the merge and never after, and landing it first is strictly better.
---
## M1 — what is actually there
**`pct config 9201`, both demo hosts, read 2026-08-02.**
| | demo-felhom (N100) | demo-hp (t740) | golden default |
|---|---|---|---|
| `rootfs` | 32 G | 32 G | `OS_SIZE_GB=32` |
| `mp0` `/var/lib/docker` | **200 G**, `backup=1` | **50 G**, `backup=1` | `GOLDEN_DOCKER_GB=16` |
| `mp1` `/mnt/sys_drive` | **50 G**, `backup=1` | **20 G**, `backup=1` | `GOLDEN_SYSDATA_GB=8` |
| `mp8` `/mnt/felhom-drives` | bind | bind | — |
| `mp9` bootstrap | bind, `ro=1` | bind, `ro=1` | — |
**Occupancy (`df`, in-guest):**
| filesystem | demo-felhom | demo-hp |
|---|---|---|
| `/` | 945 M / 32 G (4%) | 942 M / 32 G (4%) |
| `/var/lib/docker` (`mp0`) | 13 G / 197 G (7%) | 5.4 G / 50 G (12%) |
| `/mnt/sys_drive` (`mp1`) | **2.0 G / 50 G (5%)** | **92 M / 20 G (1%)** |
| data drive | 21 G / 916 G (3%) — `hdd_1` | 6.7 G / 938 G (1%) — `nvme-1tb` |
**THE FIRST FINDING IS THAT "THE LAYOUT" IS NOT ONE THING, AND THE SPREAD IS WIDER THAN §7.5 RECORDS.**
`architecture/07-backup-architecture.md` §7.5 states the default appliance as `mp0 50G / mp1 20G`
that is demo-hp exactly, and it is **not** demo-felhom, which ships **200 G / 50 G**, four and two and
a half times larger. Both differ again from the golden's own `16 G / 8 G`, because `provision` grows
the volumes after restore. **Any merge plan expressed as a fixed pair of numbers is already wrong for
one of the two boxes that exist.**
**The corollary matters more than the numbers.** §7.5's headline bound — *"≈ 19 GB of app data for a
file-only app, ≈ 10 GB for a DB-backed one"* — is derived from `mp1 = 20 G`. On demo-felhom, where
`mp1` is 50 G, the real bound is ~49 GB / ~24 GB. **The architecture doc states one bound as if it
were the fleet's, and it is one box's.** That is a documentation defect independent of the merge and
is filed as its own row.
---
## M2 — what lives on `mp1` (it is not only backups)
**`du`, in-guest, read 2026-08-02.**
```
demo-felhom /mnt/sys_drive/felhom-data 2.0 G
├── backups/primary 269 M Tier-1 units of DRIVELESS apps (~30 apps)
├── backups/secondary 1.7 G Tier-2 mirrors
└── userdata/import 12 K the canonical drop-zone (R-75)
demo-hp /mnt/sys_drive/felhom-data 92 M
├── backups/primary uptime-kuma, paperless
├── backups/secondary paperless-ngx
└── userdata/import paperless
```
**Four distinct things would move, not one**, confirming the task's warning that a plan accounting
only for the units is wrong:
1. **`backups/primary/<app>`** — the RETAINED Tier-1 recovery unit of every app with no data drive.
Thirty apps on demo-felhom.
2. **`backups/secondary/<app>`** — Tier-2 cross-drive mirrors, including the `_shares` pseudo-stack.
On demo-felhom this is **1.7 G of the 2.0 G — the majority is Tier 2, not Tier 1.**
3. **`userdata/import`** — the canonical, app-INDEPENDENT drop zone (`GetImportRoot`, R-75). It lives
on the system drive **by contract**, not by convenience.
4. **`userdata/`** more generally — the system-data userdata namespace for driveless apps.
**Observed occupancy is far below capacity on both boxes** (5% and 1%). The measured pressure today is
zero; the constraint R-163 records is a *ceiling* problem, not a *current fill* problem. That is worth
stating plainly because it bears on urgency, not on correctness.
---
## M3 — which merge shapes exist, and what each breaks
Three interacting assertions exist today, and a merge touches all three. **All three were read at
source; none had been measured before.**
**(a) The golden build ASSERTS the split and ABORTS if it is absent.**
`felhom-agent/configs/build-golden.sh:130`:
```
findmnt -no SOURCE,FSTYPE /mnt/sys_drive | grep -q . || {
echo "[golden] FATAL: /mnt/sys_drive is NOT a separate mount — the mp1 split did not take"; exit 1; }
```
There is a sibling assertion for `mp0` at :126, and **two more** at :315-324 that abort if the vzdump
log shows `excluding volume mount point mp0` or `mp1`. So the golden build fails closed on the split
in **four** places, not one.
**(b) The whole-guest archive's scope is `rootfs + mp0 + mp1`,** and only because both carry
`backup=1`. `build-golden.sh:69` records why: *"backup=1 is MANDATORY: without it vzdump EXCLUDES the
volume (extra mountpoints default backup=0)"*.
**(c) `mount_parity` compares the ARCHIVE's `mpN` set against the RESTORED guest's**
(`felhom-agent/internal/reconcile/restoretest.go:271-283`, `mountParity` at :347). Per slot it requires
the same mount path and a restored size **not smaller** than the archive's, and it checks the reverse
direction too. A mismatch fails the restore-test outright.
| shape | (a) golden assertion | (b) archive scope | (c) `mount_parity` | verdict |
|---|---|---|---|---|
| **S1 — one volume, two directories.** `mp1` stops existing; `/mnt/sys_drive` becomes a directory on the `mp0` filesystem | **BREAKS** — :130 aborts the build. Must be deleted, and :319's vzdump guard for `mp1` with it | **HOLDS** — scope becomes `rootfs + mp0`, still complete, because the data moved onto `mp0` | **HOLDS for new archives** (no `mp1` in archive ⇒ none required in restore). **A pre-merge archive restored into a merged guest is a different question — see below** | the shape D-a describes |
| **S2 — two mounts, one backing pool.** `mp0` and `mp1` remain separate `mpN` slots on the same storage | **HOLDS** — both are still separate mounts | **HOLDS** unchanged | **HOLDS** unchanged | **does NOT remove the ceiling** — two filesystems still have two independent `df`s. This is thin-provisioning, not a merge, and it converts a clean per-app refusal into a shared-pool exhaustion that neither volume can see coming |
| **S3 — grow `mp1`, keep the split** (**the shape D-a REJECTED — measured here as the baseline**) | **HOLDS** | **HOLDS** | **HOLDS** | zero structural risk, one `--sysdata-grow` value. It is *"the same wall further away"* — D-a's own words — and does not close R-163 |
**The one genuinely unmeasured item in M3, stated as unmeasured:** whether a **pre-merge archive**
(carrying `mp1`) restores cleanly into the merged world. Reading `mountParity` says it should — the
restore recreates `mp1` from the archive, so archive and restored guest agree, and parity passes. But
**that was reasoned from source and not executed**, and this project's own record is that four
production designs specced against unvalidated mechanisms were all wrong. **It is a one-command
restore-test on a Tier-0 box and should be run before the merge, not after.**
---
## M4 — the bulkhead question (the important one)
**Today `mp1` is not only a ceiling; it is a BULKHEAD.** An app whose recovery unit outgrows the space
is refused **per app**, its last good unit is preserved **byte-identical** (R-158's measurement), and
— critically — **the overflow cannot reach `/var/lib/docker`**, because it is a different filesystem.
The container runtime keeps running.
After S1 the same overflow lands on the filesystem Docker itself runs on. A runaway recovery-unit
capture would fill `/var/lib/docker`, and a full Docker data-root is not a degraded state, it is a
stopped one.
**This is the one place where "the merge is cheap" stops being true**, and it is why the warnings
shipped first rather than alongside.
Four candidate replacements. **No choice is made here — this is the operator's ruling.**
| # | replacement | what it costs | what it leaves open |
|---|---|---|---|
| **B1 — a reserved block percentage on the merged filesystem.** `tune2fs -m` reserving N% for root; the controller runs unprivileged, so a capture cannot consume the reserve while root-owned Docker can | one `tune2fs` at build time; no code | the reserve protects *root*, not *Docker's runtime need* specifically; sizing it is a guess without a measured worst case |
| **B2 — a refusal threshold in the capture path.** `captureAllRecoveryUnits` refuses when free space would drop below a floor, per app, and emits the alert R-158 just wired | small, local, testable; **reuses the alert that now exists** and restores the per-app-refusal semantics the bulkhead gave for free | the floor is a number needing justification; it protects the capture path only — a customer filling `mp0` through app data is untouched by it |
| **B3 — a filesystem quota on the backup directory.** XFS project quota / ext4 project quota on `<mp0>/sys_drive/backups` | enforces at the filesystem, so **every** writer is bounded, not only the capture path | recreates a fixed ceiling — i.e. it is R-163 again inside one volume, and D-a's objection to *"a bigger number is the same wall further away"* applies to it word for word |
| **B4 — R-167's warnings are deemed sufficient.** No hard stop; the customer is warned at 85% / 5 GiB and critically at 95% / 2 GiB | zero — it is already shipped and proven live | **a warning is not a bulkhead.** It depends on a human acting within the window, and the failure it fails to prevent is "Docker's data-root is full", which is the worst failure on the box |
**The measured input to that ruling:** on demo-felhom `mp1` holds 2.0 G against a 50 G ceiling, and
`mp0` has 175 G free. The overflow scenario is not close today on either box. **B2 is the only option
that preserves the property the bulkhead actually provided** — a per-app refusal with the last good
unit intact — and it is the one that reuses what R-158 just built.
---
## M5 — existing boxes, and what a migration costs
**Read from the hub's own registers, 2026-08-02. No box was contacted.**
The hub's `/hosts` register holds **four** hosts, of which **two are ONLINE**:
| host | customer | agent | status |
|---|---|---|---|
| `demo-felhom-8363b5` | Demo Ügyfél | 0.119.0 | **ONLINE** |
| `demo-hp-bb76ea` | Demo HP | 0.119.0 | **ONLINE** |
| `drill-r50-0a4f9a` | drill-r50 | 0.113.0 | DOWN |
| `sess-f-2670b5` | R-120 golden 0.186.0 proof | 0.116.0 | DOWN |
**The customer register lists five customers** — `david`, `demo-felhom`, `demo-hp`, `peti-felhom`,
`sess-f`.
**THE FINDING THAT CHANGES THE COST: `peti-felhom` EXISTS AS A CUSTOMER BUT HAS NO HOST IN THE
REGISTER.** Consistent with the long-standing *"guest not on agent node"* stop in the Peti-return
runbook. So:
- **The two demo boxes are the entire measurable migrated population, and both are Tier 0 —
disposable, per decision D-d.** For them a "migration" is not required at all: they can be
**reinstalled** from a merged golden, which is cheaper and lower-risk than migrating in place, and
D-d explicitly permits it.
- **D-a's condition (1) — "it must land before any external install" — is currently SATISFIED.** No
external box appears in the hub's host register. **This is the cheapest this decision will ever be,
and the window is open now.**
**UNMEASURED, and reported as unmeasured rather than omitted:** Peti's box's actual disk layout. It
does not report to the hub, so the hub holds no `pct config` for it, and it is protected by D-d and
`target-selection.md` — so it was not contacted. **Whether a merged golden implies an in-place
migration for that box, and what that costs, is not established by this spike.** Nor is whether the
box is currently restorable at every point of such a migration. Both are inputs the operator has and
this session does not.
**Also unmeasured:** the per-box in-place migration procedure itself (move `<mp1>/felhom-data` onto
`mp0`, drop the `mp1` slot, verify) has not been executed even once on a throwaway guest. If an
in-place migration is ever needed, **that rehearsal is the first thing to do, and its "is the box
restorable at every point?" answer is currently unknown.**
---
## Ranked options, and a recommendation
**Ranked by what the measurements support, not by preference.**
1. **S1 (one volume, two directories) + B2 (a refusal threshold in the capture path), shipped as a
FRESH-INSTALL shape, with the demo boxes REINSTALLED rather than migrated.**
It is what D-a describes; it genuinely removes the ceiling rather than moving it; the four golden
assertions and the two vzdump guards are a bounded, greppable edit; the archive scope stays
complete; and B2 restores the per-app refusal that is the bulkhead's real value, reusing the alert
R-158 just wired. The migration cost for the measurable population is **zero**, because both boxes
are Tier 0 and reinstallable.
2. **S1 + B4 (warnings only).** Cheapest, and everything it needs is already shipped and proven live.
Rejected as the *recommendation* only because the failure it declines to prevent — a full Docker
data-root — is the worst one on the box, and it depends on a human acting inside the window.
3. **S3 (grow `mp1`, keep the split).** Zero structural risk, one number. It is the measured baseline
and D-a rejected it; recorded so the decision is compared against something.
4. **S2 (two mounts, one pool).** Not recommended at all. It keeps both assertions satisfied while
delivering none of the benefit, and converts a clean per-app refusal into a shared-pool exhaustion
neither `df` can see coming — strictly worse than today.
**Two things to do BEFORE the merge session, both cheap, both currently unmeasured:**
- **Run one restore-test of a PRE-MERGE archive into a merged-layout guest on a Tier-0 box.** M3
reasons it passes; nothing has executed it. This project's own record is that reasoning from source
about an unvalidated mechanism has been wrong four times.
- **Rehearse the in-place migration once on a throwaway guest**, and record whether the guest is
restorable at every point of it. Only needed if Peti's box turns out to require migrating rather
than reinstalling — which is the operator's information, not the hub's.
---
## THE QUESTION FOR THE OPERATOR
**Which shape, and what replaces the bulkhead?**
The recommendation is **S1 + B2**, fresh-install shape, demo boxes reinstalled. The two open inputs
only the operator has are:
1. **Does Peti's box need an in-place migration, or can it be reinstalled?** The hub cannot answer
this — the host is not in its register. It decides whether the migration rehearsal is required work
or optional insurance.
2. **Is B2's per-app refusal the right replacement for the bulkhead, or is B4 (warnings only)
acceptable?** B2 costs a small amount of code and a justified floor; B4 costs nothing and is
already live, but accepts that a full Docker data-root is reachable.
**STOP.** The merge is next session's work, and it is a supervised one.
@@ -0,0 +1,183 @@
# SPIKE R-165 Phase 0 — the two probes the merge spike left unmeasured
**Date:** 2026-08-03 · **Author:** Claude Code · **Status:** MEASURED — **no layout changed**
`SPIKE-r165-mp1-merge-2026-08-02.md` named two things as unmeasured and both are load-bearing. This
document measures them. **It changes nothing**: no golden rebuilt, no box reinstalled, no guest config
edited outside the throwaway probe guests, which are destroyed at the end.
**Host: `felhom-pve` (N100) — Tier 0.** demo-hp would have been the default per the 2026-07-25 ruling,
but `target-selection.md` records its `local-lvm` as an **over-subscribed thin pool backing live guest
9201** (~144 GiB allocated over ~54 GiB) where filling it corrupts every guest, and it holds **no
container template**. felhom-pve has the template and 258 GiB free at 29% pool usage. Both are Tier 0;
this picked the one where a probe cannot damage a live guest. **Neither DooPlex, `ep0`, nor the
colleague's box was contacted at any point.**
---
## P1 — does a PRE-merge archive restore cleanly into the merged world?
**Verdict: PASS.**
### Method
The spike reasoned from `mountParity` that it should pass and said plainly that this had never been
executed. It is now executed, on real hardware, with the real restore-test path — not a hand-assembled
restore.
- **Archive:** `local:backup/vzdump-lxc-9201-2026_07_28-17_43_05.tar.zst` from **demo-hp**, 1.68 GB.
Confirmed pre-merge by its own vzdump log rather than by assumption:
```
including mount point rootfs ('/') in backup
including mount point mp0 ('/var/lib/docker') in backup
including mount point mp1 ('/mnt/sys_drive') in backup
```
- **Command:** `felhom-agent --selftest=restore-test -archive <volid>` on demo-hp.
### Measured result
```
"pass": true,
"verified": "boot+running",
"mount_parity": "ok",
"duration_seconds": 84.2,
"mount_inventory": [
"mp0=/var/lib/docker (50G)",
"mp1=/mnt/sys_drive (20G)",
"mp8=/mnt/felhom-drives (throwaway for the archived bind)",
"mp9=/etc/felhom-bootstrap (throwaway for the archived bind)"
]
=== selftest=restore-test OK (scratch 990000 restored+booted+verified+torn-down in 1m24s) ===
```
`mountParity` was **not** relaxed, weakened or touched in any way.
### The limit of this result, stated rather than glossed
**It was run with the CURRENT agent (v0.119.0), because the merged agent does not exist yet** — Part 2
sits after this session's STOP. What it proves is that `mountParity` compares the **archive** against
**its own restore**, so a pre-merge archive recreates its own `mp0 + mp1` in the scratch guest and the
two agree. That comparison never consults the *host's* golden layout, which is why the merge cannot
invalidate it — **provided Part 2 honours its own constraint not to touch `mountParity` or the restore
path** (§5, §12 of the task). It is an 84-second command and **should be re-run once the merged agent
exists**, which is cheap and turns a sound inference into an observation.
---
## P2 — which S1 variant actually works on this platform?
**Verdict: all three probed variants are mechanically clean. They are separated by SCOPING, not by
mechanics — and the deciding fact was not in the task's table.**
### Method
A throwaway unprivileged LXC per variant (`nesting=1,keyctl=1`, rootfs 8 G + one 10 G volume,
`backup=1`), Docker installed from the same repo with the **same `daemon.json` the golden bakes**
(`containerd-snapshotter: false`, overlay2, log caps). Per variant, measured at first boot and after
**each of three reboots**:
1. both `/var/lib/docker` and `/mnt/sys_drive` present and **writable** (write → read back → delete,
a positive observable rather than an `ls`);
2. **one** filesystem — same source device **and** the same free-space figure for both paths;
3. `dockerd` active and `docker run hello-world` succeeding;
4. then once: the **real bootstrap propagation sequence** (`mount --rbind /mnt /mnt`,
`mount --make-rshared /mnt`, `docker run -v /mnt:/mnt:rslave`);
5. and: what a container mounting `/mnt:rslave` **actually sees** — the scoping check.
No pipe hides a non-zero; every check reports its own rc and the script aborts with `MEASURED-FAIL`.
### The variants
| | volume mounted at | then |
|---|---|---|
| **V-a** | `/var/lib/docker` | `/mnt/sys_drive` = bind of `/var/lib/docker/sys_drive` |
| **V-b** | `/mnt/sys_drive` | `/var/lib/docker` = bind of `/mnt/sys_drive/docker` |
| **V-c** | `/var/lib/felhom` *(neutral)* | **both** consumer paths are binds of subdirectories |
**V-c was not in the task's table.** It was probed *because* the measurements below showed V-a and V-b
each violate a different documented invariant, and V-c is the shape that violates neither. It is
offered as a **measured option for the operator at the STOP**, not adopted — the task is explicit that
a variant is not chosen mid-session.
### Measured results
| check | V-a | V-b | V-c |
|---|---|---|---|
| both paths present + writable | **yes** | **yes** | **yes** |
| ONE filesystem, ONE free-space figure | **yes** | **yes** | **yes** |
| dockerd active + `docker run` — initial | **yes** | **yes** | **yes** |
| dockerd active + `docker run` — reboots **1/2/3** | **3/3** | **3/3** | **3/3** |
| `/mnt` propagation `shared`; container sees `/mnt` | **yes** | **yes** | **yes** |
| both paths still real mountpoints (`findmnt` non-empty — the form the golden's assertions use) | **yes** | **yes** | **yes** |
| container `statfs("/")` reports the merged volume | **yes** (10218772 KiB) | **yes** | **yes** |
| **what a container mounting `/mnt:rslave` SEES** | `sys_drive` only — **8.0K** | `sys_drive` **+ `sys_drive/docker`** — **17.9M** | `sys_drive` only — **8.0K** |
| customer data inside Docker's data-root | **YES** | no | no |
**The mechanical worry was misplaced.** The task flagged V-b's ordering risk — `/var/lib/docker` must
be bound before dockerd starts. An `/etc/fstab` bind is ordered by `local-fs.target`, which precedes
`basic.target` and therefore `docker.service`, and it held **3 reboots out of 3**. Ordering is not what
separates these variants.
### What actually separates them
**V-b breaks a documented scoping invariant, and the measurement is the proof.** The controller
container is started with `-v /mnt:/mnt:rslave`, and the bootstrap script's own comment states the
scope it relies on:
> *"scoped to /mnt, which (Model A) holds only Felhom's felhom-data-namespace mounts, never the
> customer's other on-drive data"*
Under V-b the container sees `/mnt/sys_drive/docker`**Docker's entire data-root**, 17.9 MB on an
empty probe box and growing with every image and every app volume. That sentence becomes false. The
controller already holds the Docker socket, so this is **not a capability escalation** — but it puts
Docker's internal tree inside the one path the controller's own scanners, the FileBrowser surface and
the data-migration engine (which works *"in-process over the controller's `/mnt:/mnt:rslave` RW
mount"*) treat as Felhom-only.
**V-a breaks the other one:** customer backups live inside Docker's data-root, so `du` on the data-root
stops meaning what it says, and the ordinary operator reflex for a sick Docker — clear `/var/lib/docker`
— destroys every local recovery unit on the box.
**V-c breaks neither**, at the cost of one new mount path and two fstab lines instead of one.
---
## P3 — the golden's four assertions
**Not yet run — it belongs to Part 1, which is after the STOP.** Recorded here so it is not lost:
`build-golden.sh` fails closed on the split in **four** places (`:126`, `:130` separate-mount asserts;
`:315`, `:319` vzdump-exclusion guards), and each retargeted guard must be shown to **abort** against a
deliberately wrong shape before the golden is trusted.
**One measurement already de-risks it:** under all three variants both `/var/lib/docker` and
`/mnt/sys_drive` remain **real mountpoints**, so `findmnt -no SOURCE,FSTYPE <path> | grep -q .` — the
exact form the two existing assertions use — still returns non-empty. The assertions can be
**retargeted with a changed message and an added guard for the single volume**, rather than rewritten
from scratch. They must not be deleted.
---
## Teardown
| layer | action | evidence |
|---|---|---|
| **the machine** | probe guests `9401`, `9402`, `9403` destroyed; P1's scratch `990000` was torn down by the restore-test itself | `pct list` shows only `9201` |
| **the host** | `local-lvm` **112398205 → 107204406 KiB** used (30.73% → 29.31%) — **5.19 GB actually returned**, not merely deallocated | `pvesm status` before/after |
| **the hub** | **none created, and verified rather than assumed.** No probe claimed a box, minted a customer or registered an appliance — none ever ran a controller. The registers hold the same **5 customers** (`david`, `demo-felhom`, `demo-hp`, `peti-felhom`, `sess-f`) and same **4 hosts** (`demo-felhom-8363b5`, `demo-hp-bb76ea`, `drill-r50-0a4f9a`, `sess-f-2670b5`) as before Phase 0 | hub `/` + `/hosts` |
Scratch scripts and logs removed from `felhom-pve` (`/root/p2-probe.sh`, `/tmp/p2-*.log`,
`/tmp/p1-restoretest.log`). `pct list` on both demo hosts shows only their own `9201`.
---
## What this changes about the merge
1. **P1 removes the restore risk from the decision.** A pre-merge archive restores clean with parity
ok; it should be re-confirmed against the merged agent, which is one command.
2. **The variant question is not "will it boot" — it is "which invariant do we break".** Both named
variants work perfectly and each violates one documented guarantee. That is the operator's call and
is the subject of this session's STOP.
3. **V-c exists and is measured.** It costs one extra mount path and one extra fstab line, and is the
only probed shape that keeps both guarantees.
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+12 -2
View File
@@ -116,13 +116,23 @@ root password vaulted in the hub**, `host_recovery` row `demo-hp-bb76ea` (set at
Retrieval (operator-side, and **shred the copy** — that DB holds every host's secret):
> **WAL-AWARE SINCE HUB v0.88.0 — copying `hub.db` ALONE is no longer safe.** The hub runs SQLite in
> **WAL** mode (R-172), so a committed transaction may still live in `hub.db-wal` and not yet be in
> the main file. A bare `cat /data/hub.db` therefore yields a copy that is **valid but stale** — it
> opens cleanly and silently lacks the most recent writes, which is the worst failure shape for a
> credential lookup. Copy the `-wal` beside it and let SQLite replay it on open.
```bash
sudo kubectl -n felhom-system exec <hub-pod> -- cat /data/hub.db > /tmp/x.db
sudo kubectl -n felhom-system exec <hub-pod> -- cat /data/hub.db > /tmp/x.db
sudo kubectl -n felhom-system exec <hub-pod> -- cat /data/hub.db-wal > /tmp/x.db-wal 2>/dev/null || true
python3 -c "import sqlite3;print(sqlite3.connect('/tmp/x.db').execute(
\"SELECT secret FROM host_recovery WHERE host_id='demo-hp-bb76ea'\").fetchone()[0])"
shred -u /tmp/x.db
shred -u /tmp/x.db /tmp/x.db-wal
```
The `|| true` is deliberate: an absent `-wal` is legitimate (a freshly checkpointed database), and
must not fail the retrieval. **Shred both files** — the WAL holds the same secrets as the DB.
Then `sshpass -e ssh root@demo-hp` (sshpass is on DooPlex, not on the nodes).
**This is the lockout filed as R-61**: the ISO mints a throwaway root password per build and discards
@@ -3,7 +3,8 @@
**Class:** supervised operational run. **No repo version bump** — the only commits are this record
and the capacity note. **Nothing was deleted.**
**Host:** `ep0` / `felhom-hetzner`, `167.233.158.164`, Hetzner CX23, Nuremberg.
**Host:** `ep0` / `felhom-hetzner`, `167.233.158.164`, Hetzner **CX33 (4 vCPU / 8 GB RAM)**, Nuremberg.
> **Rescaled 2026-08-03** from the CX23 (2 vCPU / 3.8 GB) this runbook was written against. **The 40 GB local disk did NOT change** — this was a CPU/RAM resize — so every disk figure below still stands. The 4 GiB swapfile added on 2026-07-27 survived the resize.
**Datastore moved:** `felhom-offsite`, `/srv/pbs-felhom`**`/mnt/pbs-datastore`** (name unchanged).
**Window:** 06:58 → 07:19 UTC. PBS down 07:00 → 07:17 UTC.
@@ -229,7 +229,9 @@ as the hub 400ing an unknown event type. `verify-new` verifies each snapshot as
`keep-last 2` that covers essentially the whole datastore and turns a dead check live, for a few
minutes of ep0 CPU per weekly backup.
> Watch item: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup
> Watch item (**superseded 2026-08-03**: ep0 is now a **CX33, 8 GB RAM**, and it HAS a 4 GiB swapfile
> which survived the resize — so the pressure below is much reduced, though the shape of the concern
> stands). As written: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup
> window. Today's full forced verify completed fine (~250 MiB/s, 0 errors), but see
> `RUNBOOK-ep0-datastore-volume-2026-07-27.md` for the rsync OOM on this same box.
@@ -307,7 +309,7 @@ Untouched. Rollback remains a two-line `datastore.cfg` revert. Volume: 98 G, 13
watching: it is the only thing that reclaims chunks, and nothing has ever exercised it here.
4. **Hub PBS-DR gauge granularity** — 0.1 GB steps mean routine incremental backups are invisible to
it. Not a fault, but it cannot be used as write-proof evidence for small deltas.
5. **ep0 has no swap** (3.7 GB CX23) — see the volume runbook's OOM.
5. ~~**ep0 has no swap** (3.7 GB CX23)~~**corrected 2026-08-03: ep0 is a CX33 with 8 GB RAM and an active 4 GiB swapfile.** See the volume runbook's OOM for the original incident.
## 11. Observations
@@ -431,7 +431,7 @@ label. Filed under E-2.
| 2 | **Assignment in the storage wizard** — suggestion by attribute, refusal of the absurd (a 32 GB FAT thumb drive), never a decision by transport or `removable` (§1.2 shows both fail on the reference hardware). |
| 3 | **Unassigned drives do nothing automatically** — §2's rule, enforced in code. A drive must never acquire a role by appearing. |
| 4 | **Stickiness** — an assigned target must not move because a new drive appeared, and must never silently retarget when absent. |
| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. |
| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. **ANNOTATION 2026-08-03 (R-185) — this happened, in the half nobody looked at.** The installer's CREATE arm did issue the grant, exactly as this item asked. Its **reuse** arm — *"the target already exists, leave it as it is"* — returned without granting, so a box whose target pre-dated the install (i.e. one moved by THIS runbook) ended up pointing `local_backup_target` at a storage its own token could not read. **CORRECTION 2026-08-03, same day, measured on the box: it DID surface as a 403, exactly as this item predicted — the earlier annotation here said otherwise and was wrong.** demo-felhom's local-api backup jobs 403'd **six times** between 09:24 and 17:34 CEST: `POST /nodes/demo-felhom/vzdump -> HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.Allocate)`. The hub raised `whole_guest_backup_failed` at the first one (*"retrying with backoff"*) and edge-triggering correctly suppressed the rest, so the operator was told once. It ALSO surfaced as the agent's **read** returning `{"data":[]}` while root saw three archives — so the tier was silently never restore-tested. Both demo boxes carried it. Closed by installer **1.24.0** (the reuse arm grants too, with a gate asserting every arm that resolves the target also grants on it) and agent **v0.123.0** (the box now asks whether it may read each tier, because an empty listing cannot distinguish forbidden from newborn). |
| 6 | **Absent-target policy** per §6: decide fallback-vs-fail, and if fallback, alarm that protection is degraded rather than reporting a healthy tier. |
| 7 | **Retention and space accounting** on a drive the customer also uses — today `keep-last=3` competes with customer data with no reservation and no ceiling. |
| 8 | The honest **single-drive label**. |
+3 -2
View File
@@ -5,7 +5,8 @@
> firewall, and the hub-driven `felhom-peersync` reconcile surface. Re-running it on a fresh VM
> re-creates the endpoint from nothing (that is the DR story, step 8).
>
> **Validated:** 2026-07-03 on the dev/test endpoint `felhom-hetzner` (Hetzner CX23, Debian 13,
> **Validated:** 2026-07-03 on the dev/test endpoint `felhom-hetzner` (Hetzner CX23 **at the time — rescaled
> to a CX33, 4 vCPU / 8 GB RAM, on 2026-08-03; the 40 GB local disk is unchanged**, Debian 13,
> `167.233.158.164` / `2a01:4f8:1c16:7aa1::1`) with hub v0.32.0. The production endpoint is a
> later re-run of this runbook on a production VM.
>
@@ -31,7 +32,7 @@ Parameters used throughout (adjust for a new endpoint):
points at nothing (live-run finding). Home-resolver propagation can lag public DNS by
minutes — a client-side `wg-quick up` that fails to resolve right after record creation
just needs a retry.
- [ ] Sanity: `ssh root@167.233.158.164 hostname``felhom-hetzner` (the throwaway CX23), not
- [ ] Sanity: `ssh root@167.233.158.164 hostname``felhom-hetzner` (the throwaway box, **CX33 since 2026-08-03**), not
any production box.
## 1. Base (on the box, as root)
+19 -12
View File
@@ -29,20 +29,23 @@ prohibition as covering the act it names and nothing more.
|---|---|---|
| **0 — disposable. Reach here first.** | Exists to be broken; reinstalling is a routine afternoon, not an incident. **A drill that needs a victim uses one of these.** | `demo-hp` (t740), `demo-felhom` (N100) |
| **1 — create and destroy freely** | Throwaway VMs, guests, scratch customers — **hosted on a Tier 0 machine** | drill VMs, scratch guests |
| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster** — and, by D-d, **nothing else** |
| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster**, **`ep0`** (operator ruling 2026-08-03) — and nothing else |
**DooPlex is Tier 2 because it *is* the recovery chain** — hub, Gitea, registry, PBS, k3s + Longhorn.
Everything else rebuilds from it; it rebuilds from nothing. A bad moment in a DR drill there costs the
thing under test, the source of truth for it, and the backups, at once.
**`ep0` + the Hetzner Storage Boxes were Tier 2 until 2026-08-02 and are no longer** — D-d's protected
list names two machines and ep0 is not one of them. **That does not make them scratch, and the
difference is an act, not a tier** (see the rule above the table): ep0 holds the **PBS-DR datastore and
the restic copy of a real customer's data**, which is the only off-premises copy that exists, so
*deleting datastores, prune jobs, tunnel config or nftables rules* remains forbidden by what it would
destroy rather than by what tier it sits in. Reads are fine; it is still never a drill target.
**Flagged for the operator: D-d did not name ep0 either way.** Confirm it explicitly — this page has
read it the narrow way (not protected, but not wipeable) rather than assume the broad one.
**`ep0` is Tier 2 — PROTECTED. Operator ruling, 2026-08-03.** D-d named two protected machines and did
not name ep0 either way, so this page carried the question in writing for two days and read it the
narrow way meanwhile (not protected, but not wipeable). The ruling settles it and **extends D-d's
protected list to three machines**: DooPlex, Peti's cluster, ep0.
The reason it was never really in doubt: ep0 holds the **PBS-DR datastore and the restic copy of a
real customer's data**, which is the only off-premises copy that exists. So *deleting datastores,
prune jobs, tunnel config or nftables rules* was already forbidden by what it would destroy; the
ruling makes the classification say so plainly instead of leaving each session to re-derive it.
**Reads are fine** — including the ordinary off-site read a restore-test performs (R-86) — and it is
never a drill target. The Hetzner Storage Boxes ride the same reasoning.
**Standing ruling, 2026-07-25 (`operations/nodes.md`):** drill and build VMs live on the **t740** — not
felhom-pve, and **moved off DooPlex**. This page exists because that ruling sat where no session reads.
@@ -95,11 +98,15 @@ still shares a device with its guest, so a drive failure is **offsite-only recov
migrated, parked until the tester reinstalls (`PETI` in `backlog/OPEN-ITEMS.md`). Currently DOWN, no
enrolled host. No access route from DooPlex, and nothing here needs one.
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **not protected by D-d; not scratch either**
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **Tier 2, PROTECTED** (operator ruling 2026-08-03)
Reads are fine. It is the **offsite of last resort** (PBS-DR datastore, WireGuard hub, operator OOB
path) and RAM-constrained (3.8 GB, R-90) so a large restore can OOM it. Do not delete datastores, prune
jobs, tunnel config or nftables rules; never a drill target. The Storage Boxes hold the restic copy —
path) and — until 2026-08-03 — RAM-constrained (3.8 GB, R-90); it is now a **CX33 with 8 GB RAM and a
4 GiB swapfile**, which is what closed R-90. A very large restore is still worth watching — the 8 GB
is comfortable, not unbounded, and the OOM that started R-90 was a 14.46 GB restore read against
3.8 GB. Do not delete datastores, prune jobs, tunnel config or nftables rules; never a drill target.
**The ordinary off-site READ a restore-test performs is permitted and unchanged by the ruling**
(R-86): the classification forbids destruction, not use. The Storage Boxes hold the restic copy —
customer documents and photos, on a credential that can still delete (R-95).
**Access: `ssh root@167.233.158.164` from DooPlex** — *not* `felhom-pve → 10.77.0.1`, the route that
produced a false "unreachable" verdict (standing rule 2).
+8 -4
View File
@@ -178,7 +178,7 @@ turns a true alarm into one the operator dismisses.
### A comment asserting an invariant needs a test pinning it, or it is a wish
**Six instances in this project have shipped guarantees the code did not provide** — each survived
**Seven instances in this project have shipped guarantees the code did not provide** — each survived
review because the comment read as settled:
| # | Comment | What it claimed | What the code did |
@@ -189,10 +189,14 @@ review because the comment read as settled:
| 4 | `classifyRunStates` I1 | *"StateStopped means deliberately stopped by the user"* | quiesce stops stacks the same way — a failed restart was silent (F-CRIT-1) |
| 5 | `inflight.go` | *"a caller that cannot acquire DEFERS"* | the backup caller recorded a failure and paged the operator (F-A1) |
| 6 | `quiesce.go` | the agent's 409 *prevents* "a spurious failure" | on the start path it produced one (F-A1) |
| 7 | `recovery_unit.go` B2 refusal (R-181) | *"the previous unit is untouched and NOTHING was deleted"* | *nothing deleted* held; **untouched was measured false** — the floor was checked ONLY in `captureAllRecoveryUnits`, while the two dump legs wrote the bulk into the same tree first and unguarded, so a 182,272 B tar became 2,147,666,432 B under a manifest that had not moved |
Two of these (4 and 5/6) were found by Campaign 8 **on live hardware**, not by review or unit tests
#4 had a green, red-proofed test suite over a production path that was broken two independent
ways. So:
Three of these (4, 5/6 and 7) were found **on live hardware**, not by review or unit tests#4 had a
green, red-proofed test suite over a production path that was broken two independent ways, and #7
survived a full green suite plus three of its own red-proofs, because every one of them asserted the
mechanism inside `captureAllRecoveryUnits` and none asserted the **consequence** across the whole
backup run. The test that would have caught it is the one #7's fix ships: fingerprint the tree before
and after, and compare. So:
- If a comment states an invariant, **name the test that pins it**, or write one.
- If an invariant has a stated dependency (*"if either invariant changes, revisit this"*), that is
@@ -139,7 +139,7 @@ is not reachable, for two reasons that are each **already-recorded deliberate po
`endpoint_id` only; per-endpoint allocation is an explicitly deferred arc (`hub/README.md:260`).
So the only two configurations are: *DR tier on* → the campaign's PBS traffic lands on **ep0**, which is
Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90) and fenced by §3 — or *DR tier off*
Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90**note added 2026-08-03: ep0 has since been rescaled to a CX33 with 8 GB RAM; this journal records what was true when it was written and is deliberately not revised**) and fenced by §3 — or *DR tier off*
no Tier 3 at all. **Chosen: DR tier OFF, offsite OFF**, which is the only option §3 permits.
Consequence, stated plainly rather than discovered later: the campaign touches **neither ep0 nor the
+261
View File
@@ -1,3 +1,264 @@
## v0.91.1 — observation may only WIDEN a tier's window, never tighten it (2026-08-03, R-86 Part 2)
**Found by checking v0.91.0 against the live box before trusting it, not by review.** demo-felhom's
offsite tier holds two retained snapshots — `2026-07-27T19:55:41Z` and `2026-07-28T04:49:43Z`
**8 h 54 m apart**, because one is a healing artefact and the other a real weekly run. The mean-gap
estimator therefore reads a **weekly** tier as nine-hourly: ×4 gives 36 h, the 7-day floor lifts it to
168 h, and a weekly tier proved weekly reaches ~8.25 days of proof age. **The false alarm this whole
task exists to prevent would have returned within a week, on the box it had just shipped to.**
`restoreProvenWindow` now takes `max(observed, declared)`. Observation refines a tier's rhythm
**upward** and is ignored downward, which is right on its own terms and not merely cautious: a gap
SHORTER than the declared rhythm is routine and means nothing — a retry, a manual run, a heal, a
catch-up after an outage — while a gap LONGER than it is real information, saying this tier genuinely
receives archives less often than the model assumes and its window must widen or it alarms.
**The cost, stated rather than hidden:** a tier that truly runs faster than its declared rhythm gets a
wider window than it strictly needs, i.e. a slower `restore_test_stale` signal. That is the right
direction for a signal whose message is *"unverified"*. *"Broken now"* is `restore_test_failed`, which
is immediate and untouched.
Three live-derived cases added to `TestRestoreProvenWindow_Contract`, including the exact 8 h 54 m
gap measured on the box; red-proved by restoring the tighten-too branch
(`window(pbs, observed=8h54m) = 168h, want 288h`).
## v0.91.0 — a tier's staleness window learns the tier's own rhythm (2026-08-03, R-86 Part 2)
**This ships WITH the agent's v0.121.0, not after it.** The agent now proves a tier once per ARCHIVE
GENERATION rather than on a 24h timer, so a tier backed up weekly is proved weekly — correctly, and
in perfect health. `restoreProvenStaleAfter` was a flat 7 days, and its own comment derived that
number from the cadence R-86 removes:
> *"the restore-test cadence is 24h and rotation is oldest-first across two tiers, so each tier is
> proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive missed opportunities."*
A weekly tier proved weekly reaches a proof age of **exactly one interval** just before its next
proof — 168h against a 168h window, false by a hair — so it did not merely sit near that line, it sat
ON it, and any ordinary delay (a late backup, one deferral behind a running backup) tipped it into a
nightly alarm about a system that is working. Shipping the agent alone would have converted the
improvement into a false alarm.
**The window is now per tier**, `restoreProvenWindow(tier, observed, ok)`:
- **the tier's own archive interval**, observed from the host-reports the hub already holds —
`pbs_snapshots[]` plus successful `backups[]` attributed by TARGET TYPE (the slice-A.4 rule: a
PBS-targeted vzdump appears in both arrays, and classifying by array membership would attribute an
offsite archive to the host tier);
- **× 4 generations** — the settle generation plus ~3 missed opportunities, deliberately the same
tolerance the flat constant expressed. The change is to the RHYTHM, not to the patience;
- **floored at 7 days** — the old constant kept as a floor, so no tier is judged more tightly than
before;
- **capped at 12 days** — strictly inside the 2-week offsite retention with two days to spare, so a
tier is never called stale against an archive PBS has already pruned;
- **falling back to the DECLARED rhythm** when a box's history is too short to observe one:
`backupStaleAfter` (26h) for the host tier and `offsiteBackupStaleAfter` (8d) for the offsite tier
— the very thresholds the backup-freshness checker already judges those tiers against. A fresh box
with one snapshot has no observable interval, and falling back to the FLOOR there would recreate
the false alarm on exactly the tier this task is about.
**Kept, because both were earned:** absence is UNKNOWN until an anchored window has passed (R-81's
structure, untouched), and the stale signal stays edge-triggered. `restore_test_failed` and
`restore_test_stale` remain DISTINCT — one says your recovery is broken, the other says it is
unverified, and the second is the one that quietly becomes the first.
**Every reason string now states the window it was judged against** — R-100's corollary: when a
verdict changes what it counts from, the alarm text has to change with it, or an operator reads
"limit 168h" under a tier actually judged at 288h and dismisses a true alarm.
**The window READ is unchanged in cost** (14 days), which is both enough to find proof inside the
widest window and enough to see two generations of a weekly tier.
## v0.90.1 — the digest's per-app lines stop repeating the filesystem figures (2026-08-03, R-182)
**Backfilled 2026-08-03 (R-86 session).** This version was built, deployed and recorded in `REPORT.md`
and the R-182 row, but never given a CHANGELOG entry — and `REPORT.md` is overwritten every session,
so the per-repo history under-reported what was actually running. The deployed image has been
`felhom-hub:0.90.1` since `f21e7ca`.
Found by reading the first REAL digest, not by design: every app row ended with the same usage clause
the mail already prints once on its own Filesystem line. On a two-app box that is untidy; down a list
of a dozen it is the same forty characters twelve times, pushing the part that DIFFERS off a phone
screen at 07:00 — the only moment that mail has to work.
The reserve's refusal message is authored for a single-app alert where naming the filesystem is
right, so the message is unchanged and the DIGEST trims the duplicate when rendering.
`trimRepeatedUsage` removes ONLY an exact "— <target path>:" suffix, so an unrelated reason is
untouched and a reason that is nothing but the usage clause is left alone rather than emptied.
Also inverted the operator half of `TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer`: it
required the operator to be e-mailed a per-app capture failure, which was correct when that event was
the only signal and is wrong now that it is the RECORD and the digest is the notification. The
customer-safety claim is unchanged — R-158's guarantee MOVED, it did not weaken.
## v0.90.0 — a dropped notification leaves a trace, and the backup digest arrives (2026-08-03, R-182)
**The smallest change on the board with the largest effect on trust: `processOperator`'s cooldown no
longer returns bare.** It used to drop the event *before* `LogNotification`, so a suppressed operator
alert and an event that never happened were indistinguishable — from the operator's side **and from
the hub's own records**. Measured 2026-08-03: nine `recovery_unit_capture_failed` events arrived, two
were mailed, and **seven left no row anywhere**. That is why the defect took a day to get the right
way round: there was nothing to read.
A suppressed operator event now writes a `suppressed` row carrying the message and **the key that
suppressed it**, so the collision is readable without reading code. This applies to **every** operator
event, not only the one that exposed it. It deliberately does **not** change the cooldown's duration
or semantics — it makes the drop visible, not absent.
**`backup_run_failures` — the per-run digest.** One operator mail at the end of a backup run listing
every app that failed, its leg and its reason, with the counts and the target filesystem's free
space. Added to `allowedEventTypes` **and** to `operatorOnlyEvents` — allowlisting alone does not make
an event operator-only, and `FormatCustomerEmail` falls back to the raw English message rather than
blocking. A test demonstrates a customer with the type in their enabled list receiving nothing.
**`recordOnlyEvents` — a third routing class.** Types that are STORED and RECORDED but never mailed.
`recovery_unit_capture_failed` moves here: it is the durable per-failure record, and the digest is the
notification. Deliberately a register rather than downgrading the severity to `info`, which would have
had the same routing effect while relabelling a genuine failure as informational in the events table,
the operator UI and every historical query.
**`cooldownRunSuffix` — the run discriminator.** A sibling of `cooldownTierSuffix` rather than a
branch inside it, so `tier` keeps byte-identical semantics and R-97a's tests are untouched. It makes
the cooldown effectively inert for the digest, **which is the intent**: a digest is already
rate-limited by construction, one per run and only when something failed, so there is nothing for a
timer to collapse — while the periodic refresh sweep sends **no** `run_id` and therefore stays under
the ordinary hourly cooldown.
**The e-mail is rendered as a list, not a JSON blob** — the one operator mail with a variable-length
payload, and a dozen apps on one line is unreadable on a phone at 07:00. An absent space reading
renders as *unavailable*, never as zeros.
## v0.89.0 — the two halves of decision D-c (2026-08-02, R-167 · R-158)
**Decision D-c routes two new signals to two different audiences, and the hub is where that routing
is enforced.** A customer can free space, delete files or add a drive, so a FILL WARNING is theirs. A
customer can do nothing about a per-app backup capture failure, so it is not.
**New operator-only event type `recovery_unit_capture_failed`** (controller v0.191.0, R-158). Added to
`allowedEventTypes` **and** to `notify.operatorOnlyEvents`. Deliberately NOT a reuse of
`backup_failed`, which carries a `customerMessages` entry and sits in the controller's
`DefaultEnabledEvents` — reusing it would email the customer, in Hungarian, about a failure they
cannot act on. R-158's original proposal named `backup_failed`; D-c overrides it.
**`disk_warning` / `disk_critical` lose their generic `customerMessages` entries.** These two types
were allowlisted here, carried Hungarian copy, sat in the controller's default enabled events and had
a UI checkbox — and **nothing in any repo emitted them**, a complete customer pipeline with no
producer. Controller v0.191.0 becomes that producer, and it sends a **dynamic** Hungarian message
naming the drive and its free space. A static entry would be actively harmful: `FormatCustomerEmail`
PREFERS the entry over the message, so keeping one would discard the label and the byte figures and
leave the customer with *"A lemezterület 90% felett van"* — a warning with nothing to act on. Same
reason `offbox_enlarge_blocked` and `disk_health_degraded` have no entry. The deletion itself is
pinned by a test.
**New `notify.IsOperatorOnly`** — a read-only accessor so the `api` package can pin BOTH registers of
a new event type in ONE test. Allowlisted-but-not-operator-only is invisible when the two are checked
separately, and it is the defect v0.78.0 actually shipped. The register stays unexported so nothing
can widen it at runtime.
**Tests:** 574 → **579**. The operator half is proven through the real dispatch path under the
*breaking* configuration — the customer has the event enabled and has an email address — because that
is the only configuration in which a missing `operatorOnlyEvents` entry is visible. Red-proof:
removing the entry shows the customer being emailed, and the `skipped/operator_only` log row is
asserted as a positive observable rather than inferred from an absent delivery.
## v0.88.0 — the WAL that never was (2026-08-02, R-172)
**The hub has never actually been in WAL mode.** `store.New` opened the database with
`?_journal_mode=WAL&_busy_timeout=5000`**mattn/go-sqlite3** syntax — while the driver is
**modernc.org/sqlite**, whose `applyQueryParams` reads only `_pragma`, `_time_format`,
`_time_integer_format`, `_txlock` and `_inttotime`. Everything else is **ignored without an error**.
So the hub ran in the default rollback-journal mode with `busy_timeout=0` for its entire life, while
its own source said otherwise — a configuration asserting an invariant the code did not provide.
**How it surfaced.** A false `HOST STALE` banner for `demo-felhom-8363b5` while the agent was up two
days and reconciling normally. In rollback-journal mode a reader excludes a writer, so rendering an
operator page can block a host report; the hub then returns **HTTP 500**, the agent logs
`hub: report failed; keeping current interval` and **waits its full 15-minute interval**, and
staleness fires at 30 minutes. **Two consecutive collisions = a false alarm + an operator e-mail.**
Measured: 13 `SQLITE_BUSY` collisions in one pod lifetime, and the alarm fired twice that day
(19:12:32 and 20:42:32 CEST) for a host that was never down.
**The observable that proved it:** a 128 MB `/data/hub.db` with **no `-wal`/`-shm` file beside it
while the database was open**. In WAL mode those files must exist.
**The fix is one DSN, and each parameter earns its place:**
```
?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate
```
- `journal_mode(WAL)` — readers and one writer proceed concurrently, so a page render can no longer
block a report. It is a property of the database FILE, so it persists once set.
- `busy_timeout(5000)` — writers still serialise; without a timeout SQLite returns `SQLITE_BUSY`
*immediately* rather than waiting.
- `_txlock=immediate` — **the one that is easy to miss, and WAL + busy_timeout alone would not cover
it.** `database/sql`'s `Begin()` is DEFERRED, so a transaction that reads then writes must upgrade
its lock, and a failed upgrade is `SQLITE_BUSY_SNAPSHOT`, which **`busy_timeout` does not retry**.
This store has 10+ `db.Begin()` sites and they are all write paths (customer delete/reset, wg,
appliance, pbsdr, telemetry, log bundles). Without this the fix would leave a known un-retryable
path open.
**Every test asserts what the DATABASE reports, never the DSN string** — a test on the string would
have passed happily for the entire life of the bug. Five tests: the runtime pragma values; the
`-wal`/`-shm` files existing beside an open DB (the production signature, pinned); a reader not
blocking a writer (the consequence, not the mechanism); concurrent writers waiting instead of
erroring; and racing read-then-write transactions. Plus `TestSQLiteDriverIgnoresMattnStyleParams`, a
guard on the ROOT CAUSE: it fails if someone "tidies" the pragmas back to the familiar mattn form,
and skips itself with instructions if a future driver starts honouring them.
**Red-proof:** restoring the shipped DSN reproduces the live failure exactly — `journal_mode = "delete"`,
the `-wal` absent, and `a write FAILED while a read was open: database is locked (5) (SQLITE_BUSY)`.
**Operational consequence, handled rather than discovered later:** a WAL database cannot be copied by
taking `hub.db` alone — a committed transaction may still be in `hub.db-wal`, so a bare `cat` yields
a copy that opens cleanly and **silently omits the newest writes**. That is the worst shape for a
credential lookup, and the break-glass retrieval in `documentation/operations/nodes.md` used exactly
that command. Both it and the `_recovery-inventory` note are now WAL-aware (copy the `-wal`, shred
both).
**Retries (options b and c in R-172) were NOT added.** With readers no longer blocking writers and
the upgrade path covered, a `SQLITE_BUSY` reaching an HTTP handler should now be rare enough to be a
real signal. If any appear after this, they mean something else and a retry would hide it. Revisit
only on evidence.
## v0.87.0 — the Setup tab stops claiming a host-install version it cannot know (2026-08-02)
**R-94, all three legs, closed by deletion rather than derivation.** The customer page's Setup
Command card read *"Day-0 host bootstrap for host-install **1.19.0**"*. The served script was
**1.22.0**, and had been since 14 July — nineteen days of an operator-facing number that was simply
wrong, with a version number's authority behind it.
**Why deriving it is not achievable honestly.** The Option-1 command downloads
`felhom-host-install.sh` from the website **at run time**, and the website git-syncs `main` every
thirty seconds (R-110). The hub therefore cannot know which version a given box will run — not at
build time, not at render time. Any literal there is a guess. The comment that guarded the old const
already half-admitted this ("Display-only… the served script is always current"). R-94(a) offered
*derive it, or delete it*; deleting removes the drift class permanently instead of automating it.
**What changed.**
- `internal/web/configs.go`: `const hostInstallVersion`, the `pageData.ScriptVersion` field and its
assignment are **gone**. A NOTE stands in their place recording why there is deliberately no
constant here, so the next person does not helpfully re-add one.
- `internal/web/templates/customer_unified.html`: the sentence now says the command always fetches
the **current** installer and renders no version at all.
- `internal/web/render_test.go`: the assertion `strings.Contains(html, hostInstallVersion)` compared
the constant to itself and **passed at any value** — demonstrated green with the const set to
`9.9.9` while the served script was 1.22.0. Deleted, not replaced: there is no longer a version to
assert. The `data-customer-id` and static-fallback assertions stay.
- `scripts/hostinstall_gates.py` gate 1 **inverts**: it used to require the hub const to EQUAL
`SCRIPT_VERSION`; it now asserts the hub carries **no host-install version literal at all**,
matched in six code shapes across every `.go`/`.html` under `hub/`. Comments are deliberately not
stripped — a `//` inside a URL string literal would truncate the scan and blind the gate — so the
patterns match declarations, fields, assignments and the template action, never prose.
- `scripts/felhom-host-install.sh`: **comment only**, `SCRIPT_VERSION` untouched. It claimed the gate
keeps the hub copy equal, an invariant that no longer exists; a comment asserting an invariant the
code does not provide is a wish.
**Red-proofs.** Restoring the const fails the rewritten gate 1 on three of its six shapes. The old
`render_test.go` assertion passes with the const at `9.9.9`.
**Live verification:** endpoint-level (no browser on DooPlex) — the customer Setup tab is fetched and
grepped for a version literal.
## v0.86.0 — Copy works without revealing, and every copy branch reports itself (2026-07-31)
**Found by the operator, in the way that matters: it cost a real login.** The v0.84.0 Console access
@@ -0,0 +1,40 @@
package api
import "testing"
// R-182 — the backup run digest needs the allowlist half, and NOT the customerMessages half.
//
// A new event type is a pair of register entries, and each half fails differently. For this type the
// pair is unusual and that is the point:
//
// - missing from `allowedEventTypes` → POST /event returns 400 and the digest VANISHES,
// which would rebuild the exact silence R-182 exists to end;
// - PRESENT in `customerMessages` → the customer would be e-mailed, in Hungarian, a
// list of which apps' backups failed and why — operator detail they can take no action on. So
// this type must be allowlisted and must NOT have a customer message.
//
// The customer-facing half of decision D-c is the FILL WARNING, which fires before this and is
// actionable (free space, delete files, add a drive). This is the operator's half.
//
// Operator-only routing itself is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a
// customerMessages entry — that assumption shipped in v0.78.0 and was wrong, because
// FormatCustomerEmail falls back to the raw message. It is pinned in
// `internal/notify/backup_run_digest_test.go`, which demonstrates a customer with the type in their
// enabled list receiving nothing.
func TestBackupRunDigestIsAllowlisted(t *testing.T) {
if !allowedEventTypes["backup_run_failures"] {
t.Fatal("backup_run_failures must be in allowedEventTypes, or POST /event 400s and the " +
"whole run digest is dropped at the door — the silence R-182 was filed against")
}
}
// The per-app event is the RECORD and must not be removed while the digest is the notification.
// Deleting it would make the digest the only trace, and a digest that fails to send would then take
// the record with it — the coupling R-182's fix exists to break.
func TestPerAppCaptureEventStaysAllowlisted(t *testing.T) {
if !allowedEventTypes["recovery_unit_capture_failed"] {
t.Fatal("recovery_unit_capture_failed was removed from allowedEventTypes — it is the " +
"durable per-failure RECORD, and the digest is only the notification; the operator " +
"register and every historical query depend on it")
}
}
+19
View File
@@ -1573,6 +1573,25 @@ var allowedEventTypes = map[string]bool{
"whole_guest_backup_failed": true,
"whole_guest_backup_recovered": true,
// R-158 / R-167 (controller v0.191.0, decision D-c): a per-app Tier-1 recovery-unit capture
// failed. Until then a `[WARN]` line in the controller reached no hub channel at all — the fifth
// instance in this project of a mechanism built and left disconnected.
//
// DELIBERATELY NOT `backup_failed`, for exactly the reason recorded above for the whole-guest
// pair: that type carries a customerMessages entry AND sits in the controller's
// DefaultEnabledEvents, so reusing it emails the CUSTOMER, in Hungarian, about a failure they
// cannot act on. R-158's original proposal named `backup_failed`; D-c routes this to the
// operator, and where the two disagree D-c wins.
//
// OPERATOR-ONLY IS ENFORCED BY `notify.operatorOnlyEvents` — see the paragraph above. This entry
// alone does NOT make it operator-only.
"recovery_unit_capture_failed": true,
// R-182. The per-run backup digest: one event at the end of a run, listing every app whose
// backup failed or was refused. Allowlisting it is NOT what keeps it away from customers —
// `notify.operatorOnlyEvents` is (see the comment there); both entries ship together and
// `backup_run_digest_event_test.go` pins the pair.
"backup_run_failures": true,
// Controller-pushed events
"controller_started": true,
"claim_lockout": true, // v0.50.0 — claim/reset code brute-force lockout tripped
@@ -0,0 +1,50 @@
package api
import (
"testing"
"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
)
// R-158 / R-167 (D-c) — the per-app Tier-1 recovery-unit capture failure alert.
//
// A new event type is a PAIR of registers, and each half fails differently:
// - missing from allowedEventTypes → POST /event returns 400 and the event VANISHES (R-97a);
// - missing from operatorOnlyEvents → it is delivered TO THE CUSTOMER, in raw English, about a
// failure they can take no action on.
//
// The second is the quiet one — delivery "works", so nothing looks broken. It is also the defect
// v0.78.0 actually shipped: adding a type to allowedEventTypes and ASSUMING that made it
// operator-only. `FormatCustomerEmail` treats a missing customerMessages entry as a fallback to the
// raw message, not a block, and the only customer gate is configuration. Both halves are pinned
// here, in the same test, because fixing one and not the other is the realistic mistake.
func TestRecoveryUnitCaptureFailedIsAllowlistedAndOperatorOnly(t *testing.T) {
const et = "recovery_unit_capture_failed"
if !allowedEventTypes[et] {
t.Fatalf("%s must be in allowedEventTypes, or POST /event 400s and a per-app Tier-1 backup "+
"failure reaches no hub channel at all — which is the R-158 gap, un-fixed", et)
}
if !notify.IsOperatorOnly(et) {
t.Fatalf("%s is allowlisted but NOT in notify.operatorOnlyEvents — the customer would be "+
"emailed about a recovery-unit capture failure they can take no action on. Adding a type "+
"to allowedEventTypes does NOT make it operator-only; that was the v0.78.0 defect "+
"(corrected in v0.79.0/R-97c) and this is the same mistake one event later", et)
}
}
// The customer's half of D-c must NOT be operator-only — a fill warning is precisely the alert a
// customer CAN act on (free space, delete files, add a drive). Pinned in the same file as its
// operator sibling so the routing decision is read as one thing, which is what D-c is.
func TestFillWarningReachesTheCustomer(t *testing.T) {
for _, et := range []string{"disk_warning", "disk_critical"} {
if !allowedEventTypes[et] {
t.Fatalf("%s must stay in allowedEventTypes — it is the customer fill warning the "+
"controller emits from v0.191.0 (R-167)", et)
}
if notify.IsOperatorOnly(et) {
t.Fatalf("%s is in operatorOnlyEvents — the customer would never be warned that their "+
"disk is filling, which is the whole customer half of decision D-c", et)
}
}
}
+195 -20
View File
@@ -32,15 +32,104 @@ import (
// is a NEW monitor written straight after the third, so it copies R-81's verdict structure rather
// than re-deriving it. A tier never proven on a newborn box is UNKNOWN, never FAILED.
// restoreProvenStaleAfter is how long a tier may go unproven before it is called stale.
// ── HOW LONG MAY A TIER GO UNPROVEN? (R-86 Part 2) ───────────────────────────────────────────
//
// Derivation, not a guess: the restore-test cadence is 24h and rotation is oldest-first across two
// tiers, so each tier is proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive
// missed opportunities before alarming — loud enough to matter, quiet enough not to fire on one
// skipped cycle (a deferral behind a long backup is normal, not a fault). It is also comfortably
// inside the 2-week offsite retention (operator ruling 2026-07-26), so a tier is never reported
// stale against an archive that is about to be pruned anyway.
const restoreProvenStaleAfter = 7 * 24 * time.Hour
// This was one flat constant, 7 days, and its comment derived that number like this:
//
// "the restore-test cadence is 24h and rotation is oldest-first across two tiers, so each tier is
// proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive missed opportunities."
//
// **That premise is exactly what R-86 removed.** The agent no longer tests on an interval at all: a
// tier is tested once per ARCHIVE GENERATION — when it holds a settled archive that has not been
// proven. A tier backed up weekly is therefore proved weekly, by design and in perfect health, and
// against a flat 7-day window it would sit on the line and alarm every night about a system that is
// working. Shipping the agent's half alone would have converted the improvement into a false alarm,
// which is why the two ship together.
//
// The window is now derived from **the tier's own backup rhythm**, which the hub can observe from
// the reports it already receives, and it keeps everything the constant had earned:
//
// - absence is UNKNOWN until an anchored window has passed (R-81's structure, untouched);
// - the signal stays edge-triggered;
// - it never exceeds the offsite retention, so a tier is never called stale against an archive
// that is about to be pruned;
// - and it is never TIGHTER than the 7 days that were already tolerated.
const (
// restoreProvenGenerations is how many archive generations may pass unproven before alarming.
// 4 = the settle lag's own generation plus ~3 missed opportunities — deliberately the same
// tolerance the flat constant expressed, so the change is to the RHYTHM, not to the patience.
restoreProvenGenerations = 4
// restoreProvenWindowFloor is the shortest window that may be applied to any tier. It is the
// old constant, kept as a FLOOR rather than deleted: a daily tier computes 4 days from its own
// rhythm, and tightening a live threshold is not what this task is for. A deferral behind a
// long backup is normal, not a fault.
restoreProvenWindowFloor = 7 * 24 * time.Hour
// restoreProvenWindowCap keeps the window strictly inside the 2-week offsite retention
// (operator ruling 2026-07-26) with two days to spare. Beyond it the hub would be judging a
// tier against an archive PBS has already pruned — an alarm nobody can act on, and the bound
// the old constant respected in its own way.
restoreProvenWindowCap = 12 * 24 * time.Hour
// restoreWindowRead is how far back the hub reads host-reports for this check: far enough to
// find proof anywhere inside the widest window, and to see at least two archive generations of
// a WEEKLY tier so its rhythm is observable at all.
restoreWindowRead = 2 * restoreProvenWindowFloor
)
// declaredArchiveInterval is the rhythm the hub ALREADY attributes to a tier — the same thresholds
// the backup-freshness checker judges it against (deadline.go / deadline_tiers.go). It is the
// fallback when a box's history is too short to observe a rhythm, and it is the right fallback
// precisely because it is not a second opinion: if these two checkers disagreed about how often a
// tier is expected to receive an archive, one of them would be alarming on the other's model.
//
// It is stated per RESTORE tier name ("local"/"pbs" — what the agent reports as source_tier), which
// is the same split the backup tiers use under different names ("host"/"offsite").
func declaredArchiveInterval(tier string) time.Duration {
if tier == "pbs" {
return offsiteBackupStaleAfter // 8 days: the weekly cadence plus a day of headroom
}
return backupStaleAfter // 26 hours: the daily cadence plus headroom
}
// restoreProvenWindow is how long THIS tier may go unproven, given its observed archive interval.
//
// observedOK=false means the box's retained history did not contain two archive generations for
// this tier, so the declared rhythm is used. That fallback matters most for exactly the tier this
// task is about: a fresh box with a weekly offsite tier has one snapshot and no observable
// interval, and falling back to the FLOOR there would recreate the false alarm.
//
// OBSERVATION MAY ONLY WIDEN, NEVER TIGHTEN — and this is not caution, it is a live measurement.
// On demo-felhom (2026-08-03) the offsite tier's two retained snapshots are `2026-07-27T19:55:41Z`
// and `2026-07-28T04:49:43Z`: **8 h 54 m apart**, because one is a healing artefact and the other a
// real weekly run. A mean-gap estimate therefore reads a WEEKLY tier as nine-hourly, ×4 gives 36 h,
// the floor lifts it to 7 days — and a weekly tier proved weekly reaches ~8.25 days of proof age, so
// the false alarm this whole task exists to prevent would have returned within a week, on the very
// box it shipped to.
//
// The asymmetry is right on its own terms too. A gap SHORTER than the declared rhythm is routine and
// means nothing — a retry, a manual run, a heal, a catch-up after an outage. A gap LONGER than the
// declared rhythm is real information: this tier genuinely receives archives less often than the
// model says, and its window must widen or it alarms. So observation refines the rhythm upward and
// is ignored downward. The cost is stated plainly: a tier that truly runs FASTER than its declared
// rhythm gets a wider window than it strictly needs, i.e. a slower stale signal. That is the right
// direction for a signal whose message is "unverified" — "broken NOW" is `restore_test_failed`, and
// that one is immediate and unaffected.
func restoreProvenWindow(tier string, observed time.Duration, observedOK bool) time.Duration {
interval := declaredArchiveInterval(tier)
if observedOK && observed > interval {
interval = observed
}
w := time.Duration(restoreProvenGenerations) * interval
if w < restoreProvenWindowFloor {
w = restoreProvenWindowFloor
}
if w > restoreProvenWindowCap {
w = restoreProvenWindowCap
}
return w
}
// Event types. Operator-tier only — see the dispatcher note in RestoreTestChecker.
const (
@@ -156,12 +245,13 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
return
}
rows, err := c.store.GetHostReportsSince(customerID, now.Add(-2*restoreProvenStaleAfter))
rows, err := c.store.GetHostReportsSince(customerID, now.Add(-restoreWindowRead))
if err != nil {
c.logger.Printf("[WARN] restore-test check: window read failed for %s: %v", customerID, err)
return
}
proven := lastProvenPerTier(rows)
intervals := observedArchiveIntervals(rows)
first, ferr := c.store.GetFirstHostReportAt(customerID)
if ferr != nil {
@@ -170,7 +260,9 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
}
for _, tier := range tiers {
v := assessRestoreProven(tier, proven[tier], first, now)
observed, observedOK := intervals[tier]
window := restoreProvenWindow(tier, observed, observedOK)
v := assessRestoreProven(tier, proven[tier], first, now, window)
key := customerID + "|" + tier
c.mu.Lock()
prev := c.staleStates[key]
@@ -192,37 +284,120 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
}
}
// assessRestoreProven is the per-tier verdict. PURE (now injected) so the policy is unit-tested —
// the property that made R-81 provable, kept deliberately.
// assessRestoreProven is the per-tier verdict. PURE (now and the window injected) so the policy is
// unit-tested — the property that made R-81 provable, kept deliberately.
//
// no proof, anchor NOT elapsed → UNKNOWN (newborn box; never an alarm)
// no proof, anchor elapsed → MISSED
// proof older than the window → MISSED
// otherwise → OK
func assessRestoreProven(tier string, provenAt, firstReportAt, now time.Time) backupAssessment {
//
// `window` is now the TIER'S OWN (R-86 Part 2) rather than one constant for every tier, and every
// reason string states the window it was judged against. That is R-100's corollary applied here:
// when a verdict changes what it counts from, the alarm text has to change with it, or an operator
// reads "limit 168h" under a tier that was actually judged at 288h and dismisses a true alarm.
func assessRestoreProven(tier string, provenAt, firstReportAt, now time.Time, window time.Duration) backupAssessment {
if provenAt.IsZero() {
if firstReportAt.IsZero() {
return backupAssessment{verdict: verdictMissed,
reason: fmt.Sprintf("%s tier: never restore-proven, and no first-contact anchor to defer against", tier)}
}
watched := now.Sub(firstReportAt)
if watched <= restoreProvenStaleAfter {
if watched <= window {
return backupAssessment{verdict: verdictUnknown,
reason: fmt.Sprintf("%s tier: not restore-proven yet, but only watching for %s (grace %s since first contact %s) — newborn, not a fault",
tier, watched.Round(time.Hour), restoreProvenStaleAfter, firstReportAt.Format(time.RFC3339))}
tier, watched.Round(time.Hour), window, firstReportAt.Format(time.RFC3339))}
}
return backupAssessment{verdict: verdictMissed,
reason: fmt.Sprintf("%s tier: NEVER successfully restore-proven in %s of watching (limit %s) — the tier is unverified, not known-broken",
tier, watched.Round(time.Hour), restoreProvenStaleAfter)}
reason: fmt.Sprintf("%s tier: NEVER successfully restore-proven in %s of watching (limit %s, this tier's own backup rhythm) — the tier is unverified, not known-broken",
tier, watched.Round(time.Hour), window)}
}
if age := now.Sub(provenAt); age > restoreProvenStaleAfter {
if age := now.Sub(provenAt); age > window {
return backupAssessment{verdict: verdictMissed,
reason: fmt.Sprintf("%s tier: last successful restore-test was %s ago (limit %s) — the tier is unverified, not known-broken",
tier, age.Round(time.Hour), restoreProvenStaleAfter)}
reason: fmt.Sprintf("%s tier: last successful restore-test was %s ago (limit %s, this tier's own backup rhythm) — the tier is unverified, not known-broken",
tier, age.Round(time.Hour), window)}
}
return backupAssessment{verdict: verdictOK}
}
// observedArchiveIntervals estimates how often each RESTORE tier actually receives an archive, from
// the host-reports the hub already holds. Keyed by restore-tier name ("local"/"pbs").
//
// Evidence is every distinct archive timestamp in the window: `pbs_snapshots[]` for the offsite
// tier (PBS enumerates its whole retention in each report, so one report usually settles the
// question) and successful `backups[]` records attributed by TARGET TYPE for both tiers — the
// slice-A.4 rule, because a PBS-targeted vzdump appears in BOTH arrays and classifying by array
// membership would attribute an offsite archive to the host tier.
//
// The estimate is the MEAN gap: (newest oldest) / (generations 1). It needs two generations;
// with fewer, ok=false and the caller falls back to the declared rhythm. It is deliberately crude,
// and can afford to be: restoreProvenWindow clamps the result between a 7-day floor and a 12-day
// cap, so the only discrimination this has to get right is "roughly daily" versus "several days or
// slower" — which is exactly the distinction that turns a healthy weekly tier into a false alarm.
func observedArchiveIntervals(rows []store.HostReportRow) map[string]time.Duration {
seen := map[string]map[int64]struct{}{ // tier → set of archive unix times
"local": {},
"pbs": {},
}
add := func(tier string, t time.Time) {
if t.IsZero() {
return
}
seen[tier][t.UTC().Unix()] = struct{}{}
}
for _, r := range rows {
var hr hostReportBackups
if json.Unmarshal([]byte(r.ReportJSON), &hr) != nil {
continue // one malformed retained report must not blind the scan
}
pbs := pbsTargetSet(hr)
for _, ps := range hr.PBSSnapshots {
if t, ok := parseBackupTime(ps.BackupTime); ok {
add("pbs", t)
}
}
for _, b := range hr.Backups {
if !b.Success {
continue
}
t, ok := parseBackupTime(b.StartedAt)
if !ok {
continue
}
if pbs[b.TargetID] {
add("pbs", t)
} else {
add("local", t)
}
}
}
out := map[string]time.Duration{}
for tier, set := range seen {
if len(set) < 2 {
continue // not observable — the caller uses the declared rhythm
}
var oldest, newest int64
first := true
for ts := range set {
if first || ts < oldest {
oldest = ts
}
if first || ts > newest {
newest = ts
}
first = false
}
span := time.Duration(newest-oldest) * time.Second
if span <= 0 {
continue
}
out[tier] = span / time.Duration(len(set)-1)
}
return out
}
// expectedRestoreTiers names the tiers this box actually HAS, so a box without an offsite tier is
// never reported stale for one. Same gate as Slice C's `expected`, and for the same reason: without
// it every box lacking a tier would alarm once the anchor elapsed — absence-is-not-failure,
+223 -9
View File
@@ -182,26 +182,33 @@ func TestRestoreTest_NewbornDoesNotAlarm(t *testing.T) {
}
}
// The boundary, pinned by name so a refactor has to delete an obviously-named contract.
// The boundary, pinned by name so a refactor has to delete an obviously-named contract. R-86 made
// the limit per-tier, so the anchor is now measured against THE TIER'S OWN window — here the local
// tier's, which clamps to the 7-day floor and so keeps this contract numerically identical to the
// one the flat constant expressed.
func TestRestoreTest_Contract_UnprovenIsUnknownUntilTheAnchorElapses(t *testing.T) {
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
window := restoreProvenWindow("local", 24*time.Hour, true)
if window != restoreProvenWindowFloor {
t.Fatalf("precondition: a daily local tier must clamp to the floor; got %s", window)
}
cases := []struct {
name string
watched time.Duration
wantMissed bool
}{
{"newborn, 1h", time.Hour, false},
{"just inside", restoreProvenStaleAfter - time.Minute, false},
{"exactly at the limit", restoreProvenStaleAfter, false},
{"just outside", restoreProvenStaleAfter + time.Minute, true},
{"just inside", window - time.Minute, false},
{"exactly at the limit", window, false},
{"just outside", window + time.Minute, true},
{"long past", 30 * 24 * time.Hour, true},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
got := assessRestoreProven("pbs", time.Time{}, now.Add(-c.watched), now)
got := assessRestoreProven("local", time.Time{}, now.Add(-c.watched), now, window)
if got.missed() != c.wantMissed {
t.Fatalf("CONTRACT VIOLATED: unproven for %s (limit %s) → missed=%v, want %v (reason %q)",
c.watched, restoreProvenStaleAfter, got.missed(), c.wantMissed, got.reason)
c.watched, window, got.missed(), c.wantMissed, got.reason)
}
if !c.wantMissed && got.verdict != verdictUnknown {
t.Fatalf("a deferred tier must be UNKNOWN (visible), not OK; got verdict=%d", got.verdict)
@@ -215,15 +222,17 @@ func TestRestoreTest_Contract_UnprovenIsUnknownUntilTheAnchorElapses(t *testing.
func TestRestoreTest_StaleIsSeparateFromFailure(t *testing.T) {
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
stale := assessRestoreProven("pbs", now.Add(-9*24*time.Hour), now.Add(-60*24*time.Hour), now)
// A DAILY tier judged on its own rhythm: the window clamps to the 7-day floor.
daily := restoreProvenWindow("local", 24*time.Hour, true)
stale := assessRestoreProven("local", now.Add(-9*24*time.Hour), now.Add(-60*24*time.Hour), now, daily)
if !stale.missed() {
t.Fatalf("a tier last proven 9 days ago (limit %s) must be stale; got %q", restoreProvenStaleAfter, stale.reason)
t.Fatalf("a daily tier last proven 9 days ago (limit %s) must be stale; got %q", daily, stale.reason)
}
// The wording must not read as "broken" — that is the other signal.
if !strings.Contains(stale.reason, "unverified, not known-broken") {
t.Fatalf("staleness must say UNVERIFIED, not broken — merging the two is the thing this avoids; got %q", stale.reason)
}
fresh := assessRestoreProven("pbs", now.Add(-2*24*time.Hour), now.Add(-60*24*time.Hour), now)
fresh := assessRestoreProven("local", now.Add(-2*24*time.Hour), now.Add(-60*24*time.Hour), now, daily)
if fresh.verdict != verdictOK {
t.Fatalf("a tier proven 2 days ago is fine; got verdict=%d reason=%q", fresh.verdict, fresh.reason)
}
@@ -283,3 +292,208 @@ func boolStr(b bool) string {
}
return "false"
}
// ── SCENARIO G — a healthy WEEKLY tier is never reported stale (R-86 Part 2) ─────────────────
//
// This is the test that pins the false alarm this change would otherwise have CREATED. The agent
// now proves a tier once per archive generation, so a weekly offsite tier is proved weekly — in
// perfect health. Against the old flat 7-day window it would sit on the line and alarm every night.
//
// COMPANION RED-PROOF (observed 2026-08-03): pin the window flat, as it was —
//
// - window := restoreProvenWindow(tier, observed, observedOK)
// - window := restoreProvenWindowFloor // the pre-R-86 flat 7 days
//
// → --- FAIL: TestRestoreTest_HealthyWeeklyTierIsNeverStale
//
// week 0: a weekly tier proved on its own archive must never be stale (proof age 172h0m0s,
// window 168h0m0s); verdict=2 reason="pbs tier: last successful restore-test was 172h0m0s ago
// (limit 168h0m0s, this tier's own backup rhythm) — the tier is unverified, not known-broken"
//
// Restored. The mutation is one line because the whole of Part 2 is one decision: whose rhythm.
//
// NOTE, because it is the finding this test nearly hid: the FIRST version of this fixture had NO
// jitter, and it PASSED under the mutation. A perfectly regular weekly tier's proof age reaches
// EXACTLY 168h just before the next proof, and `age > window` is false by a hair — a hollow test
// that would have shipped Part 1 and its false alarm together. The jitter below is what makes this
// a test, and it is also the truth about the old constant: a healthy weekly tier did not merely sit
// near the line, it sat ON it, so any ordinary delay tipped it over.
func TestRestoreTest_HealthyWeeklyTierIsNeverStale(t *testing.T) {
start := time.Date(2026, 6, 1, 3, 0, 0, 0, time.UTC)
firstContact := start.Add(-24 * time.Hour)
// The observable rhythm of a weekly tier, as the hub would compute it from the reports. No
// assertion about the window ITSELF here on purpose: that is the mechanism, and it is pinned in
// TestRestoreProvenWindow_Contract. What this test asserts is the CONSEQUENCE — does the alarm
// fire? — because R-97b proved a mechanism and shipped a broken consequence anyway.
weekly := restoreProvenWindow("pbs", 7*24*time.Hour, true)
// Walk several weeks of a HEALTHY tier, with the jitter a real one has: the backup does not land
// to the second, and a restore-test can be deferred one evaluation behind a running backup.
//
// The jitter is the point. A perfectly regular weekly tier's proof reaches an age of EXACTLY one
// interval (168h) just before the next proof, and against a flat 168h window `age > window` is
// false by a hair — so a regular fixture would pass against the very constant this task must
// change, and prove nothing. That is the brief's "sits exactly on that line": every real-world
// delay pushes it over, and the alarm is about a system that is working.
settle, evalLatency := 24*time.Hour, 6*time.Hour
archiveLate := []time.Duration{0, 4 * time.Hour, 2 * time.Hour, 6 * time.Hour, 0, 3 * time.Hour}
deferred := []time.Duration{0, 0, 6 * time.Hour, 0, 0, 6 * time.Hour} // one evaluation behind a backup
archiveAt := func(week int) time.Time {
return start.AddDate(0, 0, 7*week).Add(archiveLate[week])
}
provenAt := func(week int) time.Time {
return archiveAt(week).Add(settle + evalLatency + deferred[week])
}
var worst time.Duration
for week := 0; week+1 < len(archiveLate); week++ {
// The widest the proof's age ever gets: the instant before the NEXT week's proof lands.
now := provenAt(week + 1).Add(-time.Second)
age := now.Sub(provenAt(week))
if age > worst {
worst = age
}
v := assessRestoreProven("pbs", provenAt(week), firstContact, now, weekly)
if v.verdict != verdictOK {
t.Fatalf("week %d: a weekly tier proved on its own archive must never be stale (proof age %s, window %s); verdict=%d reason=%q",
week, age.Round(time.Hour), weekly, v.verdict, v.reason)
}
}
// The fixture must actually EXERCISE the boundary — a jitter-free walk would sit at exactly one
// interval and pass against a flat 7-day window, which is the hollow version of this test.
if worst <= restoreProvenWindowFloor {
t.Fatalf("this fixture never exceeds the old flat window (worst proof age %s) — it cannot detect the defect it exists for", worst)
}
}
// ...and a weekly tier that genuinely STOPS being proved must still alarm. A window that never
// fires is not a fix, it is a deletion.
func TestRestoreTest_WeeklyTierThatStopsBeingProvedStillAlarms(t *testing.T) {
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
weekly := restoreProvenWindow("pbs", 7*24*time.Hour, true)
v := assessRestoreProven("pbs", now.Add(-weekly-time.Hour), now.Add(-90*24*time.Hour), now, weekly)
if !v.missed() {
t.Fatalf("a weekly tier unproven for longer than its own window MUST alarm; got verdict=%d reason=%q", v.verdict, v.reason)
}
if !strings.Contains(v.reason, weekly.String()) {
t.Fatalf("the alarm must state the window it was judged against (R-100's corollary); got %q", v.reason)
}
}
// The window's own contract: derived from the tier's rhythm, floored, capped, and never dependent
// on an unobservable history for the tier that would suffer most from a wrong answer.
func TestRestoreProvenWindow_Contract(t *testing.T) {
cases := []struct {
name string
tier string
observed time.Duration
observedOK bool
want time.Duration
}{
{"daily local clamps to the floor", "local", 24 * time.Hour, true, restoreProvenWindowFloor},
{"weekly pbs widens", "pbs", 7 * 24 * time.Hour, true, restoreProvenWindowCap},
{"3-day tier sits between", "pbs", 72 * time.Hour, true, 12 * 24 * time.Hour},
{"unobservable local falls back to its declared rhythm", "local", 0, false, restoreProvenWindowFloor},
{"unobservable pbs falls back WIDE, not to the floor", "pbs", 0, false, restoreProvenWindowCap},
{"a nonsense zero interval is ignored", "pbs", 0, true, restoreProvenWindowCap},
// MEASURED ON THE LIVE BOX, and the reason observation may only WIDEN. demo-felhom's two
// retained PBS snapshots sit 8h54m apart (one is a healing artefact), so a mean-gap estimate
// reads a WEEKLY tier as nine-hourly. Taking that at face value gives 4x9h = 36h → the 7-day
// floor → and a weekly tier proved weekly (~8.25d of proof age) alarms within a week of this
// shipping, on the box it shipped to.
{"a short observed gap must NOT tighten a weekly tier", "pbs", 8*time.Hour + 54*time.Minute, true, restoreProvenWindowCap},
{"a short observed gap must not tighten the host tier either", "local", 30 * time.Minute, true, restoreProvenWindowFloor},
// ...but a tier that genuinely runs SLOWER than its declared rhythm still widens.
{"a genuinely slower tier widens", "local", 4 * 24 * time.Hour, true, restoreProvenWindowCap},
}
// The relationship Part 1 depends on: a weekly tier's window must be WIDER than a daily tier's,
// or proving weekly (which is now correct behaviour) alarms on itself.
if restoreProvenWindow("pbs", 7*24*time.Hour, true) <= restoreProvenWindow("local", 24*time.Hour, true) {
t.Fatal("a weekly tier must earn a wider window than a daily one — otherwise R-86's agent half alarms about itself")
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
got := restoreProvenWindow(c.tier, c.observed, c.observedOK)
if got != c.want {
t.Fatalf("window(%s, observed=%s ok=%v) = %s, want %s", c.tier, c.observed, c.observedOK, got, c.want)
}
if got < restoreProvenWindowFloor || got > restoreProvenWindowCap {
t.Fatalf("every window must stay inside [%s, %s]; got %s", restoreProvenWindowFloor, restoreProvenWindowCap, got)
}
})
}
}
// The rhythm must be OBSERVED from the reports, not assumed — including the slice-A.4 rule that a
// PBS-targeted vzdump appears in both arrays and must be attributed by TARGET TYPE.
func TestObservedArchiveIntervals_FromReports(t *testing.T) {
base := time.Date(2026, 7, 1, 2, 0, 0, 0, time.UTC)
mk := func(localAt []time.Time, pbsAt []time.Time) string {
type stg struct{ Name, Type, Content string }
type bk struct {
TargetID string `json:"target_id"`
Success bool `json:"success"`
StartedAt string `json:"started_at"`
}
type snap struct {
BackupTime string `json:"backup_time"`
}
payload := struct {
StorageTargets []struct {
Name string `json:"name"`
Type string `json:"type"`
Content string `json:"content"`
} `json:"storage_targets"`
Backups []bk `json:"backups"`
PBSSnapshots []snap `json:"pbs_snapshots"`
}{}
payload.StorageTargets = append(payload.StorageTargets, struct {
Name string `json:"name"`
Type string `json:"type"`
Content string `json:"content"`
}{"felhom-backup", "dir", "backup"}, struct {
Name string `json:"name"`
Type string `json:"type"`
Content string `json:"content"`
}{"felhom-pbs", "pbs", "backup"})
for _, at := range localAt {
payload.Backups = append(payload.Backups, bk{"felhom-backup", true, at.Format(time.RFC3339)})
}
for _, at := range pbsAt {
// The SAME archive appears as a vzdump record AND as a snapshot — slice A.4.
payload.Backups = append(payload.Backups, bk{"felhom-pbs", true, at.Format(time.RFC3339)})
payload.PBSSnapshots = append(payload.PBSSnapshots, snap{at.Format(time.RFC3339)})
}
b, err := json.Marshal(payload)
if err != nil {
t.Fatal(err)
}
return string(b)
}
rows := []store.HostReportRow{
{ReportJSON: mk(
[]time.Time{base, base.AddDate(0, 0, 1), base.AddDate(0, 0, 2)},
[]time.Time{base, base.AddDate(0, 0, 7)},
)},
{ReportJSON: `{{{malformed`}, // must not blind the scan
}
got := observedArchiveIntervals(rows)
if d, ok := got["local"]; !ok || d != 24*time.Hour {
t.Fatalf("a daily host tier must be observed as ~24h; got %s ok=%v", d, ok)
}
if d, ok := got["pbs"]; !ok || d != 7*24*time.Hour {
t.Fatalf("a weekly offsite tier must be observed as ~7d — and its vzdump record must NOT be "+
"counted into the host tier (slice A.4); got %s ok=%v", d, ok)
}
// One generation is not a rhythm: unobservable, so the caller falls back to the declared one.
single := []store.HostReportRow{{ReportJSON: mk(nil, []time.Time{base})}}
if d, ok := observedArchiveIntervals(single)["pbs"]; ok {
t.Fatalf("one archive cannot yield an interval; got %s", d)
}
}
@@ -0,0 +1,301 @@
package notify
import (
"io"
"log"
"strings"
"testing"
)
// R-182 — one e-mail per backup run, and nothing dropped without a trace.
//
// MEASURED, NOT SUPPOSED. On 2026-08-03 nine `recovery_unit_capture_failed` events reached the hub
// and TWO operator e-mails went out. The operator cooldown key is
// `customerID + ":" + eventType + cooldownTierSuffix(details)`, that event carries `app` but no
// `tier`, so the key held no app identifier: the first refused app took the hour's slot and every
// other app's failure was discarded — **before `LogNotification`**, so it left no row on any channel
// and could not be found afterwards.
//
// The operator ruled against the obvious fix (putting `app` in the key), because on a full disk that
// is one e-mail per app. These pin the shape that replaced it.
// ── Scenario D — a suppressed operator event leaves a trace ───────────────────────────────────────
// The bare `return` at the cooldown is the whole reason this defect took a day to get the right way
// round: there was nothing to read. A drop must be as visible in the record as a send.
//
// DELIBERATELY EXERCISED ON A DIFFERENT EVENT TYPE than the one that exposed the defect.
// `recovery_unit_capture_failed` is now record-only and never reaches the cooldown at all, so using
// it here would prove nothing. `whole_guest_backup_failed` is an ordinary operator event, and using
// it pins §2.1's actual claim: the suppression row applies to EVERY operator event, not only the one
// that happened to be measured.
func TestSuppressedOperatorEvent_LeavesARow(t *testing.T) {
st := newDispStore(t)
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
sent := captureSeam(d)
// Two events of the SAME type with no discriminator — the second must be suppressed.
d.ProcessEvent("c1", "whole_guest_backup_failed", "error", "opengist failed", `{"app":"opengist"}`, "controller")
d.ProcessEvent("c1", "whole_guest_backup_failed", "error", "privatebin failed", `{"app":"privatebin"}`, "controller")
if got := len(mailsFor(*sent, "op@felhom.eu")); got != 1 {
t.Fatalf("operator mails = %d, want 1 — the premise of this test is that the second IS suppressed", got)
}
rows, err := st.GetRecentNotifications("c1", 50)
if err != nil {
t.Fatal(err)
}
var sentRows, suppressed []store2Row
for _, r := range rows {
if r.Channel != "operator" || r.EventType != "whole_guest_backup_failed" {
continue
}
switch r.Status {
case "sent":
sentRows = append(sentRows, store2Row{r.Status, r.Message, r.ErrorMessage})
case "suppressed":
suppressed = append(suppressed, store2Row{r.Status, r.Message, r.ErrorMessage})
}
}
if len(sentRows) != 1 {
t.Fatalf("want 1 'sent' operator row, got %d", len(sentRows))
}
if len(suppressed) != 1 {
t.Fatalf("want 1 'suppressed' operator row, got %d — a cooldown drop that writes NOTHING is "+
"indistinguishable from an event that never happened, which is exactly how seven "+
"failures went missing on 2026-08-03", len(suppressed))
}
// The row must name the app that was dropped, or it records that something was suppressed
// without recording WHAT — half a fix.
if !strings.Contains(suppressed[0].message, "privatebin") {
t.Fatalf("the suppressed row does not name the dropped event: %q", suppressed[0].message)
}
// And it must carry the key, so the reason it collided is readable without reading code.
if !strings.Contains(suppressed[0].errMsg, "key=") {
t.Fatalf("the suppressed row does not carry the cooldown key: %q", suppressed[0].errMsg)
}
}
type store2Row struct{ status, message, errMsg string }
// ── Scenario E — two runs in a day each report ───────────────────────────────────────────────────
// The operator ruled explicitly on this: someone pressing the backup button is actively trying to
// get a backup, and finding out tomorrow would be worse than an extra mail in a rare case.
func TestTwoRunsInAnHour_BothReport(t *testing.T) {
st := newDispStore(t)
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
sent := captureSeam(d)
nightly := `{"run_id":"run-a","run_kind":"nightly","failed":2,"attempted":5,"apps":[{"app":"opengist","leg":"volume dump","reason":"reserve"},{"app":"privatebin","leg":"volume dump","reason":"reserve"}]}`
manual := `{"run_id":"run-b","run_kind":"manual","failed":2,"attempted":5,"apps":[{"app":"opengist","leg":"volume dump","reason":"reserve"},{"app":"privatebin","leg":"volume dump","reason":"reserve"}]}`
d.ProcessEvent("c1", "backup_run_failures", "error", "2 of 5 apps failed", nightly, "controller")
d.ProcessEvent("c1", "backup_run_failures", "error", "2 of 5 apps failed", manual, "controller")
if got := len(mailsFor(*sent, "op@felhom.eu")); got != 2 {
t.Fatalf("operator mails = %d, want 2 — the 1-hour cooldown swallowed the manual run's "+
"digest, which is the fix reappearing one level up: press the button, the run fails, "+
"and you are told nothing because the machine already wrote this hour", got)
}
}
// The run discriminator must be NARROW, exactly like its `tier` sibling — empty unless the producer
// opts in — or every existing event type's cooldown silently stops collapsing anything.
func TestCooldownRunSuffix_EmptyForEverythingElse(t *testing.T) {
cases := []struct{ name, details string }{
{"no details", ""},
{"details without run_id", `{"app":"opengist","error":"boom"}`},
{"empty run_id value", `{"run_id":""}`},
{"malformed json", `{{{nope`},
{"run_id mentioned in a STRING, not as a key", `{"error":"the run_id: abc failed"}`},
{"null details", `null`},
}
for _, c := range cases {
if got := cooldownRunSuffix(c.details); got != "" {
t.Errorf("%s: suffix must be EMPTY so every other type's cooldown is unchanged, got %q", c.name, got)
}
}
if got := cooldownRunSuffix(`{"run_id":"run-a"}`); got != ":run-a" {
t.Fatalf("suffix should be the run id, got %q", got)
}
// The two suffixes must not interfere: a tier event still keys on its tier and nothing else.
if got := cooldownTierSuffix(`{"tier":"felhom-pbs"}`) + cooldownRunSuffix(`{"tier":"felhom-pbs"}`); got != ":felhom-pbs" {
t.Fatalf("a tier-only event's key changed to %q — R-97a's behaviour must be byte-identical", got)
}
}
// ── Scenario G — the customer never receives the digest ──────────────────────────────────────────
// v0.78.0 asserted in a COMMENT that a type with no `customerMessages` entry structurally cannot
// reach a customer. It can: FormatCustomerEmail falls back to the raw English message and the only
// customer gate is configuration. So this is demonstrated, not argued.
func TestDigest_IsOperatorOnly_EvenWithAWideEnabledList(t *testing.T) {
st := newDispStore(t)
// A customer who has enabled EVERYTHING, including this type by name.
if err := st.SaveNotificationPrefs("c1", "cust@example.com",
[]string{"backup_run_failures", "node_down", "disk_warning"}, 6); err != nil {
t.Fatal(err)
}
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
sent := captureSeam(d)
details := `{"run_id":"run-a","run_kind":"nightly","failed":1,"attempted":3,"apps":[{"app":"opengist","leg":"volume dump","reason":"reserve"}]}`
d.ProcessEvent("c1", "backup_run_failures", "error", "1 of 3 apps failed", details, "controller")
if got := mailsFor(*sent, "cust@example.com"); len(got) != 0 {
t.Fatalf("the CUSTOMER received an operator digest (%d mails) — a list of which apps' "+
"backups failed is not something they can act on, and the raw body is English", len(got))
}
if got := len(mailsFor(*sent, "op@felhom.eu")); got != 1 {
t.Fatalf("operator mails = %d, want 1", got)
}
if !operatorOnlyEvents["backup_run_failures"] {
t.Fatal("backup_run_failures is not in operatorOnlyEvents — allowlisting alone does NOT " +
"keep it from a customer; that assumption shipped once and was wrong (v0.78.0)")
}
}
// ── Part 3 — the e-mail a person actually reads ──────────────────────────────────────────────────
func TestDigestEmail_ListsAppsLegsAndReasons(t *testing.T) {
details := `{"run_id":"run-a","run_kind":"nightly","failed":3,"attempted":40,` +
`"target_path":"/mnt/sys_drive","used_gb":64.3,"avail_gb":0.9,"total_gb":68.7,` +
`"used_percent":94,"space_known":true,"apps":[` +
`{"app":"opengist","leg":"volume dump","reason":"refused: below the reserve (headroom)"},` +
`{"app":"privatebin","leg":"volume dump","reason":"refused: below the reserve (headroom)"},` +
`{"app":"immich","leg":"database dump","reason":"pg_dump: connection refused"}]}`
subject, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error",
"3 of 40 apps failed to back up", details)
// The subject must carry the counts: the operator's first decision is made from it alone.
for _, want := range []string{"demo-hp", "3 of 40", "nightly"} {
if !strings.Contains(subject, want) {
t.Errorf("subject %q missing %q", subject, want)
}
}
// Every app, its leg and its reason.
for _, want := range []string{
"opengist", "privatebin", "immich",
"volume dump", "database dump",
"below the reserve", "pg_dump: connection refused",
} {
if !strings.Contains(body, want) {
t.Errorf("body missing %q:\n%s", want, body)
}
}
// The counts and the free space, so "one broken app" and "a full disk" read differently.
if !strings.Contains(body, "3 of 40") {
t.Errorf("body does not carry the failed-of-attempted count:\n%s", body)
}
if !strings.Contains(body, "0.9 GB free") {
t.Errorf("body does not carry the free space:\n%s", body)
}
// It must NOT be a JSON blob.
if strings.Contains(body, `"apps":[`) {
t.Errorf("the digest rendered as raw JSON — unreadable on a phone at 07:00:\n%s", body)
}
}
// An absent space reading must render as unavailable, never as zeros: "0 GB free" and "we could not
// look" are opposite diagnoses, and the operator acts differently on each.
func TestDigestEmail_UnknownSpaceIsNotZero(t *testing.T) {
details := `{"run_id":"r","run_kind":"nightly","failed":1,"attempted":2,"target_path":"/mnt/x",` +
`"space_known":false,"apps":[{"app":"a","leg":"capture","reason":"boom"}]}`
_, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error", "1 of 2 failed", details)
if strings.Contains(body, "0.0 GB free") {
t.Fatalf("an unreadable filesystem rendered as zeros:\n%s", body)
}
if !strings.Contains(body, "unavailable") {
t.Fatalf("an unreadable filesystem must say so:\n%s", body)
}
}
// A payload that cannot be parsed must still produce a mail — degraded, never swallowed.
func TestDigestEmail_UnparseableDetailsStillMails(t *testing.T) {
_, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error", "something failed", `{{{`)
if body == "" || !strings.Contains(body, "something failed") {
t.Fatalf("an unparseable digest lost the mail entirely:\n%s", body)
}
}
// ── Scenario C — every failure is RECORDED, e-mailed or not ──────────────────────────────────────
// The per-app event is the record; the digest is the notification. The record must not inherit the
// notification's conditions — no cooldown, no preferences, no dependence on a mail going out.
func TestPerAppFailure_IsRecordedButNotMailed(t *testing.T) {
st := newDispStore(t)
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
sent := captureSeam(d)
apps := []string{"opengist", "privatebin", "immich", "homebox", "nextcloud"}
for _, a := range apps {
d.ProcessEvent("c1", "recovery_unit_capture_failed", "error",
"Recovery unit capture FAILED for \""+a+"\"", `{"app":"`+a+`"}`, "controller")
}
// NOT mailed — the digest is the notification.
if got := len(*sent); got != 0 {
t.Fatalf("%d mail(s) sent for per-app failures — they are the RECORD; one mail per app on a "+
"full disk is the volume problem wearing the correctness problem's clothes, which is "+
"exactly what the operator ruled against", got)
}
// But ALL FIVE recorded — this is the assertion yesterday's defect would have failed: nine
// arrived, two were mailed, seven left no row anywhere.
rows, err := st.GetRecentNotifications("c1", 50)
if err != nil {
t.Fatal(err)
}
seen := map[string]bool{}
for _, r := range rows {
if r.EventType == "recovery_unit_capture_failed" && r.Status == "recorded" {
for _, a := range apps {
if strings.Contains(r.Message, a) {
seen[a] = true
}
}
}
}
if len(seen) != len(apps) {
t.Fatalf("only %d of %d per-app failures were recorded (%v) — a failure that produced no row "+
"anywhere is the measured defect of 2026-08-03", len(seen), len(apps), seen)
}
}
// The per-app reason must not repeat the filesystem figures the digest already prints once. Reviewed
// as copy against the first real digest, not designed in the abstract.
func TestDigestEmail_ReasonDoesNotRepeatTheUsageLine(t *testing.T) {
reason := "refused: below the reserve (reserve: 97% used or 1.0 GiB free) — /mnt/sys_drive: 65.0/68.7 GB used (95%), 0.2 GB free"
details := `{"run_id":"r","run_kind":"nightly","failed":1,"attempted":2,"target_path":"/mnt/sys_drive",` +
`"used_gb":65,"avail_gb":0.2,"total_gb":68.7,"used_percent":95,"space_known":true,` +
`"apps":[{"app":"opengist","leg":"whole app","reason":"` + reason + `"}]}`
_, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error", "1 of 2 failed", details)
// The figures appear ONCE, on the Filesystem line — not again on every app row.
if strings.Count(body, "65.0/68.7 GB used") != 1 {
t.Fatalf("the usage clause appears %d times; it must appear once, on its own line — repeated "+
"down a list of a dozen apps it pushes the part that DIFFERS off a phone screen:\n%s",
strings.Count(body, "65.0/68.7 GB used"), body)
}
// But the reason itself survives — trimming must not eat the diagnosis.
if !strings.Contains(body, "below the reserve") {
t.Fatalf("the reason was trimmed away entirely:\n%s", body)
}
}
// A reason naming a DIFFERENT path, or none, must be left completely alone.
func TestTrimRepeatedUsage_LeavesUnrelatedReasonsAlone(t *testing.T) {
for _, c := range []struct{ reason, target string }{
{"pg_dump: connection refused", "/mnt/sys_drive"},
{"tar failed — /mnt/other: 1/2 GB used (50%), 1 GB free", "/mnt/sys_drive"},
{"boom", ""},
{"", "/mnt/sys_drive"},
} {
if got := trimRepeatedUsage(c.reason, c.target); got != c.reason {
t.Errorf("reason %q (target %q) was altered to %q", c.reason, c.target, got)
}
}
}
+113 -1
View File
@@ -116,6 +116,18 @@ func (d *Dispatcher) ProcessEvent(customerID, eventType, severity, message, deta
return
}
// R-182: record-only types are written down and never mailed. Placed BEFORE the severity gate
// so the row is written whatever the severity — the record must not inherit the notification's
// conditions, which is the coupling this whole finding is about.
if recordOnlyEvents[eventType] {
if err := d.store.LogNotification(customerID, eventType, severity, message, "recorded",
"record-only: the per-run digest (backup_run_failures) carries the notification", "operator"); err != nil {
d.logger.Printf("[WARN] Failed to record %s for %s: %v", eventType, customerID, err)
}
d.logger.Printf("[INFO] Recorded (not mailed) %s for %s — the run digest is the notification", eventType, customerID)
return
}
// warning / error / critical trigger notifications. "info" is an intentional non-notify (status/
// recovery events). Anything else is UNRECOGNIZED — log it (don't silently drop), so a bad severity
// surfaces instead of vanishing (the felhom-pve-class lesson: a critical event must never be lost).
@@ -260,15 +272,67 @@ func cooldownTierSuffix(detailsJSON string) string {
return ":" + d.Tier
}
// cooldownRunSuffix returns ":"+run_id when the event's details carry a non-empty `run_id`, else "".
//
// R-182. `cooldownTierSuffix`'s sibling, and deliberately a SEPARATE function rather than an extra
// branch inside it: `tier` keeps byte-identical semantics for every type that uses it, so R-97a's
// behaviour and its tests are untouched by this.
//
// WHY A BACKUP RUN NEEDS ONE. The run digest describes ONE RUN, and a box can have two in a day —
// the nightly one and a manual one the operator triggered *because* something looked wrong. With no
// run-scoped discriminator the 1-hour cooldown would swallow the second, which is the failure this
// row exists to fix, reappearing one level up: the operator presses the button, the run fails, and
// they are told nothing because the machine already wrote that hour.
//
// IT MAKES THE COOLDOWN EFFECTIVELY INERT FOR THIS TYPE, AND THAT IS THE INTENT, NOT AN OVERSIGHT.
// A digest is already rate-limited by construction — one per run, emitted only when something
// failed — so there is nothing for a timer to collapse. The cooldown protects against a repeating
// identical alert; a digest cannot repeat, because each run is a different run.
//
// NARROW, LIKE ITS SIBLING: empty unless the producer opts in by sending a `run_id`, so no existing
// event type's cooldown behaviour changes.
func cooldownRunSuffix(detailsJSON string) string {
if detailsJSON == "" || !strings.Contains(detailsJSON, "\"run_id\"") {
return ""
}
var d struct {
RunID string `json:"run_id"`
}
if err := json.Unmarshal([]byte(detailsJSON), &d); err != nil || d.RunID == "" {
return ""
}
return ":" + d.RunID
}
func (d *Dispatcher) processOperator(customerID, eventType, severity, message, detailsJSON, source string) {
if !d.operatorOn || d.operatorEmail == "" {
return
}
cooldownKey := customerID + ":" + eventType + cooldownTierSuffix(detailsJSON)
cooldownKey := customerID + ":" + eventType + cooldownTierSuffix(detailsJSON) + cooldownRunSuffix(detailsJSON)
d.mu.Lock()
if last, ok := d.opCooldowns[cooldownKey]; ok && time.Since(last) < 1*time.Hour {
d.mu.Unlock()
// R-182: RECORD THE SUPPRESSION. This used to be a bare `return` — the event was dropped
// before any LogNotification, so a cooldown drop and an event that never happened were
// indistinguishable from the operator's side AND from the hub's own records.
//
// Measured 2026-08-03: nine `recovery_unit_capture_failed` events arrived, two emails were
// sent, and the other seven left NO ROW ON ANY CHANNEL. The defect that hid was serious —
// the cooldown key carries no app identifier, so the first refused app took the slot and
// every other app's failure that hour was discarded — but the reason it took a day to find
// the right way round is this line: there was nothing to read.
//
// "We chose not to e-mail you" and "nothing happened" must never look identical. This
// applies to EVERY operator event, not only the one that exposed it. It makes the drop
// visible; it deliberately does NOT change the cooldown's duration or semantics.
if err := d.store.LogNotification(customerID, eventType, severity, message,
"suppressed", "operator cooldown 1h, key="+cooldownKey, "operator"); err != nil {
d.logger.Printf("[WARN] Failed to record suppressed operator notification for %s/%s: %v",
customerID, eventType, err)
}
d.logger.Printf("[INFO] Operator email suppressed for %s/%s — cooldown (key=%s)",
customerID, eventType, cooldownKey)
return
}
d.opCooldowns[cooldownKey] = time.Now()
@@ -285,6 +349,33 @@ func (d *Dispatcher) processOperator(customerID, eventType, severity, message, d
d.store.LogNotification(customerID, eventType, severity, message, "sent", "", "operator")
}
// recordOnlyEvents are STORED and RECORDED but never e-mailed, on either channel.
//
// R-182. The distinction this register exists to make is the whole of that finding: **the record and
// the notification are different things.** A per-app backup failure must always be written down —
// every time, unconditionally, regardless of cooldowns, preferences or whether any mail went out —
// and it must NOT compete for an e-mail slot, because the per-run digest
// (`backup_run_failures`) is what a person is meant to read.
//
// Before this, `recovery_unit_capture_failed` was both at once, and it did neither well: on
// 2026-08-03 nine of them arrived, two were e-mailed, and the other seven were dropped by the
// 1-hour cooldown BEFORE anything was written down. So the operator was told about one app, the
// other apps' failures were discarded, and nothing anywhere recorded that a choice had been made.
//
// WHY A REGISTER AND NOT severity "info". Downgrading the severity would have the same routing
// effect — `severityNotifies` drops info — but it would also relabel a genuine failure as
// informational in the events table, the operator UI and every historical query, and it would
// silently drop the X-Priority handling if the type were ever promoted back. This says what it
// means: not silent, not urgent, RECORDED.
//
// IT IS NOT A WAY TO MUTE THINGS. A type belongs here only when something else carries its
// notification. Adding one with no digest behind it rebuilds the silence R-182 was filed against.
var recordOnlyEvents = map[string]bool{
// The per-app Tier-1 capture/refusal failure. Its notification is the run digest, which lists
// every failed app in one mail; this row is the durable per-failure record behind it.
"recovery_unit_capture_failed": true,
}
// operatorOnlyEvents are event types that must NEVER reach a customer, whatever their preferences say.
//
// R-97c. This register exists because the guarantee it provides was previously ASSERTED IN A COMMENT
@@ -315,8 +406,29 @@ var operatorOnlyEvents = map[string]bool{
// a consequence of another type's routing — true today, and silently untrue the moment the failed
// event becomes customer-visible. Belt, not inference.
"whole_guest_backup_recovered": true,
// R-158 / R-167 (D-c). A per-app Tier-1 recovery-unit capture failure. A customer can take no
// action on it — the causes are a full filesystem, a permission fault or a broken dump, all of
// which the operator resolves — and the alert carries operator-grade detail (target path, byte
// figures, the raw error). The customer's half of D-c is the FILL WARNING, which fires BEFORE
// this and is actionable: free space, delete files, add a drive.
"recovery_unit_capture_failed": true,
// R-182. The per-run backup digest. It is the same class as the line above and for the same
// reason — a customer can act on a full disk (that is the fill warning, which fires first and
// IS customer-facing) but not on a list of which apps' backups failed and why. It also carries
// operator-grade detail: per-app leg names, raw refusal reasons and byte figures.
//
// Listed here rather than relying on the absence of a `customerMessages` entry, which is NOT a
// block — `FormatCustomerEmail` falls back to the raw English message. That mistake shipped
// once (v0.78.0) and the comment above records it.
"backup_run_failures": true,
}
// IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the
// api package can pin BOTH registers of a new event type in one test — allowlisted-but-not-
// operator-only is the v0.78.0 defect, and it is only visible when the two are checked together.
// Read-only: the register itself stays unexported so nothing can widen it at runtime.
func IsOperatorOnly(eventType string) bool { return operatorOnlyEvents[eventType] }
func (d *Dispatcher) processCustomer(customerID, eventType, severity, message, detailsJSON, source string) {
// R-97c: operator-tier events stop here, BEFORE prefs are consulted — the point is that no
// customer configuration can opt in. Logged rather than dropped, so the skip is visible in
@@ -0,0 +1,142 @@
package notify
import (
"io"
"log"
"strings"
"testing"
)
func discardLogger() *log.Logger { return log.New(io.Discard, "", 0) }
// R-158 / R-167 Scenario G — the two new signals of decision D-c sit on OPPOSITE sides of the
// operator/customer line, and each is tested through the real dispatch path rather than by reading
// the register.
//
// D-c's rule, restated: a customer can free space, delete files or add a drive, so a FILL WARNING is
// theirs. A customer can do nothing about a recovery-unit capture failure, so it is not.
// The operator half. Run under the BREAKING configuration — the customer has the event explicitly
// enabled and has an email address — because that is the only configuration in which a missing
// operatorOnlyEvents entry is visible. This is the v0.78.0 defect, demonstrated rather than argued.
func TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer(t *testing.T) {
st := opOnlyStore(t)
if err := st.SaveNotificationPrefs("c1", "customer@example.com",
[]string{"recovery_unit_capture_failed", "backup_failed"}, 6); err != nil {
t.Fatalf("SaveNotificationPrefs: %v", err)
}
rec := &sentTo{}
d := opOnlyDispatcher(t, st, rec)
d.ProcessEvent("c1", "recovery_unit_capture_failed", "error",
`Recovery unit capture FAILED for "immich" — the app has no fresh local (Tier-1) backup`,
`{"app":"immich","used_percent":100,"space_known":true}`, "controller")
for _, to := range rec.to {
if to == "customer@example.com" {
t.Fatalf("a customer was emailed the OPERATOR-ONLY recovery_unit_capture_failed (%s) — "+
"enabled_events must not be able to opt in to a failure they cannot act on", to)
}
}
// R-182 CHANGED WHAT THIS ASSERTS, DELIBERATELY, AND THE OLD ASSERTION IS WORTH KEEPING IN VIEW.
//
// Until 2026-08-03 this test required the OPERATOR to be e-mailed here, on the grounds that "the
// alert is the whole point of R-158". That was right when this event was the only signal, and it
// is wrong now: measured, nine of these arrived at the hub and two were mailed, because the
// operator cooldown key carries no app identifier — so as an alarm it told the operator about one
// app and threw the rest away.
//
// The type is now RECORD-ONLY: written down every time, never mailed. R-158's guarantee — the
// operator learns WHICH app failed and WHY — is not weakened, it MOVED: the per-run digest
// `backup_run_failures` carries every failed app in one mail, and is pinned by
// backup_run_digest_test.go. The customer safety claim below is untouched and is the reason this
// test still exists.
for _, to := range rec.to {
if to == "operator@felhom.eu" {
t.Fatal("the operator was e-mailed a PER-APP capture failure — this type is the record " +
"now, not the alarm. One mail per failing app on a full disk is a dozen mails, which " +
"is the volume problem the operator ruled against; the digest is the notification")
}
}
// The RECORD must exist, always. It is what makes the digest trustworthy: if the digest is ever
// lost, delayed or suppressed, the failures are still individually written down. An absent row is
// equally consistent with "correctly not mailed" and "the dispatcher never ran" — the positive
// observable is the row itself (standing rule 3).
logs, err := st.GetRecentNotifications("c1", 20)
if err != nil {
t.Fatalf("GetRecentNotifications: %v", err)
}
found := false
for _, l := range logs {
if l.EventType == "recovery_unit_capture_failed" && l.Status == "recorded" &&
strings.Contains(l.ErrorMessage, "record-only") {
found = true
}
}
if !found {
t.Fatalf("the per-app failure left no 'recorded' row — a failure that is neither mailed nor "+
"written down is exactly the 2026-08-03 defect, rebuilt; got %d row(s)", len(logs))
}
}
// The customer half. The fill warning MUST be delivered, and it must render the controller's own
// Hungarian text — which names the drive and the free space — rather than a generic template or the
// raw English.
func TestDiskFillWarning_ReachesTheCustomerInHungarianWithItsNumbers(t *testing.T) {
st := opOnlyStore(t)
if err := st.SaveNotificationPrefs("c1", "customer@example.com", []string{"disk_warning"}, 6); err != nil {
t.Fatal(err)
}
// The exact shape the controller sends from v0.191.0.
const hun = `A(z) „Fotók" tároló 87%-osan megtelt — 4,2 GB szabad hely maradt. ` +
`Szabadíts fel helyet (törölj felesleges fájlokat, vagy csatlakoztass új meghajtót), ` +
`különben a biztonsági mentések hamarosan meghiúsulnak.`
var bodies []string
d := NewDispatcher(st, "test-key", "from@felhom.eu", "operator@felhom.eu", true, discardLogger())
d.sendEmailFn = func(to, subject, body string, headers map[string]string) error {
if to == "customer@example.com" {
bodies = append(bodies, subject+"\n"+body)
}
return nil
}
d.ProcessEvent("c1", "disk_warning", "warning", hun,
`{"label":"Fotók","used_percent":87,"avail_gb":4.2}`, "controller")
if len(bodies) == 0 {
t.Fatal("the customer was NOT warned that a disk is filling — this is the customer half of " +
"decision D-c, and nothing reached them")
}
got := strings.Join(bodies, "\n")
// The controller's dynamic text must survive. A customerMessages entry for disk_warning would
// OVERRIDE it (templates.go prefers the entry) and discard the drive name and the free space,
// leaving the customer with "A lemezterület 90% felett van" and nothing to act on — which is
// exactly why v0.191.0 removes those two entries.
if !strings.Contains(got, "Fotók") {
t.Fatalf("the customer email does not name the drive — a generic customerMessages entry has "+
"discarded the controller's dynamic text (templates.go priority). Got:\n%s", got)
}
if !strings.Contains(got, "4,2 GB") {
t.Fatalf("the customer email does not carry the free-space figure. Got:\n%s", got)
}
if !strings.Contains(got, "Szabadíts fel helyet") {
t.Fatalf("the customer email does not tell the customer what to DO. Got:\n%s", got)
}
}
// The generic entries must STAY REMOVED. A well-meaning re-add would silently re-break the test
// above's guarantee for every future reader — this pins the deletion itself.
func TestDiskFillTypesHaveNoGenericCustomerMessage(t *testing.T) {
for _, et := range []string{"disk_warning", "disk_critical"} {
if msg, ok := customerMessages[et]; ok {
t.Fatalf("customerMessages[%q] = %q — a static entry OVERRIDES the controller's dynamic "+
"Hungarian text (FormatCustomerEmail prefers the entry), discarding the drive name "+
"and the free-space figure the customer needs. Same reason offbox_enlarge_blocked "+
"and disk_health_degraded deliberately have none", et, msg)
}
}
}
+138 -3
View File
@@ -1,6 +1,7 @@
package notify
import (
"encoding/json"
"fmt"
"strings"
"time"
@@ -41,6 +42,17 @@ Severity: %s
Time: %s
Message: %s`, customerID, eventType, severity, now, message)
// R-182: the backup run digest gets a rendered list instead of a raw JSON blob. It is the one
// operator mail that carries a VARIABLE-LENGTH payload, and a dozen apps as one line of JSON is
// unreadable on a phone at 07:00, which is the only time it matters.
if eventType == "backup_run_failures" {
if rendered, sub, ok := renderBackupRunFailures(customerID, detailsJSON); ok {
return sub, body + rendered + fmt.Sprintf("\n\nDashboard: https://hub.felhom.eu/customers/%s", customerID)
}
// Unparseable details fall through to the raw form below rather than losing the mail. A
// digest that renders badly still tells the operator something; a swallowed one does not.
}
if detailsJSON != "" && detailsJSON != "{}" {
body += fmt.Sprintf("\nDetails: %s", detailsJSON)
}
@@ -71,9 +83,20 @@ var customerMessages = map[string]string{
"offbox_repo_orphaned": "A távoli mentési tároló elárvult: a benne lévő mentések egy korábbi, már nem elérhető kulccsal készültek (jellemzően újratelepítés után). Új mentés a tároló visszaállításáig nem készül — nyisd meg a Távoli mentés oldalt.",
"offbox_repo_reset": "A távoli mentési tároló visszaállítva: a régi előzmény félretéve (nem törölve), és egy üres, új tároló jött létre a mostani kulccsal.",
// Disk events (GUEST — the controller's own cgroup view)
"disk_warning": "A lemezterület 90% felett van — kérjük, szabadíts fel helyet.",
"disk_critical": "A lemezterület kritikusan magas (95%+) — azonnali beavatkozás szükséges!",
// Disk events (GUEST — the controller's own view) — `disk_warning` / `disk_critical`.
//
// DELIBERATELY NO ENTRY, from hub v0.89.0 / controller v0.191.0 (R-167, decision D-c). These two
// types were allowlisted here, carried generic Hungarian copy, sat in the controller's
// DefaultEnabledEvents and had a UI checkbox — and NOTHING IN ANY REPO EMITTED THEM. A complete
// customer pipeline with no producer; the sixth "built but never wired" instance in this project.
// The controller became their producer in v0.191.0.
//
// The producer sends a DYNAMIC Hungarian message naming the filesystem and its free space, so a
// static entry here would be actively harmful: FormatCustomerEmail PREFERS the entry over the
// message, so re-adding one would discard the drive name and the byte figures and leave the
// customer with "A lemezterület 90% felett van" — a warning with nothing to act on. Same reason
// `offbox_enlarge_blocked` and `disk_health_degraded` have no entry. Pinned by
// TestDiskFillTypesHaveNoGenericCustomerMessage.
// Host disk events (the Proxmox HOST root filesystem — distinct from the guest disk above)
"host_disk_warning": "A házszerver alaprendszerének (Proxmox-gazda) gyökérlemeze 90% felett van — kérjük, szabadíts fel helyet (pl. régi biztonsági mentések).",
@@ -271,3 +294,115 @@ Ha nem te kérted ezt, hagyd figyelmen kívül ezt az e-mailt.
Felhom.eu`, link)
return subject, body
}
// ──────────────────────────────────────────────────────────────────────
// R-182 — the backup run digest
// ──────────────────────────────────────────────────────────────────────
// backupRunFailure is one app's failed leg within a run.
type backupRunFailure struct {
App string `json:"app"`
Leg string `json:"leg"`
Reason string `json:"reason"`
}
// backupRunDetails is the digest payload the controller sends.
type backupRunDetails struct {
RunID string `json:"run_id"`
RunKind string `json:"run_kind"`
Failed int `json:"failed"`
Attempted int `json:"attempted"`
TargetPath string `json:"target_path"`
UsedGB float64 `json:"used_gb"`
AvailGB float64 `json:"avail_gb"`
TotalGB float64 `json:"total_gb"`
UsedPercent float64 `json:"used_percent"`
SpaceKnown bool `json:"space_known"`
Apps []backupRunFailure `json:"apps"`
}
// renderBackupRunFailures turns the digest details into an operator-readable block and a subject
// that says the count without being opened. Returns ok=false when the payload cannot be parsed or
// names no apps, so the caller can fall back to the raw rendering rather than mail an empty list.
//
// THE SUCCESS COUNT IS NOT DECORATION. "3 of 4 apps failed" is a catastrophe and "3 of 40" is a bad
// night; the list alone cannot tell them apart, and the operator's first decision — get up now, or
// look after coffee — depends entirely on which it is.
func renderBackupRunFailures(customerID, detailsJSON string) (string, string, bool) {
if detailsJSON == "" {
return "", "", false
}
var d backupRunDetails
if err := json.Unmarshal([]byte(detailsJSON), &d); err != nil || len(d.Apps) == 0 {
return "", "", false
}
kind := d.RunKind
if kind == "" {
kind = "backup"
}
subject := fmt.Sprintf("[Felhom] 🔴 %s: %d of %d apps failed to back up (%s run)",
customerID, d.Failed, d.Attempted, kind)
// Column-align the app names so the leg and reason line up and the block scans vertically.
width := 0
for _, a := range d.Apps {
if len(a.App) > width {
width = len(a.App)
}
}
legWidth := 0
for _, a := range d.Apps {
if len(a.Leg) > legWidth {
legWidth = len(a.Leg)
}
}
var b strings.Builder
fmt.Fprintf(&b, "\n\nFAILED: %d of %d apps attempted in this %s run.\n\n", d.Failed, d.Attempted, kind)
for _, a := range d.Apps {
reason := trimRepeatedUsage(a.Reason, d.TargetPath)
if reason == "" {
reason = "(no reason recorded)"
}
fmt.Fprintf(&b, " %-*s %-*s %s\n", width, a.App, legWidth, a.Leg, reason)
}
// The space figures answer "is this one broken app or a full disk" before the reasons are read.
// An absent reading renders as unavailable, never as zeros — "0 GB free" and "we could not look"
// are opposite diagnoses (the UnitSpace rule, same reasoning, other side of the wire).
if d.SpaceKnown {
fmt.Fprintf(&b, "\nFilesystem: %s — %.1f/%.1f GB used (%.0f%%), %.1f GB free\n",
d.TargetPath, d.UsedGB, d.TotalGB, d.UsedPercent, d.AvailGB)
} else {
fmt.Fprintf(&b, "\nFilesystem: %s — usage unavailable (the filesystem could not be read)\n", d.TargetPath)
}
b.WriteString("\nEvery failure above is also recorded individually in the notification log,\n")
b.WriteString("whether or not this mail was sent.")
return b.String(), subject, true
}
// trimRepeatedUsage strips the trailing "— /path: X/Y GB used (Z%), W GB free" clause from a per-app
// reason, because the digest prints those figures ONCE for the whole run on its own line.
//
// This is a copy fix, and it was made after reading the first real digest rather than from the
// design. The reserve's refusal message is authored for a single-app alert, where naming the
// filesystem is exactly right; repeated down a list of a dozen apps it is the same forty characters
// twelve times, and it pushes the part that differs off the right-hand edge of a phone screen at
// 07:00 — which is the only moment this mail has to work.
//
// It trims ONLY an exact "— <target path>:" suffix, so a reason that mentions a different path, or
// none, is left completely alone. A reason that is nothing but the usage clause is left alone too:
// removing everything would turn a bad line into an empty one.
func trimRepeatedUsage(reason, targetPath string) string {
if reason == "" || targetPath == "" {
return reason
}
marker := " — " + targetPath + ":"
i := strings.LastIndex(reason, marker)
if i <= 0 {
return reason
}
return strings.TrimSpace(reason[:i])
}
+230
View File
@@ -0,0 +1,230 @@
package store
import (
"database/sql"
"io"
"log"
"os"
"path/filepath"
"sync"
"testing"
"time"
)
// R-172 — the connection pragmas must actually be APPLIED, not merely requested.
//
// THE DEFECT THESE PIN. The DSN read `?_journal_mode=WAL&_busy_timeout=5000` — mattn/go-sqlite3
// syntax — while the driver is modernc.org/sqlite, which ignores unknown parameters WITHOUT AN
// ERROR. The hub therefore ran in rollback-journal mode with busy_timeout=0 for its entire life,
// and nothing said so. Any test that asserted the DSN STRING would have passed throughout.
//
// So every assertion below reads the value back from the DATABASE.
// newPragmaStore is a sibling of host_test.go's newTestStore that also returns the DB PATH, because
// two of the checks below are about the files SQLite creates beside it.
func newPragmaStore(t *testing.T) (*Store, string) {
t.Helper()
path := filepath.Join(t.TempDir(), "hub.db")
s, err := New(path, log.New(io.Discard, "", 0))
if err != nil {
t.Fatalf("New: %v", err)
}
t.Cleanup(func() { s.Close() })
return s, path
}
func TestStorePragmasAreActuallyApplied(t *testing.T) {
// RED-PROOF: restore the old DSN (`?_journal_mode=WAL&_busy_timeout=5000`) and this fails with
// journal_mode=delete, busy_timeout=0 — i.e. it reproduces the shipped bug exactly.
// Demonstrated in hub REPORT.md.
s, _ := newPragmaStore(t)
var journal string
if err := s.db.QueryRow("PRAGMA journal_mode").Scan(&journal); err != nil {
t.Fatalf("read journal_mode: %v", err)
}
if journal != "wal" {
t.Fatalf("journal_mode = %q, want \"wal\" — in rollback-journal mode a reader blocks a writer, "+
"so rendering an operator page can 500 a host report (R-172)", journal)
}
var busy int
if err := s.db.QueryRow("PRAGMA busy_timeout").Scan(&busy); err != nil {
t.Fatalf("read busy_timeout: %v", err)
}
if busy < 5000 {
t.Fatalf("busy_timeout = %d, want >= 5000 — without it SQLite returns SQLITE_BUSY immediately "+
"instead of waiting, and the hub turns that into an HTTP 500", busy)
}
}
func TestStoreWALFilesExistWhileOpen(t *testing.T) {
// The on-disk observable that EXPOSED the bug in production, pinned as a test: in WAL mode the
// `-wal` and `-shm` files must exist beside an open database. Their absence on the live hub is
// what proved the pragma was never applied, so it is the check to keep.
s, path := newPragmaStore(t)
// A write guarantees the WAL is materialised rather than merely configured.
if _, err := s.db.Exec(`CREATE TABLE IF NOT EXISTS r172_probe (k TEXT)`); err != nil {
t.Fatalf("probe write: %v", err)
}
if _, err := s.db.Exec(`INSERT INTO r172_probe (k) VALUES ('x')`); err != nil {
t.Fatalf("probe insert: %v", err)
}
for _, suffix := range []string{"-wal", "-shm"} {
if _, err := os.Stat(path + suffix); err != nil {
t.Fatalf("%s is missing beside an OPEN database (%v) — this is exactly the signature that "+
"proved the live hub was NOT in WAL mode", filepath.Base(path+suffix), err)
}
}
}
func TestStoreReaderDoesNotBlockWriter(t *testing.T) {
// THE CONSEQUENCE, not the mechanism. A held READ transaction — what rendering an operator page
// does — must not make a concurrent write fail. In rollback-journal mode it does, and that is the
// whole of R-172: `Failed to save host-report …: database is locked (5) (SQLITE_BUSY)`.
s, _ := newPragmaStore(t)
if _, err := s.db.Exec(`CREATE TABLE IF NOT EXISTS r172_probe (k TEXT)`); err != nil {
t.Fatalf("setup: %v", err)
}
if _, err := s.db.Exec(`INSERT INTO r172_probe (k) VALUES ('seed')`); err != nil {
t.Fatalf("seed: %v", err)
}
// Hold a read open for the whole write.
rows, err := s.db.Query(`SELECT k FROM r172_probe`)
if err != nil {
t.Fatalf("open read: %v", err)
}
defer rows.Close()
if !rows.Next() {
t.Fatal("expected a seeded row")
}
done := make(chan error, 1)
go func() {
_, err := s.db.Exec(`INSERT INTO r172_probe (k) VALUES ('concurrent')`)
done <- err
}()
select {
case err := <-done:
if err != nil {
t.Fatalf("a write FAILED while a read was open: %v — this is the live 500 (R-172)", err)
}
case <-time.After(15 * time.Second):
t.Fatal("a write BLOCKED indefinitely while a read was open")
}
}
func TestStoreConcurrentWritersDoNotReturnBusy(t *testing.T) {
// Writers still serialise under WAL; busy_timeout is what turns that into a WAIT rather than an
// error. Several concurrent writers must all succeed — the host report, the event save and a UI
// action genuinely do overlap on this hub.
s, _ := newPragmaStore(t)
if _, err := s.db.Exec(`CREATE TABLE IF NOT EXISTS r172_probe (k TEXT)`); err != nil {
t.Fatalf("setup: %v", err)
}
const writers = 8
var wg sync.WaitGroup
errs := make(chan error, writers)
for i := 0; i < writers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
if _, err := s.db.Exec(`INSERT INTO r172_probe (k) VALUES ('w')`); err != nil {
errs <- err
}
}()
}
wg.Wait()
close(errs)
for err := range errs {
t.Fatalf("a concurrent writer returned an error instead of waiting: %v", err)
}
}
func TestStoreTransactionUpgradeDoesNotReturnBusySnapshot(t *testing.T) {
// `_txlock=immediate` is the parameter that is easy to leave out, and WAL + busy_timeout alone
// would not cover this. With a DEFERRED transaction (database/sql's default), a tx that reads and
// then writes must upgrade its lock, and a failed upgrade is SQLITE_BUSY_SNAPSHOT — which
// busy_timeout does NOT retry. This store has 10+ db.Begin() sites and they are all write paths.
//
// RED-PROOF: drop `&_txlock=immediate` from sqliteDSNParams and this test becomes able to fail
// (it is inherently racy without it, which is precisely the point — an un-retryable error that
// appears only under contention is the worst kind to ship). Demonstrated in hub REPORT.md.
s, _ := newPragmaStore(t)
if _, err := s.db.Exec(`CREATE TABLE IF NOT EXISTS r172_probe (k TEXT)`); err != nil {
t.Fatalf("setup: %v", err)
}
if _, err := s.db.Exec(`INSERT INTO r172_probe (k) VALUES ('seed')`); err != nil {
t.Fatalf("seed: %v", err)
}
// Two read-then-write transactions racing is the upgrade shape.
const txs = 6
var wg sync.WaitGroup
errs := make(chan error, txs)
for i := 0; i < txs; i++ {
wg.Add(1)
go func() {
defer wg.Done()
tx, err := s.db.Begin()
if err != nil {
errs <- err
return
}
var n int
if err := tx.QueryRow(`SELECT COUNT(*) FROM r172_probe`).Scan(&n); err != nil {
tx.Rollback()
errs <- err
return
}
if _, err := tx.Exec(`INSERT INTO r172_probe (k) VALUES ('tx')`); err != nil {
tx.Rollback()
errs <- err
return
}
if err := tx.Commit(); err != nil {
errs <- err
}
}()
}
wg.Wait()
close(errs)
for err := range errs {
t.Fatalf("a read-then-write transaction failed under contention: %v — this is the "+
"SQLITE_BUSY_SNAPSHOT that busy_timeout cannot retry, and _txlock=immediate prevents", err)
}
}
// TestSQLiteDriverIgnoresMattnStyleParams is the regression guard for the ROOT CAUSE, not the symptom.
//
// It documents, executably, that the old DSN syntax is silently ignored by this driver — so anyone
// who "tidies" the pragmas back to the more familiar mattn form gets a failing test instead of a
// hub that quietly reverts to rollback-journal mode for another few months.
func TestSQLiteDriverIgnoresMattnStyleParams(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "mattnstyle.db")
db, err := sql.Open("sqlite", path+"?_journal_mode=WAL&_busy_timeout=5000")
if err != nil {
t.Fatalf("open: %v", err)
}
defer db.Close()
var journal string
if err := db.QueryRow("PRAGMA journal_mode").Scan(&journal); err != nil {
t.Fatalf("read journal_mode: %v", err)
}
if journal == "wal" {
t.Skip("this driver now honours mattn-style parameters — the R-172 trap is gone; simplify " +
"sqliteDSNParams and delete this test")
}
if journal != "delete" {
t.Fatalf("journal_mode = %q, expected the driver to IGNORE the mattn-style parameter and leave "+
"the default; if this changed, re-read sqliteDSNParams", journal)
}
}
+38 -2
View File
@@ -51,9 +51,45 @@ type CustomerSummary struct {
DiskSummary string
}
// sqliteDSNParams are the connection pragmas, and getting the SYNTAX right is the whole point.
//
// ── R-172: this DSN was WRONG for the hub's entire life, and it failed SILENTLY ──────────────────
//
// It used to read `?_journal_mode=WAL&_busy_timeout=5000`. That is **mattn/go-sqlite3** syntax. This
// hub uses **modernc.org/sqlite**, whose `applyQueryParams` reads only `_pragma`, `_time_format`,
// `_time_integer_format`, `_txlock` and `_inttotime` — anything else is **ignored without an error**.
// So the hub ran in the default rollback-journal mode with busy_timeout=0 while its own source said
// otherwise: a configuration asserting an invariant the code did not provide, the same class as the
// comments in `CLAUDE.md`'s false-invariant table.
//
// The observable that proved it: a 128 MB `/data/hub.db` with **no `-wal`/`-shm` file beside it while
// the database was open**. In WAL mode those files must exist. Consequence, measured on 2026-08-02:
// 13 `SQLITE_BUSY` collisions in one pod lifetime, each returning HTTP 500 to a host report, and two
// consecutive misses crossing the 30-minute staleness threshold — a false `host_stale` alarm plus an
// operator e-mail for a host that was up and healthy throughout.
//
// Each parameter, and why it is not optional:
//
// - journal_mode(WAL) — in rollback-journal mode a writer excludes readers and vice versa, so
// rendering an operator page could block a host report. WAL lets readers and one writer proceed
// concurrently. It is a property of the DATABASE FILE, so it persists once set.
// - busy_timeout(5000) — writers still serialise against each other. Without a timeout SQLite
// returns SQLITE_BUSY *immediately* rather than waiting; 5 s is far longer than any write here.
// - txlock=immediate — THE ONE THAT IS EASY TO MISS. `database/sql`'s Begin() is DEFERRED by
// default, so a transaction that reads and then writes must upgrade its lock, and a failed
// upgrade returns SQLITE_BUSY_SNAPSHOT, which **busy_timeout does not retry**. This store has
// 10+ `db.Begin()` sites and they are all write paths (customer delete/reset, wg, appliance,
// pbsdr, telemetry, log bundles). Taking the write lock up front converts that un-retryable
// failure into an ordinary wait covered by busy_timeout above. WAL + busy_timeout WITHOUT this
// would leave a known un-retryable path open and ship half a fix.
//
// TestStorePragmasAreActuallyApplied asserts what the DATABASE reports, never what string was passed
// — asserting the DSN would have passed happily for the entire life of the bug.
const sqliteDSNParams = "?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate"
// New creates a new store and initializes the schema.
func New(dbPath string, logger *log.Logger) (*Store, error) {
db, err := sql.Open("sqlite", dbPath+"?_journal_mode=WAL&_busy_timeout=5000")
db, err := sql.Open("sqlite", dbPath+sqliteDSNParams)
if err != nil {
return nil, fmt.Errorf("opening database: %w", err)
}
@@ -779,7 +815,7 @@ type NotificationLogEntry struct {
EventType string
Severity string
Message string
Status string // "sent", "skipped", "failed"
Status string // "sent", "skipped", "failed", "suppressed" (R-182: a cooldown drop, recorded rather than silent)
ErrorMessage string
Channel string // "operator" or "customer"
CreatedAt time.Time
+8 -11
View File
@@ -20,12 +20,14 @@ import (
var validCustomerID = regexp.MustCompile(`^[a-zA-Z0-9.\-]+$`)
// hostInstallVersion is the felhom-host-install.sh version the customer page's install-command
// generator targets. Kept in sync with scripts/felhom-host-install.sh SCRIPT_VERSION — the generator
// only ever emits flags this version parses. Display-only (the Option-1 command downloads the served
// script, which is always current); bump when the generator's flag surface follows a new script.
// Drift is now gated: scripts/hostinstall_gates.py asserts this const == SCRIPT_VERSION (drill F-1).
const hostInstallVersion = "1.19.0"
// NOTE (R-94, 2026-08-02): there is deliberately NO host-install version constant here, and the
// Setup tab renders no version number. The hub cannot know which version a box will run: the
// Option-1 command downloads felhom-host-install.sh from the website at run time, and the website
// git-syncs `main` every 30s (R-110). Any build-time literal here is a guess wearing a version
// number's authority — the previous const said 1.19.0 while the served script was 1.22.0, and had
// been wrong since 2026-07-14. The single version source is scripts/felhom-host-install.sh's
// SCRIPT_VERSION; scripts/hostinstall_gates.py gate 1 now asserts this file's ABSENCE of any
// host-install version literal.
// validSemver matches a bare X.Y.Z controller version (the floor format). Empty is also accepted by
// the floor handlers (clears the override).
@@ -331,9 +333,6 @@ func (s *Server) handleCustomerUnified(w http.ResponseWriter, r *http.Request, c
CSRFField template.HTML
CSRFToken string
// ScriptVersion drives the install-command generator's header (GL-7). Display-only.
ScriptVersion string
// Hosts (v0.47.0): the customer's enrolled hosts for the Host tab — a LIST by design
// (1 today, N for a later HA cluster). Each entry is the hostDetailData view-model map
// the shared host_detail_body sub-template renders.
@@ -484,8 +483,6 @@ func (s *Server) handleCustomerUnified(w http.ResponseWriter, r *http.Request, c
CSRFField: s.csrfField(r),
CSRFToken: s.csrfToken(r),
ScriptVersion: hostInstallVersion,
Hosts: hostViews,
}
+5 -4
View File
@@ -215,10 +215,11 @@ func TestTemplates_InstallGenerator(t *testing.T) {
t.Errorf("generator control missing: %s", id)
}
}
// targets the right script version + carries the client-side customer id
if !strings.Contains(html, hostInstallVersion) {
t.Errorf("ScriptVersion %s not rendered", hostInstallVersion)
}
// carries the client-side customer id. There is deliberately NO version assertion here: R-94
// deleted the rendered host-install version, because the hub cannot know which version a box
// will run (the script is fetched at run time). The assertion that used to sit here compared
// hostInstallVersion to itself and passed at any value — it was demonstrated green with the
// const set to "9.9.9" while the served script was 1.22.0.
if !strings.Contains(html, `data-customer-id="peti-felhom"`) {
t.Errorf("generator missing data-customer-id")
}
@@ -491,9 +491,10 @@
<section class="card">
<h2>Setup Command</h2>
<p class="text-muted" style="margin-bottom: 1rem; font-size: 0.85rem;">
Day-0 host bootstrap for host-install <strong>{{.ScriptVersion}}</strong>. Run on a
freshly-PVE-installed Proxmox <strong>host</strong> as root (create the customer in the
hub first). It enrolls the host, installs + verifies the agent, and provisions the guest;
Day-0 host bootstrap. The command always fetches the <strong>current</strong>
felhom-host-install.sh from felhom.eu &mdash; there is no version to pick here. Run it
on a freshly-PVE-installed Proxmox <strong>host</strong> as root (create the customer in
the hub first). It enrolls the host, installs + verifies the agent, and provisions the guest;
the in-guest controller then pulls its own <code>controller.yaml</code>. The retrieval
passphrase is entered at the no-echo prompt — never on the command line.
</p>
+1 -1
View File
@@ -125,7 +125,7 @@ spec:
spec:
containers:
- name: hub
image: gitea.dooplex.hu/admin/felhom-hub:0.86.0
image: gitea.dooplex.hu/admin/felhom-hub:0.91.1
ports:
- containerPort: 8080
name: http
+83 -3
View File
@@ -71,8 +71,12 @@ data:
# Host-install script. It lives at the repo's /scripts (outside the website doc-root),
# synced into .../current/scripts by git-sync (see the sparse-checkout ConfigMap). Served
# as text/plain so operators can inspect it in a browser before download-then-run.
# R-110: served from the INSTALLER TAG's tree, not the website's. The URL is unchanged
# (https://felhom.eu/scripts/felhom-host-install.sh) — it never carried a ref, so every
# producer of it (the bootstrap script, the hub's day-0 command) follows the tag with no
# edit. What changed is which tree this root points at.
location /scripts/ {
root /usr/share/nginx/html/current;
root /usr/share/nginx/scripts/current;
default_type text/plain;
}
@@ -213,8 +217,12 @@ metadata:
name: git-sync-sparse-checkout
namespace: felhom-system
data:
# R-110: TWO sparse-checkouts, because there are now two syncs with two different refs.
# The website tracks `main` (a copy edit must never need a release); /scripts/ tracks the
# installer TAG (pushing the installer must never publish it).
sparse-checkout: |
/website/
sparse-checkout-scripts: |
/scripts/
---
# ===================
@@ -246,6 +254,9 @@ spec:
- name: git-data
mountPath: /usr/share/nginx/html
readOnly: true
- name: git-data-scripts
mountPath: /usr/share/nginx/scripts
readOnly: true
- name: nginx-config
mountPath: /etc/nginx/conf.d/default.conf
subPath: default.conf
@@ -269,11 +280,14 @@ spec:
initialDelaySeconds: 3
periodSeconds: 10
# ── The WEBSITE sync — tracks `main`, unchanged cadence ──────────────────────────────
# Deliberately still a branch: the site is content, and a typo fix must reach felhom.eu in
# thirty seconds without cutting a release. Only /scripts/ moved to a tag (R-110).
- name: git-sync
image: registry.k8s.io/git-sync/git-sync:v4.4.0
args:
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
- --branch=main
- --ref=main
- --root=/git
- --link=current
- --period=30s
@@ -294,13 +308,52 @@ spec:
securityContext:
runAsUser: 65534 # nobody
# ── The INSTALLER sync — tracks a TAG (R-110, operator ruling 2026-08-03) ─────────────
# felhom-host-install.sh runs as root on a virgin machine. Before this it was served
# straight from `main`, so pushing it WAS publishing it: within thirty seconds it was what
# every new machine downloaded and ran, with no staging and no rollback but another push.
#
# Publishing is now moving this tag; rolling back is moving it back. PROVEN, not assumed:
# git-sync v4.4.0 follows a tag AND notices a moved one — measured 2026-08-03 on a
# throwaway sync against this very repo (`update required … local:<old> remote:<new>` →
# `updated successfully`, one period, ~20 s).
#
# Bump this ref when the installer's published version changes. `hostinstall_gates.py`
# gate 6 fails if this sync stops naming an `installer-v…` tag.
- name: git-sync-scripts
image: registry.k8s.io/git-sync/git-sync:v4.4.0
args:
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
- --ref=installer-v1.24.0
- --root=/git-scripts
- --link=current
- --period=30s
- --sparse-checkout-file=/etc/git-sync-scripts/sparse-checkout
volumeMounts:
- name: git-data-scripts
mountPath: /git-scripts
- name: sparse-checkout-scripts
mountPath: /etc/git-sync-scripts
resources:
requests:
memory: "32Mi"
cpu: "10m"
limits:
memory: "128Mi"
cpu: "100m"
securityContext:
runAsUser: 65534 # nobody
# Init container: wait for first sync before nginx starts
initContainers:
# BOTH trees are seeded before nginx accepts traffic. The second one is why /scripts/ has
# no 404 window across this change: a fresh pod does not become ready until the installer
# tag has been checked out, exactly as the website already worked.
- name: git-sync-init
image: registry.k8s.io/git-sync/git-sync:v4.4.0
args:
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
- --branch=main
- --ref=main
- --root=/git
- --link=current
- --one-time
@@ -312,16 +365,43 @@ spec:
mountPath: /etc/git-sync
securityContext:
runAsUser: 65534
- name: git-sync-scripts-init
image: registry.k8s.io/git-sync/git-sync:v4.4.0
args:
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
- --ref=installer-v1.24.0
- --root=/git-scripts
- --link=current
- --one-time
- --sparse-checkout-file=/etc/git-sync-scripts/sparse-checkout
volumeMounts:
- name: git-data-scripts
mountPath: /git-scripts
- name: sparse-checkout-scripts
mountPath: /etc/git-sync-scripts
securityContext:
runAsUser: 65534
volumes:
- name: git-data
emptyDir: {}
- name: git-data-scripts
emptyDir: {}
- name: nginx-config
configMap:
name: nginx-config
- name: sparse-checkout
configMap:
name: git-sync-sparse-checkout
items:
- key: sparse-checkout
path: sparse-checkout
- name: sparse-checkout-scripts
configMap:
name: git-sync-sparse-checkout
items:
- key: sparse-checkout-scripts
path: sparse-checkout
---
apiVersion: v1
kind: Service
+240
View File
@@ -1,3 +1,243 @@
## 1.25.0 — the off-site tier stops asking to prune (2026-08-04, R-191)
**A backup that worked must not report failure.** The off-site tier was written with `keep_last: 2`,
so every weekly run uploaded its snapshot successfully and then failed the whole job on a prune the
box's token is deliberately refused: `prune 'ct/9201': permission check failed - missing
Datastore.Modify|Datastore.Prune` → `TASK ERROR: job errors` → `whole_guest_backup_failed` in the
operator's inbox. Every week, on both boxes, about a backup that had already succeeded.
**R-89 moved off-site pruning SERVER-SIDE** — ep0 runs a per-namespace prune job and box tokens stay
write-only, so a box can never delete its own off-site history. The 2026-07-26 "two weeks" ruling was
not reversed; where it is ENFORCED moved, and this value did not follow. The tier now writes
`keep_last: 0`, which the agent's existing guard (`allowPBSPrune = !primary && keep_last > 0`) already
reads as "never prune from the box" — no agent change needed.
**VERIFIED BEFORE CHANGING IT** (read-only on ep0, 2026-08-04): prune jobs `prune-demo-felhom` and
`prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, schedule 03:30, keep-last 2,
and have run **every day since 2026-07-27 — 18 tasks, all `status=OK`**, the newest showing
`retention options: --ns demo-felhom --max-depth 0 --keep-last 2` and keeping exactly two. Disabling
the client-side prune without that check would have traded a weekly false alarm for unbounded growth.
A gate now asserts the off-site tier carries no client-side prune, so the value cannot drift back
quietly. The local tier's retention is untouched.
## 1.24.0 — a pre-existing backup target is granted too (2026-08-03, R-185)
**`configure_backup_target` has two arms and only one of them granted.** The Case A arm creates the
storage and calls `felhom-backup-target-apply grant` in the same breath — a box that builds its own
target has always been fine. The **Scenario-F arm** — *"the target already exists, leave it exactly as
it is"* — returned without ever granting.
So a box whose `felhom-backup` pre-dated the install (created by the vzdump-target-move runbook, or
surviving a reinstall) ended up with `local_backup_target: felhom-backup` while its token held
`FelhomAgentStore` on only `local`, `local-lvm` and `felhom-pbs`. Measured on **both** demo boxes
2026-08-03: the content API answers `{"data":[]}` through the agent's token while root lists three
archives. That tier was invisible to the agent and never restore-tested — and nothing said so,
because an empty listing is also what a brand-new tier returns.
The reuse arm now ensures the ACL through the same guarded wrapper, so both arms leave the box in the
same state. **Scenario F is unviolated:** the storage DEFINITION is still untouched — granting the
role the agent is supposed to have on the target this script is about to write into `agent.json` is
finishing the job, not retargeting the box. `pveum acl modify` is idempotent, so a box that already
has the grant is unchanged and a box whose token was rotated gets it back.
`$BACKUP_TARGET_ID` is deliberately **not** added to `PVE_STORAGES`, and the comment now says why: that
list is granted in step 4/5, before the target has been resolved in step 6, and `--acl-storages`
entries are preflight-checked for existence. The grant belongs with the resolution, which is where it
already was for a newly created target.
**A gate now asserts it** (`hostinstall_gates.py`): every arm of `configure_backup_target` that
resolves the target must also grant on it. Red-proved by reverting the reuse arm — `resolves the
backup target in 2 place(s) but grants in only 1`.
## v1.23.0 — the installer is published, not pushed (2026-08-03, R-110 + R-183)
**Two channels moved off `main` in the same change, because either one left behind makes the other
cosmetic.**
**Channel 1 — the served script.** `manifests/webpage.yaml` git-synced `/scripts/` from
`--branch=main` on a 30 s period and nginx served that working tree, so **pushing this file WAS
publishing it**: within half a minute it was what every new machine downloaded and ran as root, with
no staging and no rollback but another push. The sync is now **split in two**: the website keeps
tracking `main` at the same cadence (a copy edit must never need a release), and `/scripts/` tracks
the tag **`installer-v<SCRIPT_VERSION>`**. Publishing is moving that tag; rolling back is moving it
back.
**PROVEN, not assumed:** git-sync v4.4.0 follows a tag *and* notices a **moved** one — measured on a
throwaway sync against this repo, `update required … local:<old> remote:<new>` → `updated
successfully`, within one period (~20 s). The moved-tag half is what the whole publish model rests
on, so it was measured before the manifest was touched.
**Channel 2 — the sixteen files the installer fetches while it runs.** `fetch_raw` pulled from
`$AGENT_REPO/raw/branch/main`. It now pulls from **`raw/tag/v$ART_AGENT_VER`** — the agent version the
hub has vouched and whose binary sha this script already verifies.
**That is a correctness fix, not only a publish-channel one (→ R-183).** These are the AGENT's
configs — its systemd unit, its sudoers, its guarded wrappers — and a fresh install was fetching the
**vouched binary** while taking its configs from **whatever `main` held**. Two refs, one install, and
nothing compared them. The right ref for them was never this script's `SCRIPT_VERSION`: they do not
live in this repo and have no relationship to its version line.
**No fallback to a branch.** A vouched version whose tag is missing fails loudly rather than quietly
serving `main` — a silent fallback is the appearance of control with none of it. `felhom-agent`
carries `v<version>` tags from now on, `release-agent.sh` creates them, and `agent_gates.py` fails if
the vouched version is not downloadable.
**Channel 3 — the URL — needed no change, and that is worth recording rather than leaving as a
silence.** `https://felhom.eu/scripts/felhom-host-install.sh` never carried a ref: the ref lives in
the manifest. So both producers of that URL (`scripts/iso/felhom-bootstrap.sh`, the hub's day-0
command) follow the tag with no edit — **and no hub change, so no hub version bump.**
**Gate 6 in `hostinstall_gates.py`** pins all three structurally, with no network so it stays in
`--fast` and runs in CI on every push: no `raw/branch/` ref anywhere in the installer; `fetch_raw`
still pins to `$ART_AGENT_VER`; the manifest still syncs `/scripts/` from an `installer-v…` tag and
the website still from `main`.
**It deliberately does NOT assert "a tag exists for the current SCRIPT_VERSION".** That gate would go
red on the very push that bumps the version, before publishing — and publishing being a separate
deliberate act is the entire ruling. A gate that fails on the normal path is one people learn to
ignore.
## docs — v1.22.0 exercised end to end on two real reinstalls (2026-08-03, R-178) — **no script change**
**Nothing shipped.** `felhom-host-install.sh` stayed at **v1.22.0**; the published copy at
`https://felhom.eu/scripts/felhom-host-install.sh` was confirmed byte-identical to the repo copy
(`sha256 ed02acb2da46c8d2b5c486ce99d5b9a2747e8786c6eb03652cf755ed1abdd9f4`) before use. Both demo
boxes were uninstalled and reinstalled with it, by **two deliberately different supply paths**:
demo-hp with `--golden <local volid>` (the `:2584` alternative), demo-felhom with
`--force-gitea-golden` (the canonical C.3 customer command). The merge-aware `step_grows` produced
`data +46G (->70G, ONE volume)` and `+226G (->250G)` respectively, and `fetch_verify` was observed
succeeding against the vouched manifest for **both** artifacts on demo-felhom
(`verified sha256 a7763d31b55b5ce7…` agent, `verified sha256 54e2a4c431daf580…` golden).
**Two script-side findings, filed not fixed** (the session was a runbook; §7 forbade code):
- **R-180**`--archive-storage` is validated for existence (`:1583`) and for golden resolution
(`:1661`), but never against the ACL storage set it is about to grant (the fixed default
`local local-lvm felhom-pbs`). Staging the golden on `felhom-backup` therefore passed every
pre-flight gate and died at **step 8/8**: `HTTP 403: permission denied at /storage/felhom-backup
(missing privilege Datastore.AllocateSpace)` — *after* step 2 minted the token, step 4b **rotated
and vaulted root@pam**, and step 5 installed the agent. `ARCHIVE_STORAGE ∈ PVE_STORAGES` is a
one-line assertion over two variables both known at `:1583`.
- **R-179**`--uninstall` leaves the NAS network-storage systemd units behind
(`mnt-felhom\x2ddrives-<share>.{mount,automount}`; automount left `failed`, parent bind left
mounted). The Part E residue-diff provenance is from **v1.9.1**, which predates the feature — and
demo-felhom, which never had a share configured, left nothing, which is exactly why a diff on such
a box reported clean.
Full evidence: root `REPORT.md`.
## host-install: one data volume, derived from the disk (2026-08-03, R-165)
**Forced by a census, not planned.** `felhom-agent` v0.120.0 merges the appliance's two data volumes
into one (decision D-a). `step_grows` computed **two** numbers and the install call passed both, so
this script had to change with the agent or every install would have provisioned a half-sized box.
- **`step_grows` computes ONE total.** The old 80/20 docker-vs-sysdata split is summed: `226` where it
was `184 + 42`, `106` where it was `84 + 22`, `46` where it was `34 + 12`. **A standard appliance
keeps exactly the capacity it had — 250 G — it is simply no longer split by a wall.**
- **The size still comes from the physical disk.** `step_grows` already read the thin pool's real free
space (`lvs /dev/pve/data`); the merge only collapsed its two outputs into one. This is what makes
the merge safe to ship: an unflagged install does **not** get the golden's 24 G base.
- **`--sysdata-grow` is DEPRECATED but still honoured.** It is no longer auto-computed (set to 0), and
a hand-passed value still counts because the agent **folds** it into the single volume's grow rather
than dropping it — so an operator reproducing an old command line gets the same total.
## CI — a Gitea Actions runner, and a red run that reaches a person (2026-08-02, R-168)
**No version bump anywhere: nothing in the product repos is compiled, built or deployed by this.**
Recorded explicitly so the omission reads as a decision rather than a miss.
**What this closes.** Session 1 (same day) gave every repo one gate entry point and a
`.githooks/pre-push` that refuses a failing push. That hook is per-clone and `--no-verify` skips it,
so nothing independent of the person pushing ever saw whether the gates passed. This is the
independent half, and with it **R-29 CLOSES** — on the demonstrated alarm, not on a green run.
**`.gitea/workflows/gates.yml` (new)** — triggers on `push`, `runs-on: felhom-gates`, obtains the
source with a shallow `git fetch` of the **exact pushed SHA** from the in-cluster Gitea Service, and
runs `scripts/repo_gates.py --fast` and nothing else. **No `uses:` step anywhere** — JavaScript
actions need a node runtime the host-mode runner does not have, and probe P3 measured that a plain
`git fetch` is sufficient and lands on the pushed commit. No `|| true`; the entry point's exit code
IS the job's result.
**The alarm, which is the half that matters.** Probe P5 measured that a failed run produces **no
mail, no notification row and no log line** from Gitea. A red tick in a web UI nobody watches is
exactly the defect R-29 filed, rebuilt one layer up — so the workflow sends its own email on failure
via Resend (the hub's existing transactional path) and **prints the provider's accepted id**, making
"a message left the machine" an observable. **Demonstrated, not asserted:** a deliberately broken
commit pushed with `--no-verify` produced run #6 `failure` and
`RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`.
Two traps found while building it, both worth keeping because each looks like something else:
the runner image has **no `curl`** on purpose (python3 and git only — so the step uses `urllib`
rather than growing the image), and `api.resend.com` sits behind **Cloudflare, which 403s the default
`Python-urllib` User-Agent with error 1010** — a failure that reads exactly like an auth failure and
is not one.
**The standing limit, written into the workflow itself: it REPORTS, it cannot REFUSE.** Every repo
pushes straight to `main` with no pull request, so there is no merge for a status check to stand at.
That is not a gap in the runner; there is no gate in the road. Making it blocking needs branch
protection plus a PR workflow, which changes how the operator works → **R-169**, waiting on them.
**`documentation/audits/SPIKE-ci-runner-2026-08-02.md` (new)** — all six probes, method, measurement
and ruling; none produced a STOP. Also records a near-miss worth more than the probes: a `| tail -5`
inside my own census query silently dropped rows and looked exactly like a baseline drift big enough
to change the task. **An instrument that can drop results silently is not a measurement.**
**`CLAUDE.md`** gains the matching rule from session 1's red-proofing: a `go test -run` pattern that
matches no test prints `ok` and exits 0, so a red-proof using `-run` must first prove the filter
matched something.
**`CONTEXT.md`** gains S-8 (CI detects, does not block, and why that is structural), S-9 (a detector
that tells no one is not finished), S-10 (the runner is unprivileged because DooPlex is Tier 2), and
S-11 (CI reproduces the workspace's sibling layout, because two entry points depend on it).
## Gate enforcement — one entry point per repo, and a pre-push hook (2026-08-02)
**No version bump: `scripts/` carries no version, and this is tooling.** Recorded explicitly so the
omission reads as a decision rather than a miss.
**The census that started it.** Thirteen gate scripts exist across the four felhom repos. A full run
on 2026-08-02 found one clean correlation: **every check a `CLAUDE.md` tells a person to run was
passing, and two of the four nobody is told to run were failing** — `hostinstall_gates.py` since
14 July, and `reuse_refs_check.py` on all four repos. Both failures were harmless in effect, which
was checked line by line; nothing would have said so if they had not been.
**`scripts/repo_gates.py` (new)** — THE entry point for this repo. Runs `site_gates`,
`hostinstall_gates`, `hub_confirm_gate`, `manifest_bearer_gate` and `reuse_refs_check` on this root,
streams each gate's own output, exits worst-wins non-zero, and reports exit 2 distinctly as
INCONCLUSIVE. **A missing gate script is a FAILURE and prints the path tried** — fail-closed, because
a runner that quietly skips a gate is the inert-seam failure this project has shipped four times. It
copies `app-catalog-felhom.eu/scripts/catalog_gates.py` (R-161), **not** `site_gates.py`, which is a
gate and not a runner — copying that would have produced a ninth monolith.
**`scripts/reuse_refs_check.py` — resolution taught, not loosened.** RED on all four repos with 13
findings, of which a hand audit found **zero** genuine drift: twelve were package shorthand whose
file sits a couple of directories deeper, and `wgsync/reconciler.go`, cited by the controller, lives
in the hub. `REUSE.md` cites by package shorthand and across repos on purpose; the tool was wrong.
New order, first hit wins: exact → suffix → ambiguous (real citation, imprecise shorthand — not a
failure) → sibling repo (as-is, or with the sibling's own name stripped off the front) → FAIL.
**Every non-exact hit is printed** and every root prints a per-rule tally, because "0 failures" alone
cannot tell a working checker from a blind one. A failure lists every resolution attempted. Evidence
trees (`audits/`, `documentation/tests/`) are excluded from the suffix index — a copy of a file is
not the file. An absent sibling is never a failure; an unreadable parent says so and continues.
Result: 13/13 resolve, all four roots exit 0.
**`scripts/test_reuse_refs_check.py` (new, 13 tests)** — one per resolution row plus the kill
condition. Red-proof: making `resolve()` return `exact` for an unresolvable token turns four of them
red. **`scripts/test_repo_gates.py` (new, 3 tests)** — a SEAM test asserting each member gate's own
distinctive stdout, never the runner's summary line; red-proofed with an inert `run_gate` that still
prints "all felhom.eu gates OK" and exits 0.
**`.githooks/pre-push` (new)** — runs `repo_gates.py --fast` and refuses the push. Its honest limits
are written into the hook itself: it is **per-clone** (`core.hooksPath` is local config; arm with
`git config core.hooksPath .githooks`, and any manual entry-point run WARNS when a clone is unarmed)
and **`git push --no-verify` bypasses it on purpose** — an escape hatch that cannot be reached is one
that gets removed the first time it is inconvenient; using it must be stated in the session report.
Measured on git 2.47.3: a relative `core.hooksPath` resolves correctly and the hook's cwd is the repo
root whether `git push` is issued from the root or any subdirectory. The half that is neither
per-clone nor skippable is CI — now tracked as R-168.
## ISO v1.26.1 — the PUBLIC installer ISO, PUBLISHED (2026-07-31)
**Live at `https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso`**
+103 -18
View File
@@ -115,8 +115,8 @@
# (default: appliance → island 169.254.253.1:8443; byo → vmbr0 IP:8443)
# --no-island appliance only: keep the historical LAN bind instead of the R-50 island
# --rootfs-grow N grow OS rootfs by N GiB (default: auto-compute)
# --datavol-grow N grow Docker-data vol by N GiB (default: auto-compute)
# --sysdata-grow N grow user-data vol by N GiB (default: auto-compute)
# --datavol-grow N grow the single data volume by N GiB (default: auto-compute from the pool)
# --sysdata-grow N DEPRECATED (R-165): added to --datavol-grow; there is one volume now
#
# Guest cap (appliance: optional — protect a SHARED host's other guests; byo: BOTH REQUIRED —
# the only noisy-neighbor protection on a host you do not own; needs agent >= v0.52.0):
@@ -184,9 +184,11 @@
set -euo pipefail
SCRIPT_VERSION="1.22.0" # the SINGLE version source (F-1): -h, the run banners, and the hub
# Setup-tab copy (hub internal/web/configs.go hostInstallVersion —
# scripts/hostinstall_gates.py asserts the two stay equal) all follow it.
SCRIPT_VERSION="1.25.0" # the SINGLE version source (F-1): -h and the run banners follow it.
# The hub used to carry a copy for its Setup tab; R-94 DELETED it
# (2026-08-02) because the hub cannot know which version a box runs —
# the Setup command fetches this script at run time. scripts/
# hostinstall_gates.py now asserts the hub carries NO version literal.
# 1.16.0: the FELHOM_ESCROW sudoers alias (controller-driven escrow
# ceremony, agent v0.88.0) ships on every install/update — it rides the
# CANONICAL sudoers fetch below (configs/felhom-agent.sudoers from the
@@ -305,6 +307,14 @@ PVE_PRIVS_STORE="Datastore.Allocate Datastore.AllocateSpace"
PVE_PRIVS_BASE="Sys.Audit SDN.Use Datastore.Audit"
# Storages the agent reads/writes (archive+dump=local, restore=local-lvm, offsite DR=felhom-pbs). The
# offsite felhom-pbs MUST be included or the agent's DR backup 403s (SPIKE residual #1). --acl-storages overrides.
#
# `$BACKUP_TARGET_ID` is deliberately NOT in this list, and the reason matters because the obvious
# tidy-up is to add it (R-185, 2026-08-03). This list is granted in STEP 4/5, before
# configure_backup_target has run in STEP 6 — so at this point the target may not exist yet, and
# --acl-storages entries are preflight-checked for existence. The target's grant therefore belongs
# with the target's RESOLUTION, where it already is for a freshly created one, and now also for a
# pre-existing one. Adding it here would grant on a storage that may not exist and would still leave
# the resolution path as the single owner of that decision, split across two places.
PVE_STORAGES=(local local-lvm felhom-pbs)
# E-2: the whole-guest backup target storage id, and what configure_backup_target resolved to.
# BACKUP_TARGET_RESOLVED feeds agent.json's local_backup_target; "local" means DEGRADED (Case B).
@@ -490,12 +500,31 @@ fetch_verify() {
# exists, else anonymous). These are non-executable text (not the integrity-checked binary); the
# sudoers is `visudo -cf`-validated before install, which catches corruption/tampering that would
# matter. $1=repo-path $2=dest
#
# R-110 / R-183: PINNED TO THE AGENT VERSION BEING INSTALLED, never to a branch.
#
# These sixteen files are the AGENT's configs — its systemd unit, its sudoers, its guarded wrappers —
# so the ref that is correct for them is the agent version this run is installing, which the hub has
# vouched and whose binary sha this script verifies. It is NOT the installer's own SCRIPT_VERSION:
# these files do not live in the installer's repo and have no relationship to its version line.
#
# Before this they came from `raw/branch/main`, which is a REAL SKEW and not only a publish-channel
# defect (R-183): a fresh install fetched the vouched agent BINARY while taking its unit file and
# sudoers from whatever `main` happened to hold — two refs, one install, and nothing compared them.
#
# NO FALLBACK TO A BRANCH. A vouched version whose tag is missing must fail loudly here rather than
# quietly serving `main`, because a silent fallback is exactly the "appearance of control with none of
# it" this change exists to remove. `agent_gates.py`'s published-version gate keeps the tag and the
# vouched version in step, so this die is a backstop and not the primary control.
fetch_raw() {
local path="$1" dest="$2"
# Late steps (mgmt-watchdog, OOB) can run without step 5 having resolved the manifest.
[[ -n "$ART_AGENT_VER" ]] || resolve_artifacts
[[ -n "$ART_AGENT_VER" ]] || die "cannot pin $path: no agent version resolved from the hub manifest"
local -a _auth; _git_auth_args _auth
curl -fsS "${_auth[@]}" -o "$dest" \
"$GITEA_BASE/$GITEA_OWNER/$AGENT_REPO/raw/branch/main/$path" \
|| die "raw fetch failed: $path"
"$GITEA_BASE/$GITEA_OWNER/$AGENT_REPO/raw/tag/v$ART_AGENT_VER/$path" \
|| die "raw fetch failed: $path (agent tag v$ART_AGENT_VER — is that version tagged in $AGENT_REPO?)"
[[ -s "$dest" ]] || die "raw fetch empty: $path"
}
@@ -631,8 +660,31 @@ configure_backup_target() {
# existing id is an error, and repointing a live target is exactly the silent retarget this
# whole arc closes.
if pvesm status --storage "$BACKUP_TARGET_ID" >/dev/null 2>&1; then
log_skip " backup target '$BACKUP_TARGET_ID' already exists — leaving it exactly as it is (Scenario F)"
log_skip " backup target '$BACKUP_TARGET_ID' already exists — leaving its DEFINITION exactly as it is (Scenario F)"
BACKUP_TARGET_RESOLVED="$BACKUP_TARGET_ID"
# R-185: …but STILL ensure the ACL. "The storage already exists" says nothing about whether
# the agent may READ it, and this early return is where the two came apart.
#
# THE DEFECT THIS CLOSES, measured on both demo boxes 2026-08-03. The CASE A path below
# creates the storage and grants in the same breath, so a box that built its own target is
# fine. A box whose target ALREADY existed — created by the vzdump-target-move runbook, or
# surviving a reinstall — returned here and never granted. The result: `local_backup_target`
# pointed at `felhom-backup` while the token held FelhomAgentStore only on local, local-lvm
# and felhom-pbs, so the API answered `{"data":[]}` for that storage while root saw three
# archives. The tier was invisible to the agent and never restore-tested, and nothing said so
# — because an empty listing is also what a brand-new tier returns.
#
# Scenario F is UNVIOLATED: the storage definition is still untouched. Granting a role the
# agent is supposed to have on the target this same script is about to write into
# agent.json is not "touching the box's target", it is finishing the job. `pveum acl modify`
# is idempotent, so a box that already has the grant is unchanged, and a box that had its
# token rotated gets it back.
if $DRY_RUN; then
log_dry "felhom-backup-target-apply grant $BACKUP_TARGET_ID # R-185: ACL on a pre-existing target"
else
/usr/local/sbin/felhom-backup-target-apply grant "$BACKUP_TARGET_ID" \
|| die "backup target grant failed on the pre-existing target — the agent could not read its own backup tier (R-185)"
fi
return 0
fi
local mp
@@ -1770,23 +1822,34 @@ step_token() {
#-------------------------------------------------------------------------------
step_grows() {
log_step "3/8 compute volume grows"
# Golden base: rootfs 32G + Docker-data 16G + user-data 8G (build-golden.sh).
# Golden base since build-golden.sh v3.0.0 (R-165): rootfs 32G + ONE data volume 24G. The separate
# 8G user-data volume was MERGED AWAY — one volume, one free-space figure, no ceiling — so there is
# one number to compute here instead of two.
#
# THE SIZE IS DERIVED FROM THE PHYSICAL DISK, which is what makes the merge safe to ship: an
# unflagged install does NOT get the golden's 24G, it gets a share of the thin pool's real free
# space. (Before R-165 this same block already did the deriving; the merge only collapsed its
# 80/20 docker-vs-sysdata split into a single total.)
if [[ -z "$ROOTFS_GROW$DATAVOL_GROW$SYSDATA_GROW" ]]; then
local free_gib
free_gib=$(lvs --noheadings --units g -o lv_size,data_percent /dev/pve/data 2>/dev/null | awk '{gsub(/[^0-9.]/,"",$1); used=$2; print int($1*(100-used)/100)}' 2>/dev/null || echo 0)
# Reserve headroom; split the rest ~ docker 80% / sysdata 20%; rootfs stays golden.
# Reserve headroom; the totals below are the pre-merge pair SUMMED, so an appliance gets the
# same capacity it did before — it is simply no longer split by a wall.
ROOTFS_GROW=0
if [[ "${free_gib:-0}" -ge 300 ]]; then
DATAVOL_GROW=184; SYSDATA_GROW=42 # reproduces the standard 200G/50G appliance
DATAVOL_GROW=226 # 184+42 -> the standard 250G appliance (was 200G+50G)
elif [[ "${free_gib:-0}" -ge 150 ]]; then
DATAVOL_GROW=84; SYSDATA_GROW=22
DATAVOL_GROW=106 # 84+22
else
DATAVOL_GROW=34; SYSDATA_GROW=12 # minimal floors
DATAVOL_GROW=46 # 34+12 — minimal floor
fi
log_info " auto-computed from ~${free_gib} GiB free"
SYSDATA_GROW=0
log_info " auto-computed from ~${free_gib} GiB free (ONE volume since R-165)"
fi
ROOTFS_GROW="${ROOTFS_GROW:-0}"; DATAVOL_GROW="${DATAVOL_GROW:-0}"; SYSDATA_GROW="${SYSDATA_GROW:-0}"
log_info " grows: rootfs +${ROOTFS_GROW}G (->$((32+ROOTFS_GROW))G), docker +${DATAVOL_GROW}G (->$((16+DATAVOL_GROW))G), sys_drive +${SYSDATA_GROW}G (->$((8+SYSDATA_GROW))G)"
# A hand-passed --sysdata-grow is still ACCEPTED and still counts: the agent folds it into the one
# volume (bringup.go 4b), so an operator reproducing an old command line gets the same total.
log_info " grows: rootfs +${ROOTFS_GROW}G (->$((32+ROOTFS_GROW))G), data +$((DATAVOL_GROW+SYSDATA_GROW))G (->$((24+DATAVOL_GROW+SYSDATA_GROW))G, ONE volume)"
_state_mark grows
}
@@ -2416,8 +2479,30 @@ for _k,_v in {"unit_dir":"/etc/systemd/system","stage_dir":"/var/lib/felhom-agen
base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_seconds":15,"known_refresh_seconds":20})
# R-82: local DAILY + offsite WEEKLY. The two tiers carry SEPARATE cadences and retentions —
# keep_last=3 is three DAYS on the daily tier and three WEEKS on a weekly one, so one shared knob
# would guarantee that one of them is wrong. keep_last=2 on the offsite tier = two weeks (operator
# ruling 2026-07-26).
# would guarantee that one of them is wrong.
#
# THE OFFSITE TIER CARRIES NO CLIENT-SIDE RETENTION, AND THAT IS THE POINT (R-191, 2026-08-04).
# It used to be written `keep_last: 2` ("two weeks", operator ruling 2026-07-26). **R-89 then moved
# offsite pruning SERVER-SIDE** — ep0 runs a per-namespace prune job and box tokens stay write-only,
# deliberately, so that a box can never delete its own offsite history. The 2026-07-26 ruling was not
# reversed; where it is ENFORCED moved, and this value did not follow.
#
# The consequence was weekly and wrong in the worst direction: vzdump UPLOADED the snapshot fine and
# then failed the whole job on the prune the token is refused —
# `prune 'ct/9201': permission check failed - missing Datastore.Modify|Datastore.Prune`
# → `TASK ERROR: job errors` → `whole_guest_backup_failed` in the operator's inbox. Every week, on
# both boxes, about a backup that had already succeeded. A tier that cries wolf weekly is a tier
# whose real failure nobody will see.
#
# VERIFIED BEFORE CHANGING IT (2026-08-04, read-only on ep0): prune jobs `prune-demo-felhom` and
# `prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, schedule 03:30, keep-last 2,
# and have run EVERY DAY since 2026-07-27 — 18 tasks, all `status=OK`, the newest showing
# `retention options: --ns demo-felhom --max-depth 0 --keep-last 2` and keeping exactly two. Retention
# happens; it happens THERE. **If that ever stops being true, this zero is unbounded growth** — check
# ep0's prune jobs before assuming the offsite tier is retained.
#
# `keep_last: 0` means "never prune from the box" and is the value the agent's own guard reads
# (allowPBSPrune = !primary && keep_last > 0), so no agent change is needed to honour it.
#
# The offsite tier is written even though `felhom-pbs` does not exist yet: that storage appears only
# when the hub provisions the DR tier. The agent DEFERS a tier whose target storage is absent
@@ -2428,7 +2513,7 @@ base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_sec
# setdefault: an EXISTING box's backup block is preserved WHOLE. Upgrades never gain the tier here —
# they are migrated explicitly (R-82 Slice D.2), so an in-place upgrade can never silently start
# writing to an offsite datastore.
base.setdefault('backup', {"local_backup_target":os.environ.get('BACKUP_TARGET_RESOLVED','local'),"local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0,"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":2}]})
base.setdefault('backup', {"local_backup_target":os.environ.get('BACKUP_TARGET_RESOLVED','local'),"local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0,"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":0}]})
base.setdefault('local_api', {})
base['local_api'].setdefault('enable', True)
base['local_api']['listen_addr'] = os.environ['BRIDGE_ADDR']
+166 -17
View File
@@ -5,9 +5,17 @@ drill-swept findings (DRILL-day0-vm-2026-07-12 F-1/F-7/F-9/F-10 + the ACL-narrow
Run from the repo root: python scripts/hostinstall_gates.py
Gates (all must pass; non-zero exit on any failure):
1. version exactly ONE version source: SCRIPT_VERSION exists, the header line carries
no version literal, and the hub Setup-tab const (hub internal/web/configs.go
hostInstallVersion) equals SCRIPT_VERSION (F-1 structural fix)
1. version exactly ONE version source: SCRIPT_VERSION exists, the header line carries no
version literal, and **the hub carries no host-install version literal at all**.
The third assertion inverted on 2026-08-02 (R-94): it used to require the hub's
`hostInstallVersion` const to EQUAL SCRIPT_VERSION, which is unachievable
honestly the Option-1 install command downloads felhom-host-install.sh from
the website at RUN TIME and the website git-syncs `main` every 30 seconds
(R-110), so the hub cannot know which version a given box will run. A
build-time literal there is a guess with a version number's authority, and the
real one drifted to 1.19.0-vs-1.22.0 and stayed wrong for 19 days. The label was
deleted rather than derived; this gate now pins its absence (F-1 structural fix,
second form).
2. age the `age` package is installed by the agent-install step (F-10)
3. pbs-apply configs/felhom-pbs-apply is fetched + installed to
/usr/local/sbin/felhom-pbs-apply (F-7), and the uninstall removes it
@@ -54,21 +62,49 @@ if re.search(r'felhom-host-install\.sh\s+v\d+\.\d+\.\d+', header):
else:
ok("header has no version literal")
# hub Setup-tab const must equal SCRIPT_VERSION (the copy the drill found at 1.12.0).
if os.path.exists(HUB_CONFIGS) and script_ver:
with io.open(HUB_CONFIGS, "r", encoding="utf-8") as f:
hub_src = f.read()
hm = re.search(r'hostInstallVersion\s*=\s*"(\d+\.\d+\.\d+)"', hub_src)
if not hm:
fail("hub hostInstallVersion const not found in internal/web/configs.go")
elif hm.group(1) != script_ver:
fail("hub Setup-tab hostInstallVersion=%s != SCRIPT_VERSION=%s (F-1: bump both together)"
% (hm.group(1), script_ver))
else:
ok("hub Setup-tab hostInstallVersion matches (%s)" % hm.group(1))
# The hub must carry NO host-install version literal at all (R-94, 2026-08-02). It cannot know
# which version a box will run — the Option-1 command fetches the script from the website at run
# time and the website git-syncs `main` every 30s. The const this replaced said 1.19.0 while the
# served script was 1.22.0, and had been wrong since 2026-07-14.
#
# Matched in CODE SHAPES, never as bare prose: the deleted declarations, the struct field, the
# assignment and the template action, plus a rename-proof generic form of each. Comments are
# deliberately NOT stripped (a `//` inside a URL string literal would truncate the scan and turn
# this gate blind); a comment that merely NAMES the identifier is allowed, and configs.go carries
# exactly such a note explaining the absence.
BANNED = [
(r'\bconst\s+hostInstallVersion\b', "const hostInstallVersion"),
(r'\bhostInstallVersion\s*=', "hostInstallVersion assignment"),
(r'(?i)\bconst\s+\w*hostinstall\w*version\b', "a renamed host-install version const"),
(r'\bScriptVersion\s+string\b', "ScriptVersion struct field"),
(r'\bScriptVersion\s*:', "ScriptVersion struct assignment"),
(r'\{\{\s*\.ScriptVersion\s*\}\}', "{{.ScriptVersion}} template action"),
]
HUB_DIR = os.path.join(ROOT, "hub")
if not os.path.isdir(HUB_DIR):
fail("hub/ not found at %s — cannot assert the absence of a host-install version literal" % HUB_DIR)
else:
if not os.path.exists(HUB_CONFIGS):
fail("hub/internal/web/configs.go not found — cannot cross-check the Setup-tab version")
scanned, hits = 0, 0
for dirpath, dirs, files in os.walk(HUB_DIR):
dirs[:] = [d for d in dirs if d not in (".git", "vendor", "node_modules")]
for fn in files:
if not (fn.endswith(".go") or fn.endswith(".html")):
continue
fp = os.path.join(dirpath, fn)
scanned += 1
with io.open(fp, "r", encoding="utf-8") as f:
for lineno, line in enumerate(f, 1):
for pat, what in BANNED:
if re.search(pat, line):
hits += 1
fail("%s:%d carries %s — the hub must render NO host-install version "
"(R-94: the served script is fetched at run time, so no build-time "
"value can be true). Single source: scripts/felhom-host-install.sh "
"SCRIPT_VERSION. Line: %s"
% (os.path.relpath(fp, ROOT), lineno, what, line.strip()[:120]))
if not hits:
ok("hub carries no host-install version literal (%d .go/.html files scanned, %d shapes checked)"
% (scanned, len(BANNED)))
# ── 2. age package (F-10) ───────────────────────────────────────────────────────
# must match the REAL install invocation, not the log_dry echo (red-proof-hardened twice:
@@ -110,6 +146,119 @@ if re.search(r'^PVE_STORAGES=\([^)]*felhom-pbs[^)]*\)', src, re.M):
else:
fail("felhom-pbs missing from the default PVE_STORAGES — narrowing it 403s the PBS-DR apply-bridge")
# ── 6. the publish channel is pinned, not floating (R-110 / R-183) ──────────────
#
# WHAT THIS ASSERTS, AND WHAT IT DELIBERATELY DOES NOT.
#
# It does NOT assert "a tag exists for the current SCRIPT_VERSION". That gate would fail the very
# push that bumps SCRIPT_VERSION, before publishing has happened — and publishing being a SEPARATE
# deliberate act is the whole point of R-110's ruling. A gate that goes red on the normal path is a
# gate people learn to ignore, which is the reasoning the task's own §8.4 applies to the agent-side
# gate; it applies here identically. "Is the vouched version actually downloadable" is a real
# invariant and it lives where a missing artifact genuinely breaks day-0 — `felhom-agent`'s
# `agent_gates.py`, which has the network access to answer it.
#
# What it asserts instead are the two STRUCTURAL regressions that would silently return the
# installer to a floating channel, both answerable by reading files (no network, so this stays in
# `--fast` and therefore runs in CI on every push):
#
# 6a. no `raw/branch/` ref anywhere in the installer — one of the sixteen agent-config fetches
# slipping back to `main` is exactly how a channel stays floating unnoticed, and it is
# invisible in a diff that touches one line.
# 6b. `fetch_raw` still pins to the resolved agent version — the positive form, so the mechanism
# cannot be quietly deleted rather than regressed.
# 6c. the website manifest still syncs `/scripts/` from a TAG ref and the website from `main` —
# the split is the deploy-side half of the same channel, and reverting it is one word.
branch_refs = [l for l in lines if "raw/branch/" in l and not l.lstrip().startswith("#")]
if branch_refs:
fail("installer still fetches from a BRANCH ref — the run-time channel is floating again "
"(R-110/R-183). Offending line(s): %s" % "; ".join(l.strip()[:90] for l in branch_refs))
else:
ok("no raw/branch/ ref in the installer — every run-time fetch is pinned")
if re.search(r'raw/tag/v\$ART_AGENT_VER/', src):
ok("fetch_raw pins the agent configs to the vouched agent version")
else:
fail("fetch_raw no longer pins to $ART_AGENT_VER — the agent's configs and its binary can "
"again come from different refs in one install (R-183)")
WEBPAGE = os.path.join(ROOT, "manifests", "webpage.yaml")
try:
with io.open(WEBPAGE, "r", encoding="utf-8") as f:
wp = f.read()
except IOError as e:
fail("cannot read manifests/webpage.yaml to check the publish channel: %s" % e)
wp = None
if wp is not None:
# The scripts sync must name a tag ref; the website sync must still track main.
if re.search(r'--ref=installer-v', wp):
ok("manifest: /scripts/ syncs from an installer tag")
else:
fail("manifests/webpage.yaml has no `--ref=installer-v…` sync — /scripts/ is not served "
"from a tag, so pushing the installer publishes it again (R-110)")
if re.search(r'--(branch|ref)=main', wp):
ok("manifest: the website still tracks main (a copy edit must not need a release)")
else:
fail("manifests/webpage.yaml no longer tracks main for the website — pinning the SITE to "
"the installer tag turns every copy edit into a release")
# ── R-185: every path that RESOLVES the backup target must also grant on it ──────────────────
#
# THE DEFECT THIS WOULD HAVE CAUGHT, measured on both demo boxes 2026-08-03. `configure_backup_target`
# has two arms. The CASE A arm creates the storage and grants in the same breath. The Scenario-F arm —
# "the target already exists, leave it alone" — returned WITHOUT granting, so a box whose target
# pre-dated the install pointed `local_backup_target` at a storage its own token could not read. The
# API answered `{"data":[]}` while root saw three archives, and nothing said so, because an empty
# listing is also what a brand-new tier returns.
#
# The assertion is deliberately about the FUNCTION, not about PVE_STORAGES: the target's grant belongs
# with the target's resolution (PVE_STORAGES is granted a step earlier, before the target exists), so
# what must hold is that no arm of that function can resolve a target and skip the grant.
fn = re.search(r'^configure_backup_target\(\)\s*\{(.*?)^\}', src, re.S | re.M)
if not fn:
fail("cannot find configure_backup_target() — the backup-target ACL assertion cannot run, and a "
"check that cannot run must never report OK (R-185)")
else:
body = fn.group(1)
resolutions = len(re.findall(r'BACKUP_TARGET_RESOLVED="\$BACKUP_TARGET_ID"', body))
grants = len(re.findall(r'felhom-backup-target-apply grant', body))
if resolutions == 0:
fail("configure_backup_target no longer resolves BACKUP_TARGET_ID anywhere — re-read it")
elif grants >= resolutions:
ok("every arm that resolves the backup target also grants on it (%d resolution(s), %d grant(s))"
% (resolutions, grants))
else:
fail("configure_backup_target resolves the backup target in %d place(s) but grants in only %d "
"— an arm resolves a target the agent may not READ. That is R-185: the tier's archives are "
"invisible to the agent, it is never restore-tested, and an empty listing looks exactly "
"like a brand-new tier." % (resolutions, grants))
# ── R-191: the OFFSITE tier must not arm a client-side prune ─────────────────────────────────
#
# R-89 moved offsite pruning SERVER-SIDE — ep0 runs a per-namespace prune job and box tokens stay
# write-only, so the box is REFUSED if it asks. When this default was `keep_last: 2` the effect was a
# weekly lie: vzdump uploaded the snapshot, then failed the whole job on the prune, and the operator
# was told the offsite backup had failed when it had succeeded.
#
# The assertion is on the OFFSITE entry only. The local tier's `local_backup_retention` is untouched
# and must stay untouched — it prunes correctly and is allowed to.
m = re.search(r'"backup_targets":\s*\[(.*?)\]', src, re.S)
if not m:
fail("cannot find backup_targets in the rendered agent.json defaults — the offsite-retention "
"assertion cannot run, and a check that cannot run must never report OK (R-191)")
else:
targets = m.group(1)
kl = re.search(r'"keep_last"\s*:\s*(\d+)', targets)
if not kl:
fail("the offsite backup_target carries no keep_last at all — expected an explicit 0 "
"(R-191: 0 means 'never prune from the box'; absent is not the same statement)")
elif kl.group(1) != "0":
fail("the offsite backup_target arms a CLIENT-SIDE prune (keep_last=%s). R-89 moved offsite "
"pruning server-side to ep0 and box tokens are write-only, so every weekly run will "
"upload successfully and then FAIL the job on a refused prune (R-191)." % kl.group(1))
else:
ok("the offsite tier arms no client-side prune (keep_last=0; retention is ep0's prune jobs)")
print()
if fails:
print("hostinstall gates: %d FAILURE(S)" % len(fails))
+128
View File
@@ -0,0 +1,128 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""repo_gates.py — THE entry point for this repo's gates. Run from the repo root:
python3 scripts/repo_gates.py # every gate
python3 scripts/repo_gates.py --fast # only gates that touch no network and no container
# runtime (what .githooks/pre-push runs)
Gates, in order (all must pass; **non-zero exit on any failure**):
1. site website HTML: BOM, emoji, nav/footer, analytics, CDN, tokens, cache-busting
2. hostinstall felhom-host-install.sh's five drill-swept invariants (+ R-94's absent-version)
3. hub-confirm no native confirm()/prompt() in hub templates
4. manifest-bearer no bearer-shaped literal anywhere in manifests/
5. reuse-refs every path cited by this repo's REUSE.md still resolves
WHY THIS FILE EXISTS (2026-08-02, closing R-29 leg (a) and half of leg (b)).
A census of all thirteen gate scripts across the four felhom repos found one clean correlation:
**every check a CLAUDE.md tells a person to run was passing, and two of the four nobody is told
to run were failing** one since 14 July. Neither failure was harmful in effect, which was
checked line by line; nothing would have said so if they had been. The fix is not more gates, it
is one place to run them from. `app-catalog-felhom.eu/scripts/catalog_gates.py` is the canonical
shape (R-161) and this copies it deliberately rather than inventing a second one.
`site_gates.py` is a GATE eight assertions in one file and is NOT the model for this file. A
runner that invokes separate gates is the shape that survives; copying site_gates would just add
a ninth monolith.
FAIL-CLOSED. A gate script that is missing is a FAILURE, never a skip, and the exact path tried
is printed. A runner that quietly drops a gate is the inert-seam failure this project has shipped
four times.
EXIT CODES. Each gate returns 0 clean / 1 convicted / 2 inconclusive. This runner exits non-zero
if any gate is non-zero, and reports 2 distinctly as INCONCLUSIVE an undetermined result is
never a pass, but it is not a conviction either, and the operator needs to know which they have.
"""
import os
import subprocess
import sys
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
SCRIPTS = os.path.join(ROOT, "scripts")
# (label, absolute script path, args, fast)
GATES = [
("site", os.path.join(SCRIPTS, "site_gates.py"), [], True),
("hostinstall", os.path.join(SCRIPTS, "hostinstall_gates.py"), [], True),
("hub-confirm", os.path.join(SCRIPTS, "hub_confirm_gate.py"), [], True),
("manifest-bearer", os.path.join(SCRIPTS, "manifest_bearer_gate.py"), [], True),
("reuse-refs", os.path.join(SCRIPTS, "reuse_refs_check.py"), [ROOT], True),
]
VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"}
def hooks_armed_note(root):
"""Print a WARNING (never a failure) when this clone's pre-push hook is not switched on.
core.hooksPath is local config and a clone does not carry it, so an unarmed clone is silent
by construction this is the only place it becomes visible.
"""
try:
val = subprocess.check_output(["git", "config", "--get", "core.hooksPath"],
cwd=root, stderr=subprocess.DEVNULL).decode().strip()
except Exception:
val = ""
norm = val.replace("\\", "/").rstrip("/")
if norm == ".githooks" or norm.endswith("/.githooks"):
return
print("WARNING: this clone is UNARMED — core.hooksPath is %s, so the pre-push hook will not\n"
" run here. Switch it on once with: git config core.hooksPath .githooks"
% (("'" + val + "'") if val else "unset"))
def run_gate(label, path, args):
if not os.path.exists(path):
print("\nFAIL: gate '%s' is MISSING — tried %s" % (label, path))
print(" A missing gate is a failure, never a skip (fail-closed).")
return 1
print("\n" + "=" * 78)
print("== gate: %s (%s%s)" % (label, os.path.basename(path),
(" " + " ".join(args)) if args else ""))
print("=" * 78, flush=True)
# stream the gate's own output rather than capturing it — its diagnostics are the point,
# and a runner that swallows them makes a conviction unreadable.
return subprocess.call([sys.executable, path] + args, cwd=ROOT)
def main(argv):
fast = "--fast" in argv
unknown = [a for a in argv if a != "--fast"]
if unknown:
print("unknown argument(s): %s" % " ".join(unknown))
print("usage: python3 scripts/repo_gates.py [--fast]")
return 2
selected = [g for g in GATES if g[3] or not fast]
skipped = [g[0] for g in GATES if not (g[3] or not fast)]
print("repo_gates (felhom.eu) — %d gate(s)%s" % (len(selected), " [--fast]" if fast else ""))
if skipped:
print(" --fast SKIPPED (deliberate periodic runs, never in a hook): %s" % ", ".join(skipped))
hooks_armed_note(ROOT)
results = [(label, run_gate(label, path, args)) for label, path, args, _f in selected]
print("\n" + "=" * 78)
print("== summary")
print("=" * 78)
worst = 0
for label, rc in results:
print(" %-18s %-13s (exit %d)" % (label, VERDICT.get(rc, "ERROR"), rc))
if rc != 0:
worst = 1 if rc == 1 or worst == 1 else 2
if worst == 0:
print("\nall felhom.eu gates OK")
return 0
convicted = [l for l, rc in results if rc == 1]
undecided = [l for l, rc in results if rc not in (0, 1)]
if convicted:
print("\nCONVICTED: %s" % ", ".join(convicted))
if undecided:
print("UNDETERMINED (never a pass): %s" % ", ".join(undecided))
return worst
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+166 -20
View File
@@ -1,21 +1,142 @@
# -*- coding: utf-8 -*-
"""REUSE.md refs check — staleness defense for the per-repo reuse maps.
Usage: python scripts/reuse_refs_check.py <repo-root> [<repo-root> ...]
Usage: python3 scripts/reuse_refs_check.py <repo-root> [<repo-root> ...]
For each repo root given, parses <root>/REUSE.md, extracts every cited file path
(*.go, *.py, *.html, *.css, *.yml, *.yaml, *.sh) and verifies the file exists in
the tree. Only slash-containing (repo-relative) tokens are checked bare filenames
are conventions, not citations. Prints offending lines; exits non-zero if any cited
path is missing. Symbols are NOT checked here those are spot-verified by the
reviewer at file:line.
(*.go, *.py, *.html, *.css, *.yml, *.yaml, *.sh) and verifies the file EXISTS somewhere it
could honestly be. Only slash-containing tokens are checked bare filenames are conventions,
not citations. Symbols are NOT checked here; those are spot-verified by the reviewer at
file:line. Exits non-zero if any cited path resolves nowhere.
RESOLUTION ORDER (2026-08-02 operator ruling; first hit wins, and every non-exact hit is
PRINTED so a weakening of the check is visible rather than silent):
1. exact <root>/<token> exists no note
2. suffix exactly one indexed file under <root> ends with /<token>
3. ambiguous more than one does still OK: the citation is real, the shorthand is
imprecise. All matches are printed and marked AMBIGUOUS.
4. cross-repo resolved in an immediate SIBLING repo (a sibling dir containing .git), either
as-is or with the sibling's own name stripped from the front of the token
5. FAIL nowhere prints the file, line, token, and EVERY resolution attempted
WHY THIS SHAPE. Before this, the checker demanded repo-relative paths and was RED on all four
repos: 13 findings, and a hand audit of all 13 on 2026-08-02 found **zero** genuine drift.
Twelve were package shorthand whose file sits one or two directories deeper
(`appbackup/userdata.go` `controller/internal/appbackup/userdata.go`); one,
`wgsync/reconciler.go`, is cited by the controller's REUSE.md and lives in the HUB. REUSE.md
cites by package shorthand and across repos on purpose that convention is the useful one, and
the tool was what was wrong. Rejected alternatives, recorded so they are not revisited:
rewriting all four REUSE.md files to full paths (makes the docs worse to serve the tool), and
deleting the checker (REUSE drift across four repos is a live risk).
THE POSITIVE OBSERVABLE. Every root prints a per-rule tally. "0 failures" alone cannot tell a
working checker from a blind one a run that suddenly resolves everything by SUFFIX is telling
you something, and the counts are where you see it. The kill condition is pinned by
scripts/test_reuse_refs_check.py: a citation that exists nowhere still FAILS.
"""
import io, os, re, sys
# path-looking tokens ending in a checked extension; globs (*) are conventions, not refs
PATH_RE = re.compile(r'[A-Za-z0-9_][A-Za-z0-9_./\-]*/[A-Za-z0-9_./\-]*\.(?:go|py|html|css|yml|yaml|sh)\b')
# An EVIDENCE COPY of a file is not the file — never let an audit or a test-findings tree satisfy
# a citation. `.git`/`vendor`/`node_modules` are excluded as noise.
EXCLUDE_NAMES = {".git", "node_modules", "vendor", "audits"}
EXCLUDE_RELPATHS = {"documentation/tests"}
fails = 0
_index_cache = {}
class RepoIndex(object):
"""One walk per repo root, reused across every token and every sibling lookup."""
def __init__(self, root):
self.root = root
self.name = os.path.basename(root)
self.files = set() # posix-style relpaths
self.by_base = {} # basename -> [relpath, ...]
for dirpath, dirs, filenames in os.walk(root):
rel = os.path.relpath(dirpath, root).replace(os.sep, "/")
if rel == ".":
rel = ""
dirs[:] = [d for d in dirs
if d not in EXCLUDE_NAMES
and ((rel + "/" + d).lstrip("/") not in EXCLUDE_RELPATHS)]
for fn in filenames:
p = (rel + "/" + fn).lstrip("/")
self.files.add(p)
self.by_base.setdefault(fn, []).append(p)
def exact(self, token):
return token in self.files
def suffix_matches(self, token):
base = token.rsplit("/", 1)[-1]
return sorted(p for p in self.by_base.get(base, [])
if p != token and p.endswith("/" + token))
def index_for(root):
root = os.path.abspath(root)
if root not in _index_cache:
_index_cache[root] = RepoIndex(root)
return _index_cache[root]
def find_siblings(root):
"""Immediate sibling dirs of <root> that are themselves git working trees. One level only.
Returns (list_of_paths, error_message_or_None). A sibling repo that is simply absent is NEVER
a failure a clone in isolation must still be able to check itself.
"""
parent = os.path.dirname(os.path.abspath(root))
try:
entries = sorted(os.listdir(parent))
except OSError as e:
return [], "parent %s not readable (%s) — siblings were NOT searched" % (parent, e)
sibs = []
for e in entries:
p = os.path.join(parent, e)
if os.path.abspath(p) == os.path.abspath(root):
continue
if os.path.isdir(p) and os.path.exists(os.path.join(p, ".git")):
sibs.append(p)
return sibs, None
def resolve(token, idx, siblings):
"""(rule, note, tried) — rule is one of exact/suffix/ambiguous/cross-repo/None."""
tried = ["repo-relative %s/%s" % (idx.name, token)]
if idx.exact(token):
return "exact", "", tried
tried.append("suffix search over %d indexed files in %s" % (len(idx.files), idx.name))
m = idx.suffix_matches(token)
if len(m) == 1:
return "suffix", "resolved by suffix → %s" % m[0], tried
if len(m) > 1:
return "ambiguous", "AMBIGUOUS — %d matches: %s" % (len(m), ", ".join(m)), tried
for sib in siblings:
sidx = index_for(sib)
# a token may carry the sibling's own repo name on the front (app-catalog's REUSE.md
# cites `felhom.eu/scripts/site_gates.py` that way) — try both forms.
cands = [token]
if token.startswith(sidx.name + "/"):
cands.append(token[len(sidx.name) + 1:])
for cand in cands:
tried.append("sibling %s: %s" % (sidx.name, cand))
if sidx.exact(cand):
return "cross-repo", "cross-repo → %s/%s" % (sidx.name, cand), tried
sm = sidx.suffix_matches(cand)
if len(sm) == 1:
return "cross-repo", "cross-repo (suffix) → %s/%s" % (sidx.name, sm[0]), tried
if len(sm) > 1:
return "cross-repo", "cross-repo AMBIGUOUS in %s%d matches: %s" % (
sidx.name, len(sm), ", ".join(sm)), tried
return None, "", tried
def check_repo(root):
@@ -27,7 +148,14 @@ def check_repo(root):
print("FAIL [%s]: no REUSE.md at %s" % (name, reuse))
fails += 1
return
seen, missing = set(), 0
idx = index_for(root)
siblings, sib_err = find_siblings(root)
if sib_err:
# say so and continue — do NOT silently pretend siblings were searched
print("NOTE [%s]: %s" % (name, sib_err))
seen = set()
tally = {"exact": 0, "suffix": 0, "ambiguous": 0, "cross-repo": 0, "failed": 0}
with io.open(reuse, encoding="utf-8") as f:
for lineno, line in enumerate(f, 1):
for m in PATH_RE.finditer(line):
@@ -37,18 +165,36 @@ def check_repo(root):
if p in seen:
continue
seen.add(p)
if not os.path.isfile(os.path.join(root, p)):
print("FAIL [%s] line %d: cited path missing: %s" % (name, lineno, p))
missing += 1
if missing:
fails += missing
else:
print("OK [%s]: %d cited paths, all exist" % (name, len(seen)))
rule, note, tried = resolve(p, idx, siblings)
if rule is None:
tally["failed"] += 1
print("FAIL [%s] line %d: cited path resolves NOWHERE: %s" % (name, lineno, p))
for t in tried:
print(" tried: %s" % t)
if not siblings and not sib_err:
print(" tried: no sibling git repos found beside %s" % name)
else:
tally[rule] += 1
if note:
print("note [%s] line %d: %s (%s)" % (name, lineno, p, note))
print("%s [%s]: %d cited paths — exact %d, suffix %d, ambiguous %d, cross-repo %d, FAILED %d "
"(siblings searched: %s)" % (
"FAIL" if tally["failed"] else "OK ", name, len(seen),
tally["exact"], tally["suffix"], tally["ambiguous"], tally["cross-repo"],
tally["failed"],
", ".join(os.path.basename(s) for s in siblings) or "none"))
fails += tally["failed"]
if len(sys.argv) < 2:
print(__doc__)
sys.exit(2)
for r in sys.argv[1:]:
check_repo(r)
sys.exit(1 if fails else 0)
def main(argv):
if len(argv) < 1:
print(__doc__)
return 2
for r in argv:
check_repo(r)
return 1 if fails else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+58
View File
@@ -0,0 +1,58 @@
# -*- coding: utf-8 -*-
"""Seam test for scripts/repo_gates.py.
Run: python3 scripts/test_repo_gates.py
WHY THIS EXISTS. An entry point is a seam by definition: a runner that LISTS a gate but never
executes it is inert and fully green, and this project has shipped an inert seam four times. So
the assertion is on each member gate's OWN distinctive stdout — never on the runner's summary
line, which the runner can print without ever calling anything plus the exit code, which is a
runner's actual effect.
Red-proofed 2026-08-02: replacing run_gate's body with `return 0` (the inert runner) turns
test_every_member_gate_actually_ran red while the summary still prints "all felhom.eu gates OK".
"""
import os
import subprocess
import sys
import unittest
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
ENTRY = os.path.join(ROOT, "scripts", "repo_gates.py")
# (label, a substring only THAT gate can print)
FINGERPRINTS = [
("site", "site gates OK"),
("hostinstall", "hostinstall gates: ALL PASS"),
("hub-confirm", "hub confirm gate"),
("manifest-bearer", "manifest bearer gate"),
("reuse-refs", "cited paths — exact"),
]
class RepoGatesTest(unittest.TestCase):
@classmethod
def setUpClass(cls):
p = subprocess.run([sys.executable, ENTRY, "--fast"], cwd=ROOT,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
cls.rc = p.returncode
cls.out = p.stdout.decode("utf-8", "replace")
def test_exit_code_is_zero(self):
self.assertEqual(self.rc, 0, self.out)
def test_every_member_gate_actually_ran(self):
for label, fingerprint in FINGERPRINTS:
self.assertIn(fingerprint, self.out,
"gate %r is listed but its own output never appeared — an inert runner "
"prints the summary without calling anything:\n%s" % (label, self.out))
def test_unknown_argument_is_rejected(self):
p = subprocess.run([sys.executable, ENTRY, "--nope"], cwd=ROOT,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
self.assertEqual(p.returncode, 2, p.stdout.decode("utf-8", "replace"))
if __name__ == "__main__":
unittest.main(verbosity=2)
+191
View File
@@ -0,0 +1,191 @@
# -*- coding: utf-8 -*-
"""Fixture tests for scripts/reuse_refs_check.py — one per row of its resolution table, plus the
kill condition.
Run: python3 scripts/test_reuse_refs_check.py
THE ONE THAT MATTERS is test_absent_path_fails (Scenario E). The 2026-08-02 change taught the
checker to resolve suffixes and sibling repos, which turned 13 findings green in one step. A
checker made green by being made BLIND is a failure this project has shipped before, so the
ability to still fail is pinned here, and the assertion is on the EXIT CODE the effect not on
the summary text, which the checker can print without having decided anything.
"""
import importlib.util
import io
import os
import shutil
import sys
import tempfile
import unittest
SCRIPT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "reuse_refs_check.py")
def load_checker():
"""Fresh module per test — the checker keeps a global failure count and an index cache."""
spec = importlib.util.spec_from_file_location("reuse_refs_check_under_test", SCRIPT)
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
return mod
def write(path, text=""):
d = os.path.dirname(path)
if d and not os.path.isdir(d):
os.makedirs(d)
with io.open(path, "w", encoding="utf-8") as f:
f.write(text)
class ReuseRefsCheckTest(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.mkdtemp(prefix="reuse-refs-")
self.root = os.path.join(self.tmp, "myrepo")
write(os.path.join(self.root, ".git"), "gitdir: elsewhere\n")
def tearDown(self):
shutil.rmtree(self.tmp, ignore_errors=True)
# ── harness ──────────────────────────────────────────────────────────────────
def run_check(self, *roots):
"""Returns (exit_code, stdout). Exit code is the assertion that counts."""
mod = load_checker()
buf = io.StringIO()
real = sys.stdout
sys.stdout = buf
try:
rc = mod.main(list(roots) or [self.root])
finally:
sys.stdout = real
return rc, buf.getvalue()
def reuse(self, body):
write(os.path.join(self.root, "REUSE.md"), body)
def sibling(self, name):
p = os.path.join(self.tmp, name)
write(os.path.join(p, ".git"), "gitdir: elsewhere\n")
return p
# ── row 1: exact ─────────────────────────────────────────────────────────────
def test_exact_match_is_silent_and_passes(self):
write(os.path.join(self.root, "a", "b.go"), "package a\n")
self.reuse("see `a/b.go` for the thing\n")
rc, out = self.run_check()
self.assertEqual(rc, 0, out)
self.assertIn("exact 1", out)
self.assertNotIn("note [", out) # an exact hit prints no note
# ── row 2: suffix ────────────────────────────────────────────────────────────
def test_package_shorthand_resolves_by_suffix(self):
write(os.path.join(self.root, "controller", "internal", "pkg", "x.go"), "package pkg\n")
self.reuse("see `pkg/x.go`\n")
rc, out = self.run_check()
self.assertEqual(rc, 0, out)
self.assertIn("resolved by suffix", out)
self.assertIn("controller/internal/pkg/x.go", out)
self.assertIn("suffix 1", out)
# ── row 3: ambiguous — real citation, imprecise shorthand; NOT a failure ─────
def test_two_suffix_matches_are_ambiguous_not_fatal(self):
write(os.path.join(self.root, "one", "pkg", "x.go"), "package pkg\n")
write(os.path.join(self.root, "two", "pkg", "x.go"), "package pkg\n")
self.reuse("see `pkg/x.go`\n")
rc, out = self.run_check()
self.assertEqual(rc, 0, out)
self.assertIn("AMBIGUOUS", out)
self.assertIn("one/pkg/x.go", out)
self.assertIn("two/pkg/x.go", out)
self.assertIn("ambiguous 1", out)
# ── row 4: cross-repo, by suffix in a sibling ────────────────────────────────
def test_sibling_repo_resolution(self):
sib = self.sibling("otherrepo")
write(os.path.join(sib, "hub", "internal", "wgsync", "reconciler.go"), "package wgsync\n")
self.reuse("see `wgsync/reconciler.go`\n")
rc, out = self.run_check()
self.assertEqual(rc, 0, out)
self.assertIn("cross-repo", out)
self.assertIn("otherrepo", out)
self.assertIn("cross-repo 1", out)
# ── row 4b: cross-repo where the token CARRIES the sibling's repo name ───────
def test_sibling_repo_name_prefixed_token(self):
sib = self.sibling("felhom.eu")
write(os.path.join(sib, "scripts", "site_gates.py"), "# gate\n")
self.reuse("run `felhom.eu/scripts/site_gates.py`\n")
rc, out = self.run_check()
self.assertEqual(rc, 0, out)
self.assertIn("cross-repo", out)
def test_non_git_sibling_is_not_searched(self):
plain = os.path.join(self.tmp, "notarepo") # no .git — not a repo, must not resolve
write(os.path.join(plain, "pkg", "x.go"), "package pkg\n")
self.reuse("see `pkg/x.go`\n")
rc, out = self.run_check()
self.assertNotEqual(rc, 0, out)
# ── row 5: THE KILL CONDITION (Scenario E) ──────────────────────────────────
def test_absent_path_fails(self):
write(os.path.join(self.root, "internal", "present.go"), "package internal\n")
self.reuse("line one\nsee `internal/definitely_absent_xyz.go`\n")
rc, out = self.run_check()
self.assertNotEqual(rc, 0, "a citation that exists NOWHERE must fail:\n" + out)
self.assertIn("definitely_absent_xyz.go", out)
self.assertIn("line 2", out) # names the line
self.assertIn("FAILED 1", out)
def test_failure_lists_every_resolution_attempted(self):
"""CLAUDE.md standing rule: a 'not found' claim must name what was tried."""
self.sibling("otherrepo")
self.reuse("see `internal/definitely_absent_xyz.go`\n")
rc, out = self.run_check()
self.assertNotEqual(rc, 0, out)
self.assertIn("tried: repo-relative myrepo/internal/definitely_absent_xyz.go", out)
self.assertIn("tried: suffix search over", out)
self.assertIn("tried: sibling otherrepo", out)
# ── evidence trees are not the file ─────────────────────────────────────────
def test_evidence_copy_does_not_satisfy_a_citation(self):
write(os.path.join(self.root, "documentation", "audits", "pkg", "x.go"), "package pkg\n")
write(os.path.join(self.root, "documentation", "tests", "pkg", "y.go"), "package pkg\n")
self.reuse("see `pkg/x.go` and `pkg/y.go`\n")
rc, out = self.run_check()
self.assertNotEqual(rc, 0, "an audits/ or documentation/tests/ copy must NOT resolve:\n" + out)
self.assertIn("FAILED 2", out)
# ── a clone in isolation must still check itself ────────────────────────────
def test_no_siblings_is_not_a_failure(self):
write(os.path.join(self.root, "a", "b.go"), "package a\n")
self.reuse("see `a/b.go`\n")
rc, out = self.run_check()
self.assertEqual(rc, 0, out)
self.assertIn("siblings searched: none", out)
def test_missing_reuse_md_fails(self):
rc, out = self.run_check()
self.assertNotEqual(rc, 0, out)
self.assertIn("no REUSE.md", out)
# ── globs stay conventions, not refs ────────────────────────────────────────
def test_glob_is_not_a_citation(self):
self.reuse("the gates are `scripts/*.py`\n")
rc, out = self.run_check()
self.assertEqual(rc, 0, out)
self.assertIn("0 cited paths", out)
# ── no args → usage, exit 2 ─────────────────────────────────────────────────
def test_no_args_is_usage_exit_2(self):
mod = load_checker()
buf, real = io.StringIO(), sys.stdout
sys.stdout = buf
try:
rc = mod.main([])
finally:
sys.stdout = real
self.assertEqual(rc, 2)
if __name__ == "__main__":
unittest.main(verbosity=2)
+67 -1
View File
@@ -1,6 +1,6 @@
---
name: felhom-build-deploy
description: Build, deploy, publish, or verify ANY Felhom artifact — felhom-controller image (guest 9201 bootstrap deploy), felhom-agent binary (felhom-pve), felhom-hub (GitOps/ArgoCD), the felhom.eu website (git-sync), or the app catalog. Use whenever the task says build, deploy, ship, release, publish, bump version, restart the controller/agent/hub, or verify what version is live. Contains the exact verified commands and the gotchas that silently break deploys.
description: Build, deploy, publish, or verify ANY Felhom artifact — felhom-controller image (guest 9201 bootstrap deploy), felhom-agent binary (felhom-pve), felhom-hub (GitOps/ArgoCD), the felhom.eu website (git-sync), the app catalog, or the PUBLIC installer ISO (iso.felhom.eu). Use whenever the task says build, deploy, ship, release, publish, bump version, restart the controller/agent/hub, or verify what version is live. Contains the exact verified commands and the gotchas that silently break deploys.
---
# Felhom build & deploy runbooks
@@ -75,6 +75,72 @@ Publish to Gitea (so Day-0 self-install can fetch it): `scripts/publish-agent.sh
`REGISTRY_*` creds. The hub's Day-0 artifact manifest must then vouch the new version — that UI is
operator-password-gated (CC cannot); flag it as an operator follow-up.
## Installer ISO (felhom.eu/scripts/iso → iso.felhom.eu) — PUBLIC, irreversible
**Two modes, and picking the wrong one ships the wrong product.**
| Mode | What it is | Menu |
|---|---|---|
| `--release` | the **public** image. No `answer.toml`, no root password, no SSH key, no disk profile. Day-0 rides a `.deb`. | TWO interactive entries, graphical default, timeout 15s |
| `--pairing` / `--bootstrap-env` | operator-built **appliance** image: baked answer file, baked root hash, a disk profile pinned to one machine | ONE automated entry |
```bash
# build the public image (clean-tree gate first — an unpushed change does not exist)
export FELHOM_ISO_OUT=$FELHOM_ROOT/felhom-iso/out
bash scripts/iso/build-felhom-iso.sh \
--pve-iso $FELHOM_ROOT/drill/proxmox-ve_9.2-1.iso \
--iso-sha256 4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c --release
# -> $FELHOM_ISO_OUT/felhom-installer-<VER>-pve<PVE>.iso + .sha256 + .manifest.txt (NO .rootpw.txt)
```
**Before publishing, two hard gates — neither is optional and neither is a script yet:**
1. **`documentation/runbooks/iso-release-gate.md`** — 13 criteria, run against **the exact file you
will upload**, not the build inputs. It is a manual checklist; `repo_gates.py` does **not** cover
it, so nothing will remind you (R-29's shape — say so if you skip it).
2. **A proof install from the built image on BOTH menu entries** (graphical *and* Terminal UI), each
showing: package installed, unit `enabled`, unit fired on first boot, and a pairing code in
`/etc/felhom/appliance-pairing-code`. Spike 4 *reasoned* the graphical path follows from shared
`Install.pm`; the 1.26.0 run proved that reasoning insufficient in a different place — do both.
```bash
# publish — rclone in a container, env-only config, so NO credential file is ever written
source ~/.config/credentials # ISO_S3_CLIENT_AK / _SK / ISO_S3_URL — never echo, never log
docker run --rm -v $FELHOM_ISO_OUT:/data:ro \
-e RCLONE_CONFIG_R2_TYPE=s3 -e RCLONE_CONFIG_R2_PROVIDER=Cloudflare \
-e RCLONE_CONFIG_R2_ACCESS_KEY_ID="$ISO_S3_CLIENT_AK" \
-e RCLONE_CONFIG_R2_SECRET_ACCESS_KEY="$ISO_S3_CLIENT_SK" \
-e RCLONE_CONFIG_R2_ENDPOINT="$ISO_S3_URL" \
-e RCLONE_CONFIG_R2_REGION=auto -e RCLONE_CONFIG_R2_NO_CHECK_BUCKET=true \
rclone/rclone:latest copy /data R2:felhom-iso --include "felhom-installer-<VER>*" --s3-chunk-size 64M
# verify by ROUND TRIP — the downloaded bytes, not the local file
curl -fsSL -o /tmp/rt.iso https://iso.felhom.eu/felhom-installer-<VER>-pve<PVE>.iso && sha256sum /tmp/rt.iso
```
`ListBuckets` 403s — the token is object-scoped; list with `lsf R2:felhom-iso`, not `lsd R2:`.
### Proof-VM traps — every one of these cost a wrong diagnosis
- **Set `--boot` in a SEPARATE `qm set`, after the disk exists.** `qm set <id> --scsi0 … --boot order="scsi0;ide2"`
in one call silently yields `boot: order=net0;ide2`; the VM netboots, fails, falls through to the CD.
- **After the install, detach the CD** (`qm set <id> --delete ide2; qm set <id> --boot order="scsi0"`)
or the machine re-enters the installer on reboot — **a completed install looks exactly like a stuck one.**
Judge completion from `qm config` + disk usage, never from the screen.
- **Verify focus by screendump before every `Enter`.** TUI: red-highlighted button, tab order. GTK:
dashed focus ring, and `Enter` lands in text *fields*, not `Next`. Not checking once aborted an install.
- **Proof installs register unclaimed appliances at the hub — discard them** or R-131 grows:
`curl -u ":$HUB_PW" -X POST http://<hub-clusterIP>:8080/appliances/<id>/discard` → 303.
The verb is **`/discard`**, POST only (`hub/internal/web/server.go:345`); `/delete` 404s.
- Venue: `demo-hp`, scratch `dir` storage at **`/mnt/nvme-1tb` root** (a subdirectory reads
`disconnected` forever — the agent's `exactMount` check). Never `local-lvm`. Remove the storage at teardown.
**Why the shape is what it is** (do not re-derive; four spikes measured it):
`documentation/audits/SPIKE-universal-iso-{1,2,3,4}-2026-07-31.md`. In short — no udev property
distinguishes an internal disk from a customer's backup drive and a two-disk filter match silently
wipes one, so **there is no safe automated disk selection for unseen hardware**; and `[first-boot]`
is never placed on the system by an interactive install, so day-0 rides a `.deb` in
`/proxmox/packages/` instead (`Install.pm:1343-1372`).
## Hub (felhom.eu/hub → k3s, GitOps via ArgoCD app `felhom`)
**The manifest is the truth.** A code push + image build deploys NOTHING until `manifests/hub.yaml`'s