4f5784894e918ba4f2c170d60aeb0c4d0dfee651
172 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 admin
|
b080ecf411 |
hub v0.99.0 — the hub can see whether the operator can get in (R-260); G-1 gate closes, R-247 closes
oobDegraded tested five things and the sixth never arrived. The agent has emitted `operator_key_configured` on every heartbeat since v0.72.0 — the SAME version that introduced the `oob` stanza carrying it — and store.HostOOBRow mirrored five of the agent's eight OOB fields. With no field for it, encoding/json discarded the fact on arrival, so a box with felhom-sshd active, reachable, a valid config and a configured peer reported `ok` with NO OPERATOR KEY INSTALLED AT ALL. Not a wrong answer: an answer to a question nobody was asking. `operator_peer_configured`, which the hub did read, only says the peer IP is in desired-state — that OOB is MEANT to work, not that entry is possible. Now decoded: operator_key_configured, plus wg_handshake_age_s and healed_at. The last two ride the ALERT TEXT and are deliberately NOT in the predicate — widening a check beyond the fact that is now arriving is how a check stops being read. SCENARIO F, decided on a measurement rather than a preference. operator_key_configured decodes as a POINTER: nil = the agent never said, reported distinctly and never as ok. The version gate was rejected because the field and its stanza shipped in the SAME agent version (v0.72.0), so a stanza without the field cannot come from any released agent; the fleet is 0.113.0/0.127.0 and the vouched floor is 0.127.0. Handled explicitly anyway and pinned, because "cannot happen" is a claim this project has been burned by. THE MESSAGE NAMES THE FAULT. oobDegradedReason is the single source for both predicate and text, so the alert can never name a different fault from the one that fired. The old form derived it separately and had a vocabulary of two — unreachable, or config invalid — with no way to say the key is missing. The operator reads this at 07:00. TESTS DRIVE THE DECODE BOUNDARY. Every hub OOB test before this built a HostOOBRow by hand, and a test written that way CANNOT SEE A FIELD THAT NEVER DECODES — which is how this held a green suite for five weeks. The pre-existing fixture oobReport() also omitted the field, so those scenarios ran against a report shape no released agent produces (same family as R-262). Both fixed. Red-proofs, 8 expected outcomes and 0 wrong, each with the mutation asserted applied: dropping the field returns the false ok; an unconditional check alerts a healthy box; unknown-as-ok restores the silent pass. G-1 CLOSED — scripts/wire_contract_gate.py shipped as ranked, built BEFORE the fixes and seen failing on 40 fields (documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md). Two instrument defects the control caught first: a substring false negative (grep -F healed_at matched privsep_healed_at) and treating dr_recipe as wholly opaque when its top-level sections ARE decoded through an allow-list that already cost offsite_restic (R-122). The prompt for this session said "465 emitted tags, eight unreachable". Checked against the repo: R-260 said "at least eight DECISION-BEARING facts", never eight tags. The real count is 40. R-260 CLOSED (class gated, sharpest instance fixed). R-247 CLOSED (controller v0.209.0). R-264 MINTED and OPEN — the 21 facts with no consumer, allowlisted with reasons so that gating the class could not be mistaken for deciding them. Still open and named: R-246, R-255..R-259, R-261..R-263, and C7's test-comment half. Capability map checked: it claims OOB access is implemented, never monitored, so no row was untrue; what was untrue sat one layer down and the row now records it. repo_gates --fast: all 8 OK. go build/vet/test green in hub, run separately from this commit. |
||
|
admin
|
ac4b2a4ba9 |
hub: drop the retained recovery package when the box declares its set-aside history deleted (R-241)
The hub half of the controller's abandonment countdown, and the ONLY reason felhom.eu was touched for R-241 at all. A customer who abandons their old off-site history gets a 14-day countdown. At the end of it the controller deletes the set-aside restic store and then DECLARES offsite.abandon_purge_requested in its report until the retained sealed package that protected that store is gone too. Removing only one half leaves a state that asks a question nobody can answer: a package that opens nothing, or ciphertext nobody can ever decrypt. store.PurgeSupersededEscrowForCustomer is the one place R-198's retention is ever undone, and its doc comment says why that is legitimate here. It NEVER touches host_escrow - the current package covers the key the box is using now and is what makes its live backups recoverable. Only host_escrow_superseded rows go. The handler acts on the box's DECLARATION, never an inference, on the same principle as offsite.state: the hub cannot see that a remote store was deleted and the box can. It is placed immediately BEFORE the ACK is built, deliberately. GetEscrowStatusForCustomer is read after it runs, so the SAME response that carries the request's effect is what closes the box's two-phase commit - no second round-trip, and no window in which the box believes it is still owed. The declaration repeats on every report until that ACK stops reporting a superseded package, so a lost request retries by itself. A purge failure is logged at ERROR and never swallowed: the box keeps declaring, so it retries, but an operator must be able to see that the two halves are apart right now. An idempotent re-declaration (already purged, the box has not yet seen the confirming ACK) logs at DEBUG and is not an error. Audit event offsite_abandon_purged is hub-internal, like the pbsdr_* and offsite_selfheal_* events - allowedEventTypes governs the box-pushed POST /event surface, not this. No agent change. No deletion has been performed against any real store. Green: go build, go vet, go test ./... all clean in hub/; repo gates OK. |
||
|
admin
|
a7f1d277b1 |
hub: the held-floor REASON must match the CAUSE (CAMPAIGN-11 follow-on)
gates / gates (push) Successful in 7s
v0.97.0 introduced a second hold reason and left both surfaces printing the first. The freshly deployed hub logged, for the campaign box: managed floor HELD for c11: agent "0.125.0" < MinAgent 0.113.0 which is FALSE — 0.125.0 is ABOVE 0.113.0. That box is held because its floor sits above the vouched golden, not because of its agent. CLAUDE.md's corollary exactly: when a verdict changes which field it counts from, the alarm text has to change with it, or a true alarm reads as one to dismiss. Both the ACK log line and the Hosts-dashboard HeldReason now come from one ManagedFloorDecision.HoldReason(), and TestResolveManagedFloor_HoldReasonMatchesTheCause pins each reason to its cause. |
||
|
admin
|
7e1d2898bd |
hub v0.97.0 — the floor stops being served past the agent it depends on (CAMPAIGN-11)
gates / gates (push) Successful in 8s
R-216, the hub half. ResolveManagedFloor's own comment says it exists to "never push a controller past the agent it depends on", and it compared against ArtifactManifest.MinAgent — which by ITS own comment describes the GOLDEN's controller. publish-train-rules.md rule 3 states the rule about the FLOOR's controller. Measured live: golden 0.192.0 / MinAgent 0.113.0, floor 0.200.0, agent 0.120.0 — served, and the box was pushed onto a controller needing agent 0.125.0. A floor ABOVE the vouched golden is now HELD with its own reason (HeldBeyondGolden), reusing Part D's dashboard visibility. Nobody types a number twice: the vouched MinAgent keeps its meaning, the guard stops applying it to versions it does not describe. An uncoupled release is untouched; an unparseable golden degrades rather than gating. R-222: the report ACK's escrow object gains superseded_present / superseded_at, counting only rows that actually carry an identity blob. One boolean and one timestamp, for one message. No read path — that link is still unbuilt. Red-proof: removing the floor-above-golden branch reproduces the campaign's measurement. |
||
|
admin
|
f62a115891 |
R-204 item 4 (hub half): the hub answers a rebuilt box's request (hub v0.96.0)
New internal/offsiteheal, the sibling of pbsdrheal: it acts ONLY on the state the box declares, sustained across two distinct reports, re-staging the stored credential before ever minting a new one. A healthy box is a pure no-op; it never blind-timer-reissues and never re-runs a provisioning step. RESTAGE IS POSSIBLE because the stored value survives a consume — established from the schema and ConsumeOneTimeSecret (which stamps consumed_at and nothing else), not inherited from the PBS analogy, and pinned by a test that asserts the SAME value comes back. reportHasOffsite is TIGHTENED to require enabled:true. Its comment asserted that presence == applied-on-the-box, and the declaration deliberately breaks that premise; left alone it would have read a request for help as proof the tier was applied. Provably a no-op for every report shape that existed before, because an attached object has always carried enabled:true. R-192's guard half is CLOSED BY REPLACEMENT: the delivery checker's counting inference read the OLDEST 500 reports after a consume — all predating a rebuild, which is why demo-hp sat stranded for 108 reports under a confident regressed-shape verdict. A declaration outranks both inferred shapes, and the checker stands down with a record so the two mechanisms cannot double-issue. No escrow ceremony is ever run or requested: credential automatic, key customer-present. |
||
|
admin
|
d1a8edb332 |
R-196 / R-204 item 2: a re-issue no longer marks a healthy escrow stale (hub v0.95.0)
ReissueCredentials marked the escrow stale on every re-issue, on precautionary grounds — the box's re-apply MIGHT mint a fresh repository password. It usually does not. A stale flag withholds restic_pw_sha256 from the ACK, which stops the controller's auto-confirm, which leaves EscrowState pending, which makes OffboxRunnable false: every off-site backup refused on a box whose key was never in doubt — and the customer told to re-run the one ceremony that would have superseded the key just recovered. The case it guessed at is measured elsewhere: the controller's Scenario-F re-check compares the sealed hash against the live repo password on every ACK (and the mark was BLINDING it by emptying that hash), and R-197's offsite_repo_key_changed fires on a proven difference across a supersession. offsite_reissued is unchanged. MarkEscrowStale is kept without a caller so a future EVIDENTIAL writer has the mechanism, with a test pinning it live. TestReissue_InvalidatesEscrow is replaced by its exact inverse. |
||
|
admin
|
435f4a5229 |
hub v0.94.0: a box can fetch its own sealed recovery package (R-199 link 6)
gates / gates (push) Successful in 7s
Link 6 of the recovery chain had no client. The hub has served the identity blob since
slice 10D from handleReEnroll / handleGetRestoreDirective, gated on operator-armed recovery
mode and the global key -- and nothing in the agent, the hub UI, any script or any runbook
ever called either. The only documented retrieval was sqlite3 writefile() by hand on a
kubectl cp-ed database.
GET /api/v1/hosts/{host_id}/escrow is the box-authenticated mirror of the PUT that put the
blob there. Self-scoped (a per-host key reads only its own; global may read any). A host with
no bundle gets 200 {present:false} -- a 404 is indistinguishable from an unknown host and a
bare empty 200 from a zero-length blob.
THE TRADE IS RECORDED IN THE HANDLER, not inferred: obtaining the blob used to require the
operator to arm recovery mode; now whoever controls a rebuilt box can obtain it with that
box's own credential. They still cannot open it -- the hub has never held R and a wrong code
fails closed at age's scrypt KDF. The mitigation is that every retrieval raises
escrow_blob_served (warning, operator-only), recorded before the bytes leave.
escrowSelfServiceRetrieval is the single decision point: flip it to false and the endpoint
additionally requires recovery mode, changing nothing else.
The operator-driven DR path is untouched, pinned by a test. Red-proofs observed: removing the
ownership check serves host B's blob to host A; removing the record makes it silent.
|
||
|
admin
|
91cabdde1b |
hub v0.93.0: the retention keeps the key it was built to keep (R-198) + three honesty fixes (R-197, R-192, R-196)
gates / gates (push) Successful in 7s
R-198 — host_escrow_superseded shipped with `blob` (the K-escrow / PBS datastore key) and
identity_blob was added to host_escrow LATER, never here. The offsite restic REPOSITORY
password lives in identity_blob. So demoteCurrentEscrowTx -- whose own comment calls it "THE
ONE escrow row-copy routine" -- retained the whole-guest key and silently dropped the off-site
data key, which is the secret the retention was built to preserve. And because the copy happens
as the new blob overwrites the old, the destroying act was the ESCROW CEREMONY: the exact thing
a rebuilt box tells its customer to run, on a card promising in Hungarian that the old backups
stay recoverable. Both demo boxes crossed that line on 2026-08-04.
- identity_blob added to the table (CREATE + additive ALTER) and carried in the shared copy
routine, so BOTH callers are fixed at once: re-escrow and host-delete demotion.
- ListSupersededEscrow reads it back; store.HostEscrow gains IdentityBlob.
- CountCurrentEscrowWithIdentity is the census of who the fix protects.
- Nothing is backfillable: pre-v0.93.0 retained rows have no blob and their sources are gone.
- Tests assert the CONSEQUENCE (a retained row can still yield a repo password), which is why
the pre-existing retention test stayed green for two months asserting the mechanism.
R-197 — SaveHostEscrow returns the hash it replaced; the escrow PUT raises
offsite_repo_key_changed (warning, operator-only, edge-triggered) when both hashes are known and
differ. No hash value travels. Severity chosen for the world v0.93.0 creates: with the identity
blob retained, a changed key is "this history now depends on an older recovery code", not a loss.
R-192 (half) — the stuck alert now reports the two shapes it actually covers, burned and
regressed, each stating its own measurement; the regressed text withdraws the Re-issue
recommendation. Every self-heal refusal leaves a notification_log row with its reason. The
guard's logic is unchanged; its 500-oldest-reports scoping stays OPEN and the window is named in
the alert text so the limitation travels with the number. offsite_delivery_stuck and
offsite_credential_restaged are added to operatorOnlyEvents -- neither was registered and neither
has a customerMessages entry, which is not a block.
R-196 — five comments (not the three the spec expected) claimed ReissueCredentials rotates the
restic repo password. It resets the PROVIDER password and cannot touch the repo password, which
is generated on the box. All five corrected; the staleness mark documented as precautionary. The
BEHAVIOUR stays open.
Not in this release: R-199, R-200, R-201 remain open -- the chain that hands the key back is
still unassembled. Part 5 hit its gate; the orphan card is untouched (R-202).
|
||
|
admin
|
7fff45d688 |
R-195: a customer with no machine ever bound does not alarm (hub v0.92.0) + R-193/R-192 spike
gates / gates (push) Successful in 7s
Part 4 (ships): `david` — a prospective customer with hosts=0, host_deletions=0, reports=0 — e-mailed an expected_dbdump_missed ERROR at 03:00 UTC three mornings running. The existing down-skip could never cover it: it reads the staleness checker's state, which is seeded from a query over the `reports` table, so a customer that never reported has no state at all and GetState() returns "" rather than "down". store.HasEverBoundHost (hosts row OR host_deletions tombstone) is consulted once per customer at the top of the deadline loop. The discriminator is "was a host EVER bound", never "has a report arrived" — a box installed and never heard from is a real fault and keeps alarming. Fail-OPEN on a read error. Red-proof observed: removing the guard fails with `got [expected_dbdump_missed]`, verbatim the event david sent. Parts 0-3 (spike, NO production code for R-193/R-192): audits/SPIKE-offsite-credential-recovery-2026-08-04.md establishes that the one-shot provider password is the RECOVERABLE secret and the restic repository password is the irreplaceable one — and that a guest rebuild mints a fresh one, orphaning the previous off-site history. Measured without touching a box, by comparing host_escrow.restic_pw_sha256 against host_escrow_superseded: BOTH demo boxes changed (demo-hp 15 snapshots / 40.9 MB, demo-felhom 36 snapshots / 1.14 GB). demo-felhom's "lucky" 76-second recovery restored delivery and not the repository, silently, for 13h. ReissueCredentials does NOT rotate the restic password (R-39's record and two hub comments are wrong -> R-196); candidate (b) is not implementable against a zero-knowledge escrow; candidate (a) already exists as F3 and is wired to the wrong event. Ends in ranked options and an unanswered question for the operator. R-195 SHIPPED; R-196 + R-197 filed; R-192 + R-193 updated, neither closed. |
||
|
admin
|
046df303b6 |
hub v0.91.1 — observation may only WIDEN a tier's window, never tighten it (R-86)
gates / gates (push) Successful in 7s
Found by checking v0.91.0 against the live box, not by review. demo-felhom's two retained PBS snapshots sit 8h54m apart (one is a healing artefact), so the mean-gap estimator reads a WEEKLY tier as nine-hourly: x4 = 36h, the 7-day floor lifts it to 168h, and a weekly tier proved weekly reaches ~8.25d of proof age. The false alarm this task exists to prevent would have returned within a week, on the box it had just shipped to. restoreProvenWindow now takes max(observed, declared). A gap SHORTER than the declared rhythm is routine and means nothing (a retry, a manual run, a heal, a catch-up); a gap LONGER than it is real information. Cost stated: a tier running faster than its declared rhythm gets a slower stale signal — the right direction for a signal that means 'unverified', since 'broken now' is a different event. |
||
|
admin
|
323f45a5ef |
hub v0.91.0 — the staleness window learns each tier's own rhythm (R-86 Part 2)
gates / gates (push) Successful in 7s
Ships WITH agent v0.121.0, not after it. The agent now proves a tier once per
ARCHIVE GENERATION, so a weekly tier is proved weekly — in perfect health. The
flat 7-day restoreProvenStaleAfter derived its number from the 24h cadence R-86
removes, and a healthy weekly tier's proof age reaches EXACTLY 168h just before
its next proof: it sat ON the line, so any ordinary delay tipped it into a
nightly alarm about a working system.
restoreProvenWindow(tier, observed, ok):
- the tier's own archive interval, OBSERVED from reports the hub already holds
(pbs_snapshots + successful backups attributed by TARGET TYPE, slice A.4)
- x4 generations = the same tolerance the flat constant expressed
- floored at 7d (never tighter than before), capped at 12d (strictly inside the
2-week offsite retention)
- falls back to the DECLARED rhythm (26h host / 8d offsite — the thresholds the
backup-freshness checker already uses) when history is too short to observe
one; falling back to the FLOOR would recreate the false alarm on a fresh box
Kept: absence is UNKNOWN until the anchored window passes; the signal stays
edge-triggered; failed and stale remain distinct events. Every reason string now
states the window it was judged against (R-100's corollary).
Also backfills the missing v0.90.1 CHANGELOG entry (deployed since
|
||
|
admin
|
f21e7caed1 |
hub v0.90.1 — the digest's per-app lines stop repeating the filesystem figures (R-182)
gates / gates (push) Successful in 7s
Found by reading the first REAL digest, not by design. Every app row ended with the same usage clause the mail already prints once on its own Filesystem line. On a two-app box that is untidy; down a list of a dozen it is the same forty characters twelve times, pushing the part that DIFFERS off a phone screen at 07:00 — the only moment this mail has to work. The reserve's refusal message is authored for a single-app alert where naming the filesystem is right, so the message is unchanged; the digest trims the duplicate when rendering. trimRepeatedUsage removes ONLY an exact "— <target path>:" suffix, so an unrelated reason is untouched and a reason that is nothing but the usage clause is left alone rather than emptied. Also updates TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer, which required the OPERATOR to be emailed a per-app capture failure. That was correct when the event was the only signal and is wrong now that it is the record and the digest is the notification. Its customer-safety claim is unchanged and is why the test still exists; the operator assertion is inverted with the reasoning written in place, and R-158's guarantee is shown to have MOVED, not weakened. |
||
|
admin
|
dd40f85bb8 |
hub v0.90.0 — a dropped notification leaves a trace, and the backup digest arrives (R-182)
gates / gates (push) Successful in 7s
processOperator's cooldown no longer returns bare. It dropped the event BEFORE LogNotification, so a suppressed operator alert and an event that never happened were indistinguishable — from the operator's side and from the hub's own records. Measured 2026-08-03: nine recovery_unit_capture_failed events arrived, two were mailed, seven left no row anywhere. That is why the defect took a day to get the right way round: there was nothing to read. A suppressed operator event now writes a `suppressed` row carrying the message and the key that suppressed it. This applies to EVERY operator event, not only the one that exposed it. It does NOT change the cooldown's duration or semantics. backup_run_failures: the per-run digest. In allowedEventTypes AND in operatorOnlyEvents — allowlisting alone does not make an event operator-only, and FormatCustomerEmail falls back to the raw English message rather than blocking. A test demonstrates a customer with the type enabled receiving nothing. recordOnlyEvents: a third routing class — stored and recorded, never mailed. recovery_unit_capture_failed moves here: it is the record, the digest is the notification. A register rather than downgrading severity to info, which would relabel a genuine failure as informational everywhere it is queried. cooldownRunSuffix: a sibling of cooldownTierSuffix, not a branch inside it, so tier keeps byte-identical semantics and R-97a's tests are untouched. It makes the cooldown effectively inert for the digest, which is the intent — a digest is already rate-limited by construction; the refresh sweep sends no run_id and so stays under the ordinary hourly cooldown. The email renders as a list, not a JSON blob. An absent space reading renders as unavailable, never as zeros. |
||
|
admin
|
179dd79882 |
hub v0.89.0 — the two halves of decision D-c (R-167, R-158)
gates / gates (push) Successful in 7s
New OPERATOR-ONLY event type recovery_unit_capture_failed (controller v0.191.0, R-158): in allowedEventTypes AND notify.operatorOnlyEvents. Deliberately not a reuse of backup_failed, which carries customer copy and sits in the controller's DefaultEnabledEvents — reusing it would email the customer in Hungarian about a failure they cannot act on. R-158's own proposal said backup_failed; D-c overrides it. disk_warning/disk_critical lose their generic customerMessages entries. Both were allowlisted, copy'd, default-enabled and checkbox'd with NO producer anywhere; controller v0.191.0 becomes that producer and sends a DYNAMIC Hungarian message naming the drive and its free space. FormatCustomerEmail prefers the entry over the message, so keeping a static entry would discard the label and the byte figures — the same reason offbox_enlarge_blocked and disk_health_degraded have none. The deletion is pinned by a test. New notify.IsOperatorOnly so the api package can pin BOTH registers of a new event type in ONE test; allowlisted-but-not-operator-only is invisible when they are checked separately, and it is the defect v0.78.0 shipped. The register itself stays unexported. REUSE.md's "new event type" extension point rewritten: it told readers to always add a customerMessages entry, which is wrong for operator-only types and harmful for dynamic-message ones. Tests 574 -> 579. Red-proof: removing the operatorOnlyEvents entry shows the customer being emailed; the skipped/operator_only row is asserted as a positive observable. |
||
|
admin
|
0fc54e0122 |
hub v0.88.0 — the WAL that never was (R-172)
gates / gates (push) Successful in 7s
store.New opened the DB with `?_journal_mode=WAL&_busy_timeout=5000`, which is mattn/go-sqlite3 syntax. The driver is modernc.org/sqlite, whose applyQueryParams reads only _pragma/_time_format/_time_integer_format/_txlock/_inttotime and IGNORES anything else WITHOUT AN ERROR. So the hub ran in rollback-journal mode with busy_timeout=0 for its entire life while its own source said otherwise. Surfaced as a false HOST STALE banner: in rollback-journal mode a reader excludes a writer, so rendering an operator page blocks a host report; the hub 500s, the agent waits its full 15-minute interval without retrying, and staleness fires at 30 minutes — two collisions is a false alarm plus an operator email. 13 collisions in one pod lifetime; the alarm fired twice on 2026-08-02 for a host that was up two days and reconciling throughout. The observable that proved it: a 128 MB /data/hub.db with no -wal/-shm beside it while the DB was open. Fix: ?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate. _txlock=immediate is not optional — database/sql's Begin() is DEFERRED, so a read-then-write tx must upgrade its lock and a failed upgrade is SQLITE_BUSY_SNAPSHOT, which busy_timeout does NOT retry; this store has 10+ db.Begin() sites and they are all write paths. Every test asserts what the DATABASE reports, never the DSN string — a string test would have passed for the whole life of the bug. Red-proof: restoring the shipped DSN reproduces journal_mode="delete", the missing -wal, and the live "database is locked (5) (SQLITE_BUSY)". Operational consequence handled: a WAL DB cannot be copied by taking hub.db alone — a bare `cat` opens cleanly and silently omits the newest writes. The break-glass retrieval in operations/nodes.md used exactly that; it and the recovery-inventory note are now WAL-aware. |
||
|
admin
|
4cc123809c |
revert the Scenario B breakage — main is green again
gates / gates (push) Successful in 7s
The deliberate hostInstallVersion const is removed. It existed only to produce a real red run (#3-#6) and the demonstrated alarm; R-94's deletion stands. |
||
|
admin
|
3252d51104 |
SCENARIO B: deliberately break the hostinstall gate (reverted immediately)
gates / gates (push) Failing after 7s
Pushed with --no-verify ON PURPOSE: this simulates exactly the bypass that CI exists to catch. The local pre-push hook would have refused this commit. |
||
|
admin
|
d319ae573e |
hub: delete the host-install version label (R-94) + invert hostinstall gate 1
The Setup tab said 'host-install 1.19.0' while the served script was 1.22.0, and had been wrong since 2026-07-14. Deriving the number honestly is not possible: the Option-1 command downloads felhom-host-install.sh from the website at RUN TIME and the website git-syncs main every 30s (R-110), so no build-time value in the hub can be true. R-94(a) offered derive-or-delete; deleted, which removes the drift class instead of automating it. - configs.go: hostInstallVersion const, pageData.ScriptVersion field and its assignment all removed; a NOTE in their place records why there is no constant here. - customer_unified.html: the sentence now says the command always fetches the current installer, and renders no version. - hostinstall_gates.py gate 1: the third assertion INVERTS — it used to require the hub const to equal SCRIPT_VERSION, it now asserts the hub carries no host-install version literal at all, matched in six code shapes across every .go/.html under hub/ (comments are deliberately not stripped: a // inside a URL literal would blind the scan). - render_test.go: the assertion 'html contains hostInstallVersion' compared the constant to itself and passed at ANY value — demonstrated green with the const at 9.9.9 while the script was 1.22.0. Deleted, not replaced: there is no longer a version to assert. - felhom-host-install.sh: COMMENT ONLY (SCRIPT_VERSION untouched) — it claimed the gate keeps the hub copy equal, an invariant that no longer exists. Red-proofs: restoring the const fails the rewritten gate 1 (3 shapes hit); the old render_test assertion passes at 9.9.9. |
||
|
admin
|
670ec35ece |
hub v0.86.0 — Copy works without revealing, and every copy branch reports itself
Found by the operator, in the way that matters: it cost a real login.
The v0.84.0 Console access card shipped its Copy button DISABLED until a Reveal.
Clicking it did nothing, silently, so the clipboard kept whatever was already in
it — another host's console password from an earlier reveal. That got pasted into
demo-hp's PVE login, which failed with no explanation: the box logged a plain
`password check failed for user (root)`, the credential was never at fault, and
nothing on screen said the copy had not happened.
A copy button that silently no-ops is worse than no copy button. The operator
cannot tell "copied" from "did nothing", and the stale value left behind is a
VALID secret for a DIFFERENT machine — so the failure looks like a stale
credential and sends you diagnosing the wrong thing.
Copy now works without revealing, and that is the safer default rather than a
concession: the secret goes straight to the clipboard and never renders on
screen, so it cannot be shoulder-surfed or caught in a screenshot. Reveal remains
for when it must be read.
Three silent-failure branches closed, all in the same eight-line function:
- not yet revealed -> was a disabled no-op; now fetches and copies
- navigator.clipboard absent -> was silently skipped; now shows it and says why
- writeText() REJECTED -> promise was ignored, so the operator believed it
copied; now shows it and reports the refusal
The success path names the host ("Copied demo-hp-bb76ea's root@pam password"),
because the clipboard is fleet-wide and every box has a different console
password — "copied" alone cannot say for WHICH box, which is the confusion that
produced the incident.
One retrieval path, shared: the endpoint is defined once (data-reveal-url) and
read back with getAttribute, so Copy cannot drift onto a different, unaudited URL
than Reveal. Server-side is unchanged — both buttons hit the same CSRF-gated
endpoint and both write the same recovery_credential_revealed event, which is
correct: the register records accesses, and a copy is an access.
Tests 566 -> 568. Red-proof: re-adding `disabled` reproduces the shipped bug.
|
||
|
admin
|
e07d90f0f4 |
hub v0.85.0 — Network card: a host's addresses are visible at last
Pairs with agent v0.119.0 and is useless without it. A managed box's LAN IP was not shown anywhere in the hub, because nothing reported it — the host report carried no address of any kind. The only IP reachable from the UI at all was the WireGuard one, on /offsite's peer table keyed by pubkey, so an operator could go peer->host and never host->peer, which is the direction anyone actually asks in. The host page grows a Network card: every routable address the box holds, one row per (interface, address), plus a WireGuard row. On demo-felhom that is vmbr0 192.168.0.162/24 and tailscale0 100.70.170.35/32 — with the PVE web console at https://<the LAN address>:8006, the thing the operator wanted and could not get. WireGuard is rendered as TWO facts, deliberately. WGAssignedIP is the hub's own allocation (wg_peers, authoritative desired state); WGConfirmed is whether the box reports actually holding it. Showing the allocation alone would make a peer that was never applied look healthy — the same shape as reading a timestamp that records an attempt as if it recorded a result. The split is keyed on the ALLOCATION, not the interface name: wg-felhom is the agent's current unit name, and a UI keyed on that string would silently mis-render the day it changes. An old agent renders UNKNOWN, never "no addresses". Below agent 0.119.0 the field is absent from the wire, and an absent signal is not a negative result — the page says so and names the version needed. Rendering an empty list there would have stated something false about the host. No new store table and no new ingest path: the report is already stored opaquely and GetWGPeerForHost already existed with no UI consumer. This is parse + render. The report fixture in the tests is the REAL wire — the addresses block copied out of `felhom-agent --selftest=hub` on demo-felhom running 0.119.0. Tests 559 -> 566; four red-proofs (inert view-model, unconditional confirmation, the old-agent branch, and the drift case) each run, observed failing, reverted. |
||
|
admin
|
1956e5d390 |
hub v0.84.0 — break-glass console credential on the host page
The credential existed and was not reachable when it was wanted. Every box has
had a strong random root@pam password since TASK G1, vaulted in the hub at day 0
and used for real during the sshd incident — but the only way to read it back was
a hand-written curl carrying the global operator key, a secret kept out-of-band.
In practice the PVE web console on a demo box felt locked.
The host page grows a Console access card: presence + username + set_at by
default, Reveal fetches the plaintext on demand for 60 s with a Copy button.
Masking clears the JS variable, and also fires on a second click and on
visibilitychange. A host with nothing vaulted says so, and says why.
The secret is NEVER rendered into the page, and that constraint shapes the
change. The render path uses a new store.GetHostRecoveryMeta whose struct and
SELECT both omit the secret column, so it is structurally incapable of carrying
one. The plaintext crosses the wire only in the response to POST
/hosts/{id}/reveal-recovery-credential (Cache-Control: no-store, CSRF-gated at
the ServeHTTP level; POST precisely so that gate applies and so no secret is
retrievable by URL alone). Deliberately NOT the customer page's data-secret
widget, which embeds the plaintext on every load.
A delivered reveal writes one recovery_credential_revealed event on the host's
customer timeline (info, source hub, Hungarian) via SaveEvent alone — no
dispatcher, nobody emailed, the log_tail_requested shape. Two reveals write two
events: the register records accesses, not states. A 404 is not an access. An
unbound host reveals fine and writes no event; the [INFO] hub line, carrying the
username and a length only, is then the record.
The global-key API path is untouched by design — it is the route for when the
hub UI itself is broken, and coupling it to the session layer would delete the
independence that makes it a fallback.
Recorded as a real trade: the hub session password alone now unlocks console root
fleet-wide, where retrieval previously also needed the global key. Accepted for a
single-operator, HU-geo-fenced hub that already stores these passwords in
plaintext at rest (CONTEXT.md ruling S-4). The plaintext-at-rest half is filed as
R-133 — every hub DB backup is a fleet-wide console-credential dump.
Tests 550 -> 559; four red-proofs (page leak, audit event, CSRF gate, route
order) each run, observed failing, and reverted. The route-order proof is a seam
test driving ServeHTTP: a handler-level test cannot see that defect, because the
handler is correct and simply never runs.
|
||
|
admin
|
acfc2b7e95 |
R-109 + R-122: the recipe assembly stops dropping sections (hub v0.83.0)
AssembleDRRecipe's hostHalfShape/appHalfShape are ALLOW-LISTS, not the forward-compat their comment advertised: a section an emitter adds is silently discarded until it is named in both the shape struct and AssembledRecipe. No error, no log, no failing test. R-122 (found this session): that already happened and shipped. The controller has emitted offsite_restic since fork-4 — the offsite recovery LOCATION — the hub stored it for all three real customers, and appHalfShape never listed the key, so no delivered recipe has ever contained it. It stayed green because the fixture drAppHalf is hand-written and omits the field. R-109: the agent's new backup_target is a new top-level host-half section and would have been dropped identically, making the fix read as shipped while changing nothing an operator can see. 3 tests built on halves read verbatim out of the live dr_recipe table, plus 2 red-proofs (each mutation asserted to have landed). vet rc=0, suite rc=0, 17 ok. Registers: R-106 + R-109 dispositioned; R-105/R-106 were READY in ROADMAP with no OPEN-ITEMS row (→ R-123, registered); R-124 filed on the "root" spelling. |
||
|
admin
|
1a68b53b06 |
hub v0.82.0 (R-120): the vouch path REFUSES a golden the fleet has already outrun
The golden's version IS the controller it bakes (build-golden.sh:345 defaults GOLDEN_VERSION to the controller tag), so a golden behind the newest deployed controller means every FRESH install lands on stale application code. On the R-120 occurrence that stale code shipped a customer-facing falsehood: a box from the 0.185.1 golden told a customer whose backup drive had fallen out that the backup was on the same disk as the system -- false, the drive was gone -- and offered a different drive as the remedy. WHY A GATE, NOT A REMINDER. The gap has opened three times: R-111 (golden's agent 17 releases behind), R-115 (agent built and deployed, never published), R-120 (this). The first two were closed by re-baking and remembering; remembering then failed again. R-29 is the standing proof that a check nobody runs is worse than none because it reads as coverage -- hostinstall_gates.py sat RED and uninvoked across three version bumps and hub_confirm_gate.py has never run at all. So the property that matters is not whether a check exists but whether it BLOCKS. - Wired into handleSetArtifacts (internal/web/configs.go), immediately before the only write, on the sole UI path to SetArtifactManifest -- it runs on every vouch without anyone choosing to. A script in scripts/ would have been a fourth orphan. - It REFUSES (operator ruling, 2026-07-30), with a flash naming the remedy. - Signal: store.NewestReportedControllerVersion() over reports.controller_version, SEMVER-compared in Go -- MAX() in SQL ranks 0.99.0 above 0.186.0, a pair this fleet has shipped. No outbound call, no new credential. - Fail-open in exactly two deliberate cases: an empty golden field (clearing the manifest is legitimate) and an unknown fleet version (a new hub must vouch its first golden). NEAR-MISS RECORDED: the first draft read guests.controller_version, a column that exists in the schema and that NOTHING writes -- it would always have seen "" and failed open, i.e. inert, this gate's own failure shape. Caught by grepping for a writer before trusting the column. Blind spot stated rather than papered over: a controller no box has ever run is invisible to this signal. Not the failure that has bitten -- all three instances were deployed-newer-than-baked. 4 tests through the PRODUCTION handler over httptest, never an injected seam. The refusal asserts both the flash and that the manifest was NOT written, because a gate that redirects and saves anyway reads as enforcement while providing none. Red-proof: deleting the block makes the stale golden vouchable and both assertions fail. ROADMAP R-29's audit list now records this as the FIRST enforced gate, so the contrast with its three orphans is kept rather than lost. The orphans are unchanged. Suite rc=0 read separately from this commit. |
||
|
admin
|
1257014c2b |
hub v0.81.0 — E-2: backup_target_absent gets its own signal (ships first)
An event type the hub does not allowlist makes POST /event return 400 and the
event vanishes (R-97a). The controller cannot emit backup_target_absent until
this is live, so the hub half ships first.
E-2 Phase 0 established that an absent backup target has NO prompt signal today.
The controller's drive-gate path stops apps and logs a WARN but emits nothing:
NotifyStorageDisconnected is defined and never called anywhere (verified against
the gitignored-cmd/ trap with a positive control). A drive that is ONLY a backup
target has no apps to stop, so it is entirely silent. The sole signal is the
tier's own failure at its next due cycle -- up to ~24h on the daily local tier,
which is the R-100 shape: a real fault visible only after a deadline elapses.
Added to BOTH registers, because each half fails differently:
allowedEventTypes -- without it the event is lost at the door;
customerMessages -- without it the event IS delivered but in the controller's
raw operator English, and nothing looks broken.
backup_target_absent is deliberately NOT folded into storage_disconnected: that
says "a drive went away and some apps may have stopped"; this says "the thing
that makes your backup survive a disk failure is gone".
Hungarian copy names the consequence, not just the fact. backup_target_restored
is the paired recovery at info severity -- severityNotifies NOT widened.
Three tests pin the pair and the copy's substance. All red-proofed with the
mutation VERIFIED to have landed first: the initial attempt silently no-op'd
(gofmt had realigned the map) and the test "passed" -- a false proof that would
have been reported as evidence.
Green gate: build + vet + test rc=0, run separately from this commit.
|
||
|
admin
|
b505ee9125 |
R-100: offsite staleness counts from the last SUCCESS (hub v0.80.0)
isStale counted from last_run, written unconditionally on failure, so a nightly-failing tier read as fresh forever. Now anchored on last_success with an explicit legacy degrade (logged once) and the never-ran branch untouched. emitStale states the real reason. |
||
|
admin
|
2c0e43e0d0 |
hub v0.79.0 — R-97c: make the operator-only claim true
v0.78.0 asserted in a comment that a type with no customerMessages entry cannot reach a customer. It can: templates.go falls back to the raw message when the entry is missing, and the only customer gate is prefs.EnabledEvents — pure configuration. A customer with whole_guest_backup_failed enabled would have been emailed raw English operator text about a backup they cannot act on. The new test proves it against the v0.78.0 shape. operatorOnlyEvents is now an explicit register checked before prefs, logging a skipped/operator_only row so the skip is visible. NOT implemented as 'missing customerMessages blocks delivery' — several types rely on that fallback on purpose. The handler comment now names the real mechanism. |
||
|
admin
|
331193b898 |
hub v0.78.0 — R-97a: whole-guest backup events, operator-only
internal/quiesce had no route to the hub at all: three failed whole-guest backups on 2026-07-27 produced zero events. Hub half of the fix. whole_guest_backup_failed / _recovered are allowlisted with NO customerMessages entry. Deliberately not backup_failed/backup_completed — those have customer Hungarian templates AND sit in demo-felhom's live enabled_events, so reusing them would email the customer that their backup failed while it is still retrying behind the R-88 breaker. The recovery joins recoveredPairedDownTypes because it is severity info and severityNotifies drops info — otherwise the operator hears it break and never hears it heal. Its customer leg is pairing-gated and can never fire. Operator cooldown gains a per-tier dimension from the event details, so one tier cannot mask another for an hour. Narrow: empty suffix unless a tier is sent, so no existing event type changes. |
||
 Claude Code
|
ce4e03dcd8 |
hub v0.77.0 — R-85 Part 2: a restore-test result becomes a SIGNAL
A failed restore-test was a [WARN] line in the ingest handler and nothing else — no event, no notification, no gauge. True for the LOCAL tier that was already being tested, so the loudest DR signal this system produces was inaudible. Rotating tiers without this would only mean two tiers can fail silently instead of one. Two signals, deliberately NOT merged: restore_test_failed (error) — a run completed and did NOT pass restore_test_stale (warning) — a tier not PROVEN within its interval Merging them collapses 'your DR is broken' into 'your DR is unverified', and the second is the one that quietly becomes the first. The staleness wording says 'unverified, not known-broken' and a test asserts that phrasing. Anchored per R-81, not re-derived: a never-proven tier on a newborn box is UNKNOWN, not FAILED, until the window elapses. This family has made the opposite mistake three times; this monitor was written straight after the third, so it copies R-81's structure rather than inventing a fourth shape. restoreProvenStaleAfter = 7d is derived: oldest-first over two tiers at a 24h cadence proves each ~every 2 days, so 7d tolerates ~3 missed opportunities and sits inside the 2-week offsite retention. Per-tier proof comes from the hub's retained host-report window — the agent reports only its latest run, so the latest report alone cannot answer 'when was the OTHER tier last proven?'. Reused R-81's mechanism instead of a wire change. Both types registered in allowedEventTypes (R-77's inert-seam lesson) and operator-tier only — no customerMessages entry. FIXED a time bomb I introduced in Slice C: the restart-blind-window test hard-coded 2026-07-18T18:31:06Z while comparing against the real clock. Harmless under one 26h threshold; once the offsite tier got an 8-day limit it passed all day and began failing at 18:31 UTC, exactly 8 days later. Now relative. Red-proofs B and D observed. Full suite green (17 packages, rc=0). |
||
|
Claude Code
|
b11607b26b |
hub v0.76.0 — R-82 Slice C: tier-aware backup thresholds
R-81 merged every backup signal into one 'newest' against a single 26h limit. backupStaleAfter's own comment recorded why that stops being right under a weekly offsite tier. Each tier is now judged against its own threshold; R-81's structure (three verdicts, anchored absence, distinct reasons) and its boundary test are preserved intact. - offsiteBackupStaleAfter = 8d (7d cadence + headroom); backupStaleAfter keeps 26h and now names the HOST tier only - splitTiers / assessTier / newestBackupEvidenceByTier Slice-A.4 rule implemented: a PBS-targeted vzdump appears in BOTH arrays, so classification is by TARGET TYPE (target_id -> storage_targets[].name -> type), never by array membership — otherwise a PBS backup makes a stale host tier look fresh. storage_targets is used rather than pbs_dr.storage_id because the latter is null on a box with a PBS storage but no DR descriptor. A tier is only judged when the box HAS it, else every box without an offsite tier would alarm once the anchor elapsed — R-81's mistake one level down. With neither tier identifiable (old agent) the pre-Slice-C path runs unchanged. Intended behaviour change: a 30h offsite snapshot no longer alarms. Three fixtures asserted the merged threshold; each still asserts an alarm at the correct limit. No assertion was weakened. RECORDED LIMITATION: the hub infers 'PBS => weekly' from storage type. defaultBackupTarget is felhom-pbs, so a box that never sets local_backup_target would run PBS as its DAILY tier and be judged against 8 days — 7 days of blindness. No box is in that shape today; the real fix is the agent reporting per-tier cadences. Own task. Red-proof observed. Replayed live: demo-felhom OK, demo-hp UNKNOWN (defers correctly), drill-r50 MISSED (true positive). No customer email would be sent. |
||
|
Claude Code
|
f5a5e2b911 |
hub v0.75.0: R-81 — "no signal" is not "bad signal" (anchor the backup deadline check)
Third instance of one class (hub v0.12.0, v0.73.0, this), fixed as a class. On 2026-07-26 03:00 UTC expected_backup_missed fired on demo-felhom, demo-hp and drill-r50 at once; the demo-felhom one reached the CUSTOMER channel claiming "newest backup is 176h0m0s old". Nothing was wrong — three vzdump archives were on disk. Cause: the agent backup store is in-memory, so the R-50 fleet restart emptied `backups` until the next run, and the hub read empty as "no backup exists". - assessBackupFreshness returns OK/UNKNOWN/MISSED instead of `missed bool`; absence is UNKNOWN until it outlives an anchored window. Still pure. - store.GetHostReportsSince + monitor.newestBackupEvidence read the hubs own retained history (bounded 7-day lookback, early-exit on fresh evidence) — "when did I last SEE evidence of a backup?" The anchor was free: the hub already retains 90 days. No agent change, no new persisted state. - store.GetFirstHostReportAt anchors absence at first contact, reusing the existing 26h threshold as the grace (no new knob, the v0.73.0 shape). - Deferrals logged + counted; reason strings kept distinct. - backupStaleAfter untouched; landmine recorded (a weekly PBS snapshot would alarm six days in seven) and owned by R-82. Tests 493->508. Red-proofs A/B/C observed and restored; A reproduces the live message verbatim. Replayed the real 03:00 reports (600/417/77 rows): all three now silent. Source: documentation/audits/DIAG-backup-missed-2026-07-26.md |
||
|
admin
|
9cfa619ec3 |
hub v0.74.0: allow local_api_endpoint_drift; R-77 docs + R-78/79/80
The allowlist entry is REQUIRED, not cosmetic: handleEvent 400s an unknown event_type, so controller v0.173.0's new drift alert would be silently inert without it. Shipped with the controller that emits it. Docs: - RUNBOOK-local-api-endpoint-drift.md — how to repair a drift, including the step everyone will want to skip (establish which value is CORRECT from what the agent is actually bound to, rather than assuming bootstrap.json wins) and what success looks like (SILENCE, not a "recovered" line, because a fresh controller's healthy first observation is not logged). Records both 2026-07-26 repairs. - ROADMAP: R-77 shipped; R-78 the local_api authority ruling, with the clobber-a-working-channel risk spelled out in BOTH directions so it is not resolved opportunistically; R-79 the whole-surface English-strings sweep; R-80 expected_backup_missed, flagged as likely outranking R-77 because 7.3 days of stale backup materially exceeds the ~1.5-day channel outage, so the causal link the DIAG hedged on cannot be the whole story. - Capability map: note against the drive-wizard row (every agent-backed capability rides this channel) that a silent drift class is now detected. NO row status flips — detection is not prevention. |
||
|
admin
|
485321f694 |
R-50 Phase A: host-install v1.19.0 island default + hub version sync
- felhom-host-install v1.19.0: portless vmbr9 island bridge, appliance binds local_api on 169.254.253.1:8443, writes island_bridge/island_guest_addr, pins lan_resolver.host_ip to the LAN IP (Finding-1). --no-island opt-out. - hub hostInstallVersion 1.16.0 -> 1.19.0 (F-1 sync). hostinstall_gates PASS. - Pairs with agent v0.96.0 (attaches guest net1). byo unchanged. Coupling: island install requires agent >= 0.96.0 (vouch first). |
||
|
admin
|
19dbb02ee3 |
hub v0.73.1: allowlist disk_health_degraded (controller v0.169.0 disk-health)
Adds disk_health_degraded to allowedEventTypes so the controller's per-disk SMART degradation notification is ingested, not 400-rejected. Deliberately no customerMessages entry (like offbox_enlarge_blocked) — the controller's dynamic Hungarian message (disk label + attributes) is preserved by the templates.go fallback. Test + red-proof. |
||
|
admin
|
b03a53ddcf |
hub v0.73.0 — offsite_stale anchored on newborn tiers (never-ran = applied-only + consumed_at/escrow anchor; one state one owner)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR |
||
|
admin
|
1133aade73 |
hub v0.72.0 — R-70 + R-71c: offsite delivery-state detector, card, stuck event, R-39(a)-guarded self-heal restage
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR |
||
|
admin
|
c766c8af82 | hub v0.71.0: paired recovery mails (F11), prefs seeding at claim + empty-email no-clobber (F12), priority headers + operator test leg (F14-light) | ||
|
admin
|
f4c2c4151b | hub v0.70.1: the ghost customer's Delete button must exist (Danger-zone render gate split) | ||
|
admin
|
9b3381be0a |
feat(hub): v0.70.0 — a deleted customer actually disappears (residue leg + ghost cleanup)
Found validating v0.69.0 against the live hub. demo-vm-felhom was deleted on 07-18 and was still on the Customers list AND still raising offsite_stale (10 events, latest 07-21 17:34, operator email at 19:34) — because GetCustomers() is report-derived and no lifecycle tier ever deleted a report. New leg 3 (residue), before the record purge: reports, app_telemetry, app_log_tails, log_tail_requests, customer_notifications, plus the credential-bearing appliance_registrations and selfbind_tokens. Audit (events, notification_log) and F-14 provenance still survive. Ghost customers are now deletable: 404 means "nothing here", not "no config row". With no config row the offsite descriptor is unknowable, so the Hetzner and descriptor legs record skipped_no_config rather than a bare "skipped". Two more red-proofs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS |
||
|
admin
|
61dbd870c3 |
feat(hub): v0.69.0 — customer DELETE is the guided full-teardown cascade (R-25b)
POST /configs/{id}/delete now runs hosts -> RESET -> purge behind three
acknowledgements, a typed customer-id, a stale-preview check and the
ONLINE-host refusal (every gate before any write, so a refusal has zero
side effects). The shallow handleConfigDelete is gone.
Two invariants are asserted, not just commented: ruling 3 is preserved by
construction (leg 2 never sees a host row) and retained escrow custody is
purged exactly once, in leg 3 (leg 2 runs with purgeEscrow=false).
handleCustomerReset's committed half was extracted as commitCustomerReset;
the standalone RESET path is byte-identical to v0.68.1 and its suite is
untouched. Five red-proofs run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS
|
||
|
admin
|
4a4233059d |
hub v0.68.1 — fix the Configuration layout broken by the wrapper-sha field
The v0.68.0 row wrapped itself in a <div>, but the artifacts <form> IS the CSS grid (display:grid, no inner container). The stray </div> closed the surrounding card from inside the form and the new <div> was never closed — it swallowed the submit button and ran to </form>, so the row rendered outside the card and Save landed inline. Reported by the operator on first use. The field still submitted (it stayed inside the form), so this was layout damage rather than data loss, but the unbalanced markup put every section below it in the wrong container. Fixed as plain grid cells (grid-column: 2/4), no nested elements. There was no render assertion on this form at all, which is why a hand-edit broke it silently. The new test asserts the field is inside the form, the button has not escaped, the form contains ZERO divs, whole-page div balance holds, and the sections after it survive. Red-proofed against the broken shape. |
||
|
admin
|
107f74ea3c |
hub v0.68.0 — auth_failed self-heal, consumed_at honesty gauge, wrapper drift (R-39 + R-50b(a))
Completes the hub half of R-39's fleet fix on top of the generation core (
|
||
|
admin
|
c484aa204e |
hub: R-39 core — stamp a secret GENERATION into the pbs_dr descriptor
The fleet half of R-39. An ep0 credential re-issue re-keys the SECRET of an existing token, so token_id, fingerprint, datastore and namespace all come back byte-identical. The agent re-applies on the descriptor's CONTENT HASH, so a re-issue was invisible to a converged box: it short-circuited, never consumed the fresh secret, and served a revoked credential while reporting `applied` — the N100 failure of 2026-07-18. host_pbs_secrets gains a monotonic per-host `generation`, advanced by every fresh MINT and by nothing else, stamped into the descriptor as `secret_generation`. That is now the only field a re-key moves, and it is what re-arms the agent. DEVIATION FROM SPEC, deliberate: the brief said to return "the new row's id (int64) … no schema change". There is no row id — host_pbs_secrets is keyed by host_id and UPSERTed last-write-wins, so a new row never exists, and created_at collides for two mints in the same second. An additive counter column is the only monotonic source; it uses the repo's existing idempotent ALTER-TABLE idiom. RestageHostPBSSecret deliberately does NOT advance it: a re-stage re-arms the SAME secret, the descriptor content genuinely has not changed, and a bump would cause a pointless agent refetch loop (that method's own contract says so). Also corrects a comment that asserted the re-issue refreshes the descriptor "with the NEW token_id/fingerprint". That is false for a re-key, and believing it is why the descriptor was never expected to be identical in the first place. omitempty is load-bearing: a zero generation must not start emitting a new key into every pre-existing descriptor, which would itself be a fleet-wide spurious re-apply. Compatibility: agents below 0.91.0 drop the unknown JSON key and behave exactly as today — inert, not breaking (Scenario C). Tests: store-level monotonicity + per-host isolation + restage-leaves-it-alone; descriptor byte-change, omitempty, and sibling-key round-trip; and a FLOW-level test driving ReissuePBSDR against a fake that models a real re-key. Red-proof run at the assertion level (not the compiler): commenting out the stamp makes the flow test fail with both byte-identical blocks printed. |
||
|
admin
|
b6d537d86c |
hub v0.67.0 — auto-minted self-bind link, post-RESET staleness, unprovisioned-offsite warning
Four small items, each a case where the hub already knew something and said
nothing. Green: build, vet, tests all pass.
(a) Self-bind link is minted automatically at customer creation AND at RESET
completion (R-36 sub-item). The console banner tells the customer to open
"az e-mailben kapott link"; until now that email existed only once the
operator remembered the button, so the banner could point at something that
did not exist — during the 2026-07-18 rehearsal the box waited ~11.7 min on
exactly that. handleSelfBindLinkSend's body was extracted into a shared
mintAndSendSelfBindLink core so the button and the auto-mint callers cannot
drift apart on the honesty rules: F1 (no address -> mint nothing) and F2
(send failed -> delete the token, never leave it live). The wrapper NEVER
fails the operation it rides on — a create that provisioned Cloudflare,
offsite and PBS must not 500 over a courtesy email.
Gap found and closed while wiring it: PurgeCustomerResetDBState does NOT
clear selfbind_tokens, so a link minted BEFORE a reset would have stayed
live across it. A successful mint already replaces it (delete-then-insert,
single-active); the skip paths would not have, so they now clear stale
tokens too. Invariant: after auto-mint runs the only live link is one it
just issued, or none.
(b) Post-RESET staleness banner (R-37). When a RESET COMPLETED after the newest
report, every health figure on the page describes a lifecycle that no longer
exists, and the page kept showing pre-RESET warnings as current. Narrow on
purpose: an in-flight reset does not trigger it, and it clears itself when a
report arrives. Ties resolve to STALE — SQLite timestamps are second-
resolution and a same-second report almost certainly predates the reset;
erring the other way would hide the banner exactly when it matters.
(c) Unprovisioned-offsite warning (R-36 interim). enabled==true with type=="" is
a real, stable, silent state: provisioning is Save-triggered and the
re-enroll auto-re-issue deliberately skips an unprovisioned target, so
nothing self-heals it. Reuses the exact predicate the offsite re-issue
handler already refuses on.
(d) pbsdr_reissued rendered an EMPTY flash box — the key had no template branch,
so re-issuing PBS credentials showed a success box with no words (observed
live 2026-07-18). Now describes what was staged plus the R-39 caveat:
confirm `pvesm status` shows the entry active, because a converged agent can
report `applied` while the storage still 401s.
New .flash-warn (amber, --warn tokens) for the deviation tier between success
and error — exception-color principle: only on deviation, never on a healthy
page.
Tests assert each banner is ABSENT in the nominal cases as well as present in
the deviating one — a banner that always renders is worse than none. Both
red-proofed: deleting the pbsdr_reissued branch reproduces the original empty
box; neutering the staleness predicate fails the banner assertion. New
read-only store accessor CountSelfBindTokens makes the single-active invariant
assertable.
NOT in this train: the R-39 hub-side generation-bump fix the pre-travel task
made conditional. Its condition was REFUTED (SetHostDesired bumps
unconditionally; applyPBSDR is idempotent as documented) — the real mechanism is
the agent's descriptor-hash convergence and needs its own spec.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
|
||
|
admin
|
592818492c |
hub v0.66.0 + ISO v1.20.0: customer self-bind (R-27 slice 1)
Let a customer bind their own freshly-installed appliance without the operator: operator "Send self-bind link" mints a 7-day tokenized capability link, emailed (Hungarian, sibling sender) to the customer, who opens a public /bind/<token> page and proves two factors — the console pairing code shown on the box screen + their retrieval passphrase — and the hub stages the bind via the same BindAppliance (provenance customer_selfbind). The box's ~30s appliance poll delivers. Viktor's three rulings verbatim: console pairing code (no appliance list ever rendered), operator-sent tokenized link, 5-attempt lockout -> "call support". Wrong code == wrong passphrase (one generic failure, no oracle, both factors compared unconditionally); expiry falls back to operator-bind unchanged. THE TRAP: one public prefix /bind/, exempt from auth+CSRF at both /login gate sites via a single isPublicBindPath predicate (tight trailing-slash match; ServeMux ..-cleans; handler rejects '/' in token). 9 tests (Scenarios A-F + F1/F2); 4 red-proofs verified red-then-green (lockout, oracle, widened-prefix, single-active). GC verdict: no appliance GC -> the 7-day TTL stands alone. Controller/agent untouched; R-27b deferred. Green: full hub build/vet/test (17 ok) + bash -n + hub confirm gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017qDiBqKKQ5vPB5fXBqu7Kp |
||
|
admin
|
7f11cfb36c |
hub v0.65.0 — PBS DR storage visibility (ep0 usage op) + Offsite tab split + dual dashboard gauges (R-5)
Makes PBS DR storage visible like the restic pool box (v0.64.0), differentiated. Scoping
correction: restic = subaccounts on the shared Hetzner Storage Box (Hetzner API); PBS DR =
the felhom-offsite PBS datastore on the ep0 endpoint VM (NO Hetzner API). Option A
(Viktor-ruled): a read-only `usage` op on the felhom-tenantsync ep0 forced command (twin of
fingerprint), polled by a new hub checker on the 15-min throttle. READ-ONLY throughout.
Phase-0 (gate PASSED): on ep0 (PBS 4.2.3), df -B1 --output=size,used,avail <datastore path>
yields bytes (39990112256/7627939840/... ~19%), read-only, existing sudo context, no admin token.
- scripts/felhom-tenantsync.sh -> v1.2.0: read-only `usage` short-circuit (df on the datastore
path), no customer_id, no admin token, NO mutation. + a bash harness proving zero mutation.
- tenantsync.Client.Usage() + BoxUsage; unknown-op -> typed ErrUsageUnsupported (graceful).
- monitor.PBSDRBoxChecker: OffsiteBoxChecker clone over a usageReader seam; 15-min throttle,
cached PBSBoxSnapshot, escalation-only pbsdr_box_fill on the "pbsdr-box" scope (operator only,
no SaveEvent), recovery re-arm. Fill only. THREE states: ok / unavailable (ep0 <=v1.1.0,
neutral no-alert) / degraded (exec failed, keep last).
- config: Alerting.PBSDRBoxFill{Warn,Crit}Percent (80/90); built with the tenantsync client,
60s sweep, SetPBSDRBox. Hub deploy INDEPENDENT of the ep0 update (graceful degradation).
- web: /offsite splits into Restic + PBS DR hash tabs (endpoint cards under PBS DR); PBS panel;
the single dashboard tile becomes two gauges (RESTIC pct.ratio, PBS DR pct / n/a).
- runbook offsite-endpoint.md 10: v1.2.0 update steps (no sudoers/authorized_keys change).
Tests: 10 Go + the harness; 3 red-proofs (usage mutation, escalation-only, unavailable-drives-band)
confirmed red then restored. go build/vet/test + bash -n + hub confirm gate all pass.
|
||
|
admin
|
a4268c3707 | hub: offsite pool-box checker logs the pool trend on each refresh (operator visibility) | ||
|
admin
|
4bb2df0dc4 |
hub v0.64.0 — offsite pool-box aggregate: fill, oversubscription, per-customer bars, operator alert (R-5)
The operator sees the shared pool box's real state on the hub: total box fill vs
capacity, Σ(shared soft quotas) vs capacity (the oversubscription ratio), per-customer
usage/quota bars, and a box-level operator alert (fill % + oversub ratio) on the existing
dispatcher's operator channel. Per-customer fill alerts already existed; the box-level
aggregate was the gap. READ-ONLY against Hetzner (GET only).
Phase-0 probe (gate PASSED): the live pool box 611714 returns capacity via
storage_box_type.size (1 TiB / bx11) and usage via a stats object (size/size_data/
size_snapshots), all bytes; our token reads it (200).
- hetznerapi: additive StorageBoxType + StorageBoxStats on StorageBox (no existing field/
method changed); fake carries them + a GetBoxCalls counter; golden decode test.
- monitor.OffsiteBoxChecker: OffsiteChecker-sibling for the box; fetch-throttled (1 GET/
15min), cached BoxSnapshot, escalation-only + recovery re-arm. FILL (used/capacity 80/90)
+ OVERSUB (Σ shared+enabled quotas / capacity, 2.0x) — independent. Σ from the ConfigJSON
Descriptor (offsite.ReadDescriptor, new), never the report echo; dedicated+disabled
excluded. Scope "pool-box" -> operator channel only, no SaveEvent. Failed fetch keeps the
last snapshot degraded; missing data never becomes 0% and never transitions a band.
- config: Alerting.OffsiteBoxFill{Warn,Crit}Percent + OffsiteOversubWarnRatio (80/90/2.0
defaults; thresholds pending Viktor's ruling). Constructed in the HETZNER_TOKEN branch,
60s sweep, snapshot handed to the web server.
- web: Offsite-tab panel (fill bar, Σ+ratio, per-customer usage/quota rows) + a compact
dashboard tile; reads the cached snapshot only, never fetches; nil -> "not configured".
Tests: 10 new + 4 red-proofs (throttle, Σ filter, escalation-only, failed-fetch honesty),
all confirmed red then restored. go build/vet/test all pass; hub confirm gate OK.
|
||
|
admin
|
30972d8f54 |
hub v0.63.0 — system-initiated immediacy: wire poke/bump at every mutation site that lacked one
The immediate-sync arc covered only operator-initiated desired-state changes; system-initiated mutations bumped the generation silently, so a freshly onboarded box waited a full agent tick for state the hub had already minted (observed live at slice-C onboarding). Wire the existing, live-proven notifiers into every system site on the correct plane — call-site wiring only, no new mechanism. Agent plane (poke.Notifier): - web/pbsdr.go: PBSDRAutoProvision (the observed lag), ReissuePBSDR (also lifts the pbsdrheal reconciler escalation, zero reconciler changes), handlePBSDRReissue — each pokes AFTER the successful SetHostDesired, never on a blocked/error path. - api: new nil-safe Poker seam (PokeHost/PokeAllHosts + SetPoker); handleAdminSetDesiredState pokes the target host; handleAdminSetOperatorPeer fires PokeAllHosts only when the fleet generation bump succeeded (fire-after-commit). - main.go: one poke.Notifier now feeds both planes (SetPoke + SetPoker). Controller plane (intent.Hub.Bump): - api/reissueOnReenroll: one nil-guarded bump so a long-polling controller wakes in seconds instead of on the 15-min cycle. Deliberate non-sites (unchanged): WG register (undeliverable pre-tunnel — the agent fast-tick SECONDARY owns it), WG delete (transport removed), pbsdrheal Restage (no generation bump → the 60s ticker is the pickup path). internal/pbsdrheal byte-unchanged. Tests: 10 non-hollow tests (web async channel-synchronized fake sender; api synchronous fake Poker) with explicit zero-count negatives; representative red-proofs per group (A/B/C/D) run-fail-restored. Green: go build/vet/test all pass. |
||
|
admin
|
36c5cd5fdf |
hub v0.62.0 + scripts v1.19.0 — R-21 slice C: the universal secret-free ISO
A generic ISO carries NO customer secret. The box registers itself at the hub as an unclaimed appliance; the operator binds it to a customer; the hub delivers the customer-id + retrieval passphrase ONCE; day-0 completes via the slice-A path. Hub (v0.62.0): - store/appliance.go: appliance_registrations keyed by (uuid, mac_set) — MAC set is the tiebreaker (duplicate SMBIOS UUIDs); token stored as sha256 only. Idempotent register (sticky-discard), atomic one-shot delivery, bind/discard. - api/appliance.go: POST /appliance/register (the one unauth endpoint, per-IP rate-limited, 256-bit token); GET /appliance/poll (404 no-oracle / 204 unbound / 200 deliver-once / 410 delivered). Passphrase read live, never logged. - web/appliances.go: Hosts-page "Unclaimed appliances" section + BIND (customer picker, host count display-only) + DISCARD; SSH host-key fingerprints; events. - Red-proofs: one-shot delivery + register idempotency (both proven red); 404-no-oracle, sticky-discard, bind staging, render. Green + confirm gate. Scripts (v1.19.0): - felhom-bootstrap.sh: ONE unit, TWO modes. Direct (env has customer/passphrase) = slice-A path, byte-identical, only branched around. Pairing (generic) = register + poll (RestartSec=30 is the poll timer); on delivery write the env 0600 and fall through to direct. Secrets + token shredded on success. - build-felhom-iso.sh --pairing: generic secret-free ISO, -generic filename, manifest mode=pairing. profiles/generic.profile (new). - test/bootstrap-modes.sh: Scenario D (direct = zero appliance calls) + pairing register/poll + delivery handoff — all green in a debian container. |
||
|
admin
|
4009401f46 |
hub v0.61.0 + felhom-tenantsync v1.1.0: Customer RESET (middle lifecycle tier)
One operator action returns a customer to pre-first-install: all operational state dies (offsite repo, PBS namespace+backups, DR recipe, one-time secret, claim state, retained escrow custody); identity + basic config + provenance + events survive. Sits between host delete and customer Delete. - store/customer_reset.go: customer_resets journal, live inventory, ack-gated purge (never touches identity/provenance/events), DeleteClaim. - claim.ResetToUnclaimed: delete claim row -> fresh code next onboarding. - offsite.Deprovision (idempotent) + OffsiteIdentifier + ClearProvisionedDescriptor. - tenantsync.Deprovision + felhom-tenantsync.sh deprovision op (destroys ns + backup groups + token; shared user untouched; idempotent). - web/customer_reset.go: GET reset -> inventory JSON; POST -> orchestration (external teardown FIRST, DB purge LAST; refuse-while-hosts; typed-id + separate escrow ack). Amber RESET card distinct from red Danger-zone Delete. - Red-proofs: ack-gate + partial-failure resumability (both proven red); store ack-gating + journal round-trip; offsite idempotency + descriptor clear; RESET-card render. Green: build + vet + test. |