R-198 — host_escrow_superseded shipped with `blob` (the K-escrow / PBS datastore key) and
identity_blob was added to host_escrow LATER, never here. The offsite restic REPOSITORY
password lives in identity_blob. So demoteCurrentEscrowTx -- whose own comment calls it "THE
ONE escrow row-copy routine" -- retained the whole-guest key and silently dropped the off-site
data key, which is the secret the retention was built to preserve. And because the copy happens
as the new blob overwrites the old, the destroying act was the ESCROW CEREMONY: the exact thing
a rebuilt box tells its customer to run, on a card promising in Hungarian that the old backups
stay recoverable. Both demo boxes crossed that line on 2026-08-04.
- identity_blob added to the table (CREATE + additive ALTER) and carried in the shared copy
routine, so BOTH callers are fixed at once: re-escrow and host-delete demotion.
- ListSupersededEscrow reads it back; store.HostEscrow gains IdentityBlob.
- CountCurrentEscrowWithIdentity is the census of who the fix protects.
- Nothing is backfillable: pre-v0.93.0 retained rows have no blob and their sources are gone.
- Tests assert the CONSEQUENCE (a retained row can still yield a repo password), which is why
the pre-existing retention test stayed green for two months asserting the mechanism.
R-197 — SaveHostEscrow returns the hash it replaced; the escrow PUT raises
offsite_repo_key_changed (warning, operator-only, edge-triggered) when both hashes are known and
differ. No hash value travels. Severity chosen for the world v0.93.0 creates: with the identity
blob retained, a changed key is "this history now depends on an older recovery code", not a loss.
R-192 (half) — the stuck alert now reports the two shapes it actually covers, burned and
regressed, each stating its own measurement; the regressed text withdraws the Re-issue
recommendation. Every self-heal refusal leaves a notification_log row with its reason. The
guard's logic is unchanged; its 500-oldest-reports scoping stays OPEN and the window is named in
the alert text so the limitation travels with the number. offsite_delivery_stuck and
offsite_credential_restaged are added to operatorOnlyEvents -- neither was registered and neither
has a customerMessages entry, which is not a block.
R-196 — five comments (not the three the spec expected) claimed ReissueCredentials rotates the
restic repo password. It resets the PROVIDER password and cannot touch the repo password, which
is generated on the box. All five corrected; the staleness mark documented as precautionary. The
BEHAVIOUR stays open.
Not in this release: R-199, R-200, R-201 remain open -- the chain that hands the key back is
still unassembled. Part 5 hit its gate; the orphan card is untouched (R-202).
The deliberate hostInstallVersion const is removed. It existed only to produce a real red
run (#3-#6) and the demonstrated alarm; R-94's deletion stands.
The Setup tab said 'host-install 1.19.0' while the served script was 1.22.0, and had
been wrong since 2026-07-14. Deriving the number honestly is not possible: the Option-1
command downloads felhom-host-install.sh from the website at RUN TIME and the website
git-syncs main every 30s (R-110), so no build-time value in the hub can be true. R-94(a)
offered derive-or-delete; deleted, which removes the drift class instead of automating it.
- configs.go: hostInstallVersion const, pageData.ScriptVersion field and its assignment
all removed; a NOTE in their place records why there is no constant here.
- customer_unified.html: the sentence now says the command always fetches the current
installer, and renders no version.
- hostinstall_gates.py gate 1: the third assertion INVERTS — it used to require the hub
const to equal SCRIPT_VERSION, it now asserts the hub carries no host-install version
literal at all, matched in six code shapes across every .go/.html under hub/ (comments
are deliberately not stripped: a // inside a URL literal would blind the scan).
- render_test.go: the assertion 'html contains hostInstallVersion' compared the constant
to itself and passed at ANY value — demonstrated green with the const at 9.9.9 while the
script was 1.22.0. Deleted, not replaced: there is no longer a version to assert.
- felhom-host-install.sh: COMMENT ONLY (SCRIPT_VERSION untouched) — it claimed the gate
keeps the hub copy equal, an invariant that no longer exists.
Red-proofs: restoring the const fails the rewritten gate 1 (3 shapes hit); the old
render_test assertion passes at 9.9.9.
Found by the operator, in the way that matters: it cost a real login.
The v0.84.0 Console access card shipped its Copy button DISABLED until a Reveal.
Clicking it did nothing, silently, so the clipboard kept whatever was already in
it — another host's console password from an earlier reveal. That got pasted into
demo-hp's PVE login, which failed with no explanation: the box logged a plain
`password check failed for user (root)`, the credential was never at fault, and
nothing on screen said the copy had not happened.
A copy button that silently no-ops is worse than no copy button. The operator
cannot tell "copied" from "did nothing", and the stale value left behind is a
VALID secret for a DIFFERENT machine — so the failure looks like a stale
credential and sends you diagnosing the wrong thing.
Copy now works without revealing, and that is the safer default rather than a
concession: the secret goes straight to the clipboard and never renders on
screen, so it cannot be shoulder-surfed or caught in a screenshot. Reveal remains
for when it must be read.
Three silent-failure branches closed, all in the same eight-line function:
- not yet revealed -> was a disabled no-op; now fetches and copies
- navigator.clipboard absent -> was silently skipped; now shows it and says why
- writeText() REJECTED -> promise was ignored, so the operator believed it
copied; now shows it and reports the refusal
The success path names the host ("Copied demo-hp-bb76ea's root@pam password"),
because the clipboard is fleet-wide and every box has a different console
password — "copied" alone cannot say for WHICH box, which is the confusion that
produced the incident.
One retrieval path, shared: the endpoint is defined once (data-reveal-url) and
read back with getAttribute, so Copy cannot drift onto a different, unaudited URL
than Reveal. Server-side is unchanged — both buttons hit the same CSRF-gated
endpoint and both write the same recovery_credential_revealed event, which is
correct: the register records accesses, and a copy is an access.
Tests 566 -> 568. Red-proof: re-adding `disabled` reproduces the shipped bug.
Pairs with agent v0.119.0 and is useless without it.
A managed box's LAN IP was not shown anywhere in the hub, because nothing
reported it — the host report carried no address of any kind. The only IP
reachable from the UI at all was the WireGuard one, on /offsite's peer table
keyed by pubkey, so an operator could go peer->host and never host->peer, which
is the direction anyone actually asks in.
The host page grows a Network card: every routable address the box holds, one row
per (interface, address), plus a WireGuard row. On demo-felhom that is vmbr0
192.168.0.162/24 and tailscale0 100.70.170.35/32 — with the PVE web console at
https://<the LAN address>:8006, the thing the operator wanted and could not get.
WireGuard is rendered as TWO facts, deliberately. WGAssignedIP is the hub's own
allocation (wg_peers, authoritative desired state); WGConfirmed is whether the box
reports actually holding it. Showing the allocation alone would make a peer that
was never applied look healthy — the same shape as reading a timestamp that
records an attempt as if it recorded a result.
The split is keyed on the ALLOCATION, not the interface name: wg-felhom is the
agent's current unit name, and a UI keyed on that string would silently
mis-render the day it changes.
An old agent renders UNKNOWN, never "no addresses". Below agent 0.119.0 the field
is absent from the wire, and an absent signal is not a negative result — the page
says so and names the version needed. Rendering an empty list there would have
stated something false about the host.
No new store table and no new ingest path: the report is already stored opaquely
and GetWGPeerForHost already existed with no UI consumer. This is parse + render.
The report fixture in the tests is the REAL wire — the addresses block copied out
of `felhom-agent --selftest=hub` on demo-felhom running 0.119.0.
Tests 559 -> 566; four red-proofs (inert view-model, unconditional confirmation,
the old-agent branch, and the drift case) each run, observed failing, reverted.
The credential existed and was not reachable when it was wanted. Every box has
had a strong random root@pam password since TASK G1, vaulted in the hub at day 0
and used for real during the sshd incident — but the only way to read it back was
a hand-written curl carrying the global operator key, a secret kept out-of-band.
In practice the PVE web console on a demo box felt locked.
The host page grows a Console access card: presence + username + set_at by
default, Reveal fetches the plaintext on demand for 60 s with a Copy button.
Masking clears the JS variable, and also fires on a second click and on
visibilitychange. A host with nothing vaulted says so, and says why.
The secret is NEVER rendered into the page, and that constraint shapes the
change. The render path uses a new store.GetHostRecoveryMeta whose struct and
SELECT both omit the secret column, so it is structurally incapable of carrying
one. The plaintext crosses the wire only in the response to POST
/hosts/{id}/reveal-recovery-credential (Cache-Control: no-store, CSRF-gated at
the ServeHTTP level; POST precisely so that gate applies and so no secret is
retrievable by URL alone). Deliberately NOT the customer page's data-secret
widget, which embeds the plaintext on every load.
A delivered reveal writes one recovery_credential_revealed event on the host's
customer timeline (info, source hub, Hungarian) via SaveEvent alone — no
dispatcher, nobody emailed, the log_tail_requested shape. Two reveals write two
events: the register records accesses, not states. A 404 is not an access. An
unbound host reveals fine and writes no event; the [INFO] hub line, carrying the
username and a length only, is then the record.
The global-key API path is untouched by design — it is the route for when the
hub UI itself is broken, and coupling it to the session layer would delete the
independence that makes it a fallback.
Recorded as a real trade: the hub session password alone now unlocks console root
fleet-wide, where retrieval previously also needed the global key. Accepted for a
single-operator, HU-geo-fenced hub that already stores these passwords in
plaintext at rest (CONTEXT.md ruling S-4). The plaintext-at-rest half is filed as
R-133 — every hub DB backup is a fleet-wide console-credential dump.
Tests 550 -> 559; four red-proofs (page leak, audit event, CSRF gate, route
order) each run, observed failing, and reverted. The route-order proof is a seam
test driving ServeHTTP: a handler-level test cannot see that defect, because the
handler is correct and simply never runs.
The golden's version IS the controller it bakes (build-golden.sh:345 defaults
GOLDEN_VERSION to the controller tag), so a golden behind the newest deployed
controller means every FRESH install lands on stale application code. On the R-120
occurrence that stale code shipped a customer-facing falsehood: a box from the
0.185.1 golden told a customer whose backup drive had fallen out that the backup was
on the same disk as the system -- false, the drive was gone -- and offered a
different drive as the remedy.
WHY A GATE, NOT A REMINDER. The gap has opened three times: R-111 (golden's agent 17
releases behind), R-115 (agent built and deployed, never published), R-120 (this).
The first two were closed by re-baking and remembering; remembering then failed
again. R-29 is the standing proof that a check nobody runs is worse than none because
it reads as coverage -- hostinstall_gates.py sat RED and uninvoked across three
version bumps and hub_confirm_gate.py has never run at all. So the property that
matters is not whether a check exists but whether it BLOCKS.
- Wired into handleSetArtifacts (internal/web/configs.go), immediately before the
only write, on the sole UI path to SetArtifactManifest -- it runs on every vouch
without anyone choosing to. A script in scripts/ would have been a fourth orphan.
- It REFUSES (operator ruling, 2026-07-30), with a flash naming the remedy.
- Signal: store.NewestReportedControllerVersion() over reports.controller_version,
SEMVER-compared in Go -- MAX() in SQL ranks 0.99.0 above 0.186.0, a pair this
fleet has shipped. No outbound call, no new credential.
- Fail-open in exactly two deliberate cases: an empty golden field (clearing the
manifest is legitimate) and an unknown fleet version (a new hub must vouch its
first golden).
NEAR-MISS RECORDED: the first draft read guests.controller_version, a column that
exists in the schema and that NOTHING writes -- it would always have seen "" and
failed open, i.e. inert, this gate's own failure shape. Caught by grepping for a
writer before trusting the column.
Blind spot stated rather than papered over: a controller no box has ever run is
invisible to this signal. Not the failure that has bitten -- all three instances were
deployed-newer-than-baked.
4 tests through the PRODUCTION handler over httptest, never an injected seam. The
refusal asserts both the flash and that the manifest was NOT written, because a gate
that redirects and saves anyway reads as enforcement while providing none. Red-proof:
deleting the block makes the stale golden vouchable and both assertions fail.
ROADMAP R-29's audit list now records this as the FIRST enforced gate, so the
contrast with its three orphans is kept rather than lost. The orphans are unchanged.
Suite rc=0 read separately from this commit.
Found validating v0.69.0 against the live hub. demo-vm-felhom was deleted
on 07-18 and was still on the Customers list AND still raising offsite_stale
(10 events, latest 07-21 17:34, operator email at 19:34) — because
GetCustomers() is report-derived and no lifecycle tier ever deleted a report.
New leg 3 (residue), before the record purge: reports, app_telemetry,
app_log_tails, log_tail_requests, customer_notifications, plus the
credential-bearing appliance_registrations and selfbind_tokens. Audit
(events, notification_log) and F-14 provenance still survive.
Ghost customers are now deletable: 404 means "nothing here", not "no config
row". With no config row the offsite descriptor is unknowable, so the Hetzner
and descriptor legs record skipped_no_config rather than a bare "skipped".
Two more red-proofs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS
POST /configs/{id}/delete now runs hosts -> RESET -> purge behind three
acknowledgements, a typed customer-id, a stale-preview check and the
ONLINE-host refusal (every gate before any write, so a refusal has zero
side effects). The shallow handleConfigDelete is gone.
Two invariants are asserted, not just commented: ruling 3 is preserved by
construction (leg 2 never sees a host row) and retained escrow custody is
purged exactly once, in leg 3 (leg 2 runs with purgeEscrow=false).
handleCustomerReset's committed half was extracted as commitCustomerReset;
the standalone RESET path is byte-identical to v0.68.1 and its suite is
untouched. Five red-proofs run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS
The v0.68.0 row wrapped itself in a <div>, but the artifacts <form> IS the CSS grid
(display:grid, no inner container). The stray </div> closed the surrounding card from
inside the form and the new <div> was never closed — it swallowed the submit button and
ran to </form>, so the row rendered outside the card and Save landed inline. Reported by
the operator on first use.
The field still submitted (it stayed inside the form), so this was layout damage rather
than data loss, but the unbalanced markup put every section below it in the wrong
container.
Fixed as plain grid cells (grid-column: 2/4), no nested elements.
There was no render assertion on this form at all, which is why a hand-edit broke it
silently. The new test asserts the field is inside the form, the button has not escaped,
the form contains ZERO divs, whole-page div balance holds, and the sections after it
survive. Red-proofed against the broken shape.
Completes the hub half of R-39's fleet fix on top of the generation core (c484aa2).
pbsdrheal gains an auth_failed TRIGGER — a new trigger in the existing machine, not a
new machine. A box whose credential PBS rejects escalates to a fresh mint, never a
re-stage (which would re-feed the secret PBS just rejected), through the EXISTING damper:
a 401 flap must not become a secret-minting chain. With the generation stamp this closes
the loop end to end — agent proves the 401, hub re-keys, generation advances, descriptor
hash moves, agent re-consumes.
consumed_at honesty gauge: a staged secret still unconsumed past a 15-minute grace while
the box reports `applied` is surfaced with its own event. That is the exact 2026-07-18
fingerprint and a disagreement no single tier can see alone. Deliberately a SURFACE, not
a heal — auto-re-issuing on it would mint a second secret on top of an unconsumed one,
which is the mint/consume race R-39(a) already recorded. One event per distinct report,
and an honestly-stuck box does not double-report (its unconsumed secret is the symptom
being healed, not a contradiction).
R-50b(a): ArtifactManifest.WrapperSHA256 + operator field + host-page drift surface. The
PBS wrapper is root-owned 0755 and the pinned sudoers vector, yet installed unversioned
from raw/branch/main and absent from every manifest. Agents >=0.91.0 report the installed
hash; a mismatch is surfaced. An unknown on EITHER side reads as quiet, never as drift —
lighting every host amber on rollout day is how a warning becomes background noise. The
delivery channel itself stays R-50b(b)/(c).
Compatibility unchanged: safe for 0.90.0 agents (unknown JSON key dropped); the re-arm
and auth-honesty guarantees need agent >=0.91.0, so MinAgent moves only after the fleet
has self-updated.
Tests: auth_failed escalate/debounce/recovery-forgets-streak; honesty gauge incl. grace
window, the restage edge (consumed_at deliberately NULLed), consumed-never-alarms, and
honest-stuck-no-double-report; wrapper drift incl. both unknown directions. Red-proof run
at the assertion level: removing the auth_failed arm fails the escalation tests with
reissues=0.
The fleet half of R-39. An ep0 credential re-issue re-keys the SECRET of an existing
token, so token_id, fingerprint, datastore and namespace all come back byte-identical.
The agent re-applies on the descriptor's CONTENT HASH, so a re-issue was invisible to a
converged box: it short-circuited, never consumed the fresh secret, and served a revoked
credential while reporting `applied` — the N100 failure of 2026-07-18.
host_pbs_secrets gains a monotonic per-host `generation`, advanced by every fresh MINT and
by nothing else, stamped into the descriptor as `secret_generation`. That is now the only
field a re-key moves, and it is what re-arms the agent.
DEVIATION FROM SPEC, deliberate: the brief said to return "the new row's id (int64) …
no schema change". There is no row id — host_pbs_secrets is keyed by host_id and UPSERTed
last-write-wins, so a new row never exists, and created_at collides for two mints in the
same second. An additive counter column is the only monotonic source; it uses the repo's
existing idempotent ALTER-TABLE idiom.
RestageHostPBSSecret deliberately does NOT advance it: a re-stage re-arms the SAME secret,
the descriptor content genuinely has not changed, and a bump would cause a pointless agent
refetch loop (that method's own contract says so).
Also corrects a comment that asserted the re-issue refreshes the descriptor "with the NEW
token_id/fingerprint". That is false for a re-key, and believing it is why the descriptor
was never expected to be identical in the first place.
omitempty is load-bearing: a zero generation must not start emitting a new key into every
pre-existing descriptor, which would itself be a fleet-wide spurious re-apply.
Compatibility: agents below 0.91.0 drop the unknown JSON key and behave exactly as today —
inert, not breaking (Scenario C).
Tests: store-level monotonicity + per-host isolation + restage-leaves-it-alone; descriptor
byte-change, omitempty, and sibling-key round-trip; and a FLOW-level test driving
ReissuePBSDR against a fake that models a real re-key. Red-proof run at the assertion
level (not the compiler): commenting out the stamp makes the flow test fail with both
byte-identical blocks printed.
Four small items, each a case where the hub already knew something and said
nothing. Green: build, vet, tests all pass.
(a) Self-bind link is minted automatically at customer creation AND at RESET
completion (R-36 sub-item). The console banner tells the customer to open
"az e-mailben kapott link"; until now that email existed only once the
operator remembered the button, so the banner could point at something that
did not exist — during the 2026-07-18 rehearsal the box waited ~11.7 min on
exactly that. handleSelfBindLinkSend's body was extracted into a shared
mintAndSendSelfBindLink core so the button and the auto-mint callers cannot
drift apart on the honesty rules: F1 (no address -> mint nothing) and F2
(send failed -> delete the token, never leave it live). The wrapper NEVER
fails the operation it rides on — a create that provisioned Cloudflare,
offsite and PBS must not 500 over a courtesy email.
Gap found and closed while wiring it: PurgeCustomerResetDBState does NOT
clear selfbind_tokens, so a link minted BEFORE a reset would have stayed
live across it. A successful mint already replaces it (delete-then-insert,
single-active); the skip paths would not have, so they now clear stale
tokens too. Invariant: after auto-mint runs the only live link is one it
just issued, or none.
(b) Post-RESET staleness banner (R-37). When a RESET COMPLETED after the newest
report, every health figure on the page describes a lifecycle that no longer
exists, and the page kept showing pre-RESET warnings as current. Narrow on
purpose: an in-flight reset does not trigger it, and it clears itself when a
report arrives. Ties resolve to STALE — SQLite timestamps are second-
resolution and a same-second report almost certainly predates the reset;
erring the other way would hide the banner exactly when it matters.
(c) Unprovisioned-offsite warning (R-36 interim). enabled==true with type=="" is
a real, stable, silent state: provisioning is Save-triggered and the
re-enroll auto-re-issue deliberately skips an unprovisioned target, so
nothing self-heals it. Reuses the exact predicate the offsite re-issue
handler already refuses on.
(d) pbsdr_reissued rendered an EMPTY flash box — the key had no template branch,
so re-issuing PBS credentials showed a success box with no words (observed
live 2026-07-18). Now describes what was staged plus the R-39 caveat:
confirm `pvesm status` shows the entry active, because a converged agent can
report `applied` while the storage still 401s.
New .flash-warn (amber, --warn tokens) for the deviation tier between success
and error — exception-color principle: only on deviation, never on a healthy
page.
Tests assert each banner is ABSENT in the nominal cases as well as present in
the deviating one — a banner that always renders is worse than none. Both
red-proofed: deleting the pbsdr_reissued branch reproduces the original empty
box; neutering the staleness predicate fails the banner assertion. New
read-only store accessor CountSelfBindTokens makes the single-active invariant
assertable.
NOT in this train: the R-39 hub-side generation-bump fix the pre-travel task
made conditional. Its condition was REFUTED (SetHostDesired bumps
unconditionally; applyPBSDR is idempotent as documented) — the real mechanism is
the agent's descriptor-hash convergence and needs its own spec.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Let a customer bind their own freshly-installed appliance without the
operator: operator "Send self-bind link" mints a 7-day tokenized
capability link, emailed (Hungarian, sibling sender) to the customer, who
opens a public /bind/<token> page and proves two factors — the console
pairing code shown on the box screen + their retrieval passphrase — and
the hub stages the bind via the same BindAppliance (provenance
customer_selfbind). The box's ~30s appliance poll delivers.
Viktor's three rulings verbatim: console pairing code (no appliance list
ever rendered), operator-sent tokenized link, 5-attempt lockout ->
"call support". Wrong code == wrong passphrase (one generic failure, no
oracle, both factors compared unconditionally); expiry falls back to
operator-bind unchanged.
THE TRAP: one public prefix /bind/, exempt from auth+CSRF at both /login
gate sites via a single isPublicBindPath predicate (tight trailing-slash
match; ServeMux ..-cleans; handler rejects '/' in token). 9 tests
(Scenarios A-F + F1/F2); 4 red-proofs verified red-then-green (lockout,
oracle, widened-prefix, single-active). GC verdict: no appliance GC ->
the 7-day TTL stands alone. Controller/agent untouched; R-27b deferred.
Green: full hub build/vet/test (17 ok) + bash -n + hub confirm gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qDiBqKKQ5vPB5fXBqu7Kp
Makes PBS DR storage visible like the restic pool box (v0.64.0), differentiated. Scoping
correction: restic = subaccounts on the shared Hetzner Storage Box (Hetzner API); PBS DR =
the felhom-offsite PBS datastore on the ep0 endpoint VM (NO Hetzner API). Option A
(Viktor-ruled): a read-only `usage` op on the felhom-tenantsync ep0 forced command (twin of
fingerprint), polled by a new hub checker on the 15-min throttle. READ-ONLY throughout.
Phase-0 (gate PASSED): on ep0 (PBS 4.2.3), df -B1 --output=size,used,avail <datastore path>
yields bytes (39990112256/7627939840/... ~19%), read-only, existing sudo context, no admin token.
- scripts/felhom-tenantsync.sh -> v1.2.0: read-only `usage` short-circuit (df on the datastore
path), no customer_id, no admin token, NO mutation. + a bash harness proving zero mutation.
- tenantsync.Client.Usage() + BoxUsage; unknown-op -> typed ErrUsageUnsupported (graceful).
- monitor.PBSDRBoxChecker: OffsiteBoxChecker clone over a usageReader seam; 15-min throttle,
cached PBSBoxSnapshot, escalation-only pbsdr_box_fill on the "pbsdr-box" scope (operator only,
no SaveEvent), recovery re-arm. Fill only. THREE states: ok / unavailable (ep0 <=v1.1.0,
neutral no-alert) / degraded (exec failed, keep last).
- config: Alerting.PBSDRBoxFill{Warn,Crit}Percent (80/90); built with the tenantsync client,
60s sweep, SetPBSDRBox. Hub deploy INDEPENDENT of the ep0 update (graceful degradation).
- web: /offsite splits into Restic + PBS DR hash tabs (endpoint cards under PBS DR); PBS panel;
the single dashboard tile becomes two gauges (RESTIC pct.ratio, PBS DR pct / n/a).
- runbook offsite-endpoint.md 10: v1.2.0 update steps (no sudoers/authorized_keys change).
Tests: 10 Go + the harness; 3 red-proofs (usage mutation, escalation-only, unavailable-drives-band)
confirmed red then restored. go build/vet/test + bash -n + hub confirm gate all pass.
The operator sees the shared pool box's real state on the hub: total box fill vs
capacity, Σ(shared soft quotas) vs capacity (the oversubscription ratio), per-customer
usage/quota bars, and a box-level operator alert (fill % + oversub ratio) on the existing
dispatcher's operator channel. Per-customer fill alerts already existed; the box-level
aggregate was the gap. READ-ONLY against Hetzner (GET only).
Phase-0 probe (gate PASSED): the live pool box 611714 returns capacity via
storage_box_type.size (1 TiB / bx11) and usage via a stats object (size/size_data/
size_snapshots), all bytes; our token reads it (200).
- hetznerapi: additive StorageBoxType + StorageBoxStats on StorageBox (no existing field/
method changed); fake carries them + a GetBoxCalls counter; golden decode test.
- monitor.OffsiteBoxChecker: OffsiteChecker-sibling for the box; fetch-throttled (1 GET/
15min), cached BoxSnapshot, escalation-only + recovery re-arm. FILL (used/capacity 80/90)
+ OVERSUB (Σ shared+enabled quotas / capacity, 2.0x) — independent. Σ from the ConfigJSON
Descriptor (offsite.ReadDescriptor, new), never the report echo; dedicated+disabled
excluded. Scope "pool-box" -> operator channel only, no SaveEvent. Failed fetch keeps the
last snapshot degraded; missing data never becomes 0% and never transitions a band.
- config: Alerting.OffsiteBoxFill{Warn,Crit}Percent + OffsiteOversubWarnRatio (80/90/2.0
defaults; thresholds pending Viktor's ruling). Constructed in the HETZNER_TOKEN branch,
60s sweep, snapshot handed to the web server.
- web: Offsite-tab panel (fill bar, Σ+ratio, per-customer usage/quota rows) + a compact
dashboard tile; reads the cached snapshot only, never fetches; nil -> "not configured".
Tests: 10 new + 4 red-proofs (throttle, Σ filter, escalation-only, failed-fetch honesty),
all confirmed red then restored. go build/vet/test all pass; hub confirm gate OK.
The immediate-sync arc covered only operator-initiated desired-state changes;
system-initiated mutations bumped the generation silently, so a freshly onboarded
box waited a full agent tick for state the hub had already minted (observed live at
slice-C onboarding). Wire the existing, live-proven notifiers into every system site
on the correct plane — call-site wiring only, no new mechanism.
Agent plane (poke.Notifier):
- web/pbsdr.go: PBSDRAutoProvision (the observed lag), ReissuePBSDR (also lifts the
pbsdrheal reconciler escalation, zero reconciler changes), handlePBSDRReissue —
each pokes AFTER the successful SetHostDesired, never on a blocked/error path.
- api: new nil-safe Poker seam (PokeHost/PokeAllHosts + SetPoker); handleAdminSetDesiredState
pokes the target host; handleAdminSetOperatorPeer fires PokeAllHosts only when the
fleet generation bump succeeded (fire-after-commit).
- main.go: one poke.Notifier now feeds both planes (SetPoke + SetPoker).
Controller plane (intent.Hub.Bump):
- api/reissueOnReenroll: one nil-guarded bump so a long-polling controller wakes in
seconds instead of on the 15-min cycle.
Deliberate non-sites (unchanged): WG register (undeliverable pre-tunnel — the agent
fast-tick SECONDARY owns it), WG delete (transport removed), pbsdrheal Restage (no
generation bump → the 60s ticker is the pickup path). internal/pbsdrheal byte-unchanged.
Tests: 10 non-hollow tests (web async channel-synchronized fake sender; api synchronous
fake Poker) with explicit zero-count negatives; representative red-proofs per group
(A/B/C/D) run-fail-restored. Green: go build/vet/test all pass.
A generic ISO carries NO customer secret. The box registers itself at the hub
as an unclaimed appliance; the operator binds it to a customer; the hub delivers
the customer-id + retrieval passphrase ONCE; day-0 completes via the slice-A path.
Hub (v0.62.0):
- store/appliance.go: appliance_registrations keyed by (uuid, mac_set) — MAC set
is the tiebreaker (duplicate SMBIOS UUIDs); token stored as sha256 only.
Idempotent register (sticky-discard), atomic one-shot delivery, bind/discard.
- api/appliance.go: POST /appliance/register (the one unauth endpoint, per-IP
rate-limited, 256-bit token); GET /appliance/poll (404 no-oracle / 204 unbound
/ 200 deliver-once / 410 delivered). Passphrase read live, never logged.
- web/appliances.go: Hosts-page "Unclaimed appliances" section + BIND (customer
picker, host count display-only) + DISCARD; SSH host-key fingerprints; events.
- Red-proofs: one-shot delivery + register idempotency (both proven red);
404-no-oracle, sticky-discard, bind staging, render. Green + confirm gate.
Scripts (v1.19.0):
- felhom-bootstrap.sh: ONE unit, TWO modes. Direct (env has customer/passphrase)
= slice-A path, byte-identical, only branched around. Pairing (generic) =
register + poll (RestartSec=30 is the poll timer); on delivery write the env
0600 and fall through to direct. Secrets + token shredded on success.
- build-felhom-iso.sh --pairing: generic secret-free ISO, -generic filename,
manifest mode=pairing. profiles/generic.profile (new).
- test/bootstrap-modes.sh: Scenario D (direct = zero appliance calls) + pairing
register/poll + delivery handoff — all green in a debian container.
- DeleteHost(deleteEscrow) demotes current host_escrow into host_escrow_superseded (copy-before-delete, same tx), spares existing; one shared demoteCurrentEscrowTx (reused by SaveHostEscrow). F-14 provenance/gate unchanged.
- DeleteCustomerConfig now purges both escrow tables for all the customer's hosts incl. already-deleted (F-14 provenance UNION) — the one true purge point.
- Wording: checkbox/refusal/Danger-zone → demotion. S6b OBSOLETE. Red-proofs TestDeleteHost_Demotes + TestDeleteCustomer_Purges + wording guard.
GET /api/v1/wait long-poll: the box holds an authed hanging GET; the hub
completes it the instant any operator intent bumps that customer's in-memory
generation, then the box fires its ordinary report and the ACK delivers
everything through the unchanged machinery. 240s hold with a 25s heartbeat
newline defeats the nginx 60s proxy_read_timeout with no ingress annotation;
WriteTimeout lifted per-connection via ResponseController.
- internal/intent: per-customer generation counter + waiter registry
(Bump/Wait/Close), coalescing to latest, race-closer, in-memory by design.
Red-proofs: counter-vs-queue + race-closer (run-fail-reverted).
- api/wait.go: the endpoint (per-customer only; global key 400; A cannot see B).
- web bumps after every intent write (fire-after-commit): config CRUD, claim
resend, offsite re-issue/freeze, password regen, block/unblock, floors
(global bumps all config-managed), controller log-tail + log-bundle.
- main.go: one intent hub shared by web+api; Close() before server.Shutdown.
Pairs with controller v0.140.0 (the long-poll client). Grounding:
documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md.
F2 claim re-issue on clean-slate re-enroll (ReissueForReenroll, host-enroll mint path,
single-bump, reset code; hub never stores the password so fork B). F3 offsite re-issue on
re-enroll (ReissueOffsiteForCustomer, same machinery as the manual button). 2.3 escrow honesty
(red-proofed): re-issuing offsite marks the escrow stale (MarkEscrowStale), withholds the
mismatched restic hash from auto-confirm, DR checklist shows stale not done. Events:
claim_reissued_reenroll / offsite_reissued / escrow_stale.
Controller + scripts unchanged (source contradicted both premises): the controller reads escrow
prereqs live from the agent; the installer can't know the descriptor-provisioned storage id. F4
root fix is agent-side -> ROADMAP R-22; demo unblocked live (Part 0 ACL grant). VALIDATION doc
F2 erratum + F3/F4 dispositions. Green gate + Scenario-C red-proof pass.
Implements SPIKE-pbsdr-selfheal-2026-07-15 (e8f8c44). A box re-installed/rolled
back onto its stable host_id loses its agent-side converged marker; the hub
keeps the enabled descriptor + a CONSUMED one-time secret, the WG peer persists
(changed==false, cascade can't re-fire), so the agent sits in waiting_secret
forever. The missing piece is a consumable secret, not the descriptor.
New internal/pbsdrheal reconciler (5m, wgsync shape): for enabled+provisioned
hosts whose latest report pbs_dr.state is a stuck state past a >=2-distinct-report
debounce, re-stage the stored secret (store.RestageHostPBSSecret: clear
consumed_at, no ep0 call, NO generation bump); escalate to Re-issue (web
ReissuePBSDR) only when no secret is stored or the agent reports consumed_failed.
Converged/disabled/verify_failed/DR-OFF = no-op. PBSDRHEAL_ONLY_HOST scopes a
supervised rollout. Scenarios A-F + all six red-proofs verified. No agent change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
hub v0.54.0: change operator login password from the Configuration UI
Adds a "Login password" card on /configuration. The password was previously
settable only via the hub-config ConfigMap (auth.password_hash) + redeploy.
- store: hub_settings key operator_password_hash + Get/SetOperatorPasswordHash
- server: passwordHash field -> configPasswordHash (seed); new
effectivePasswordHash() (DB override wins, else seed) is now the single
source for the CSRF gate, RequireAuth, and handleLogin
- POST /configuration/password (handleChangePassword): requires current
password, 8-72 byte new + confirm, bcrypt cost 10, persists DB override;
existing sessions kept valid; ConfigMap stays the break-glass reset path
- UI: current/new/confirm form + inline mismatch pre-check + 6 flashes
- tests + red-proofs: override precedence, happy-path via handleLogin,
wrong-current rejection, mismatch/too-short/no-op, template render
- docs: CHANGELOG, README (auth+config), REUSE, REPORT
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LbMm4T7Ayzs1unB9pN6Uqd
@
Customer status (dashboard row, /configs list, detail header + strip) is
now worst(controllerDerived, hostStatusOf(each expected host)) via the ONE
staleness definition (Server.hostStatus, hosts.go - shared with the
HostStalenessChecker; no second threshold). Any host down/stale caps the
customer at WARN with a cause chip naming the host ("host down: <id>");
pending (never-reported) hosts worsen only once the customer has reported
(onboarding exclusion). The three previously-inlined controller-status
chains collapse into controllerStatus() (rollup.go). Display + derivation
only - checker alerting untouched.
Live shape pinned (drill-1 / Peti cluster): host down 23h + controller
report minutes old rendered a GREEN row - TestRollup_DeadHostMasking now
fails that exact outcome. Red-proof: short-circuiting foldHostStatus to
controller-only flips C + two D subtests red ("dashboard row is GREEN
over a 23h-dead host").
pbsdrProvisionAtom, on tenantsync token_exists: consult the customer's
MOST RECENT host-deletion record. escrow_acked=true -> invoke the
EXISTING tenantsync Reissue op, store the audit event (pbsdr_auto_reissue,
hub-source, "Previous key destroyed (acknowledged deletion) - credentials
re-issued automatically."), proceed to secret+descriptor as a normal
provision. No record / un-acked -> the pre-existing refusal, byte-unchanged
(never-silently-re-key law; operator ruling 2026-07-13).
Scenario A test drives the REAL handler path over a REAL escrow-ack delete;
scenario B pins the exact non-effect (zero Reissue calls, no state).
Red-proof: bypassing the gate flips both B subtests to 303 (silent re-key).
- Auto-refresh: the 60s reload fires only while a live tab is active
(nav data-live-tabs="overview,applications,events,host") AND no form is
dirty (delegated input/change listener; never reset — a reload clears it).
Skipped ticks reschedule; a muted (paused) hint shows next to the toggle on
non-live tabs / dirty forms. Toggle, localStorage key, 60s cadence, and
default-on behavior unchanged. The refresh script resolves the legacy
settings→edit hash alias like the tabs script.
- Rider 4a: every template's stylesheet link is /style.css?v={{hubVersion}}
(the v0.47.0 gotcha: max-age=3600 served stale styling for up to an hour
after a deploy). Red-proof run: a reverted bare link fails the test.
- Rider 4b: CLAUDE.md standing rule — never git add -A in this repo
(the 146d165 sweep incident); explicit paths + pull-rebase + one writing
session per clone.
- Tests: Group C structural pins (attribute read, dirty listeners, alias x2,
hint element, cadence/key survivors) + Group D cache-bust over six pages.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TZc5w5jDhFLv6qDC32KN5v
- Settings tab renamed Edit; embeds config_form_body (.ConfigForm via the
builder) + Controller Update + Geo + a new Danger zone card holding the
relocated Block/Unblock/Delete forms (endpoints + confirm() unchanged).
All cards are SIBLINGS after </form> — never nested in the config form.
- Customer Info header loses the Edit link and Block/Delete forms; only the
config-less Create Config action stays.
- GET /configs/{id}/edit is a 302 to /customers/{id}#tab=edit; tabs JS gains
the settings→edit legacy-hash alias.
- Post-action redirects land back on their tab: update/block/unblock/
offsite-reissue/offsite-freeze/pbsdr-reissue → #tab=edit, regen-password
→ #tab=setup; delete unchanged (/configs).
- handleConfigUpdate gains the server-side twin of the form's required
fields; the error path re-renders the STANDALONE page with the SUBMITTED
overrides (B3 red-proof: nil overrides → typed values reset → test FAILS;
header red-proof: restored header buttons → count=2 → test FAILS; both run).
- Tests: Group A (panel surface, sibling forms, header cleaned by COUNT),
Group B (B1 302, B2 create unchanged, B3 typed-values, B4/B5 anchor table).
Amended pins: customer_tabs_test settings→edit; pbsdr_test postUpdate now
supplies the required fields + FormRendersState asserts the embedded render.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TZc5w5jDhFLv6qDC32KN5v
Behavior-neutral extraction (the host_detail_body pattern): the config form's
<form> + in-flight script move to a {{define}} sub-template; config_form.html
keeps the chrome. renderConfigForm/handleConfigNewForm/handleConfigEditForm
now go through the one configFormData builder (nil overrides = parse stored
ConfigJSON). Prepares the customer page Edit tab embed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TZc5w5jDhFLv6qDC32KN5v
- /offsite lists ALL wg_endpoints rows as cards (id, address, pubkey, subnet,
PBS addr, peers-in-subnet count) + add/edit/delete forms
- store: ListWGEndpoints (id order) + DeleteWGEndpoint (plain delete; the
peers-in-subnet guard lives in the handler where the refusal is built);
SetWGEndpoint upsert reused, single-expected comment updated
- guards: subnet edit refused 409 while peers sit in the current subnet;
endpoint delete refused 409 while peers sit in its subnet; full form
validation (CIDR, pbs ip in subnet, port 1-65535, pubkey, id charset) -> 400
- peer table gains an Endpoint column (first id-ordered subnet match; em dash
when none); pubkey-change edit gets a type-to-confirm noting pull-based
convergence
- allocation/reconciler/desired-state STAY lowest-endpoint-id (page notes the
deferral); GetWGEndpoint semantics untouched
- tests: E1-E6 incl. guard red-proofs; store list/delete test
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vvz1NCu22p8dGkRCpeX9re
The parallel session's git add -A swept the v0.47.0 Part-4 WIP mid-red-proof:
hosts.go landed with the RED-PROOF 2 mutation (deleteEscrow := true), i.e. the
escrow acknowledgement bypassed. This restores the real gate; the swept store/
template/test files were captured in their correct final state (full go test
green). Red-proofs 1/2/5 all ran and FAILED as required before this point.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vvz1NCu22p8dGkRCpeX9re
- host_detail_body.html: {{define}}'d body sections extracted from
host_detail.html; the standalone page is now chrome + the sub-template
- hosts.go: hostDetailData(host, r) view-model builder extracted from
handleHostDetail (reused by both surfaces)
- store: ListHostsByCustomer (host_id order; the Host tab is a list by
design - N hosts for a future HA cluster)
- customer Host tab renders one host_detail_body per host + cross-link;
empty state when no host is enrolled
- tests: TestTemplates_CustomerHostTab(+_Empty), TestListHostsByCustomer
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vvz1NCu22p8dGkRCpeX9re
- style.css: .data-table td a -> :not(.btn) so <a class=btn> keeps the .btn
palette (View/Download buttons were blue-on-blue invisible)
- customer_unified.html: 8 client-side hash tabs (#tab=...) + sticky summary
strip; all sections preserved in DOM, hiding is a JS-added body class only
(no-JS = everything visible); Events tab gets a red error-count badge
- new render tests: TestTemplates_CustomerTabs (+_NoReports)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vvz1NCu22p8dGkRCpeX9re
admin