hub v0.147.0 (code): the operator's answers S4, S6, S10, S29, S30, S32, S33
gates / gates (push) Successful in 6m0s

- S4 / R-242: GoldenVouchChecker - a published golden unvouched for 2 h raises golden_not_vouched (operator only)
- S6 / R-244: the delete cascade de-references app_log_issues; one-time sweep flag -sweep-app-log-residue=count|apply
- S10 / R-266: read_failed storage entry shows "read failed", not 0 %
- S29 / R-250: host-key scan IPv4 first, ~2 min ladder, "safe to press Save again" error
- S30 / R-246: stale_at retired (reader, setter, badge)
- S32 / R-521: storage_reconnected ends the storage_disconnected cooldown on both channels
- S33 / R-402: host page "Last full check (off-site)" line; two wire-gate allowlist entries removed
Each with a test seen failing first. Not built or deployed yet (operator-attended deploy).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
2026-10-12 14:54:14 +02:00
parent c9b9432986
commit 655f5f66f8
25 changed files with 869 additions and 121 deletions
+34
View File
@@ -1,3 +1,37 @@
## v0.147.0 — the operator's answers S4, S6, S10, S29, S30, S32, S33 (2026-10-12)
Operator rulings 2026-10-12 (`09` §3 decisions 196–233). Every item has a test seen failing first.
- **S4 / R-242 — an alarm for a published golden nobody vouched.** New `monitor.GoldenVouchChecker`: every 15 min it
lists Gitea's `felhom-golden` versions and compares the newest with the manifest's `golden_version`. A gap that lasts
2 h raises one operator-only `golden_not_vouched` (warning) per version. Gitea unreachable = no verdict, never an
alarm. Off (logged) without registry credentials. `TestGoldenVouchChecker`, `TestGoldenNotVouchedIsOperatorOnly`.
- **S6 / R-244 — a deleted customer leaves no id in shared app-log rows.** The delete cascade's residue leg now
de-references the customer in `app_log_issues`: out of `affected_customers`, its sample lines dropped when they were
its own, and a row deleted only when nobody is left. Rows shared with a live customer keep that customer's history.
One-time sweep for customers deleted earlier: `hub -sweep-app-log-residue=count` (changes nothing) then `=apply`;
it touches only ids the hub knows by no config, host or report. `TestDereferenceCustomerFromAppIssues`,
`TestSweepDeletedAppIssueRefs`.
- **S10 / R-266 (hub half) — „disk reading failed" is not an empty disk.** A storage entry with `read_failed: true`
(controller v0.308.0) shows „read failed" in the dashboard's Disk column instead of „0%". Older boxes read as
before. `TestDiskSummaryShowsAFailedRead`.
- **S29 / R-250 — the first off-site setup.** The host-key scan dials IPv4 first (a fresh name answered AAAA before
the pod could route IPv6), waits up to ~2 min (was ~1), and when it still runs out the error says the store was
created and it is **safe to press Save again** (provisioning is idempotent). `TestScannerDialsIPv4First`,
`TestScanOutOfRetriesSaysSafeToPressAgain`. Known limit: the Ingress proxy reads for 60 s, so a very slow first
try can show a gateway timeout while the provisioning finishes in the background; pressing again is safe.
- **S30 / R-246 — `stale_at` retired.** Nothing set it since v0.95.0, nothing showed it, and a leftover stamp withheld
the escrow hash (every off-site run refused). Reader, setter (`MarkEscrowStale`) and the config-form badge removed;
the column stays in old databases, unread. `TestStaleAtNoLongerWithholdsTheHash`.
- **S32 / R-521 (hub half) — a lost drive mails again after a re-plug.** `storage_reconnected` now ends the
`storage_disconnected` cooldown on both channels (measured 2026-09-14: the second unplug mailed nobody). The
household's mail stays gated by its own list: only a household that switched the option on gets it.
`TestDriveLostAgainAfterReplugMailsAgain`.
- **S33 / R-402 — the host page says how the last off-site integrity check went.** „Last full check (off-site): OK,
depth 100%, 2026-10-12 01:14 UTC" in the DR / Backup card, from the newest controller report; „never checked" or
„not reported" otherwise, never a verdict without its depth. The two fields leave the wire gate's unread list.
`TestOffsiteIntegrityLine`, `TestHostPageShowsOffsiteIntegrityLine`.
## v0.146.0 — the System page answers „is anything wrong?" and „is anything waiting for me?" first (2026-10-10)
Released 2026-10-10 (operator present for the deploy). View only — the same data, buttons, routes and CSRF
+41 -3
View File
@@ -140,6 +140,7 @@ func main() {
// R-879 roll-back: open the sealed box secrets back to plaintext so a pre-R-879 hub (<= 0.137.0)
// can run on this database, then exit. Same -config and OFFSITE_SECRET_KEY as the server.
unsealBox := flag.Bool("unseal-box-secrets", false, "R-879 roll-back: unseal box API keys, passphrases and PBS tokens to plaintext, then exit")
sweepAppLog := flag.String("sweep-app-log-residue", "", "R-244 one-time sweep of DELETED customers' ids in app_log_issues: \"count\" (changes nothing) or \"apply\", then exit")
flag.Parse()
if *showVersion {
@@ -197,6 +198,27 @@ func main() {
dataStore.Close()
os.Exit(code)
}
if *sweepAppLog != "" {
// R-244 (S6, decision 201): the operator sees the count first ("count"), then says yes ("apply").
// Only ids the hub no longer knows by config, host or report are touched. Prints ids and counts only.
if *sweepAppLog != "count" && *sweepAppLog != "apply" {
fmt.Fprintln(os.Stderr, `-sweep-app-log-residue takes "count" or "apply"`)
os.Exit(2)
}
res, err := dataStore.SweepDeletedAppIssueRefs(*sweepAppLog == "apply")
dataStore.Close()
if err != nil {
fmt.Fprintf(os.Stderr, "sweep failed (nothing committed): %v\n", err)
os.Exit(1)
}
totDer, totDel := 0, 0
for _, r := range res {
fmt.Printf("%s\tdereferenced=%d\tdeleted=%d\n", r.CustomerID, r.Dereferenced, r.Deleted)
totDer, totDel = totDer+r.Dereferenced, totDel+r.Deleted
}
fmt.Printf("%s: %d deleted customer id(s), %d shared row(s) de-referenced, %d orphan row(s) deleted\n", *sweepAppLog, len(res), totDer, totDel)
os.Exit(0)
}
offsiteKeyReady := false
if v := os.Getenv("OFFSITE_SECRET_KEY"); v == "" {
logger.Printf("[ERROR] OFFSITE_SECRET_KEY unset — off-site passwords cannot be sealed; provisioning and the key registrar are DISABLED")
@@ -355,12 +377,13 @@ func main() {
webServer.SetAssetManager(assetsMgr)
// MAIL-HOLD banner on every operator page + the release button on Configuration.
webServer.SetMailHold(mailHold)
webServer.SetClaimEngine(claimEngine) // v0.50.0 — Setup-tab claim chip + resend button
webServer.SetSelfBindMailer(dispatcher) // v0.66.0 (R-27) — customer self-bind link button (sibling of claim mailer)
webServer.SetClaimEngine(claimEngine) // v0.50.0 — Setup-tab claim chip + resend button
webServer.SetSelfBindMailer(dispatcher) // v0.66.0 (R-27) — customer self-bind link button (sibling of claim mailer)
webServer.SetEventEmitter(dispatcher.ProcessEvent) // v0.135.0 (R-604) — the "floor raise skipped boxes" mail
// Day-0 artifact version dropdowns: let the operator pick a version and have the hub derive the
// sha256 from Gitea (no hand-copied checksums). Reuses the registry creds; degrades to manual text
// entry when they're absent.
var giteaClient *gitea.Client // also read by the golden-vouch checker (R-242); nil without registry creds
if cfg.Registry.Username != "" && cfg.Registry.Token != "" {
// IN-CLUSTER, not the public ingress (v0.100.0). The hub and Gitea share this cluster, and
// every one of these is a small metadata call made dozens of times per Configuration page
@@ -377,7 +400,8 @@ func main() {
if giteaAPI == "" {
giteaAPI = "http://gitea.gitea-system.svc.cluster.local:3000"
}
webServer.SetGiteaClient(gitea.New(giteaAPI, "admin", cfg.Registry.Username, cfg.Registry.Token))
giteaClient = gitea.New(giteaAPI, "admin", cfg.Registry.Username, cfg.Registry.Token)
webServer.SetGiteaClient(giteaClient)
logger.Printf("[INFO] Gitea artifact browser enabled (Day-0 version dropdowns) via %s", giteaAPI)
}
@@ -851,6 +875,17 @@ func main() {
// R-523: the agent's in-guest controller supervisor → controller_restarted_by_agent / controller_crashloop.
controllerSupervisorChecker := monitor.NewControllerSupervisorChecker(dataStore, dispatcher.ProcessEvent, logger)
offsiteDeliveryChecker := monitor.NewOffsiteDeliveryChecker(dataStore, offsiteHealReissuer, dispatcher.ProcessEvent, logger)
// R-242 second half (S4, decision 199): published golden vs vouched golden. Needs the registry creds
// (the same Gitea client as the Day-0 dropdowns); without them the check is off and says so.
var goldenVouchChecker *monitor.GoldenVouchChecker
if giteaClient != nil {
goldenVouchChecker = monitor.NewGoldenVouchChecker(
func(c context.Context) ([]string, error) { return giteaClient.ListVersions(c, "felhom-golden") },
func() string { return dataStore.GetArtifactManifest().GoldenVersion },
dispatcher.ProcessEvent, logger)
} else {
logger.Printf("[WARN] Golden vouch check OFF: no registry credentials, so the hub cannot list published goldens")
}
go func() {
ticker := time.NewTicker(60 * time.Second)
defer ticker.Stop()
@@ -871,6 +906,9 @@ func main() {
restoreTestChecker.Check() // R-85: restore-test failure + per-tier staleness
offsiteDeliveryChecker.Check()
controllerSupervisorChecker.Check()
if goldenVouchChecker != nil {
goldenVouchChecker.Check() // R-242: throttled to one Gitea listing per 15 min
}
if offsiteBoxChecker != nil {
offsiteBoxChecker.Check() // R-5: restic pool-box aggregate (fetch-throttled internally)
}
+97
View File
@@ -0,0 +1,97 @@
package monitor
import (
"context"
"fmt"
"log"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/semver"
)
// GoldenVouchChecker is R-242's second half (operator ruling 2026-10-12, S4, `09` §3 decision 199):
// an operator alarm when the newest PUBLISHED golden is not the one the hub VOUCHES.
//
// The bake half has a gate (scripts/golden_currency_gate.py), but that gate cannot see the vouch — it
// lives only in the hub's settings — so a golden baked and uploaded but never vouched passed every check
// and a fresh install would still land on the older controller. Here the hub compares the two things it
// can see itself: Gitea's `felhom-golden` package versions and its own manifest's golden_version.
//
// It waits a grace period after it first sees the gap, because a bake and its vouch are normally minutes
// apart in the same session; an alarm then would be noise. Gitea unreachable or an empty list = no
// verdict (logged), never an alarm and never a clear. One alarm per unvouched version per process.
// Pinned by TestGoldenVouchChecker.
type GoldenVouchChecker struct {
list func(ctx context.Context) ([]string, error) // newest-first published golden versions
vouched func() string // the hub manifest's golden_version
onEvent EventNotifyFunc
logger *log.Logger
now func() time.Time
grace time.Duration
every time.Duration
mu sync.Mutex
lastRun time.Time
gapSince map[string]time.Time // unvouched newest version → when first seen
alertedOn string
}
const (
eventGoldenNotVouched = "golden_not_vouched"
goldenVouchGrace = 2 * time.Hour
goldenVouchCheckEvery = 15 * time.Minute
goldenVouchListDeadline = 20 * time.Second
)
// NewGoldenVouchChecker: list = the Gitea package listing, vouched = the manifest's golden_version.
func NewGoldenVouchChecker(list func(ctx context.Context) ([]string, error), vouched func() string, onEvent EventNotifyFunc, logger *log.Logger) *GoldenVouchChecker {
return &GoldenVouchChecker{list: list, vouched: vouched, onEvent: onEvent, logger: logger,
now: time.Now, grace: goldenVouchGrace, every: goldenVouchCheckEvery, gapSince: map[string]time.Time{}}
}
// Check runs on the hub's 60 s sweep and throttles itself to one Gitea listing per 15 minutes.
func (gc *GoldenVouchChecker) Check() {
gc.mu.Lock()
defer gc.mu.Unlock()
now := gc.now()
if !gc.lastRun.IsZero() && now.Sub(gc.lastRun) < gc.every {
return
}
gc.lastRun = now
ctx, cancel := context.WithTimeout(context.Background(), goldenVouchListDeadline)
defer cancel()
vers, err := gc.list(ctx)
if err != nil || len(vers) == 0 {
gc.logger.Printf("[WARN] Golden vouch check: no verdict (published list: %d version(s), err=%v)", len(vers), err)
return
}
newest, vouched := vers[0], gc.vouched()
if vouched != "" && semver.Compare(newest, vouched) <= 0 {
if gc.alertedOn != "" {
gc.logger.Printf("[INFO] Golden vouch check: vouched %s is current again (was alerting on %s)", vouched, gc.alertedOn)
}
gc.gapSince, gc.alertedOn = map[string]time.Time{}, ""
gc.logger.Printf("[DEBUG] Golden vouch check: newest published %s, vouched %s — ok", newest, vouched)
return
}
first, seen := gc.gapSince[newest]
if !seen {
gc.gapSince[newest] = now
gc.logger.Printf("[INFO] Golden vouch check: golden %s is published, the hub vouches %q — alarm in %s if still so", newest, vouched, gc.grace)
return
}
if now.Sub(first) < gc.grace || gc.alertedOn == newest {
return
}
gc.alertedOn = newest
vouchedTxt := vouched
if vouchedTxt == "" {
vouchedTxt = "none"
}
msg := fmt.Sprintf("Golden %s is published but NOT vouched (the hub vouches %s, for %s now) — a fresh install still gets the older controller. Vouch it under Configuration → Day-0 artifacts, or record why not.",
newest, vouchedTxt, now.Sub(first).Round(time.Minute))
gc.logger.Printf("[INFO] Golden vouch check: %s", msg)
gc.onEvent("", eventGoldenNotVouched, "warning", msg,
fmt.Sprintf(`{"published":%q,"vouched":%q}`, newest, vouched), "hub")
}
+61
View File
@@ -0,0 +1,61 @@
package monitor
import (
"context"
"errors"
"io"
"log"
"testing"
"time"
)
// R-242 second half (S4, decision 199). The CONSEQUENCE: a published-but-unvouched golden raises one
// operator warning after the grace; a vouch clears it; Gitea down says nothing.
// RED-PROOF: make Check return before onEvent (the pre-v0.147.0 hub had no such checker) → "want 1 alarm".
func TestGoldenVouchChecker(t *testing.T) {
now := time.Date(2026, 10, 12, 12, 0, 0, 0, time.UTC)
published := []string{"0.308.0", "0.301.0"}
var listErr error
vouched := "0.301.0"
var events []string
gc := NewGoldenVouchChecker(
func(context.Context) ([]string, error) { return published, listErr },
func() string { return vouched },
func(cust, typ, sev, msg, det, src string) { events = append(events, typ+"/"+sev) },
log.New(io.Discard, "", 0))
gc.now = func() time.Time { return now }
step := func(d time.Duration) { now = now.Add(d); gc.Check() }
step(0) // first sight of the gap — no alarm yet
step(time.Hour)
if len(events) != 0 {
t.Fatalf("alarm inside the grace: %v", events)
}
step(61 * time.Minute) // past 2 h
if len(events) != 1 || events[0] != "golden_not_vouched/warning" {
t.Fatalf("want 1 alarm golden_not_vouched/warning, got %v", events)
}
step(time.Hour)
if len(events) != 1 {
t.Fatalf("the same version alarmed twice: %v", events)
}
vouched = "0.308.0" // the operator vouches
step(time.Hour)
published = []string{"0.309.0", "0.308.0"}
listErr = errors.New("gitea down")
step(5 * time.Hour)
if len(events) != 1 {
t.Fatalf("an unreachable Gitea must not alarm: %v", events)
}
// Throttle: two calls within 15 min list once.
calls := 0
gc2 := NewGoldenVouchChecker(func(context.Context) ([]string, error) { calls++; return published, nil },
func() string { return "0.309.0" }, func(_, _, _, _, _, _ string) {}, log.New(io.Discard, "", 0))
t0 := now
gc2.now = func() time.Time { return t0 }
gc2.Check()
gc2.Check()
if calls != 1 {
t.Fatalf("throttle: %d listings in one window, want 1", calls)
}
}
+32
View File
@@ -205,6 +205,13 @@ func (d *Dispatcher) processEvent(customerID, eventType, severity, message, mess
return
}
// R-521 F6 (S32, decision 227): an all-clear ENDS its incident's cooldown, whatever its severity,
// so the NEXT loss mails again. Measured 2026-09-14: unplug → re-plug → unplug 40 min later, and
// the second loss mailed nobody because the first one's cooldown outlived the re-plug.
if downs, ok := cooldownEndedBy[eventType]; ok {
d.forgetIncidentCooldowns(customerID, downs)
}
// Recovery branch (v0.71.0, audit F11) — BEFORE the severity gate, as an explicit eventType
// branch: *_recovered stays severity "info" (semantics frozen), but is no longer silent.
// Operator always hears both edges; the customer hears recovery iff they heard the down.
@@ -753,6 +760,8 @@ var operatorOnlyEvents = map[string]bool{
"offsite_snapshots_dropped": true,
// R-243 (2026-10-08): off-site ON, the escrow never done, no off-site copy for 7 days. The household already sees
// the reminder on every page; this is the operator's „they have not acted" line. Listed in the SAME commit.
// R-242 (S4, decision 199, 2026-10-12): a published golden the hub does not vouch — the operator's own step.
"golden_not_vouched": true,
"offsite_escrow_pending": true,
"offsite_escrow_pending_cleared": true,
// R-304 option C (2026-10-08, decision 183): a household's recovery code opens — or may open — an OLDER sealed
@@ -997,6 +1006,29 @@ func (d *Dispatcher) replyToFor(to string) string {
}
// forgetCustomerCooldown gives back a customer cooldown armed for a mail the hold dropped.
// cooldownEndedBy maps an all-clear event to the incident types whose cooldowns it ends (R-521 F6).
// Pinned by TestDriveLostAgainAfterReplugMailsAgain.
var cooldownEndedBy = map[string][]string{
"storage_reconnected": {"storage_disconnected"},
}
// forgetIncidentCooldowns drops both channels' cooldowns for customerID and each down type, including
// the tier/run-suffixed keys ("cust:type:…").
func (d *Dispatcher) forgetIncidentCooldowns(customerID string, downs []string) {
d.mu.Lock()
defer d.mu.Unlock()
for _, m := range []map[string]time.Time{d.opCooldowns, d.custCooldowns} {
for k := range m {
for _, down := range downs {
base := customerID + ":" + down
if k == base || strings.HasPrefix(k, base+":") {
delete(m, k)
}
}
}
}
}
func (d *Dispatcher) forgetCustomerCooldown(key string) {
d.mu.Lock()
delete(d.custCooldowns, key)
@@ -0,0 +1,10 @@
package notify
import "testing"
// R-242 (S4): the golden-vouch alarm is the operator's own step — never a household mail.
func TestGoldenNotVouchedIsOperatorOnly(t *testing.T) {
if !operatorOnlyEvents["golden_not_vouched"] {
t.Fatal("golden_not_vouched must be operator-only")
}
}
@@ -0,0 +1,42 @@
package notify
import (
"io"
"log"
"testing"
)
// R-521 F6 (S32, `09` §3 decision 227). MEASURED 2026-09-14 (BIGNIGHT): a drive was unplugged, plugged
// back (`storage_reconnected`), and unplugged again 40 minutes later — and the second loss mailed
// NOBODY: the per-key cooldown armed by the first `storage_disconnected` outlived the recovery.
// A re-plug ends the incident, so it ends that incident's cooldown, on both channels.
//
// Also pins the S32 rule: the household gets the lost-drive mail ONLY when its own list carries
// `storage_disconnected`; a household that did not switch it on gets none.
// RED-PROOF: without the reset in processEvent the second unplug gives 1 operator / 1 customer mail.
func TestDriveLostAgainAfterReplugMailsAgain(t *testing.T) {
st := newDispStore(t)
if err := st.SaveNotificationPrefs("on", "house-on@example.com", []string{"storage_disconnected"}, 6); err != nil {
t.Fatal(err)
}
if err := st.SaveNotificationPrefs("off", "house-off@example.com", []string{"backup_failed"}, 6); err != nil {
t.Fatal(err)
}
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
sent := captureSeam(d)
for _, c := range []string{"on", "off"} {
d.ProcessEvent(c, "storage_disconnected", "error", "drive lost", `{}`, "controller")
d.ProcessEvent(c, "storage_reconnected", "info", "drive back", `{}`, "controller")
d.ProcessEvent(c, "storage_disconnected", "error", "drive lost again", `{}`, "controller")
}
if got := len(mailsFor(*sent, "house-on@example.com")); got != 2 {
t.Fatalf("household that switched it on: %d lost-drive mails, want 2 (one per loss)", got)
}
if got := len(mailsFor(*sent, "house-off@example.com")); got != 0 {
t.Fatalf("household that did not switch it on got %d mails, want 0", got)
}
if got := len(mailsFor(*sent, "op@felhom.eu")); got != 4 {
t.Fatalf("operator: %d mails, want 4 (two losses × two customers)", got)
}
}
+11 -3
View File
@@ -105,8 +105,16 @@ func (p *Provisioner) begin(customerID string) (func(), error) {
}, nil
}
// defaultScanBackoff: 5 retries, ~60s total — sized to the observed DNS propagation lag.
var defaultScanBackoff = []time.Duration{2 * time.Second, 4 * time.Second, 8 * time.Second, 16 * time.Second, 30 * time.Second}
// defaultScanBackoff: 7 retries, ~120s of waiting. R-250 (S29, decision 224): the old ~60s ladder was
// sized to DNS existence, but a fresh name measured ~100s to settle (2026-08-07, walk5: the first create
// ran out, an identical second press succeeded ~70s later). The handler's own context (3 min) still
// bounds the whole provisioning.
var defaultScanBackoff = []time.Duration{2 * time.Second, 4 * time.Second, 8 * time.Second, 16 * time.Second, 30 * time.Second, 30 * time.Second, 30 * time.Second}
// ErrScanNotYetReachable wraps a host-key scan that ran out of retries. The provisioning before it is
// idempotent (the resource is found by its label, no second sub-account, no second password), so the
// operator can safely press again; the message says so. R-250.
var ErrScanNotYetReachable = errors.New("the off-site store was created but its address does not answer yet (a new address can take a few minutes to settle) — it is SAFE to press Save again in a minute: nothing is created twice")
func (p *Provisioner) logf(f string, a ...any) {
if p.Logger != nil {
@@ -159,7 +167,7 @@ func (p *Provisioner) ProvisionOffsite(ctx context.Context, customerID string, i
}
fp, err := p.scanWithRetry(ctx, d.Host, port)
if err != nil {
return nil, fmt.Errorf("offsite: host-key scan %s: %w", d.Host, err)
return nil, fmt.Errorf("offsite: %w (host-key scan %s: %v)", ErrScanNotYetReachable, d.Host, err)
}
d.HostFingerprint = fp
return d, nil
+6 -34
View File
@@ -230,7 +230,8 @@ func TestReissue_RefusesAmbiguousLookup(t *testing.T) {
// if esc, _ := p.Store.GetHostEscrow(host.HostID); esc != nil { p.Store.MarkEscrowStale(host.HostID) }
// }
//
// in ReissueCredentials → es.Stale becomes true, the hash is withheld, and this test fails on both.
// in ReissueCredentials → the hash is withheld, and this test fails. (The stale flag itself was retired in
// v0.147.0, R-246 — store.TestStaleAtNoLongerWithholdsTheHash.)
func TestReissue_DoesNotMarkAHealthyEscrowStale(t *testing.T) {
p, _, st := newTestProvisioner(t)
const cust = "cust-esc"
@@ -248,8 +249,8 @@ func TestReissue_DoesNotMarkAHealthyEscrowStale(t *testing.T) {
if err != nil || es == nil {
t.Fatalf("escrow status (before): %v", err)
}
if es.Stale || es.ResticPwSHA256 != "OLDHASH" {
t.Fatalf("pre-reissue escrow must be current: stale=%v hash=%q", es.Stale, es.ResticPwSHA256)
if es.ResticPwSHA256 != "OLDHASH" {
t.Fatalf("pre-reissue escrow must be current: hash=%q", es.ResticPwSHA256)
}
// Re-issue the offsite PROVIDER credential. The box still holds its repo_password file, so the
@@ -262,9 +263,6 @@ func TestReissue_DoesNotMarkAHealthyEscrowStale(t *testing.T) {
if err != nil || es == nil {
t.Fatalf("escrow status (after): %v", err)
}
if es.Stale {
t.Fatal("a re-issue marked a HEALTHY escrow stale — this blocks every off-site backup and asks the customer for a ceremony that would supersede a good key (R-196)")
}
if es.ResticPwSHA256 != "OLDHASH" {
t.Fatalf("the sealed hash must keep flowing to the ACK so the controller can auto-confirm; got %q", es.ResticPwSHA256)
}
@@ -281,34 +279,8 @@ func TestReissue_DoesNotMarkAHealthyEscrowStale(t *testing.T) {
}
}
// The other half of Scenario C: the stale MECHANISM is intact and still does its job when something
// legitimately sets it. Without this, "nothing marks stale any more" could decay into "stale no longer
// works", and the next evidential caller would ship inert (the seam-built-but-never-wired shape).
func TestEscrowStaleMechanism_StillWithholdsAndClears(t *testing.T) {
_, _, st := newTestProvisioner(t)
const cust = "cust-stale-mech"
if err := st.UpsertHost(&store.Host{HostID: cust + "-01", CustomerID: cust, APIKey: "k"}); err != nil {
t.Fatal(err)
}
if _, _, err := st.SaveHostEscrow(cust+"-01", []byte("blob"), "SHA256:fp", "zero_knowledge", "2026-07-16T00:00:00Z", "OLDHASH"); err != nil {
t.Fatal(err)
}
if err := st.MarkEscrowStale(cust + "-01"); err != nil {
t.Fatalf("MarkEscrowStale: %v", err)
}
es, _ := st.GetEscrowStatusForCustomer(cust)
if es == nil || !es.Stale || es.ResticPwSHA256 != "" {
t.Fatalf("a marked escrow must read stale AND withhold the hash: %+v", es)
}
// A fresh ceremony (new blob sealing the new password) clears stale + serves the new hash.
if _, _, err := st.SaveHostEscrow(cust+"-01", []byte("blob-2"), "SHA256:fp", "zero_knowledge", "2026-07-16T01:00:00Z", "NEWHASH"); err != nil {
t.Fatal(err)
}
es, _ = st.GetEscrowStatusForCustomer(cust)
if es == nil || es.Stale || es.ResticPwSHA256 != "NEWHASH" {
t.Fatalf("a fresh ceremony must clear stale + serve the new hash: %+v", es)
}
}
// The stale MECHANISM's own test was removed with the mechanism (R-246, v0.147.0, operator ruling 2026-10-12):
// the flag is retired, and store.TestStaleAtNoLongerWithholdsTheHash pins that an old stamp no longer withholds.
// Scenario E (SLICE 4) — the freeze lever flips ONLY readonly on the exactly-1 labelled sub-account
// (SSH stays on — a freeze must not cut access, just writes); ambiguity refuses; unfreeze reverses.
+16 -3
View File
@@ -16,6 +16,8 @@ import (
// container. The captured fingerprint is non-secret (a public host-key identifier).
type SSHHostKeyScanner struct {
Timeout time.Duration
// dial is a test seam (nil = a net.Dialer with Timeout). TestScannerDialsIPv4First.
dial func(ctx context.Context, network, addr string) (net.Conn, error)
}
// errCaptured aborts the handshake once we have the host key (we never intended to authenticate).
@@ -36,10 +38,21 @@ func (s SSHHostKeyScanner) Fingerprint(ctx context.Context, host string, port in
return errCaptured
},
}
d := net.Dialer{Timeout: timeout}
conn, err := d.DialContext(ctx, "tcp", fmt.Sprintf("%s:%d", host, port))
dial := s.dial
if dial == nil {
d := net.Dialer{Timeout: timeout}
dial = d.DialContext
}
// R-250 (S29, decision 224): IPv4 FIRST. A fresh Storage Box name answers AAAA before the hub pod
// can route IPv6 (measured 2026-08-07: "network is unreachable" on the v6 address while v4 worked).
// Any family only as the fallback; the IPv4 error is the one reported when both fail.
addr := fmt.Sprintf("%s:%d", host, port)
conn, err := dial(ctx, "tcp4", addr)
if err != nil {
return "", fmt.Errorf("dial: %w", err)
var err2 error
if conn, err2 = dial(ctx, "tcp", addr); err2 != nil {
return "", fmt.Errorf("dial: %w", err)
}
}
defer conn.Close()
// ssh.NewClientConn runs the handshake; our callback captures the key then returns errCaptured, so this
+48
View File
@@ -0,0 +1,48 @@
package offsite
import (
"context"
"errors"
"net"
"strings"
"testing"
)
// R-250 (operator ruling 2026-10-12, S29, `09` §3 decision 224): a fresh Storage Box name resolves
// AAAA-first, and the hub pod has no IPv6 route — measured 2026-08-07: `dial tcp [2a01:…]:23: connect:
// network is unreachable` while the A record already worked. The scan dials IPv4 FIRST and falls back to
// any family only when IPv4 fails.
// RED-PROOF: with the old single `DialContext(ctx, "tcp", …)` the first network recorded is "tcp".
func TestScannerDialsIPv4First(t *testing.T) {
var nets []string
s := SSHHostKeyScanner{dial: func(_ context.Context, network, _ string) (net.Conn, error) {
nets = append(nets, network)
return nil, errors.New("refused in test")
}}
if _, err := s.Fingerprint(context.Background(), "box.example", 23); err == nil {
t.Fatal("a failed dial must fail the scan")
}
if len(nets) != 2 || nets[0] != "tcp4" || nets[1] != "tcp" {
t.Fatalf("dial order = %v, want [tcp4 tcp]", nets)
}
}
// R-250 second half: when the scan runs out, the operator is told it is safe to press again, and a
// second press really creates nothing twice. RED-PROOF: with the old wrap ("offsite: host-key scan …")
// errors.Is fails and the text has no „SAFE to press".
func TestScanOutOfRetriesSaysSafeToPressAgain(t *testing.T) {
p, fake, _ := newTestProvisioner(t)
p.Scanner = &fakeScanner{err: errors.New("dial tcp: lookup box: no such host")}
_, err := p.ProvisionOffsite(context.Background(), "cust-r250", Input{Enabled: true, Type: "shared", QuotaGB: 10})
if !errors.Is(err, ErrScanNotYetReachable) || !strings.Contains(err.Error(), "SAFE to press Save again") {
t.Fatalf("err = %v, want the safe-to-press-again error", err)
}
n := len(fake.Subaccounts)
p.Scanner = &fakeScanner{fp: "SHA256:ok"}
if _, err := p.ProvisionOffsite(context.Background(), "cust-r250", Input{Enabled: true, Type: "shared", QuotaGB: 10}); err != nil {
t.Fatalf("second press: %v", err)
}
if len(fake.Subaccounts) != n {
t.Fatalf("the second press created another sub-account (%d → %d)", n, len(fake.Subaccounts))
}
}
+182
View File
@@ -0,0 +1,182 @@
package store
import (
"database/sql"
"encoding/json"
"fmt"
"sort"
"strings"
)
// R-244 (operator ruling 2026-10-12, S6, `09` §3 decision 201): a deleted customer's id must not
// survive in `app_log_issues`.
//
// The table is a FLEET-WIDE aggregate keyed on app_name+fingerprint, with the customers who hit an
// issue in `affected_customers` (a JSON list) and, for the row's sample log lines, `context_customer`
// + `context`. So a delete must DE-REFERENCE, never `DELETE … WHERE customer LIKE`: on 2026-08-08, 27
// of the 71 rows naming a torn-down venue were shared with a LIVE customer, whose history a naive
// delete would have destroyed. The rule, per row that names the customer:
// - remove the id from affected_customers;
// - if the sample lines were that customer's (context_customer = id), drop them (context, context_customer);
// - delete the row only when no customer is left in affected_customers.
// Pinned by TestDereferenceCustomerFromAppIssues and TestSweepDeletedAppIssueRefs.
// appIssueRefs are the rows that name a customer, with what de-referencing would do to each.
type appIssueRef struct {
id int64
remaining []string // affected_customers without the id
ownsCtx bool // context_customer == id
}
func appIssueRefsFor(q interface {
Query(string, ...any) (*sql.Rows, error)
}, customerID string) ([]appIssueRef, error) {
rows, err := q.Query(`SELECT id, COALESCE(affected_customers,'[]'), COALESCE(context_customer,'')
FROM app_log_issues WHERE context_customer = ? OR instr(affected_customers, ?) > 0`,
customerID, `"`+jsonEscape(customerID)+`"`)
if err != nil {
return nil, err
}
defer rows.Close()
var out []appIssueRef
for rows.Next() {
var id int64
var affected, ctxCust string
if err := rows.Scan(&id, &affected, &ctxCust); err != nil {
return nil, err
}
var list []string
_ = json.Unmarshal([]byte(affected), &list)
ref := appIssueRef{id: id, ownsCtx: ctxCust == customerID}
named := ref.ownsCtx
for _, c := range list {
if c == customerID {
named = true
continue
}
ref.remaining = append(ref.remaining, c)
}
if named { // instr() is a pre-filter; only an exact list member or the context owner counts
out = append(out, ref)
}
}
return out, rows.Err()
}
func jsonEscape(s string) string {
b, _ := json.Marshal(s)
return strings.Trim(string(b), `"`)
}
// dereferenceCustomerFromAppIssues applies the rule above inside tx. Returns (rows changed, rows deleted).
func dereferenceCustomerFromAppIssues(tx *sql.Tx, customerID string) (changed, deleted int, err error) {
refs, err := appIssueRefsFor(tx, customerID)
if err != nil {
return 0, 0, err
}
for _, r := range refs {
if len(r.remaining) == 0 {
if _, err := tx.Exec(`DELETE FROM app_log_issues WHERE id = ?`, r.id); err != nil {
return changed, deleted, err
}
deleted++
continue
}
left, _ := json.Marshal(r.remaining)
if r.ownsCtx {
_, err = tx.Exec(`UPDATE app_log_issues SET affected_customers = ?, context = '', context_customer = '' WHERE id = ?`, string(left), r.id)
} else {
_, err = tx.Exec(`UPDATE app_log_issues SET affected_customers = ? WHERE id = ?`, string(left), r.id)
}
if err != nil {
return changed, deleted, err
}
changed++
}
return changed, deleted, nil
}
// AppIssueResidue is the one-time sweep's plan (or result): per DELETED customer id, how many shared
// rows would be de-referenced and how many would be deleted.
type AppIssueResidue struct {
CustomerID string
Dereferenced int
Deleted int
}
// SweepDeletedAppIssueRefs finds every customer id named in app_log_issues that the hub no longer
// knows at all — no customer_configs row, no hosts row, no reports row — and, when apply is true,
// de-references it with the rule above. apply=false changes nothing (the count shown to the operator
// first, S6). A customer the hub still knows by ANY of the three is never touched.
func (s *Store) SweepDeletedAppIssueRefs(apply bool) ([]AppIssueResidue, error) {
ids := map[string]bool{}
rows, err := s.db.Query(`SELECT COALESCE(affected_customers,'[]'), COALESCE(context_customer,'') FROM app_log_issues`)
if err != nil {
return nil, err
}
for rows.Next() {
var affected, ctxCust string
if err := rows.Scan(&affected, &ctxCust); err != nil {
rows.Close()
return nil, err
}
var list []string
_ = json.Unmarshal([]byte(affected), &list)
for _, c := range list {
ids[c] = true
}
if ctxCust != "" {
ids[ctxCust] = true
}
}
rows.Close()
var gone []string
for id := range ids {
if id == "" {
continue
}
var n int
if err := s.db.QueryRow(`SELECT (SELECT COUNT(*) FROM customer_configs WHERE customer_id = ?)
+ (SELECT COUNT(*) FROM hosts WHERE customer_id = ?)
+ (SELECT COUNT(*) FROM reports WHERE customer_id = ?)`, id, id, id).Scan(&n); err != nil {
return nil, fmt.Errorf("sweep: is %q known: %w", id, err)
}
if n == 0 {
gone = append(gone, id)
}
}
sort.Strings(gone)
tx, err := s.db.Begin()
if err != nil {
return nil, err
}
defer tx.Rollback()
var out []AppIssueResidue
for _, id := range gone {
if !apply {
refs, err := appIssueRefsFor(tx, id)
if err != nil {
return nil, err
}
r := AppIssueResidue{CustomerID: id}
for _, ref := range refs {
if len(ref.remaining) == 0 {
r.Deleted++
} else {
r.Dereferenced++
}
}
out = append(out, r)
continue
}
ch, del, err := dereferenceCustomerFromAppIssues(tx, id)
if err != nil {
return nil, fmt.Errorf("sweep %s: %w", id, err)
}
out = append(out, AppIssueResidue{CustomerID: id, Dereferenced: ch, Deleted: del})
}
if !apply {
return out, nil
}
return out, tx.Commit()
}
@@ -0,0 +1,84 @@
package store
import (
"encoding/json"
"testing"
)
func seedIssue(t *testing.T, s *Store, app, fp, affected, ctxCust, ctx string) {
t.Helper()
if _, err := s.db.Exec(`INSERT INTO app_log_issues (app_name, fingerprint, severity, message, first_seen, last_seen, affected_customers, context, context_customer)
VALUES (?, ?, 'error', 'm', datetime('now'), datetime('now'), ?, ?, ?)`, app, fp, affected, ctx, ctxCust); err != nil {
t.Fatal(err)
}
}
func issueRow(t *testing.T, s *Store, app, fp string) (affected []string, ctxCust, ctx string, found bool) {
t.Helper()
var a string
err := s.db.QueryRow(`SELECT affected_customers, COALESCE(context_customer,''), COALESCE(context,'') FROM app_log_issues WHERE app_name=? AND fingerprint=?`, app, fp).Scan(&a, &ctxCust, &ctx)
if err != nil {
return nil, "", "", false
}
_ = json.Unmarshal([]byte(a), &affected)
return affected, ctxCust, ctx, true
}
// R-244 (S6): the customer delete de-references the shared rows. The CONSEQUENCE: the gone customer is
// named nowhere, a row only they had is gone, a row shared with a live customer keeps that customer's
// history, and a near-namesake ("gone" vs "gone2") is untouched.
// RED-PROOF: remove the dereferenceCustomerFromAppIssues call from PurgeCustomerResidue → the orphan row
// survives ("orphan row survived").
func TestDereferenceCustomerFromAppIssues(t *testing.T) {
s := newTestStore(t)
seedIssue(t, s, "immich", "a", `["gone"]`, "gone", `["line from gone"]`)
seedIssue(t, s, "immich", "b", `["gone","live"]`, "gone", `["line from gone"]`)
seedIssue(t, s, "immich", "c", `["live","gone"]`, "live", `["line from live"]`)
seedIssue(t, s, "immich", "d", `["gone2"]`, "gone2", `["x"]`)
if err := s.PurgeCustomerResidue("gone"); err != nil {
t.Fatal(err)
}
if _, _, _, found := issueRow(t, s, "immich", "a"); found {
t.Fatal("orphan row survived — a row only the deleted customer had must go")
}
if a, cc, ctx, _ := issueRow(t, s, "immich", "b"); len(a) != 1 || a[0] != "live" || cc != "" || ctx != "" {
t.Fatalf("shared row b: affected=%v context_customer=%q context=%q — want [live], the gone customer's lines dropped", a, cc, ctx)
}
if a, cc, ctx, _ := issueRow(t, s, "immich", "c"); len(a) != 1 || a[0] != "live" || cc != "live" || ctx == "" {
t.Fatalf("shared row c: affected=%v context_customer=%q — the live customer's lines must stay", a, cc)
}
if a, _, _, found := issueRow(t, s, "immich", "d"); !found || len(a) != 1 || a[0] != "gone2" {
t.Fatalf("near-namesake row d was touched: %v found=%v", a, found)
}
}
// The one-time sweep: count first (changes nothing), then apply; a customer the hub still knows by any
// of config / host / report is never touched.
func TestSweepDeletedAppIssueRefs(t *testing.T) {
s := newTestStore(t)
if err := s.SaveReport("live", []byte(`{}`)); err != nil {
t.Fatal(err)
}
seedIssue(t, s, "x", "1", `["old-venue"]`, "old-venue", `["l"]`)
seedIssue(t, s, "x", "2", `["old-venue","live"]`, "live", `["l"]`)
plan, err := s.SweepDeletedAppIssueRefs(false)
if err != nil || len(plan) != 1 || plan[0].CustomerID != "old-venue" || plan[0].Deleted != 1 || plan[0].Dereferenced != 1 {
t.Fatalf("dry run plan = %+v err=%v, want old-venue deleted 1 dereferenced 1", plan, err)
}
if _, _, _, found := issueRow(t, s, "x", "1"); !found {
t.Fatal("the dry run changed the table")
}
if _, err := s.SweepDeletedAppIssueRefs(true); err != nil {
t.Fatal(err)
}
if _, _, _, found := issueRow(t, s, "x", "1"); found {
t.Fatal("apply left the orphan row")
}
if a, cc, _, _ := issueRow(t, s, "x", "2"); len(a) != 1 || a[0] != "live" || cc != "live" {
t.Fatalf("apply damaged the live customer's row: %v %q", a, cc)
}
again, _ := s.SweepDeletedAppIssueRefs(false)
if len(again) != 0 {
t.Fatalf("the sweep is not idempotent: %+v", again)
}
}
+13 -1
View File
@@ -36,6 +36,9 @@ type CustomerResidue struct {
NotificationPrefs int
SelfBindTokens int
ApplianceRegistrations int
// AppLogIssueRefs (R-244, v0.147.0): shared app_log_issues rows naming the customer. De-referenced
// (never blindly deleted) by the purge — see app_log_residue.go.
AppLogIssueRefs int
}
// Total is the single number the "is there anything left at all?" decision hangs on.
@@ -44,7 +47,7 @@ func (r *CustomerResidue) Total() int {
return 0
}
return r.Reports + r.AppTelemetry + r.AppLogTails + r.LogTailRequests +
r.NotificationPrefs + r.SelfBindTokens + r.ApplianceRegistrations
r.NotificationPrefs + r.SelfBindTokens + r.ApplianceRegistrations + r.AppLogIssueRefs
}
// residueQueries is the ONE list both the counter and the purge walk, so a table can never be
@@ -71,6 +74,11 @@ func (s *Store) CustomerResidue(customerID string) (*CustomerResidue, error) {
return nil, fmt.Errorf("CustomerResidue %s: count %s: %w", customerID, q.table, err)
}
}
refs, err := appIssueRefsFor(s.db, customerID)
if err != nil {
return nil, fmt.Errorf("CustomerResidue %s: count app_log_issues refs: %w", customerID, err)
}
res.AppLogIssueRefs = len(refs)
return res, nil
}
@@ -92,5 +100,9 @@ func (s *Store) PurgeCustomerResidue(customerID string) error {
return fmt.Errorf("PurgeCustomerResidue %s: purge %s: %w", customerID, q.table, err)
}
}
// R-244: de-reference (not delete) the customer in the shared app-log issue rows.
if _, _, err := dereferenceCustomerFromAppIssues(tx, customerID); err != nil {
return fmt.Errorf("PurgeCustomerResidue %s: app_log_issues: %w", customerID, err)
}
return tx.Commit()
}
@@ -0,0 +1,21 @@
package store
import "testing"
// R-266 (operator ruling 2026-10-12, S10, `09` §3 decision 205): a box whose root statfs failed sends
// its "/" entry with zeros AND `read_failed: true` (controller v0.308.0). The dashboard's Disk column
// is the hub's one reader of that list; it must say the reading failed, never show the zeros as 0 %.
// An older box sends no field — its entries read as before.
// RED-PROOF: without the read_failed branch in parseDiskSummary this returns "0%/41%".
func TestDiskSummaryShowsAFailedRead(t *testing.T) {
cases := map[string]string{
`{"storage":[{"mount":"/","percent":0,"read_failed":true},{"mount":"/mnt/hdd","percent":41.2}]}`: "read failed/41%",
`{"storage":[{"mount":"/","percent":12.4},{"mount":"/mnt/hdd","percent":41.2}]}`: "12%/41%",
`{"storage":[]}`: "–",
}
for in, want := range cases {
if got := parseDiskSummary(in); got != want {
t.Errorf("parseDiskSummary(%s) = %q, want %q", in, got, want)
}
}
}
@@ -0,0 +1,38 @@
package store
import "testing"
// R-246 (operator ruling 2026-10-12, S30, `09` §3 decision 225): `stale_at` is RETIRED.
//
// The column changed behaviour — a stamped value withheld `restic_pw_sha256` from the report ACK, so
// the box could never auto-confirm and every off-site run was refused — while nothing in the product
// set it (its one caller was removed in v0.95.0) and nothing showed it. On demo-hp a leftover stamp
// silently did exactly that from 2026-08-04 until it was cleared by hand on 2026-08-08.
//
// The test asserts the CONSEQUENCE: a row that still carries an old stamp (any database written
// before this release may) serves its hash like any other row.
//
// RED-PROOF: with the pre-retirement reader (stale_at → Stale=true, hash emptied) this fails with
// „a stamped stale_at still withholds the hash".
func TestStaleAtNoLongerWithholdsTheHash(t *testing.T) {
s := newTestStore(t)
const cust = "cust-r246"
if err := s.UpsertHost(&Host{HostID: cust + "-01", CustomerID: cust, APIKey: "k"}); err != nil {
t.Fatal(err)
}
if _, _, err := s.SaveHostEscrow(cust+"-01", []byte("blob"), "SHA256:fp", "zero_knowledge", "2026-07-16T00:00:00Z", "HASH"); err != nil {
t.Fatal(err)
}
// An old database's leftover stamp. The column stays in the schema (SQLite keeps it); nothing
// in the product writes it any more.
if _, err := s.db.Exec(`UPDATE host_escrow SET stale_at = datetime('now') WHERE host_id = ?`, cust+"-01"); err != nil {
t.Fatal(err)
}
es, err := s.GetEscrowStatusForCustomer(cust)
if err != nil || es == nil {
t.Fatalf("escrow status: %v %+v", err, es)
}
if es.ResticPwSHA256 != "HASH" {
t.Fatalf("a stamped stale_at still withholds the hash (got %q) — R-246: the flag is retired", es.ResticPwSHA256)
}
}
+27 -53
View File
@@ -488,14 +488,10 @@ func (s *Store) migrate() error {
// instead of trusting blob-presence. NULL/'' = a legacy or password-less blob (never auto-confirms).
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN restic_pw_sha256 TEXT`)
// v0.57.0 (2.3, escrow honesty on offsite re-issue) — stale_at marks an escrow blob that may no
// longer cover the box's live repository password. While set, the hub stops advertising "ceremony
// done" and withholds restic_pw_sha256 from the auto-confirm ACK. NULL = current; a fresh ceremony
// (SaveHostEscrow) clears it.
// ⚠ NOTHING SETS IT as of v0.95.0 (R-196 / R-204 item 2). The only writer was the PRECAUTIONARY
// mark on offsite re-issue, which guessed rather than measured and so blocked off-site backups on
// boxes whose key had not changed. The column and its readers stay; see MarkEscrowStale for what a
// legitimate future writer would have to prove first.
// stale_at (v0.57.0) — RETIRED in v0.147.0 (R-246, operator ruling 2026-10-12, `09` §3 decision 225).
// It withheld restic_pw_sha256 from the ACK while nothing set it (its one caller went in v0.95.0)
// and nothing showed it. The ALTER stays so an old database keeps opening the same way; no code
// reads or writes the column. Pinned by TestStaleAtNoLongerWithholdsTheHash.
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN stale_at DATETIME`)
// dr_recipe (SPIKE-dr-recipe-2026-06-16): the secret-free DR reconstruction recipe, stored
@@ -1249,6 +1245,17 @@ func reportHasOffsite(reportJSON string) bool {
return p.Offsite != nil && p.Offsite.Enabled
}
// LatestControllerReportJSON returns the customer's most recent CONTROLLER report as stored ("" when none).
// Read-only; the host page reads `offsite.last_integrity_*` from it (R-402).
func (s *Store) LatestControllerReportJSON(customerID string) (string, error) {
var reportJSON string
err := s.db.QueryRow(`SELECT report_json FROM reports WHERE customer_id = ? ORDER BY id DESC LIMIT 1`, customerID).Scan(&reportJSON)
if err == sql.ErrNoRows {
return "", nil
}
return reportJSON, err
}
// LatestReportOffsitePresence reports whether the customer's most recent controller report exists
// and whether it carries an offsite status object (R-70 detector input).
func (s *Store) LatestReportOffsitePresence(customerID string) (found bool, receivedAt time.Time, hasOffsite bool, err error) {
@@ -2771,6 +2778,10 @@ func parseDiskSummary(reportJSON string) string {
Storage []struct {
Mount string `json:"mount"`
Percent float64 `json:"percent"`
// ReadFailed (R-266, controller v0.308.0): the box could not read this filesystem; its
// numbers are zeros, not an empty disk. Absent on older boxes = read as before.
// Pinned by TestDiskSummaryShowsAFailedRead.
ReadFailed bool `json:"read_failed"`
} `json:"storage"`
}
// ignore parse errors — show "–" on failure
@@ -2778,6 +2789,10 @@ func parseDiskSummary(reportJSON string) string {
var parts []string
for _, s := range report.Storage {
if s.ReadFailed {
parts = append(parts, "read failed")
continue
}
parts = append(parts, fmt.Sprintf("%.0f%%", s.Percent))
}
if len(parts) == 0 {
@@ -3218,9 +3233,6 @@ type HostEscrow struct {
// ResticPwSHA256 (SLICE 3) — the non-reversible hash of the offsite repo password the identity blob
// covers ("" = legacy/password-less blob). Safe to store/serve; the password itself never reaches the hub.
ResticPwSHA256 string
// StaleAt (v0.57.0, 2.3) — non-empty when the offsite password was re-issued after this blob was
// sealed: the blob is stale (seals a password that no longer opens the repo). Cleared by a fresh ceremony.
StaleAt string
// IdentityBlob (R-198, v0.93.0) — the age-wrapped identity bundle, which is where the offsite
// restic REPOSITORY password lives. Populated by ListSupersededEscrow so a retained blob is
// reachable from Go at all; nil for pre-v0.93.0 retained rows and for hosts that never uploaded
@@ -3373,37 +3385,14 @@ func (s *Store) CountCurrentEscrowWithIdentity() (total, withIdentity int, err e
return total, withIdentity, err
}
// MarkEscrowStale flags a host's escrow blob as stale (v0.57.0, 2.3).
//
// ⚠ IT HAS NO CALLER as of hub v0.95.0 (R-196 / R-204 item 2), and that is deliberate, not an
// oversight. Its ONE caller was `offsite.ReissueCredentials`, which called it on every re-issue on
// the PRECAUTIONARY grounds that the box's re-apply might mint a fresh repository password. It
// usually does not, so the call marked healthy escrows stale — and because a stale flag WITHHOLDS
// restic_pw_sha256 from the ACK (GetEscrowStatusForCustomer, below), it blocked every off-site backup
// on those boxes and asked the customer for a ceremony that would supersede a perfectly good key. The
// full reasoning, and the two measured signals that cover the real case, are at
// offsite.ReissueCredentials.
//
// KEPT, not deleted, because the FLAG is still live and correct — `stale_at` is read by the ACK, the
// operator card and the PBS-DR view, and a future EVIDENTIAL caller (one that has measured a key
// change rather than guessed at one) is the right way to set it. Pinned by
// TestReissue_DoesNotMarkAHealthyEscrowStale: if a caller reappears without that evidence, it fails.
//
// No-op when no escrow row exists; idempotent (only stamps the first mark since the last ceremony; a
// fresh ceremony clears stale_at via SaveHostEscrow's ON CONFLICT).
func (s *Store) MarkEscrowStale(hostID string) error {
_, err := s.db.Exec(`UPDATE host_escrow SET stale_at = datetime('now') WHERE host_id = ? AND stale_at IS NULL`, hostID)
return err
}
// GetHostEscrow returns the stored opaque escrow for a host (nil if none). Used by tests and
// (future, slice 10) restore-mode serving. The hub returns bytes verbatim; it never decrypts.
func (s *Store) GetHostEscrow(hostID string) (*HostEscrow, error) {
var e HostEscrow
err := s.db.QueryRow(`
SELECT host_id, blob, key_fingerprint, posture, created_at, updated_at, COALESCE(restic_pw_sha256, ''), COALESCE(stale_at, '')
SELECT host_id, blob, key_fingerprint, posture, created_at, updated_at, COALESCE(restic_pw_sha256, '')
FROM host_escrow WHERE host_id = ?`, hostID).
Scan(&e.HostID, &e.Blob, &e.KeyFingerprint, &e.Posture, &e.CreatedAt, &e.UpdatedAt, &e.ResticPwSHA256, &e.StaleAt)
Scan(&e.HostID, &e.Blob, &e.KeyFingerprint, &e.Posture, &e.CreatedAt, &e.UpdatedAt, &e.ResticPwSHA256)
if err == sql.ErrNoRows {
return nil, nil
}
@@ -3419,14 +3408,6 @@ type EscrowStatus struct {
IdentityBlobPresent bool `json:"identity_blob_present"`
ResticPwSHA256 string `json:"restic_pw_sha256,omitempty"`
CreatedAt string `json:"created_at,omitempty"`
// Stale (v0.57.0, 2.3) — true when `stale_at` is stamped on the host's escrow row. ⚠ SINCE hub
// v0.95.0 (R-196 / R-204 item 2) NOTHING STAMPS IT: the one caller was the precautionary
// re-issue mark, and it was removed for marking healthy escrows stale. See MarkEscrowStale.
// When stale the ResticPwSHA256 is WITHHELD (emptied) so the controller cannot auto-confirm against
// a hash that may no longer match the live repo password — the ceremony must run again. That
// withholding is exactly why the precautionary caller had to go: it BLINDED the controller's own
// hash comparison, which is the measurement that actually detects a changed repository password.
Stale bool `json:"escrow_stale,omitempty"`
// SupersededPresent (hub v0.97.0, R-222) — this host has at least one RETAINED superseded escrow
// row carrying an identity blob. It exists so the recovery screen can distinguish two situations
// that are otherwise identical from the box's side: a genuinely wrong recovery code, and a code
@@ -3450,13 +3431,12 @@ type EscrowStatus struct {
func (s *Store) GetEscrowStatusForCustomer(customerID string) (*EscrowStatus, error) {
var st EscrowStatus
var identityPresent int
var staleAt string
err := s.db.QueryRow(`
SELECT (e.identity_blob IS NOT NULL), COALESCE(e.restic_pw_sha256, ''), e.created_at, COALESCE(e.stale_at, '')
SELECT (e.identity_blob IS NOT NULL), COALESCE(e.restic_pw_sha256, ''), e.created_at
FROM host_escrow e JOIN hosts h ON h.host_id = e.host_id
WHERE h.customer_id = ?
ORDER BY e.updated_at DESC LIMIT 1`, customerID).
Scan(&identityPresent, &st.ResticPwSHA256, &st.CreatedAt, &staleAt)
Scan(&identityPresent, &st.ResticPwSHA256, &st.CreatedAt)
if err == sql.ErrNoRows {
return nil, nil
}
@@ -3464,12 +3444,6 @@ func (s *Store) GetEscrowStatusForCustomer(customerID string) (*EscrowStatus, er
return nil, err
}
st.IdentityBlobPresent = identityPresent == 1
// v0.57.0 (2.3): a stale blob must NOT auto-confirm — withhold the hash and flag it so the
// controller stays pending and the escrow wizard is offered again.
if staleAt != "" {
st.Stale = true
st.ResticPwSHA256 = ""
}
// R-222 (v0.97.0): does a RETAINED earlier package exist for this host? Only rows that actually
// carry an identity blob count — the pre-v0.93.0 rows have `identity_blob` NULL and retain nothing
// the recovery screen could ever be talking about, so counting them would make the screen offer an
+2 -2
View File
@@ -283,9 +283,9 @@ func (s *Server) handleCustomerDelete(w http.ResponseWriter, r *http.Request, cu
return
}
_ = s.store.UpdateResetLeg(journalID, "residue", "ok")
s.logger.Printf("[INFO] delete %s: residue purged (reports=%d app_telemetry=%d app_log_tails=%d log_tail_requests=%d notif_prefs=%d selfbind_tokens=%d appliance_registrations=%d)",
s.logger.Printf("[INFO] delete %s: residue purged (reports=%d app_telemetry=%d app_log_tails=%d log_tail_requests=%d notif_prefs=%d selfbind_tokens=%d appliance_registrations=%d app_log_issue_refs=%d)",
customerID, residue.Reports, residue.AppTelemetry, residue.AppLogTails, residue.LogTailRequests,
residue.NotificationPrefs, residue.SelfBindTokens, residue.ApplianceRegistrations)
residue.NotificationPrefs, residue.SelfBindTokens, residue.ApplianceRegistrations, residue.AppLogIssueRefs)
// ── Leg 4: the one true purge point — customer record + ALL escrow ciphertext ────────────────
if derr := s.store.DeleteCustomerConfig(customerID); derr != nil {
+14
View File
@@ -842,6 +842,7 @@ func (s *Server) hostDetailData(host *store.Host, r *http.Request) map[string]in
"NeedsDRMigration": capabilitiesNeedDRMigration(capabilities),
"DRPresent": drBundle != nil,
"EscrowPresent": escrow != nil,
"OffsiteIntegrity": s.offsiteIntegrityForHost(host.CustomerID),
// v0.60.0 Part B: retained superseded escrow blobs (data-first — old passphrases stay
// R-recoverable). Operator-only surface.
"SupersededEscrowCount": func() int { n, _ := s.store.CountSupersededEscrow(host.HostID); return n }(),
@@ -1106,3 +1107,16 @@ func (s *Server) handleHostDetail(w http.ResponseWriter, r *http.Request, hostID
s.logger.Printf("[ERROR] host_detail.html template: %v", err)
}
}
// offsiteIntegrityForHost reads the customer's newest controller report for the host page's
// „Last full check" line (R-402). "" when the host has no customer (the line is then not rendered).
func (s *Server) offsiteIntegrityForHost(customerID string) string {
if customerID == "" {
return ""
}
rj, err := s.store.LatestControllerReportJSON(customerID)
if err != nil {
s.logger.Printf("[WARN] host page: latest controller report for %s: %v", customerID, err)
}
return offsiteIntegrityLine(rj)
}
+38
View File
@@ -193,3 +193,41 @@ func (s *Server) offsiteCustomerRows() []offsiteCustomerRow {
sort.SliceStable(rows, func(i, j int) bool { return rows[i].UsageBytes > rows[j].UsageBytes })
return rows
}
// offsiteIntegrityLine (R-402, operator ruling 2026-10-12, S33, `09` §3 decision 228) is the host page's
// one line about the off-site integrity check: „OK/FAILED, depth N, date". It reads the controller
// report's `offsite.last_integrity_*` (controller v0.227.0 verdict, v0.228.0 depth).
//
// The timestamp decides whether the verdict means anything: no timestamp = never checked, and the
// bool (omitempty on the box) is then not read at all — the R-331 card rendered a verdict from fields
// nothing wrote. The depth is shown with every verdict; an absent depth is said to be unknown, never
// taken for "structure". Pinned by TestOffsiteIntegrityLine.
func offsiteIntegrityLine(reportJSON string) string {
var r struct {
Offsite *struct {
LastIntegrityCheck string `json:"last_integrity_check"`
LastIntegrityOK bool `json:"last_integrity_ok"`
LastIntegrityDepth string `json:"last_integrity_depth"`
} `json:"offsite"`
}
if json.Unmarshal([]byte(reportJSON), &r) != nil || r.Offsite == nil {
return "not reported (no off-site status in the box's report)"
}
o := r.Offsite
if o.LastIntegrityCheck == "" {
return "never checked"
}
verdict := "FAILED"
if o.LastIntegrityOK {
verdict = "OK"
}
depth := o.LastIntegrityDepth
if depth == "" {
depth = "unknown (controller older than v0.228.0)"
}
when := o.LastIntegrityCheck
if t, err := time.Parse(time.RFC3339, when); err == nil {
when = t.UTC().Format("2006-01-02 15:04") + " UTC"
}
return verdict + ", depth " + depth + ", " + when
}
@@ -0,0 +1,46 @@
package web
import (
"strings"
"testing"
)
// R-402 (operator ruling 2026-10-12, S33, `09` §3 decision 228): the host page shows ONE line about
// the off-site integrity check — „Last full check: OK/FAILED, depth N, date" — from the fields the
// controller has sent since v0.227.0/v0.228.0. The verdict is never shown without its depth, and an
// absent timestamp means NEVER CHECKED, not FAILED (the R-331 lesson: a card that rendered a verdict
// from fields nothing wrote).
func TestOffsiteIntegrityLine(t *testing.T) {
cases := []struct{ name, report, want string }{
{"ok, full depth", `{"offsite":{"last_integrity_check":"2026-10-12T01:14:00Z","last_integrity_ok":true,"last_integrity_depth":"100%"}}`,
"OK, depth 100%, 2026-10-12 01:14 UTC"},
{"failed (bool absent), structure", `{"offsite":{"last_integrity_check":"2026-10-11T01:00:00Z","last_integrity_depth":"structure"}}`,
"FAILED, depth structure, 2026-10-11 01:00 UTC"},
{"old controller: no depth", `{"offsite":{"last_integrity_check":"2026-10-11T01:00:00Z","last_integrity_ok":true}}`,
"OK, depth unknown (controller older than v0.228.0), 2026-10-11 01:00 UTC"},
{"never checked", `{"offsite":{"repo_size_bytes":5}}`, "never checked"},
{"no off-site object", `{"version":"0.307.0"}`, "not reported (no off-site status in the box's report)"},
{"no report", ``, "not reported (no off-site status in the box's report)"},
}
for _, c := range cases {
if got := offsiteIntegrityLine(c.report); got != c.want {
t.Errorf("%s: got %q, want %q", c.name, got, c.want)
}
}
}
// The line reaches the host page (both branches of the template gate: a host with a customer renders
// it, with the verdict from the newest controller report). RED-PROOF: delete the "OffsiteIntegrity"
// key from hostDetailData → the id and the verdict are missing.
func TestHostPageShowsOffsiteIntegrityLine(t *testing.T) {
s, st, _ := newRevealServer(t)
cookie, _ := newRevealSession(t, s)
seedNetHost(t, st, "r402-host", "0.150.0", liveReportJSON, "")
if err := st.SaveReport("c-r402-host", []byte(`{"offsite":{"last_integrity_check":"2026-10-12T01:14:00Z","last_integrity_ok":true,"last_integrity_depth":"100%"}}`)); err != nil {
t.Fatal(err)
}
body := getHostPage(t, s, cookie, "r402-host")
if !strings.Contains(body, `id="offsite-integrity"`) || !strings.Contains(body, "OK, depth 100%, 2026-10-12 01:14 UTC") {
t.Fatal("the host page does not show the off-site integrity line")
}
}
-2
View File
@@ -524,7 +524,6 @@ type pbsDRView struct {
TokenID string
WGPeer bool // cascade stage 2: the host has registered its tunnel peer
EscrowPresent bool // cascade stage 4: the ceremony ran (blob in custody)
EscrowStale bool // v0.57.0 (2.3): the blob is stale (offsite password re-issued since) — re-run the ceremony
}
// pbsDRViewFor loads the section state for the form. Read-only; every error degrades to a
@@ -547,7 +546,6 @@ func (s *Server) pbsDRViewFor(customerID string, drTier bool) pbsDRView {
}
if escrow, err := s.store.GetHostEscrow(host.HostID); err == nil && escrow != nil {
v.EscrowPresent = true
v.EscrowStale = escrow.StaleAt != ""
}
if d := readPBSDR(host.DesiredJSON); d != nil {
v.Enabled = d.Enabled
@@ -183,7 +183,7 @@
<li>{{if not .PBSDR.NoHost}}<span class="badge badge-ok">done</span> host enrolled ({{.PBSDR.HostID}}){{else}}<span class="badge badge-neutral">waiting</span> no host enrolled yet — the Day-0 install enrolls it{{end}}</li>
<li>{{if .PBSDR.WGPeer}}<span class="badge badge-ok">done</span> WG tunnel peer registered{{else}}<span class="badge badge-neutral">waiting</span> the host has not reported a WG key yet — the tunnel peer must exist before the PBS DR tier (registration is hands-free once the agent starts){{end}}</li>
<li>{{if .PBSDR.Provisioned}}<span class="badge badge-ok">done</span> descriptor provisioned (namespace {{.PBSDR.Namespace}}, token {{.PBSDR.TokenID}}){{else}}<span class="badge badge-neutral">waiting</span> provisions automatically when the WG peer registers{{end}}</li>
<li>{{if .PBSDR.EscrowStale}}<span class="badge badge-warn">stale</span> key escrow STALE — offsite credentials were re-issued since the ceremony; the sealed repo password no longer matches. Re-run the ceremony with the customer (R-moment){{else if .PBSDR.EscrowPresent}}<span class="badge badge-ok">done</span> key escrow present (ceremony done){{else}}<span class="badge badge-neutral">waiting</span> ceremony possible once the descriptor is applied on the box (run it with the customer — R-moment){{end}}</li>
<li>{{if .PBSDR.EscrowPresent}}<span class="badge badge-ok">done</span> key escrow present (ceremony done){{else}}<span class="badge badge-neutral">waiting</span> ceremony possible once the descriptor is applied on the box (run it with the customer — R-moment){{end}}</li>
</ul>
{{end}}
{{if and .PBSDR.Provisioned (not .PBSDR.DRTier)}}
@@ -498,6 +498,10 @@
<span class="label">Key Escrow</span>
<span class="value">{{if .EscrowPresent}}<span style="color: var(--green)">present</span>{{else}}<span class="text-muted">none</span>{{end}}{{if gt .SupersededEscrowCount 0}} <span class="text-muted">· {{.SupersededEscrowCount}} superseded escrow blob(s) retained</span>{{end}}</span>
</div>
{{if .OffsiteIntegrity}}<div class="info-item" id="offsite-integrity">
<span class="label">Last full check (off-site)</span>
<span class="value">{{.OffsiteIntegrity}}</span>
</div>{{end}}
</div>
</section>
+1 -19
View File
@@ -208,16 +208,6 @@ ALLOWLIST = {
(_AH, "storage_targets.smart.model_name"): (
"redundant for a verdict: the hub decodes smart.health plus every counter it bands on. The "
"model string is a display label with no threshold attached to it."),
(_CH, "offsite.last_integrity_ok"): (
"R-359, controller v0.227.0: the off-site integrity VERDICT is published so a hub surface can "
"read it, and no hub surface reads it yet. That is deliberate and it is recorded rather than "
"quietly skipped. Building the display is a HUB change, and R-331 (2026-08-30) ruled exactly "
"that class a decision for the operator, not a thing to fold into a controller task: the "
"previous integrity display was REMOVED because it rendered `Integrity Unknown` for every "
"customer forever from fields nothing wrote. Publishing the value first, and consuming it when "
"someone decides what the screen should say, is the opposite order to the one that produced "
"that card. The sibling `offsite.last_integrity_check` is NOT allowlisted and passes on its "
"own — the string already occurs hub-side. **When a hub surface is built, delete this entry.**"),
# ---- controller v0.271.0 (`09` §6.4 part 7): the automatic update leg's nightly summary ----
(_CH, "update_leg"): (
@@ -243,7 +233,7 @@ ALLOWLIST = {
(_CH, "offsite.last_proof_result"): (
"R-87, controller v0.231.0: the nightly off-site content-proof VERDICT (pass / fail / "
"cannot_judge). Published for a hub surface that does not exist yet — same class and same "
"ruling as offsite.last_integrity_ok below (R-331, R-402). ABSENT means NOT RECORDED (a "
"ruling as the integrity pair had until hub v0.147.0 read it on the host page (R-331, R-402). ABSENT means NOT RECORDED (a "
"controller older than v0.231.0), never 'failed', so any future reader must degrade to "
"unknown. **Delete this entry when a hub surface reads it.**"),
(_CH, "offsite.last_proof_reason"): (
@@ -265,14 +255,6 @@ ALLOWLIST = {
"reached, never on a skip, a missing snapshot or a restore error, so 'presence is not success' "
"does not bite here. **Delete with its siblings.**"),
(_CH, "offsite.last_integrity_depth"): (
"R-399, controller v0.228.0: the DEPTH the verdict was reached at, published beside "
"last_integrity_ok above and unconsumed for the identical reason. It matters because 'checked, "
"OK' means two different things at structure depth and at 100%, so a hub surface that shows the "
"verdict without the depth shows the same words for a check that re-read every byte and one "
"that only read the index. Absent = the box cannot answer (a controller older than v0.228.0), "
"never 'structure'. **Delete this entry with its sibling when a hub surface is built** — filed "
"as R-402 in OPEN-ITEMS.md."),
(_AH, "wireguard.last_handshake_age_s"): (
"redundant: hub-side wgsync reconciles peers from its own state, and the OOB path's own "