Commit Graph

340 Commits

Author SHA1 Message Date
Claude Code 945b7818b5 docs(arch): 07 §9.1 — record measured PBS whole-guest capacity state (R-82 Phase 0)
Per the operator's 2026-07-26 ruling: datastore growth deferred, R-82 proceeds.
Records the measurements so the constraint is written down rather than carried
in a session: 37.2 GB total / 10.8 GB used, no cross-customer dedup (per-tenant
encryption keys), 80% warn reached at roughly the second additional customer,
and the pvesm 0/0/0 reporting artifact that means operators must read fill from
the hub gauge. Also records what the tier does and does not carry, and the
conditional on the P0.1 weekly verdict (Tier-3 offsite must be healthy).

Doc NOT marked ratified — that stays Viktor's review (R-83).
2026-07-26 12:11:42 +02:00
Claude Code 65e30f2e42 docs(audits): R-82 Phase 0 — P0.1 weekly CONFIRMED, P0.2 benign, P0.3 STOP (capacity)
Read-only gate. No code, no backup triggered, no config changed.

P0.1 weekly CONFIRMED: the only 7-day-exposed state is the non-SMB half of
settings.json (storage_paths, app_backup toggles, notification prefs,
password_hash, launcher token) — none catastrophic, all recoverable. The two
items that would have overturned it, encryption.key and the offbox
credentials, are STABLE files unchanged since first boot, so a 7-day-old copy
is byte-identical. Everything referentially coupled to app state is carried
daily by Tier-1/2/3, and Tier-3 offsite was verified running and ok on both
production boxes. CONDITIONAL: drill-r50 has offsite:null, so the verdict does
not hold there.

P0.2 RESOLVED, benign: PBS returns HTTP 200 with zeroed usage to the
namespace-scoped token (DatastoreBackup, not Datastore.Audit), so PVE prints
0/0/0. Ground truth via the hub ep0 df op: 28.9% of 37.2 GB. Writes work —
the 07-18 snapshot is owned by this token, 9.74 GB, verify ok.

P0.3 STOP: 37.2 GB datastore, 10.8 GB used, 80% alert at 29.8 GB. Encrypted
per-tenant backups do not dedup across customers. Current 3 boxes project to
~15-21 GB weekly keep-3; each additional customer costs ~5-10 GB, so the 80%
alert fires at roughly the SECOND additional customer — inside the alpha
horizon. Needs a ruling on datastore size / retention before recurring writes.
2026-07-26 12:08:32 +02:00
Claude Code f5a5e2b911 hub v0.75.0: R-81 — "no signal" is not "bad signal" (anchor the backup deadline check)
Third instance of one class (hub v0.12.0, v0.73.0, this), fixed as a class.
On 2026-07-26 03:00 UTC expected_backup_missed fired on demo-felhom, demo-hp
and drill-r50 at once; the demo-felhom one reached the CUSTOMER channel
claiming "newest backup is 176h0m0s old". Nothing was wrong — three vzdump
archives were on disk. Cause: the agent backup store is in-memory, so the
R-50 fleet restart emptied `backups` until the next run, and the hub read
empty as "no backup exists".

- assessBackupFreshness returns OK/UNKNOWN/MISSED instead of `missed bool`;
  absence is UNKNOWN until it outlives an anchored window. Still pure.
- store.GetHostReportsSince + monitor.newestBackupEvidence read the hubs own
  retained history (bounded 7-day lookback, early-exit on fresh evidence) —
  "when did I last SEE evidence of a backup?" The anchor was free: the hub
  already retains 90 days. No agent change, no new persisted state.
- store.GetFirstHostReportAt anchors absence at first contact, reusing the
  existing 26h threshold as the grace (no new knob, the v0.73.0 shape).
- Deferrals logged + counted; reason strings kept distinct.
- backupStaleAfter untouched; landmine recorded (a weekly PBS snapshot would
  alarm six days in seven) and owned by R-82.

Tests 493->508. Red-proofs A/B/C observed and restored; A reproduces the live
message verbatim. Replayed the real 03:00 reports (600/417/77 rows): all
three now silent.

Source: documentation/audits/DIAG-backup-missed-2026-07-26.md
2026-07-26 11:44:15 +02:00
Claude Code add5b9bbbb docs(audits): R-80 DIAG — expected_backup_missed root-caused to in-memory agent backup store
Read-only diagnostic. No fix applied.

Verdict: FALSE ALARM for demo-felhom + demo-hp, true-but-premature for
drill-r50. Fired ONCE (2026-07-26 03:00 UTC), not nightly. No real
external customer notified.

Root cause: felhom-agent/internal/backup/store.go is in-memory by design;
an agent restart empties the host-report 'backups' array until the next
backup runs. The R-50 island migration restarted the fleet on 2026-07-25
12:44 UTC, opening an ~18h blind window that the 03:00 UTC deadline check
fell into. Hub's assessBackupFreshness reads the empty array as 'no
backup exists'.

Separate real finding: the PBS/offsite-DR tier has no schedule at all
(local_backup_target=local, no jobs.cfg) — demo-felhom has one snapshot
from 07-18, demo-hp has zero since pbs_dr was applied 07-21.
2026-07-26 09:59:35 +02:00
admin 9cfa619ec3 hub v0.74.0: allow local_api_endpoint_drift; R-77 docs + R-78/79/80
The allowlist entry is REQUIRED, not cosmetic: handleEvent 400s an unknown
event_type, so controller v0.173.0's new drift alert would be silently inert
without it. Shipped with the controller that emits it.

Docs:
- RUNBOOK-local-api-endpoint-drift.md — how to repair a drift, including the
  step everyone will want to skip (establish which value is CORRECT from what
  the agent is actually bound to, rather than assuming bootstrap.json wins) and
  what success looks like (SILENCE, not a "recovered" line, because a fresh
  controller's healthy first observation is not logged). Records both
  2026-07-26 repairs.
- ROADMAP: R-77 shipped; R-78 the local_api authority ruling, with the
  clobber-a-working-channel risk spelled out in BOTH directions so it is not
  resolved opportunistically; R-79 the whole-surface English-strings sweep;
  R-80 expected_backup_missed, flagged as likely outranking R-77 because 7.3
  days of stale backup materially exceeds the ~1.5-day channel outage, so the
  causal link the DIAG hedged on cannot be the whole story.
- Capability map: note against the drive-wizard row (every agent-backed
  capability rides this channel) that a silent drift class is now detected.
  NO row status flips — detection is not prevention.
2026-07-26 09:14:40 +02:00
admin 9e94479449 DIAG: agent-channel banner is a stale controller.yaml endpoint, not island fallout
Read-only diagnostic; nothing restarted, reconfigured or deployed.

CAUSE (H2, supported): the island migration rewrote bootstrap.json
(169.254.253.1:8443) but the controller reads controller.yaml, which still
holds the pre-island LAN address on both boxes -- mtimes 2026-07-21 and
2026-07-23, i.e. days BEFORE the migration. ensureLocalAPI
(bootstrap.go:254) returns early when local_api.endpoint is non-empty, so it
fills a MISSING endpoint and never reconciles a STALE one. The agent no longer
binds the LAN address, hence "connection refused".

H1 refuted: every island leg is healthy on both hosts -- agent active and bound
169.254.253.1:8443, vmbr9 up, guest eth1 up, TCP from the guest to the island
is OPEN. The island works; nothing is using it.

H3 refuted: hub-persisted events put the first alarm at 2026-07-25 12:44/12:46,
~17.5h before the v0.172.0 deploy. Today's re-fires are restart artifacts --
channel state is in-memory by design.

H4 refuted: felhom-agent is active on both hosts.

Also: the logged "up->down" is an ARTIFACT -- the debounce branch seeds an
unseeded state to "up", so the channel was born-down and has never been up.

Symptom B confirmed: EffectiveProtected gates cloudflared on a token predicate
but samba on smb.Enabled alone, so sharing-without-password reports FAIL for a
deliberately-undeployed stack. The function's own doc comment asserts the parity
that is broken. Caused by leg 4 of the R-75 task. Blast radius: zero emails so
far, but demo-hp flipped ok->fail and the health_critical path has reached the
CUSTOMER channel historically. The English issue strings are whole-surface, not
a one-off.

Fixes described, none applied.
2026-07-26 08:51:45 +02:00
admin 89d64f64b3 R-75 SHIPPED (controller v0.172.0): docs, capability-map note, ROADMAP collapse
New documentation/controller/import-and-data-paths.md: the canonical import root
(and why it is NOT a registered StoragePath), the three data_paths roles, the
Fork-3 validation asymmetry, the class-driven copy rule, and the seven
invariants a future change must not break.

Capability map "File access via browser" — status DELIBERATELY UNCHANGED. The
drop-zone now has its own FileBrowser source and the app page carries a deep
link, both verified live, but nothing drove the FileBrowser HTTP UI (no browser
on DooPlex), so the row's standing "browse is exercised in no doc" caveat still
holds and PROVEN-LIVE remains unearned.

R-75 collapsed to its shipped one-liner. R-76 left open — this task does not fix
it, and nothing built here assumes an import/* directory stays 2775.
2026-07-26 08:30:45 +02:00
admin 2d2050c362 ROADMAP: R-75 catalog-derived userdata skeleton + import surfaces; R-76 setgid-chain break
R-75 (spiked, GO) names the capability-map row it would flip: "File access via
browser" (00-capability-map.md line 96), currently IMPLEMENTED with the caveat
that browse/download through FileBrowser is exercised in no doc. Carries the
mandatory determinism constraint from P6 (sort + red-proof, or FileBrowser
force-recreates on every sync pass), the zero-removals invariant for
`documents`, the url.PathEscape-not-QueryEscape trap, and the four design forks
with evidence + recommendation, all awaiting operator ruling.

R-76 is minted for the two PRE-EXISTING defects the spike surfaced and
deliberately did not fix: FileBrowser Quantum creating 0644/0755 without
propagating setgid (breaking the shared-group chain one level below any
customer-created folder -- latent only because every userdata-touching app runs
uid 1000), and import/calibre living at 755 on demo-felhom where its same-app
sibling media/books is 2775.

Source: audits/SPIKE-catalog-data-paths-2026-07-26.md
2026-07-26 06:59:57 +02:00
admin 0c20c91e85 SPIKE: catalog-derived userdata skeleton + import surfaces (2026-07-26)
Verdict GO, with one mandatory constraint.

Phase 0 HOLDS: the customer-facing path set is fully derivable from data the
controller already parses (ParseComposeClassifiableBinds), and
ValidateBackupSpec's refusal set already covers the path-safety class
data_paths: needs -- so the annotation-only design introduces no new
filesystem-write primitive. No catalog folder is reachable only via env
indirection; zero templates use long `type: bind` syntax.

Found off-brief: the derivation is ALREADY LIVE at deploy time
(stacks/manager.go:183 ensureUserdataMounts, "the deploy belt"), proven by a
clean two-box control -- media/podcasts exists on demo-felhom where
audiobookshelf is deployed, not on demo-hp, and is in no skeleton.

P0 GO: derived set 14 dirs vs hardcoded 14 (+media/podcasts, -documents); all
three hand-verified anchors match. P1 GO: 4 sources incl. nested + accented
names all index ready. P2 GO: deep-link template constructible in Go from
(sourceName, relPath) alone; login redirect preserves the target. P3 GO for the
feature, with a pre-existing setgid-chain break recorded. P4 GO: userdata/import
is shareable, guard unchanged, no live share created. P5: before-state inventory
captured read-only on both boxes. P6 GO-conditional: the naive derivation
produced 20 distinct outputs from 20 identical runs -- a guaranteed FileBrowser
force-recreate loop -- fixed by one sort. P7: 12 sources safe server-side.

No production code, no version bump, no live mutation.
2026-07-26 06:59:46 +02:00
admin 2d78c283c6 R-50 SHIPPED: fleet migrated to the island; Phase 5 docs
- B2 demo-hp + B3 demo-felhom migrated to the island (agent 0.96.0), apps
  served throughout (0 container restarts), island /storage 200, LAN DNS pinned
  to the LAN IP, hub reports 0.96.0. No rollback.
- capability-map 'site/network change' row PARTIAL -> PROVEN-LIVE
- ROADMAP R-50 -> SHIPPED (fleet-migrated); add R-74 (island on Peti's cluster)
- nodes.md: both boxes island-bound, agent 0.96.0
2026-07-25 14:47:15 +02:00
admin a2500532c3 R-50: A4 fresh-provision validated (agent auto-net1 live); spike caveat closed
- A4: scratch guest provisioned from golden on the island-configured drill came
  up with net1 automatically (zero manual edits) — the v0.96.0 provisioning path
  proven live. Spike method caveat CLOSED.
- REPORT: Phase A done + vouched 0.96.0; B0/B1/A4 done; B2/B3 await operator go.
2026-07-25 14:36:56 +02:00
admin 46504938ba R-50 Phase B: island migration runbook (B0) + drill validation (B1 PASS)
Idempotent LAN->island migration procedure with rollback table + abort criteria
(firewall LAST). Validated verbatim on drill VM 300: rolled to r50pre, migrated,
island /storage 200, LAN DNS held on the LAN IP (Finding-1 pin), apps healthy,
hub reports 0.96.0. No rollback fired.
2026-07-25 14:32:26 +02:00
admin 7fffde3e13 R-50 runbook: Part C (agent 0.95.0 -> demo-hp) + qm300 forensic
- deployed agent 0.95.0 to demo-hp via break-glass (0.93.0->0.95.0), binary
  sha-verified, caps 68/68 ok, guest 9201 untouched, hub confirms 0.95.0
- forensic: qm300 had no qmdestroy; it died with the mid-July N100 reprovision
- nodes.md: fleet agents 0.95.0; demo-hp deploy note
2026-07-25 12:25:04 +02:00
admin 515e0c3cc7 R-50 island-bridge: SPIKED -> GO (probes P1-P8 pass live on t740 drill)
Provisioned nested-PVE drill 'drill-r50' (qm300 on demo-hp) via the v1.25.0
nested-vm ISO through the real day-0, then ran the R-50 empirical spike:
- vmbr9 portless island bridge + guest island NIC hot-add (LAN undisturbed)
- F1 replay money shot: LAN move survives on the island; LAN-literal bind
  reproduces the 2026-07-20 daemon-exit bug verbatim
- dnsmasq trap confirmed live + lan_resolver.host_ip fix proven
- pin address-independent (leaf SHA-256 unchanged, HTTP 200 over island)
- survival matrix: agent/guest/host-cold-reboot all return on the island

Docs: SPIKE verdict BLOCKED->GO, ROADMAP R-50 SPIKED->GO, nodes.md drill VM,
REPORT overwrite.
2026-07-25 12:19:09 +02:00
admin 5d56f93755 docs: R-50 island-bridge spike attempt 2 (t740) — STILL BLOCKED (no drill VM there); nodes.md drill-host ruling; REPORT 2026-07-25 09:59:10 +02:00
admin 1d65f44205 docs: SPIKE R-50 island-bridge — drill env ABSENT (empirical blocked); source design findings + dnsmasq trap; ROADMAP R-50 spiked 2026-07-25 08:30:25 +02:00
admin be2d90f243 docs: disk-health capability row → PROVEN-LIVE (v0.95.0/v0.171.0 SMART-coverage fix, real verdicts+models live) 2026-07-25 08:26:49 +02:00
admin 770ee5a5f6 docs: SPIKE — SMART coverage on real hw (why the card shows Nincs adat); findings-only
Read-only probes on demo-felhom: both disks (system SSD /dev/sda, USB /dev/sdb) report
PASSED via the exact allowlisted 'smartctl -a -j <dev>'. Card shows Nincs adat because the
agent never reads: 'local' is a dir on LVM pve-root (backing='' + smartDeviceFor has no
dm/LVM branch), and the USB is surfaced via the non-enriched driveTargets.Known union path.
-d sat NOT needed (bridge passes SMART through; sudoers grants only -a -j). Graded fix
directions B(low-risk, USB) > A(system SSD, medium) > C(reject). STOP — no fix implemented.
2026-07-25 07:56:02 +02:00
admin e49f20f581 docs: disk-health capability-map row (IMPLEMENTED) + ROADMAP R-73 (SMART history/trending, parked) 2026-07-24 21:42:49 +02:00
admin f379ba0b34 docs: capability map + ROADMAP — Indítópult megosztás (guest capability link, controller v0.165.0)
New capability-map row (IMPLEMENTED; §13 endpoint-level live on 9201). Records the
ruling: member accounts are superseded by the capability-URL guest share for
launcher sharing; per-member tile visibility parked under the SSO/members arc (R-15).
Updated the launcher row's member-coupling note and R-15 accordingly.
2026-07-24 12:19:21 +02:00
admin 5df15bea9f docs: launcher capability row (IMPLEMENTED) + ROADMAP R-15 coupling + R-72 brand_color curation (controller v0.163.0) 2026-07-24 09:17:46 +02:00
admin 8462338ad4 docs(R-71): SHIPPED (a)+(c) — the apply-bridge settle-gate (controller v0.162.0)
R-71(a) day-0 ordering fix shipped: offsiteapply settle-gate defers the offsite
one-time-password consume past a managed floor-update, reading the updater's own
floor/update-running state (no second floor path). Ordering-only; (b) two-phase
consume rejected-by-design. ROADMAP R-71 -> SHIPPED (a)+(c); DIAG-f10 final
annotation added. Layering: v1.25.0 build gate prevents, (a) defers, (c) heals.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N7Drmtm2RzoqbkJZCNSFNQ
2026-07-24 07:49:11 +02:00
admin 6cc875cc36 drill: R-63 ő/ű captured live on the KVM HP (+ pairing + rootpw); HP SVM-enable fix; evidence PNG
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR
2026-07-23 20:46:27 +02:00
admin 6dab0f6a2c docs: ISO train v1.25.0 — REPORT (belt/apt/R-63/gate/vault live), F9 resolved, R-63 shipped, nodes belt+vault, F8 checklist; critical golden<floor finding
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR
2026-07-23 15:48:01 +02:00
admin b620435afe ISO train v1.25.0 code: OOB belt default appliance leg (F9) + apt no-subscription (Part2) + R-63 Lat2 console font + R-71 golden>=floor build gate
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR
2026-07-23 15:38:14 +02:00
admin 527d81cf70 docs: v0.72.0 ship report + live validation, CONTEXT rulings, DIAG-f10 R-70/R-71 status annotations
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR
2026-07-23 13:05:03 +02:00
admin 1133aade73 hub v0.72.0 — R-70 + R-71c: offsite delivery-state detector, card, stuck event, R-39(a)-guarded self-heal restage
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR
2026-07-23 12:59:04 +02:00
admin c801cee647 F10 offsite leg resolved on demo-hp: day-0 update-vs-bridge race diagnosed, designed-path repair + tier round-trip proven; R-70/R-71 minted
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR
2026-07-23 12:21:05 +02:00
admin c766c8af82 hub v0.71.0: paired recovery mails (F11), prefs seeding at claim + empty-email no-clobber (F12), priority headers + operator test leg (F14-light) 2026-07-22 20:57:29 +02:00
admin 5b35023574 docs(audits): power-outage recovery audit 2026-07-22 — power cut confirmed, full self-heal in 3m15s, dead-man's-switch fired on schedule; F8-F13 2026-07-22 19:57:44 +02:00
admin e57dd8b0e8 docs: R-67 SHIPPED (coupled to R-64) + NAS doc Browsing (FileBrowser) section
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
2026-07-22 14:18:30 +02:00
admin 8ecab3cd64 docs: R-66 shipped (ROADMAP + NAS naming caveat); mint R-64 pairing-blessed + R-65 buddy-box replication
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
2026-07-22 13:57:20 +02:00
admin a12c6f9730 docs(v1.24.0): R-59/R-60/R-61 SHIPPED — CHANGELOG, README, ROADMAP (+R-62), runbook, capability map, drill evidence, REPORT
Virgin-ISO nested drill closed the train: dead-NIC install baked the
fallback (incl. the dead default gateway), the R-59 screen painted
(capture committed beside the spike doc), the cable move healed +
registered at the hub in 23s unaided, and the build's rootpw file
matched the installed box's shadow hash. R-59 SHIPPED with the recorded
deviation (first-boot gate; installer-initrd abort out of scope by
operator ack). R-60 SHIPPED (spike + drill cited; F-P9 route-flush fix
included). R-61 slice 1 SHIPPED. New R-62 row (hub delete-dialog
cosmetics, XS). Capability map: new PROVEN-LIVE row (nested != metal,
said so). Cleanup verified: felhom-pve interfaces byte-identical,
bridge/VMs/ISO removed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
2026-07-22 11:45:55 +02:00
admin c043ba9abe docs(spike): R-59/R-60 Phase-0 — the NIC sweep mechanics are GO, measured on a nested wrong-NIC install
Nested VM 9310 on felhom-pve (throwaway vmbr9 rig): re-point
bridge-ports -> ifreload (0.36s, raises the port itself) -> dhclient
lease 3.3s -> hub HTTPS 302 in 160ms; a dead candidate costs exactly
its timeout bound (rc=124 at 20s). Bonus finding: the auto-installer
picks the NIC that leases and bakes the lease as STATIC — so wrong-NIC
boxes come in two variants (192.168.100.2 fallback bake vs
truth-changed-after-install), and the gate must trigger on
hub-unreachable, never on the fallback signature. Eight design inputs
(F-P1..F-P8) recorded for the Part-3 gate. STOP-0 discharged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
2026-07-22 10:50:43 +02:00
admin f4c2c4151b hub v0.70.1: the ghost customer's Delete button must exist (Danger-zone render gate split) 2026-07-22 09:23:37 +02:00
admin 2cf37f263b docs(pilot): agent 0.93.0 publish train record + onboarding §A5 gate discharged 2026-07-22 09:03:39 +02:00
admin 05dfaa1a10 docs(runbooks): onboarding draft v3 -> v4 — post-ship refresh (R-36/R-39 workarounds deleted, HP t740 second datapoint)
Per the 2026-07-21 refresh brief: R-39 interim blocks (B4/E1) and the R-36
manual-Save block (C4) deleted — both shipped and proven live; freemail.hu
gate proven (R-4 COMPLETE); golden/floor-lift note now cites two shapes
(rehearsal + virgin HP t740 day-0 lift 0.153.0->0.156.0); A3 loader table
per operations/nodes.md (N100=mkimage/SB-off per record, HP t740=shim/SB
ENABLED); B2 multi-NIC cabled-port gotcha (R-59/R-60 pending); new A5 gate
(agent >=0.93.0 deployed box-side before the first escrow ceremony); D
offboarding pointer to §G (R-25b). DRAFT status and the C7 graduation gate
unchanged. ROADMAP R-25b pointer follows the rename.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
2026-07-22 08:14:50 +02:00
admin 9b3381be0a feat(hub): v0.70.0 — a deleted customer actually disappears (residue leg + ghost cleanup)
Found validating v0.69.0 against the live hub. demo-vm-felhom was deleted
on 07-18 and was still on the Customers list AND still raising offsite_stale
(10 events, latest 07-21 17:34, operator email at 19:34) — because
GetCustomers() is report-derived and no lifecycle tier ever deleted a report.

New leg 3 (residue), before the record purge: reports, app_telemetry,
app_log_tails, log_tail_requests, customer_notifications, plus the
credential-bearing appliance_registrations and selfbind_tokens. Audit
(events, notification_log) and F-14 provenance still survive.

Ghost customers are now deletable: 404 means "nothing here", not "no config
row". With no config row the offsite descriptor is unknowable, so the Hetzner
and descriptor legs record skipped_no_config rather than a bare "skipped".

Two more red-proofs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS
2026-07-21 20:28:06 +02:00
admin 61dbd870c3 feat(hub): v0.69.0 — customer DELETE is the guided full-teardown cascade (R-25b)
POST /configs/{id}/delete now runs hosts -> RESET -> purge behind three
acknowledgements, a typed customer-id, a stale-preview check and the
ONLINE-host refusal (every gate before any write, so a refusal has zero
side effects). The shallow handleConfigDelete is gone.

Two invariants are asserted, not just commented: ruling 3 is preserved by
construction (leg 2 never sees a host row) and retained escrow custody is
purged exactly once, in leg 3 (leg 2 runs with purgeEscrow=false).

handleCustomerReset's committed half was extracted as commitCustomerReset;
the standalone RESET path is byte-identical to v0.68.1 and its suite is
untouched. Five red-proofs run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS
2026-07-21 19:31:48 +02:00
admin 8ec45f3b42 docs: node inventory for the two-host fleet; demo-hp on the tailnet
New documentation/operations/nodes.md: HP t740 hardware, disks (SanDisk system
SSD 182195804614; Toshiba 1TB NVMe 58BS11AFT8MQ PRESENT AND UNENROLLED, still
NTFS, do not touch), the five-NIC map and the trap that cost the first install,
and the access path - no SSH key is baked, auth is the hub-vaulted G1
break-glass password.

tailscale.md gains demo-hp, the operator-lab-exception warning so a future
product-shape audit does not conclude the product ships tailscale, and the
--accept-dns evidence: the join omitted the flag, MagicDNS rewrote
/etc/resolv.conf, and it was reverted. Harmless at the vacation site, would
have bitten silently at home where split-horizon matters.

OPEN: key expiry still enabled on demo-hp (2027-01-17) - needs an admin-console
toggle or an API token; a pre-auth key cannot do it.
2026-07-21 18:51:05 +02:00
admin 460cc73bad docs: R-59/R-60/R-61 from the HP install + second-hardware pairing proof
R-59 no-DHCP install must hard-abort (it baked 192.168.100.2 static and
completed - a box that can never call home). R-60 first-boot NIC sweep
self-heal. R-61 the baked root password must be knowable; a fixed well-known
password is explicitly rejected.

Positive evidence same-session: R-21 slice C PROVEN on a SECOND, virgin board
(HP t740) - and the shim loader booted with Secure Boot ENABLED, retiring the
assumption that Felhom installs need SB off. Fresh-box floor lift
0.153.0 -> 0.156.0 during day-0 cited on the publish-train row.

ISO README gains the t740 five-NIC trap: the 4-port igb card gets no lease,
the onboard r8169 port does.
2026-07-21 18:44:54 +02:00
admin 2143ae2d8d docs: R-58 assisted disk-picker install mode + HP mini ISO profile notes
R-58 records the operator ruling (2026-07-21) with the argument verbatim: the
installer should list available storage devices, excluding the install media,
and let one be selected. Third ISO mode alongside unattended-serial and
match-nothing-safety; unattended stays the appliance/factory mode. Slice 1 is
the abort-screen candidate table, same enumeration code, and it collapses the
two-boot dance on its own. Matters most for BYO/reinstall, where the serial is
unknown and a wrong guess is destructive.

ISO README gains the HP section: shim proven on this board by the safety boot,
the uncommitted-armed-profile pattern, verify-from-inside-the-ISO, and a
pointer to the prior-LVM abort that is the one likely failure on a
second-hand disk.
2026-07-21 18:00:21 +02:00
admin 3b41a982fe docs: R-55 PROVEN LIVE, new R-57 (app lifecycle), R-56 plumbing note, capability map
R-55's reboot leg ran operator-present on 9201: immich UI-stopped -> stayed
stopped across pct reboot, calibre-web recreated, zero alerts, ~15s.

R-57 records the lifecycle mechanism with the operator's abandoned-app
requirements verbatim and plant-it as the motivating case, including why the
retired/ directory move was wrong and the v0.158.1 pointer-receiver defect.
2026-07-21 16:35:28 +02:00
admin af8a7a58a4 docs+scripts: TASK-E parts 4-7 - operator one-pager, rulings, hub build.sh adoption
PROMPT-TEMPLATE: standard 'For the operator' plain-language section, mandatory
for M+ tasks and anything with a STOP.

ROADMAP rulings (operator, 2026-07-21): R-25b full-teardown cascade with three
acks + typed name (M-sized, spec to follow, no longer blocks R-3); R-11 channel
= direct Messenger, doc is the architect's; R-42 option (a), sidecars follow the
app; R-17 delete the archive - spike-lite found NO tooling verb targets it, so
it is an operator console action; R-4 complete (freemail.hu verified).
R-55 + R-41 slice 1 marked shipped; new R-56 (app difficulty classification -
the constructive half of the glance ruling).

scripts/build-hub.sh v1.23.0: the hub build script was outside any repo. Adopted
verbatim + versioned; the build-dir path is now a symlink to it.

felhom-testing skill: the ~1/5 recovery-code 'known flake' is retired - it was a
real defect the test was correctly detecting.
2026-07-21 15:32:56 +02:00
admin d4ac52d9e0 docs(roadmap): R-54 confirmed hub-side; record the in-memory damping limitation 2026-07-21 13:39:45 +02:00
admin befaa795e0 docs(roadmap): R-54 negative leg completed — 30 healthy cycles, zero heals 2026-07-21 13:17:36 +02:00
admin 3d28862db3 docs: STOP-1 + STOP-2 evidence — all three legs PROVEN LIVE; new R-55
Dead primary: degraded in 13 s, exactly one app_start_failed, banner rendered and
self-cleared. Boot orphan: recovered in one attempt with zero alerts. Dead dhclient:
detected in 57 s on process liveness while the lease was still live, healed 120 s after the
kill — the tunnel never dropped, so the outage was prevented rather than observed.

P1 answered as a by-product: bookstack StartedAt == the moment bootrecon StartStack
returned, so unless-stopped did NOT resurrect it. F5 hypothesis confirmed.

New R-55, surfaced by the leg designed to prove the opposite: the boot bind gate recreates
and STARTS every deployed drive-backed app unconditionally, so a customer Stop does not
survive a reboot for those apps. Predates R-52 and does not implicate it, but it narrows
R-52's practical scope and needs a ruling.
2026-07-21 12:56:39 +02:00
admin 907e5ce65c docs(TASK-D): R-51/R-52 SHIPPED + new R-54 row; capability-map row; seam-discipline rider
R-51's roadmap diagnosis is corrected at the source: aggregation returned StateRunning
("partial") for a running/stopped mix, so the stack read RUNNING and IsDownState was never
consulted about  at all — the constraint that row protects was never in tension
with the fix.

New R-54 row closes the INCIDENT-guest-dhclient-killed-2026-07-20 §5 OPEN RISK, and records
the design fact that makes it work: liveness of the DHCP client is itself a probe, because
the damage is timed and the address outlives its cause by 1-2 hours. The static-guest leg is
deliberately deferred to R-50.

New capability-map row is IMPLEMENTED, not PROVEN-LIVE: one leg is live (the watchdog's
healthy cycle on felhom-pve), the three that matter are destructive and operator-present and
have not run.

PROMPT-TEMPLATE §10 gains the seam-discipline row, including that a strings.Contains source
assertion is NOT sufficient — a commented-out call still contains the string.
2026-07-21 12:40:00 +02:00
admin f761f69d46 docs: R-39 CLOSED — STOP-2 proven live; DR-tier row -> PROVEN-LIVE
The operator pressed Re-issue PBS credentials and the chain closed in 13 seconds. The
identical click on 2026-07-18 did nothing at all.

  hub  08:39:31Z  fresh mint, generation 0 -> 1; descriptor gains secret_generation: 1
                  (token_id + fingerprint BYTE-IDENTICAL — the invisible re-key shape)
  agent 10:39:34  felhom-pbs-apply read felhom-pbs      (leg b: the impossible read)
  agent 10:39:38  ERROR REJECTED ... applied and DEAD, previous_state=applied
                                                        (leg c: the R-39 state, loud)
  hub  08:39:45Z  consumed_at stamped
  agent 10:39:45  one-time token secret consumed        (leg a: NO short-circuit)
  agent 10:39:45  reconcile (set-only, no --server)
  agent 10:39:47  pbsdr: converged state=applied

Corroboration: marker hash moved to afbb3b41… (it was byte-identical to the pre-reissue
marker in the failure); secret mtime 2026-07-18 -> 2026-07-21 10:39:45; new credential
probes 200; three consecutive reports trace applied -> auth_failed -> applied; ZERO
self-heal escalations, one mint, one consume, no consumed-failed.json — the box healed
through the descriptor path before the damper was ever needed.

Recorded for future runbooks: the operator first pressed the OFFSITE re-issue (two
distinct Re-issue actions exist). Harmless to PBS-DR, but it rotated the restic password
and correctly marked the escrow STALE, so the ceremony had to be re-run. Name the surface
explicitly next time.
2026-07-21 10:52:52 +02:00
admin 54a4644721 docs: R-39 fleet fix SHIPPED (hub 0.68.0 + agent 0.91.2); R-50b(a) SHIPPED; (b)/(c) open
R-39's three legs are closed and deployed: the hub stamps a monotonic secret_generation
so a re-key finally moves the descriptor hash; the wrapper gains a narrow read verb so
the non-root agent can read the credential it writes; and ProbeAuth turns a 401 into a
loud auth_failed the existing damper escalates to a fresh mint. Plus a consumed_at
honesty gauge for the applied-but-never-consumed disagreement.

Recorded in the R-39 row, because both are the kind of thing a future reader needs:

- A load-bearing fact the spec did not flag, checked rather than trusted: Apply bails out
  if the storage status probe ERRORS and adopt converges without consuming when the
  storage reads active, so the fix depended on PVE's 401 behaviour. PVE's storage_info
  wraps activation in eval{} and leaves active=0, so a 401 returns HTTP 200 with
  active:0 — never an API error. The chain is sound by proof, not inference.

- A defect I shipped and caught: v0.91.0 built the probe seam and main.go never wired it,
  so the leg was inert while every test passed. Same class as controller v0.154.0 the day
  before. Fixed in v0.91.1 (artifact superseded, not overwritten); v0.91.2 made a healthy
  probe observable so "no auth_failed" can never again be confused with "never probed".

The DR-tier capability row is deliberately NOT upgraded to PROVEN-LIVE: the decisive
evidence is STOP-2, the operator pressing Re-issue and the box converging where the
identical click did nothing on 2026-07-18.

R-50b(a) shipped — wrapper sha256 in the manifest + agent reporting + host drift surface,
with unknown-on-either-side reading as quiet rather than drift. (b)/(c) remain open: the
wrapper is still fetched unversioned from raw/branch/main.
2026-07-21 10:24:11 +02:00