Commit Graph

58 Commits

Author SHA1 Message Date
admin d4c07873ca docs: correct the installer-channel record — R-94 retracted and re-scoped, R-110 opened
The 2026-07-29 R-94/E-2d finding was written from an unverified claim and was
false. `felhom-bootstrap.sh:96` fetches the installer from the WEBSITE, not the
hub; the website git-syncs /scripts/ from main on a 30s period; every install
since 1.22.0 hit main this morning already runs 1.22.0. Confirmed by live fetch.

- OPEN-ITEMS.md: merge the two duplicate R-94 rows into one, retract the false
  framing, re-scope to what it actually is (a drifting hand-synced constant plus
  two pieces of dead safety equipment), unblock it from E-2d.
- OPEN-ITEMS.md: de-rank R-94 in the ranked list — the "high-consequence" reason
  was the false claim in its most load-bearing form.
- OPEN-ITEMS.md: E-2d — the ISO is the STRONGER proof route, not an obstacle.
  Phase 0 question answered at source: PAIRING falls through to run_direct in
  the same invocation (:495-499), so it reaches the identical installer call.
- ROADMAP.md:149: same retraction; the original diagnosis (a hand-synced
  constant in a second repo drifts every time the first ships) survives.
- ROADMAP.md + OPEN-ITEMS.md: new R-110 — main is the installer's publish
  channel and there is no staging, tag, pinned path or rollback, for the one
  artifact that runs as root on a virgin box. Operator ruling, not a defect.
- day0-install.md C.1: one sentence recording the same about the fetch URL.

Documentation only. No version bump, no CHANGELOG entry, no code, no box touched.
2026-07-29 10:54:59 +02:00
admin b5a73e050b Move the local whole-guest backup off the guest's own device (demo-hp + demo-felhom)
Supervised operational run. No code, no version bump, nothing deleted.

Primary backup tier on both demo boxes moved from `local` (a dir storage on
/var/lib/vz -- the SAME physical device as the guest) to `felhom-backup`, a dir
storage on each box's secondary drive:

  demo-hp      /mnt/nvme-1tb  uuid:91d2dc2d-...  archive 2,256,044,492 B
  demo-felhom  /mnt/hdd_1     uuid:47a3361a-...  archive 5,957,878,962 B

Both proven end to end via the real UI path: archive lands on the secondary
drive (df delta matches the archive byte-for-byte), restore-test auto-selects it
and passes with mount_parity: ok, and freshness survives an agent restart with
an empty in-memory store -- so the age can only have come from the new storage.

Phase 0: the target is CONFIGURATION, not converged (the sole writer of
agent.json touches only escrow.pbs_storage_id and preserves unknown keys), so
the runbook's STOP did not fire. No consumer hardcodes "local" on the backup path.

Findings:
- F-1  the storage path must BE the mountpoint; a subdirectory fails exactMount
       and the target reports disconnected permanently (observe.go:321)
- F-2  --is_mountpoint 1 is load-bearing; proven live, an unguarded storage on a
       non-mounted path reports active with the ROOT filesystem's free space and
       had already created dump/ on pve-root -- a silent retarget onto the very
       device this change escapes
- F-3  FelhomAgentStore is granted per storage path; without it every backup
       403s. felhom-host-install.sh must issue it for new installs
- R-109 (new) the DR recipe records no backup target, and each box now carries
       two content=backup dir storages, one live and one frozen
- R-105 narrowed and TRACED: dr_recipe drives was [] fleet-wide because the
       enrolled drives were never PVE storages, so isUserDataDrive never saw
       them. Both boxes now populate drives; SMART on the backup drives too

Absent-drive behaviour today is fail-loudly with no silent retarget (PVE half
live-proven; agent half source-traced). That is NOT the intended fall-back-and-
alarm design -- filed as E-2 with the honest single-drive label.

Reported in full in the record: the agent was restarted with a felhom-pbs backup
in flight, producing a spurious tier failure. The backup had in fact succeeded
(PVE task OK, 6,264,034,053 B snapshot) and the spurious failure reached no
channel -- R-84 ground truth superseded it.

Outstanding: full drive-loss recovery (needs physical access) and the agent half
of the absent-drive behaviour.
2026-07-28 21:38:13 +02:00
admin f47b0a61d7 R-101 + F-DIAG closed, F-OPS documented (manual-restore runbook) 2026-07-28 16:45:37 +02:00
admin b505ee9125 R-100: offsite staleness counts from the last SUCCESS (hub v0.80.0)
isStale counted from last_run, written unconditionally on failure, so a nightly-failing
tier read as fresh forever. Now anchored on last_success with an explicit legacy degrade
(logged once) and the never-ran branch untouched. emitStale states the real reason.
2026-07-28 13:17:04 +02:00
admin e168600148 docs: F-CRIT-1 + F-A1 shipped (controller v0.179.0); invariant rule
Both marked SHIPPED + PROVEN-LIVE in OPEN-ITEMS and the campaign doc. All three
of Campaign 8's alarm findings are now closed (F-CRIT-1, F-CRIT-2, F-A1).

Adds the standing rule earned by this arc to the versioned workspace CLAUDE.md:
a comment asserting an invariant needs a test pinning it, or it is a wish — with
all six shipped-false-guarantee instances catalogued, and the corollary that a
test should assert the CONSEQUENCE (does the alarm fire?) not the MECHANISM
(does suppression expire?).
2026-07-28 09:48:51 +02:00
admin a0a1556ce6 docs: R-88b shipped; standing rule 4; READY rows re-ranked
R-88b closed (agent v0.105.0 + controller v0.178.0) — age_state gives 'unknown'
its own representation, with empty meaning legacy rather than unknown so the
first-backup valve keeps working on un-upgraded boxes.

R-97 note updated: hub v0.79.0 (R-97c) replaced a FALSE operator-only comment
with a real register — the comment claimed a guarantee the code did not provide.

Standing rule 4 (R-96): a recommendation that is not followed gets one line
saying why. Added to the live CLAUDE.md and this versioned copy — the live file
is not in a git repo, so committing to it alone would leave the rule as durable
as the chat it came from.

READY re-ranked: R-95 now leads.
2026-07-27 18:12:57 +02:00
admin 9ea5675950 docs: sync workspace-CLAUDE.md with the live file, carrying R-96's three rules
The workspace root /mnt/5_hdd/felhom.eu/git/CLAUDE.md is NOT a git repo — this is
its only version-controlled copy, and it had drifted since 2026-07-19. Committing
the three standing rules to the live file alone would have left them exactly as
undurable as the chat log they came from, which is the whole point of R-96.
2026-07-27 17:08:24 +02:00
admin a31872ea24 docs(pbs): move PBS prune server-side, close the write proof, schedule GC
Supervised runbook execution. No code, no version bump.

The felhom-pbs tier had reported `job errors` on EVERY demo-hp backup
since the tier was created on 07-26, while the data landed correctly
every time: `DatastoreBackup` grants Datastore.Backup but not
Datastore.Prune, so the box's keep_last=2 prune was denied.

Operator ruling: retention is a COMMERCIAL attribute owned by the hub;
ep0 executes. Box tokens therefore stay write-only - a compromised box
must not be able to delete its own offsite backups. No grant was widened
and felhom-tenantsync.sh is unchanged (the ruling makes it correct).

Increment 1:
- boxes stop attempting prune. allowPBSPrune is DERIVED
  (`!t.Primary && t.KeepLast > 0`), so keep_last: 0 on the PBS tier
  disables both the --prune-backups value and the gate in one config
  edit, and the tier stays armed. Verified prune_pbs_allowed=false on
  both boxes with no tier REJECTED line.
- per-namespace prune jobs on ep0, keep-last 2, daily 03:30 UTC
  (05:30 CEST), dry-run gated. demo-hp 3->2, demo-felhom untouched,
  chunk count unchanged (prune removes indexes, not chunks).

Write proof CLOSED: 08:25:47 job errors -> 09:37:29 TASK OK, snapshot
2026-07-27T09:37:29Z, chunks 9787->9813, prune step absent entirely.
Driven through POST /api/guest-backup/trigger (the UI path), not
--selftest and not raw vzdump. Hub gauge evidence explicitly NOT
satisfied - the delta is below its 0.1 GB display granularity.

GC scheduled sun 04:30 UTC and deliberately NOT run: every chunk still
carries a fresh atime from the migration copy, so a run today would
reclaim nothing. verify-new enabled per operator ruling, turning an
inert hub alarm live.

Legacy demo-felhom-01 namespace deleted with its two ACL entries and its
token (operator ruling, confirmed twice) so nothing dangles.

R-89 records the target architecture and carries the unanswered parallel
question: does the restic key on storage-box-pool-1 have DELETE rights?
If so the daily app-data tier has the identical exposure and append-only
is the equivalent answer.

ep0 is Etc/UTC, not CEST - corrected in the record.
2026-07-27 15:16:11 +02:00
admin 2b24c70536 docs(ep0): hub PBS-DR capacity gauge verified correct after the volume move
The last open item from the datastore relocation. Hub operator UI
(Offsite -> PBS DR) reports felhom-offsite (ep0) at 97.9 GB capacity,
12.6 GB used, 13% full - agreeing with the on-box df (98 G / 13 G / 13%).

The gauge follows the datastore's CONFIGURED PATH, so the relocation
required no hub-side change. RUNBOOK section 10.3 warned that a stale
37.2 GB reading would mean the gauge reads the wrong filesystem and
would be a real bug worth a roadmap item - it does not, and there is
no bug.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018ARoadHBf8rHoscfiqeVZn
2026-07-27 09:39:44 +02:00
admin ad8057c4e3 docs(ep0): relocate the felhom-offsite PBS datastore onto the 100 GB volume
Supervised runbook execution. No code change, no version bump.

felhom-offsite moved from ep0's 40 GB root disk (/srv/pbs-felhom) to a
dedicated 100 GB Hetzner Cloud Volume (/mnt/pbs-datastore, ext4 -m 0,
by-id fstab, relatime). Datastore NAME unchanged, so the PBS-DR
descriptors, per-box storage ids, ACLs and namespaces are untouched.

Capacity: 37.2 GB -> 98 GB total, 28.9% -> 13% used, headroom to the 80%
warn 19 GB -> ~65 GB. This CLEARS the R-82 Phase 0 P0.3 STOP. Per-tenant
encryption still precludes cross-customer dedup, so the slope is
unchanged - the volume buys runway, not a better cost model.

Verified: byte totals and chunk counts identical (9748), 7/7 snapshots
across all three namespaces, backup:backup ownership, clean itemised
dry-run, full verify job TASK OK with 0 errors, and a restore round-trip
(source_tier pbs, pass true, mount_parity ok, clean teardown).

Nothing deleted - the original 13 GB stays at /srv/pbs-felhom as the
rollback until a new weekly backup lands. GC deliberately not run.

Three findings recorded:
- the `scratch` datastore points at a non-existent path (pre-existing;
  now logs ENOENT every start) - operator decision
- the runbook's S6 guard test proves the wrong proposition:
  RequiresMountsFor re-mounts rather than refusing, so the test only
  bites when the device is genuinely unavailable (re-run that way, and
  the refusal was observed) - amendment recommended
- S11: storage box u629193 has no live backup path, BUT ep0 carries an
  enabled sshfs mount unit against it that must be removed before the
  box is deleted

Deviations: the volume arrived pre-formatted and mounted; S8 ran on
demo-felhom rather than demo-hp (no SSH key for demo-hp); the window was
contended by a stale in-memory 10-minute restore-test cadence whose
config had already been reverted on disk.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018ARoadHBf8rHoscfiqeVZn
2026-07-27 09:36:11 +02:00
admin 9cfa619ec3 hub v0.74.0: allow local_api_endpoint_drift; R-77 docs + R-78/79/80
The allowlist entry is REQUIRED, not cosmetic: handleEvent 400s an unknown
event_type, so controller v0.173.0's new drift alert would be silently inert
without it. Shipped with the controller that emits it.

Docs:
- RUNBOOK-local-api-endpoint-drift.md — how to repair a drift, including the
  step everyone will want to skip (establish which value is CORRECT from what
  the agent is actually bound to, rather than assuming bootstrap.json wins) and
  what success looks like (SILENCE, not a "recovered" line, because a fresh
  controller's healthy first observation is not logged). Records both
  2026-07-26 repairs.
- ROADMAP: R-77 shipped; R-78 the local_api authority ruling, with the
  clobber-a-working-channel risk spelled out in BOTH directions so it is not
  resolved opportunistically; R-79 the whole-surface English-strings sweep;
  R-80 expected_backup_missed, flagged as likely outranking R-77 because 7.3
  days of stale backup materially exceeds the ~1.5-day channel outage, so the
  causal link the DIAG hedged on cannot be the whole story.
- Capability map: note against the drive-wizard row (every agent-backed
  capability rides this channel) that a silent drift class is now detected.
  NO row status flips — detection is not prevention.
2026-07-26 09:14:40 +02:00
admin 46504938ba R-50 Phase B: island migration runbook (B0) + drill validation (B1 PASS)
Idempotent LAN->island migration procedure with rollback table + abort criteria
(firewall LAST). Validated verbatim on drill VM 300: rolled to r50pre, migrated,
island /storage 200, LAN DNS held on the LAN IP (Finding-1 pin), apps healthy,
hub reports 0.96.0. No rollback fired.
2026-07-25 14:32:26 +02:00
admin b620435afe ISO train v1.25.0 code: OOB belt default appliance leg (F9) + apt no-subscription (Part2) + R-63 Lat2 console font + R-71 golden>=floor build gate
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR
2026-07-23 15:38:14 +02:00
admin a12c6f9730 docs(v1.24.0): R-59/R-60/R-61 SHIPPED — CHANGELOG, README, ROADMAP (+R-62), runbook, capability map, drill evidence, REPORT
Virgin-ISO nested drill closed the train: dead-NIC install baked the
fallback (incl. the dead default gateway), the R-59 screen painted
(capture committed beside the spike doc), the cable move healed +
registered at the hub in 23s unaided, and the build's rootpw file
matched the installed box's shadow hash. R-59 SHIPPED with the recorded
deviation (first-boot gate; installer-initrd abort out of scope by
operator ack). R-60 SHIPPED (spike + drill cited; F-P9 route-flush fix
included). R-61 slice 1 SHIPPED. New R-62 row (hub delete-dialog
cosmetics, XS). Capability map: new PROVEN-LIVE row (nested != metal,
said so). Cleanup verified: felhom-pve interfaces byte-identical,
bridge/VMs/ISO removed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
2026-07-22 11:45:55 +02:00
admin 2cf37f263b docs(pilot): agent 0.93.0 publish train record + onboarding §A5 gate discharged 2026-07-22 09:03:39 +02:00
admin 05dfaa1a10 docs(runbooks): onboarding draft v3 -> v4 — post-ship refresh (R-36/R-39 workarounds deleted, HP t740 second datapoint)
Per the 2026-07-21 refresh brief: R-39 interim blocks (B4/E1) and the R-36
manual-Save block (C4) deleted — both shipped and proven live; freemail.hu
gate proven (R-4 COMPLETE); golden/floor-lift note now cites two shapes
(rehearsal + virgin HP t740 day-0 lift 0.153.0->0.156.0); A3 loader table
per operations/nodes.md (N100=mkimage/SB-off per record, HP t740=shim/SB
ENABLED); B2 multi-NIC cabled-port gotcha (R-59/R-60 pending); new A5 gate
(agent >=0.93.0 deployed box-side before the first escrow ceremony); D
offboarding pointer to §G (R-25b). DRAFT status and the C7 graduation gate
unchanged. ROADMAP R-25b pointer follows the rename.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
2026-07-22 08:14:50 +02:00
admin 9b3381be0a feat(hub): v0.70.0 — a deleted customer actually disappears (residue leg + ghost cleanup)
Found validating v0.69.0 against the live hub. demo-vm-felhom was deleted
on 07-18 and was still on the Customers list AND still raising offsite_stale
(10 events, latest 07-21 17:34, operator email at 19:34) — because
GetCustomers() is report-derived and no lifecycle tier ever deleted a report.

New leg 3 (residue), before the record purge: reports, app_telemetry,
app_log_tails, log_tail_requests, customer_notifications, plus the
credential-bearing appliance_registrations and selfbind_tokens. Audit
(events, notification_log) and F-14 provenance still survive.

Ghost customers are now deletable: 404 means "nothing here", not "no config
row". With no config row the offsite descriptor is unknowable, so the Hetzner
and descriptor legs record skipped_no_config rather than a bare "skipped".

Two more red-proofs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS
2026-07-21 20:28:06 +02:00
admin 61dbd870c3 feat(hub): v0.69.0 — customer DELETE is the guided full-teardown cascade (R-25b)
POST /configs/{id}/delete now runs hosts -> RESET -> purge behind three
acknowledgements, a typed customer-id, a stale-preview check and the
ONLINE-host refusal (every gate before any write, so a refusal has zero
side effects). The shallow handleConfigDelete is gone.

Two invariants are asserted, not just commented: ruling 3 is preserved by
construction (leg 2 never sees a host row) and retained escrow custody is
purged exactly once, in leg 3 (leg 2 runs with purgeEscrow=false).

handleCustomerReset's committed half was extracted as commitCustomerReset;
the standalone RESET path is byte-identical to v0.68.1 and its suite is
untouched. Five red-proofs run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS
2026-07-21 19:31:48 +02:00
admin 29b2b8ad7c docs: DooPlex-local commands in PROMPT-TEMPLATE and the manual-build runbook 2026-07-19 12:27:44 +02:00
admin fa8c0d5972 docs(workspace): record the migrated project-memory location 2026-07-19 12:22:07 +02:00
admin ee48a48288 docs: migrate workflow to DooPlex-local execution 2026-07-19 12:16:52 +02:00
admin aa9c08f0dd docs(rehearsal): N100 S8 wrap — VALIDATION, map flips, ROADMAP R-30..R-39, R-3 draft
The 2026-07-18 N100 rehearsal ran the complete final-product flow on real metal
in one pass: RESET -> generic pairing ISO v1.20.0 -> customer self-bind -> day-0
-> managed-floor lift -> escrow ceremony -> offsite snapshots. No code changed;
every finding is recorded and ranked, none fixed.

VALIDATION-n100-rehearsal-2026-07-18.md — run context, a UTC-normalised timeline
built from the hub events stream / hub DB / controller log / bootstrap + agent
journals, per-ledger verdicts for S1-S8 + ledgers 8 and 9, 12 findings, the
not-exercised list, and 7 discrepancies against the brief.

Headline wall-clocks: bind -> credential 26 s; bind -> controller running the
current version 2 min 44 s; managed floor 0.143.0 -> 0.145.0 in 5 s unattended
(initiated_by: auto-floor); escrow ceremony -> offsite enabled 12 s; drive enrol
30.3 s. No post-bind leg stalled, which is the immediacy row's real-onboarding
proof.

Capability map (10 citations added):
- Bare-metal Felhom ISO           PARTIAL     -> PROVEN-LIVE (F1 closed on metal)
- Customer self-bind (slice 1)    IMPLEMENTED -> PROVEN-LIVE (customer_selfbind)
- Guest RAM resize (R-24)         IMPLEMENTED -> PROVEN-LIVE (shrink AND grow)
- Customer RESET                  two real firings + verified external teardown
- Escrow ceremony                 first live wizard firing
- Immediacy row                   "real-onboarding proof pending" cleared
- Publish train                   box-side floor lift proven on a fresh install
- Customer claim                  R-4 gmail half (Inbox under p=quarantine)
- Offsite orphan guard            staged live leg fired on its own
- DR tier by default              candidate PROVEN-LIVE upgrade WITHDRAWN (R-39)
Not flipped, as instructed: customer-performs-restore, BYO, DLNA, multi-user.

ROADMAP — collapsed R-1 (appliance half done, Peti half survives), R-21
(physically closed), R-24, R-27 slice 1, R-4. New ranked items:
  P2-HIGH  R-39 PBS DR applied-but-dead   R-30 liveness from the wait channel
           R-31 async offsite + status    R-32 RESET base-dir purge
           R-33 bootstrap quiet-poll
  P2       R-34 backup lifecycle          R-35 config-apply session survival
           R-36 post-RESET offsite prompt R-27c console-passphrase bind
  P3       R-37 post-RESET health card    R-38 installer GRUB slice
Plus a pre-invite checklist (golden 0.145.x rebuild, freemail.hu, C6, R-11).

R-39 is NEW and was not on the brief: the PBS DR descriptor auto-provisions and
the agent converges state=applied, but pvesm reports 401 Unauthorized/inactive
and a direct probe 401s on every endpoint including /version while WG is healthy.
The hub minted a second token secret two minutes after the agent applied the
first and consumed_at is still NULL; the converged state machine will not
re-apply, and the agent's verify loop cannot read the credential to notice it
(non-root read of a file it writes through a root wrapper). Rank is provisional
pending Viktor.

R-3 draft: all four [REFINE] slots filled, self-bind made the default path with
"send the link BEFORE the customer sees the console", the measured wall-clock
table added, and interim operator workarounds for R-31/R-36/R-39. C6 (renumbered
C7) is marked as the single unexecuted step and keeps the doc a DRAFT.

Evidence bundle: 180:~/n100-rehearsal/ (10 files + MANIFEST.md), collected before
the box was unplugged for travel. Secrets read only to run probes; recorded as
lengths and metadata, never values.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
2026-07-18 19:56:17 +02:00
admin bbaa844f8c onboarding runbook 2026-07-18 19:32:00 +02:00
admin 22d04967b4 docs: relocate felhom build root to /mnt/5_hdd/felhom.eu (moved off DooPlex SSD 2026-07-18) 2026-07-18 09:51:29 +02:00
admin c6d7a69e6d runbook updated 2026-07-17 23:01:26 +02:00
admin a344dfdd54 runbook 2026-07-17 22:42:52 +02:00
admin 7f11cfb36c hub v0.65.0 — PBS DR storage visibility (ep0 usage op) + Offsite tab split + dual dashboard gauges (R-5)
Makes PBS DR storage visible like the restic pool box (v0.64.0), differentiated. Scoping
correction: restic = subaccounts on the shared Hetzner Storage Box (Hetzner API); PBS DR =
the felhom-offsite PBS datastore on the ep0 endpoint VM (NO Hetzner API). Option A
(Viktor-ruled): a read-only `usage` op on the felhom-tenantsync ep0 forced command (twin of
fingerprint), polled by a new hub checker on the 15-min throttle. READ-ONLY throughout.

Phase-0 (gate PASSED): on ep0 (PBS 4.2.3), df -B1 --output=size,used,avail <datastore path>
yields bytes (39990112256/7627939840/... ~19%), read-only, existing sudo context, no admin token.

- scripts/felhom-tenantsync.sh -> v1.2.0: read-only `usage` short-circuit (df on the datastore
  path), no customer_id, no admin token, NO mutation. + a bash harness proving zero mutation.
- tenantsync.Client.Usage() + BoxUsage; unknown-op -> typed ErrUsageUnsupported (graceful).
- monitor.PBSDRBoxChecker: OffsiteBoxChecker clone over a usageReader seam; 15-min throttle,
  cached PBSBoxSnapshot, escalation-only pbsdr_box_fill on the "pbsdr-box" scope (operator only,
  no SaveEvent), recovery re-arm. Fill only. THREE states: ok / unavailable (ep0 <=v1.1.0,
  neutral no-alert) / degraded (exec failed, keep last).
- config: Alerting.PBSDRBoxFill{Warn,Crit}Percent (80/90); built with the tenantsync client,
  60s sweep, SetPBSDRBox. Hub deploy INDEPENDENT of the ep0 update (graceful degradation).
- web: /offsite splits into Restic + PBS DR hash tabs (endpoint cards under PBS DR); PBS panel;
  the single dashboard tile becomes two gauges (RESTIC pct.ratio, PBS DR pct / n/a).
- runbook offsite-endpoint.md 10: v1.2.0 update steps (no sudoers/authorized_keys change).

Tests: 10 Go + the harness; 3 red-proofs (usage mutation, escalation-only, unavailable-drives-band)
confirmed red then restored. go build/vet/test + bash -n + hub confirm gate all pass.
2026-07-17 21:13:30 +02:00
admin 7c9647327f docs: hub v0.61.0 Customer RESET — REPORT/CONTEXT + ep0 runbook §10 (tenantsync v1.1.0 deprovision, live-drilled) 2026-07-17 13:15:57 +02:00
admin eb227486d0 hub v0.59.0: Direction-2a agent-plane immediate-sync poke sender + ep0 felhom-poke surface
- internal/poke: pinned-host-key SSH poke sender (wgsync sibling) + fire-and-forget Notifier (PokeHost/PokeAllHosts). Poke refuses non-WG targets pre-dial; contentless via ep0 forced command to the box WG /32:51822.
- wiring: Server.SetPoke; applyPBSDR pokes the host after each descriptor gen-bump; handleSetArtifacts (MinAgent floor) pokes all hosts. main.go env POKE_SSH_KEY_FILE (reuses peersync endpoint/hostkey).
- scripts/felhom-poke.sh (non-root forced command) + offsite-endpoint.md §11; manifests/hub.yaml Secret/agent-poke + POKE_SSH_KEY_FILE (image tag bump follows the build).
2026-07-16 22:48:15 +02:00
admin 375cb082e2 host-install v1.16.0 (FELHOM_ESCROW via canonical sudoers fetch; hub hostInstallVersion synced, no deploy) + RUNBOOK-escrow-ceremony rewrite: controller wizard PRIMARY, CLI operator fallback, F1 threat-model paragraph (R over CF tunnel accepted 2026-07-13), stale-blob/supersede/void semantics 2026-07-13 19:01:46 +02:00
admin 3682303131 scripts: manifest bearer gate lands as manifest_bearer_gate.py (the .gitignore *secret* pattern silently un-tracks any 'secret'-named gate file - the previous commit's gate never made it into git); references updated 2026-07-13 14:59:31 +02:00
admin 80aca3a3a1 hub + manifests: operator bearer out of git (secretKeyRef report-api)
The global bearer (api.report_api_key) leaves manifests/hub.yaml: the
ConfigMap field is an empty placeholder; the Deployment injects
REPORT_API_KEY from out-of-band Secret/report-api (deliberately NOT
optional - a missing Secret fails Ready rather than booting a hub with an
empty bearer). main.go gains the REPORT_API_KEY env override (the
RESEND_API_KEY twin). scripts/manifest_secret_gate.py blocks bearer-shaped
(64-hex) literals across manifests/ (felhom.secret.yaml's pre-existing
umami APP_SECRET is a visible non-fatal KNOWN-BACKLOG line - secrets.md
backlog, out of scope here).

Ordering (load-bearing): create Secret/report-api with the CURRENT value
BEFORE syncing; this manifest change rides the same sync as the 0.53.0
image (older images ignore the env override). The exposed git-history copy
dies only with the supervised ROTATION - procedure + consumer list in
documentation/runbooks/secrets.md ("Operator/global bearer key"); both
publish runbooks' ROTATION notes now point at that disposition. The
controller repo's example-config copy of the literal is scrubbed
(felhom-controller 51c871a).

Red-proof E: reintroducing a 64-hex literal into manifests/hub.yaml ->
gate exit 1 (recorded); restored clean.
2026-07-13 14:57:09 +02:00
admin fda783bdaf docs: DR-tier-by-default batch — decisions 1-5 in CONTEXT, day0 ACL advice reversed, ceremony decision-4 policy block, WG disclosure (tester agreement + GO-LIVE D3), drill F-disposition table, REPORT
Claude-Session: https://claude.ai/code/session_01NptTCFtu7dz2Ru89qHRagN
2026-07-12 20:43:25 +02:00
admin 8ded485a58 docs: customer-claim arc — REPORT/CONTEXT, G9 supersede, tester-agreement claim step, day0 D.3/D.4 gate, drill F-4/F-5 RESOLVED
Claude-Session: https://claude.ai/code/session_01NptTCFtu7dz2Ru89qHRagN
2026-07-12 18:56:46 +02:00
admin 3e949bc513 DRILL-day0-vm 2026-07-12: full report + runbook corrections (F-4/F-6 headline findings)
Audit doc for the Day-0 VM drill: appliance install, floor-at-first-report,
escrow ceremony + auto-confirm FIRST LIVE FIRING (~7.5 min, zero clicks),
offsite backup + restore round-trip, PBS-DR/WG fork (Peti-sequence rehearsal).
Corrects day0-install.md A.2 (git creds optional since v1.11.2, anonymous
fetch is the normal shape) and RUNBOOK-escrow-ceremony.md (identity-only mode
does NOT exist — F-6; age prereq — F-10). REPORT.md overwritten; CONTEXT.md
one-liner added.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NptTCFtu7dz2Ru89qHRagN
2026-07-12 17:16:25 +02:00
admin c322bac21a docs(break-glass): document operator-set FIXED root@pam credential (demo-felhom-01)
§5: add the standing note that a host's root@pam may be pinned to a known
operator-chosen password instead of the random day-0 one — set via chpasswd +
re-vault (PUT /hosts/<id>/recovery-credential, host api_key), guarded by the
break_glass state marker so plain installer re-runs skip it. Caveats: never
--rotate-recovery such a host; a full from-scratch reinstall wipes state.json
and re-randomizes (re-run the set-and-vault). Captures why demo-felhom-01's
root@pam "kept changing" until it was pinned 2026-07-12.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
2026-07-12 11:15:07 +02:00
admin d02a216d0f hub v0.48.0 CHANGELOG + logging-conventions: TRACE level + ring sizing/spill (controller fix-3/fix-6, CAMPAIGN-3 closed)
🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
2026-07-12 10:31:26 +02:00
admin 10a7bf09f7 build runbook 2026-07-11 21:43:09 +02:00
admin 7c038df63c hub 0.46.0 manifest bump + logging-conventions runbook + CLAUDE logging rule
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-11 17:02:54 +02:00
admin bbecf0592e hub v0.45.0: floor-UI separation + effective-floor source + per-box MinAgent conditional floor
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-11 15:33:14 +02:00
admin 4eaabb21d4 docs: publish-train rules — coupled-release gates codified (option-2)
manifest-before-floor; floor field LAST (hub_settings DB row overrides env, acts
immediately — the 0.81/0.113 9-minute skew incident); MinAgent fleet gate
(controller CHANGELOG header convention from ctrl v0.114.0); the controller
capability gate as box-level backstop.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-11 13:55:37 +02:00
admin 3ea2b481da docs: PBS DR slice-1 runbook §10 addendum + REPORT/CONTEXT (hub v0.44.0 live)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-10 21:03:51 +02:00
admin 4703b9ec4e added runbooks 2026-07-10 19:41:04 +02:00
admin fb8e5111cf docs: GL-8 wrap — runbook F3/F6 notes, findings FIXED, REPORT, decision log
day0-install: F3 (no --allow-new-leaf on a populated BYO host) + F6 (:53 must
be free, refusal + remediation). GO-LIVE-PACKAGE + DRILL-GL6 ledger flip
F1/F3/F6 -> FIXED v1.12.0; GL-7 unblocked; F2/F7 stay open. CONTEXT + REPORT.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-08 19:40:57 +02:00
admin f7cc6a720e GL-4: uninstaller gap-closure + operator-key day-0 fold (host-install v1.11.0)
Uninstall: NEW 4b4 removes the self-update artifacts (guarded wrapper,
.prev/.new.* A/B slots, rollback unit, limits drop-in — derived from
configs/felhom-selfupdate-guarded); enrolled drives under /mnt/felhom-drives
unmounted plain-umount-only (busy = warn+guidance, never -l/-f; root bind
guarded); both modes end with a KEPT-vs-WIPED statement (drives/PBS/hub
record/escrow/vaulted recovery credential live on); guest-only mode prints
the vmid's bind-store drives.

Key-pin: OPERATOR_KEY_* constants (empty until the operator ceremony) +
--operator-pubkey-file (argv-validated; comment=key_id required; file
overrides constants) -> authz.signers per the agent SignerKey schema;
no-keys-resolved KEEPS preserved signers; verify reports armed/dormant WARN.

Harness: +13 static cases + GL4-D parity + GL4-INV + PVE-tier GL4 H-U
(full-uninstall dry transcript vs live 9201). 28/28 on felhom-pve;
red-proofs RP-1..3 run->fail->revert. GO-LIVE-PACKAGE.md ABSENT AGAIN ->
status in CONTEXT.md. Live teardown/armed-pin = GL-6.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-08 08:14:07 +02:00
admin 02d63ed070 GL-2: --mode appliance|byo install profile (host-install v1.10.0) + harness
Mandatory install profile (no default): byo = a host the operator does not
own. Break-glass gated OFF at its call site in byo (root@pam never touched),
mandatory --cores/--memory, argv-time refusals (--enable-oob/--rotate-recovery,
non-9.x PVE, missing --acl-storages), host-mutation disclosure + typed-hostname
ack, byo config asserts (lan_resolver/wg_tunnel/oob off; byo flips the
lan_resolver write default to off), pool+ACL verify asserts in BOTH modes (R2),
--preflight-only (no state, PASS/FAIL verdict), resume mode-mismatch refusal,
FELHOM_INSTALL_STATE_DIR harness override.

NEW scripts/hostinstall-mode-harness.sh: static refusal matrix C1-C4 + grep
invariants + PVE tier (C5 + A/B dry transcripts). 16/16 PASS on felhom-pve
(C5 live); red-proofs RP-1..RP-3 run->fail->revert. shellcheck clean at
severity=warning. Docs: day0-install SC.5 byo section + trust model; REUSE row;
CONTEXT + REPORT. Live drill = GL-6 (supervised); STOP honored (no non-dry run).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-07 20:26:32 +02:00
admin 61f4898d30 docs(H1): doc06 §4.5/§4.6 amendment + endpoint runbook §9 + scripts CHANGELOG + REPORT + CONTEXT
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 23:03:33 +02:00
admin 28ac5f575c docs(runbook): break-glass management-plane recovery (TASK G1)
The KEXINIT-reset diagnostic, the three-layer model, PVE-web-console recovery with
the vaulted root@pam credential, the /run/sshd one-line fix, finding a recurring
clobber, and the keep-the-180-token-fresh standing note.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 19:04:34 +02:00
admin 73e5a003f2 docs(offsite): S4.1 — restore-test unattended (tier-aware deadline; VM.Allocate phantom)
Offsite restore-test now runs unattended on the agent-token path (agent v0.68.0):
tier-aware restore-task timeout (pbs→120m, local→10m). The scratch-band
VM.Allocate follow-up was a phantom — the scratch restores INTO /pool/felhom
(ACL already grants it); the earlier 403 was the 10m-timeout consequence. No
ACL/host-install change. doc-06 S4 row + runbook §4b updated.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-04 19:54:48 +02:00
admin 1907dab296 docs(offsite): S4 SHIPPED — per-customer PBS tenancy + v4-pin/watchdog (live 2026-07-04)
- runbook offsite-endpoint.md: new §4a (endpoint per-customer tenancy) + §4b
  (box-side storage + agent wiring). Confirmed minimal ACL: DatastoreBackup on
  /datastore/<ds>/<ns> (NOT /ns/<ns>) to BOTH user felhom@pbs AND token
  felhom@pbs!<ns> (PBS privsep = intersection); cross-tenant 403; ns-scoped
  verify works with DatastoreBackup; DatastoreBackup can't prune (safety).
  Box: box-born encryption key, argv-safe .pw/.enc + storage.cfg, agent PVE
  FelhomAgentStore grant, pbs-secrets WARN fix, local_backup_target retarget.
- doc-06 §3.4 SHIPPED (ns-aware PBS client v0.67.0 rationale); §4.2 v4-pin +
  watchdog SHIPPED v0.66.0 (closes the dual-stack trap); S4 roadmap row done +
  follow-ups (restore-test WAN deadline + scratch VM.Allocate; Tier-1/Tier-2
  target split).

Live: real vzdump of 9201 → ciphertext in ns over the tunnel; verify=ok under
the box's own scoped token; WARN gone; restore decrypts with the box-born key.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-04 17:21:05 +02:00