From 9cfa619ec367aba3a5c6ab5ba1438d92e5aad62d Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 26 Jul 2026 09:14:40 +0200 Subject: [PATCH] hub v0.74.0: allow local_api_endpoint_drift; R-77 docs + R-78/79/80 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The allowlist entry is REQUIRED, not cosmetic: handleEvent 400s an unknown event_type, so controller v0.173.0's new drift alert would be silently inert without it. Shipped with the controller that emits it. Docs: - RUNBOOK-local-api-endpoint-drift.md — how to repair a drift, including the step everyone will want to skip (establish which value is CORRECT from what the agent is actually bound to, rather than assuming bootstrap.json wins) and what success looks like (SILENCE, not a "recovered" line, because a fresh controller's healthy first observation is not logged). Records both 2026-07-26 repairs. - ROADMAP: R-77 shipped; R-78 the local_api authority ruling, with the clobber-a-working-channel risk spelled out in BOTH directions so it is not resolved opportunistically; R-79 the whole-surface English-strings sweep; R-80 expected_backup_missed, flagged as likely outranking R-77 because 7.3 days of stale backup materially exceeds the ~1.5-day channel outage, so the causal link the DIAG hedged on cannot be the whole story. - Capability map: note against the drive-wizard row (every agent-backed capability rides this channel) that a silent drift class is now detected. NO row status flips — detection is not prevention. --- .../architecture/00-capability-map.md | 2 +- documentation/backlog/ROADMAP.md | 4 + .../RUNBOOK-local-api-endpoint-drift.md | 108 ++++++++++++++++++ hub/CHANGELOG.md | 15 +++ hub/internal/api/handler.go | 18 +-- manifests/hub.yaml | 2 +- 6 files changed, 140 insertions(+), 9 deletions(-) create mode 100644 documentation/runbooks/RUNBOOK-local-api-endpoint-drift.md diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 0e4d7cc..d244add 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -76,7 +76,7 @@ | Scenario | Components | Status | Evidence | Gap / roadmap | |---|---|---|---|---| -| Drive wizard: scan/format/mount/enroll, incl. legacy-boot LVM-root hosts | controller, agent v0.87 | **PROVEN-LIVE** | `DISPOSITION-ia-finding2-systemdisks-2026-07-13` (legacy EFI+LVM host, root not offered, byte-identical); enroll/format live in `storage-lifecycle-acceptance-2026-06-15` (E10 re-enroll, data intact); agent fence self-test refuses `/dev/sda` | (Cited `CAMPAIGN-2` T-STG-ENROLL/SEC-FORMAT were auth-hollow CSRF-403.) **Fresh-USB wizard enroll+format through the customer UI PROVEN-LIVE (controller v0.141.0, 2026-07-17):** a 64 GB scratch USB driven through the real `/api/storage/init` endpoints (login+CSRF) → confirm → detached format (~27 s mkfs) → mount → register → mounted+registered at `/mnt/felhom-drives/scratch1`. **F6 (initialize-to-usable) now covered:** the wizard runs the chain as a detached, disconnect-safe, pollable job (3-step progress) with an agent format-status poll for a slow mkfs | +| Drive wizard: scan/format/mount/enroll, incl. legacy-boot LVM-root hosts | controller, agent v0.87 | **PROVEN-LIVE** | `DISPOSITION-ia-finding2-systemdisks-2026-07-13` (legacy EFI+LVM host, root not offered, byte-identical); enroll/format live in `storage-lifecycle-acceptance-2026-06-15` (E10 re-enroll, data intact); agent fence self-test refuses `/dev/sda` | (Cited `CAMPAIGN-2` T-STG-ENROLL/SEC-FORMAT were auth-hollow CSRF-403.) **Fresh-USB wizard enroll+format through the customer UI PROVEN-LIVE (controller v0.141.0, 2026-07-17):** a 64 GB scratch USB driven through the real `/api/storage/init` endpoints (login+CSRF) → confirm → detached format (~27 s mkfs) → mount → register → mounted+registered at `/mnt/felhom-drives/scratch1`. **F6 (initialize-to-usable) now covered:** the wizard runs the chain as a detached, disconnect-safe, pollable job (3-step progress) with an agent format-status poll for a slow mkfs **2026-07-26 — a SILENT failure class on the channel every agent-backed capability depends on (this row, data migration, USB enrollment, guest RAM, quiesce/PBS) is now DETECTED (controller v0.173.0, R-77). No row status flips.** `controller.yaml` and `bootstrap.json` could disagree on `local_api.endpoint` indefinitely with no signal: the R-50 island migration rewrote the latter, the fleet kept dialling the former, and for 17.5 h the only alert was a generic "agent unreachable" that read as an infrastructure blip. Drift now raises its own event type (`local_api_endpoint_drift`) naming both values. It is DETECTION ONLY — the authority ruling is R-78 — so the class is now loud, not prevented. Evidence: `audits/DIAG-agent-channel-2026-07-26.md`. | | Data migration between drives (all / per-app), crash-safe | controller | **PROVEN-LIVE** | `CAMPAIGN-6C` 4P-5 (scope=app round-trip, byte-identical); `storage-lifecycle-acceptance-2026-06-15` (two migrate-all runs via dashboard UI, sha256 byte-identical) | (Cited `CAMPAIGN-2` T-STG-MIGRATE-* were auth-hollow.) "crash-safe" is design-level (copy→verify→remove) — no clean live crash-during-migration PASS | | NAS (NFS/SMB-client) verify-before-commit, uid-1000 probe, categorized Hungarian errors, DSM-validated | controller v0.113–117, agent v0.81/84/85 | **PROVEN-LIVE** | `SPIKE-nas-verify-2026-07-11`, `SPIKE-nas-dsm-2026-07-11`, `CAMPAIGN-3-2026-07-11` (boot/reassert fixes) | | | USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | `storage-lifecycle-acceptance-2026-06-15` E4 (yanked-while-running → agent auto-rebind) + E10 (re-enroll, data intact); `CAMPAIGN-4`/`6A` (3 USB re-establish across device-letter reshuffle) | (Cited `RUNBOOK-usb` could NOT complete a wizard enrollment; `CAMPAIGN-2` legs were auth-hollow.) Fresh-USB **wizard enrollment** specifically still unproven | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index bdae190..34d0e52 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -105,6 +105,10 @@ | R-73 | **SMART history / trending (hub-side)** | M | idea (parked) | Parked follow-up to the v0.169.0 disk-health card + degradation alert (agent v0.94.0 serializes SMART into `/disks`; controller renders „Lemezek állapota" + fires `disk_health_degraded` on a degradation). Today the verdict is point-in-time and the baseline is in-memory (a controller restart re-baselines silently). Hub-side would let the operator SEE a disk's SMART counters trend over time (reallocated/pending sectors climbing, NVMe wear %), catch slow degradation the 6h transition check can miss between reboots, and persist the baseline across restarts. Needs a hub ingest surface for periodic SMART snapshots + a small time-series store + an operator trend view. **No demo hardware exposes real SMART** (virtualized), so this only earns evidence on a real box | | R-74 | **Island control plane on a CLUSTER (Peti's 2 nodes)** — bring R-50's island bridge to a multi-node PVE cluster. | M | idea (Phase C of R-50, parked) | R-50 shipped the island for the ONE-host fleet (demo-hp, demo-felhom). A cluster needs **bridge parity on every node**: either per-node identical `/etc/network/interfaces` `vmbr9` stanzas (simplest, drift-prone) or — preferred at ≥2 nodes — a Proxmox **SDN zone/vnet** defined cluster-wide (one definition, auto-applied per node). The guest island IP is per-guest + node-independent; the **agent-follows-guest** rule holds (each node's agent binds its own `vmbr9` `169.254.253.1`). Migration order per the spike: drill-proven → demo (done) → **Peti (this row)**. Its own supervised runbook, coordinated with Peti (a live customer). Completes the capability-map "site/network change" row for clustered installs. Source: `audits/SPIKE-island-bridge-2026-07-25.md` (cluster-parity finding) + `RUNBOOK-island-migration.md` (single-host procedure to generalise) | | R-75 | **Catalog-derived userdata skeleton + import surfaces** | M | **SHIPPED (controller v0.172.0 + catalog, 2026-07-26)** | `${IMPORT_PATH}` = `/userdata/import` — ONE canonical drop-zone per box on the system drive, with **no per-drive fallback** (unresolvable leaves the var unset so compose fails loudly). *Operator ruling overriding the spike's Fork-1 per-drive recommendation:* each drop-zone app has exactly one ingest bind, so a per-drive `import/` would put a dead lookalike on every other drive — unbacked too, since `import/*` is `class: excluded`. Third `BindRoot` (`RootImport`) threaded through the classifier AND every resolver (`resolveAbs` would otherwise have named a directory on the WRONG DRIVE); catalog-derived skeleton, **sorted** (the naive map-order derivation measured 20 distinct outputs from 20 identical runs — a fleet-wide FileBrowser restart loop via `fbNeedsRecreate`); one authoritative compose parser; FileBrowser „Beolvasás" source; app-page „Hova tegyem a fájlokat?" with `PathEscape` deep links and **class-driven** copy; `data_paths:` annotation (malformed path ⇒ whole-block reject, unknown role ⇒ fails OPEN); system-owned `beolvasas` SMB share refused server-side at handler AND store. **Live-proven on demo-hp:** consume bind resolves to the system drive, a file dropped through FileBrowser's mount was ingested and deleted in ~30 s, the share auto-appeared on enable and survived a direct delete POST, and the before/after directory diff on BOTH boxes shows **zero removals** (demo-hp gained `media/podcasts`, the one genuinely derived entry). Two defects found and fixed during the live legs: `EnsureImportRoot` left the parent `userdata` at 755, and the carry-list initially re-created the per-drive drop-zone forever. Capability-map row *File access via browser* **deliberately NOT upgraded** — nothing drove the FileBrowser HTTP UI. Docs: `controller/import-and-data-paths.md`, spike `audits/SPIKE-catalog-data-paths-2026-07-26.md` | +| R-77 | **Endpoint-drift detection, samba protected-set gate, channel log honesty** | S | **SHIPPED (controller v0.173.0 + hub v0.74.0, 2026-07-26)** | Fixes the two defects in `audits/DIAG-agent-channel-2026-07-26.md`. `bootstrap.DetectEndpointDrift` names a `controller.yaml` vs `bootstrap.json` `local_api.endpoint` divergence with its own event type `local_api_endpoint_drift` and its own banner — and **writes nothing** (see R-78). `EffectiveProtected` now gates samba on `Enabled && UserSet`, mirroring both of `reconcileSambaAt`'s early returns, plus the doc-comment correction. Channel log distinguishes born-down (`unseeded->down`) from a real transition, logging-only. **Part 0 repaired both production boxes** (endpoint → `169.254.253.1:8443`; fingerprint+token agreed). Tests 951→959; red-proofs A/E/F. | +| R-78 | **`local_api` authority ruling — auto-reconcile vs detect-only** | M | idea (deferred OUT of R-77 on purpose) | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | +| R-79 | **`report.Issues` / `report.Warnings` are English on customer-facing surfaces** | M | idea | **Whole-surface, not a one-off** (DIAG §6): every producer is English — `"SSD/HDD disk usage critical"`, `"Docker: %v"`, `"Protected container not running: %s"`, and all six `Warnings` strings. They render on the customer's Hungarian dashboard, and the `health_critical` path has reached the **customer** email channel three times historically. Deliberately NOT bundled into R-77: a copy sweep across every producer would have buried two safety fixes in string churn, and the seam is not obvious — translate at the producer, or at the render/notification boundary where operator-English and customer-Hungarian already diverge? Pick the seam in a spike; the strings are mechanical after. | +| R-80 | **`expected_backup_missed` firing nightly on all three customers — one reaching a CUSTOMER channel** | M | idea — **likely outranks R-77** | 2026-07-26 03:00 fired for demo-felhom, demo-hp and drill-r50; the demo-felhom one went to the **customer** channel reporting `newest backup is 176h0m0s old` — **7.3 days**. **The causal link the DIAG hedged on is unlikely to be the whole story: 7.3 days materially exceeds the ~1.5-day channel outage (2026-07-25 12:44 → 2026-07-26 07:00), so backups were already stale for ~6 days BEFORE the channel broke.** The channel outage may compound it but cannot explain it. This is a customer-visible claim about their data protection and needs its own diagnostic: is it a real backup gap, a stale-report artefact, or a threshold/derivation defect? Do not assume it resolves with R-77's repair — verify against PBS snapshot reality. | | R-76 | **FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable** | S | idea (surfaced by the R-75 spike, 2026-07-26) | Two related findings from `audits/SPIKE-catalog-data-paths-2026-07-26.md` P3/P5, both **pre-existing** and deliberately left alone by that spike. **(a)** FileBrowser Quantum 1.3.3 creates files `0644` and folders `0755` and does **not** propagate the setgid bit — even though the entrypoint wrapper's `umask 002` really is in effect (`/proc/1/status` `Umask: 0002`). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's *group* half holds and only its *mode* half is lost. The consequence is proven with a control: inside a UI-created `0755` folder a gid-1000 process's file landed group **1000**, while the identical write into the 2775 parent landed group **100**. So **any folder a customer creates through FileBrowser breaks the shared-group chain one level down.** Latent today — every userdata-touching catalog app that declares an identity declares uid/gid **1000**, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at `infra/infra.go:156` is right that the image ignores `-e UMASK` but does not say the wrapper fails to achieve the intended mode either. **(b)** `import/calibre` is live on demo-felhom at `755 1000:1000` where every sibling is `2775 root:1000` — with `media/books` on the same box, same app, same deploy, at `2775` as the control, and no parser asymmetry (checked: `ParseComposeUserdataMounts` picks up both calibre-web binds). Consistent with the consuming app rewriting the mode of its own ingest dir after the deploy belt sets it; **not confirmed causally**. Together they mean **a drop-zone directory's mode is not stable against either the customer or the consuming app** — verify before building anything that assumes `import/*` stays 2775. Flips no capability-map row today (latent); would become customer-visible the moment a non-1000 content app enters the catalog | | R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → universal secret-free / operator-bind (option C) | XL | **SHIPPED + PHYSICALLY CLOSED (slices A+B+C; rehearsal executed 2026-07-18)** | **PHYSICAL CLOSURE 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the generic pairing ISO v1.20.0 (`--loader mkimage`, SB off) **booted the very AMI board that F1 blocked**, installed unattended, and the box self-registered as an unclaimed appliance the same second it first booted (16:17:14) → self-bind → credential → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware and the rehearsal dependency on this item is discharged.** Two residual notes stay open, neither blocking: PXE/network-boot is still unbuilt (the third F1 option, for boards where even USB-mkimage fails), and mkimage remains unsigned → **SB must be OFF**. The installer's GRUB menu still offers interactive installers → new item **R-38**. — Prior: **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17):** the GENERIC secret-free universal ISO (`build-felhom-iso.sh --pairing`). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (`/api/v1/appliance/register` + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. **One unit, two modes** (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest `secret-bearing: no`). The **bind is operator-password-gated** → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — **which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0**. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). **Customer-facing self-bind page = R-27 (future).** Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* | **SECOND-HARDWARE PROOF 2026-07-21 (slice C, demo-hp): the pairing flow is no longer a one-board result.** The universal secret-free ISO was booted on a completely different machine — an **HP t740 (Ryzen V1756B, AMI M42 firmware)** versus the N100 it was proven on — and the whole chain ran on **virgin hardware in one pass**: armed install → self-registration as an unclaimed appliance → operator bind → day-0, ending with a running customer guest 9201 and agent 0.92.1 checking in as host `demo-hp-bb76ea`. **Two things generalise from the second board specifically:** the **shim** loader booted with **Secure Boot ENABLED** (`mokutil --sb-state` → `SecureBoot enabled`), confirming the mkimage/SB-off dance is an N100-firmware workaround and NOT a Felhom requirement; and the exact-serial disk filter selected the SanDisk system SSD while leaving the box's **1TB NVMe untouched and unenrolled** (its prior NTFS partition is still intact, unmounted, in no LVM/ZFS) — the destructive path stayed inside its filter on hardware it had never seen. **Not clean, and the failures are filed:** the install got no DHCP on the 4-port NIC and baked a static fallback rather than aborting (**R-59**), which cost a cable move and a hand-repair; and the console was unreachable because the baked root password is unknowable (**R-61**) | R-29 | **The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases.** `controller/scripts/docker_run_volume_path_gate.py` has failed continuously since **2026-07-14 (v0.129.0)** and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. **(a) The finding itself is benign and the fix is 3 lines.** The flagged call is `internal/appexport/estimate.go:179` `docker run --rm -v :/vol:ro alpine du` — a **NAMED-VOLUME** mount, i.e. daemon-side with no host path, which is the *safe* shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (`export.go` `volName+":/vol"`, `backup.go` `volName+":/vol:ro"`, `restore.go` `volName+":/vol"`). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, **not** a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. **(b) The systemic half is the real item:** the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note *'Windows green gate silently red (read-only fsync)'*. Decide where they run (pre-push hook, `build.sh` step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, `felhom-controller` REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller `template_id_gate` / `emoji_gate` / `native_confirm_gate` / `offbox_rename_gate` / `mojibake_gate` / `app_row_dedup_gate` / `docker_run_volume_path_gate`, hub `hub_confirm_gate`, manifests `manifest_bearer_gate`, website `site_gates`. **Do not bundle (a) into an unrelated feature commit** — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff. **2026-07-18 rehearsal note:** the run's finding list independently re-raised "assign the pre-existing `docker_run_volume_path_gate` failure its ID so red stops normalizing" — **that is this item; no second ID was minted.** | diff --git a/documentation/runbooks/RUNBOOK-local-api-endpoint-drift.md b/documentation/runbooks/RUNBOOK-local-api-endpoint-drift.md new file mode 100644 index 0000000..18da80d --- /dev/null +++ b/documentation/runbooks/RUNBOOK-local-api-endpoint-drift.md @@ -0,0 +1,108 @@ +# RUNBOOK — repairing a `local_api` endpoint drift + +**When to run:** the controller logs +`[ERROR] bootstrap: local_api endpoint DRIFT — …` at startup, or the operator receives a +`local_api_endpoint_drift` event, or the „A tárolókezelő ügynök címe elavult a beállításokban." +banner appears. Detection shipped in controller **v0.173.0** (R-77). + +**What drift means:** `controller.yaml` and `bootstrap.json` disagree on `local_api.endpoint`. The +controller dials **`controller.yaml`**. The agent listens wherever it was last configured — which, +after the R-50 island migration, is `169.254.253.1:8443`. + +**Why the controller does not fix this itself:** auto-reconcile would clobber a *correct* +`controller.yaml` from a *stale* `bootstrap.json` — a half-completed re-provision or a hand-repaired +guest would lose a working channel on its next restart, fleet-wide and silently. Which file is +authoritative is **R-78**, unresolved. Until it is ruled, repair is a deliberate operator action. + +> **Operator-present.** This edits a live guest's config and restarts its controller. The channel +> being down also degrades the drive gate, guest-reboot recovery and the quiesce/backup loop, so it +> is worth doing promptly — but not blind. + +--- + +## 1. Establish which value is correct — do NOT assume `bootstrap.json` + +`bootstrap.json` is right *when the drift came from a migration or re-provision* — the common case. +It is **wrong** if someone hand-repaired `controller.yaml` after a failed provision. Decide with +evidence, not by default: + +```bash +# What is the agent ACTUALLY listening on? This is the ground truth. +ssh "ss -lntH 'sport = :8443'" # post-island: 169.254.253.1:8443 +``` + +**The correct endpoint is whatever the agent is bound to.** Both files are just claims about it. + +## 2. Compare all three fields, not only the endpoint + +`mergeLocalAPI` replaces the **whole** block, so a migration may have moved more than the address. + +```bash +pct exec -- python3 - <<'PY' +import json, re, hashlib +CY="/var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml" +BJ="/etc/felhom-bootstrap/bootstrap.json" +def yget(path): + blk={}; inb=False + for line in open(path): + if re.match(r'^local_api:', line): inb=True; continue + if inb and re.match(r'^\S', line): break + if inb: + m=re.match(r'\s+(\w+):\s*(.*)\s*$', line) + if m: blk[m.group(1)]=m.group(2).strip().strip('"').strip("'") + return blk +cy=yget(CY); bj=(json.load(open(BJ)).get("local_api") or {}) +print("endpoint cy/bj :", repr(cy.get("endpoint")), repr(bj.get("endpoint"))) +print("fingerprint AGREE :", cy.get("fingerprint")==bj.get("fingerprint")) +print("token AGREE :", cy.get("token")==bj.get("token")) +PY +``` + +**Never print the fingerprint or token values** — booleans only, and never into a committed file. + +> **STOP if `fingerprint` or `token` disagree.** A moved pin or a rotated token is a *different* +> failure: correcting only the address would then fail closed on the pin (`agent_channel_pin_mismatch`) +> or 401 (`agent_channel_unauthorized`). Diagnose that separately before touching anything. + +## 3. Back up, edit the endpoint only, restart + +```bash +CY=/var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml +pct exec -- cp -a $CY $CY.pre-fix.bak +# edit ONLY local_api.endpoint to the value from step 1 +pct exec -- systemctl restart felhom-controller-bootstrap.service +``` + +## 4. Verify — and know what success looks like + +```bash +pct exec -- docker logs felhom-controller 2>&1 | grep -i "\[channel\]" +``` + +**Expect SILENCE, not a "recovered" line.** On a freshly restarted controller the first probe +succeeds and `Check` returns early (*healthy first-obs → no notify*), so nothing is logged. A +`recovered` line only appears when a *running* controller transitions down→up. + +Positive confirmation is therefore: + +- **zero** `[channel]` lines after ~90 s (a broken channel logs `transient down` within 60 s and + `DOWN` within 120 s); +- the dashboard banner cleared; +- **zero** new `agent_channel_*` events hub-side; +- no `local_api endpoint DRIFT` line at startup (the two files now agree — if they still disagree + because you corrected `controller.yaml` rather than `bootstrap.json`, the drift alert will keep + firing and that is **correct**: it is telling you the two sources still disagree. Align the other + file too, or accept the alert until R-78). + +--- + +## Executed instances + +| date | boxes | before → after | fingerprint/token | outcome | +|---|---|---|---|---| +| 2026-07-26 | demo-felhom 9201 | `192.168.0.162:8443` → `169.254.253.1:8443` | both AGREE | channel healthy; 0 `[channel]` lines in 90 s; banner cleared | +| 2026-07-26 | demo-hp 9201 | `192.168.0.87:8443` → `169.254.253.1:8443` | both AGREE | channel healthy; 0 `[channel]` lines in 90 s | + +Root cause of both: the R-50 island migration (2026-07-25 12:42–12:46) rewrote `bootstrap.json` and +restarted the controllers, which re-read the untouched `controller.yaml`. Full analysis: +`audits/DIAG-agent-channel-2026-07-26.md`. diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 6ef92d8..87fcc5b 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,5 +1,20 @@ # Felhom Hub — Changelog +## v0.74.0 — allow `local_api_endpoint_drift` (controller v0.173.0 / R-77) (2026-07-26) + +One line in `allowedEventTypes`. It is **not optional**: `handleEvent` 400s an unknown `event_type` +("Invalid event_type"), so the controller's new drift alert would have been **silently inert** without +it — the exact seam-wiring failure class this project has hit four times. Shipped with the controller +that emits it, not after. + +Operator-only, `error` severity (drift never self-heals), and deliberately **not** an `agent_channel_*` +type: during the 2026-07-25 island-migration outage the generic "agent unreachable" event was the only +signal for 17.5 h and it hid a specific, fixable config fault. Naming the cause separately from the +symptom is the whole point. No customer notification toggle, matching the other `agent_channel_*` and +`host_*` operator events. + +Source: `documentation/audits/DIAG-agent-channel-2026-07-26.md`. + ## v0.73.2 — sync `hostInstallVersion` → 1.19.0 (R-50 island host-install) (2026-07-25) `hostInstallVersion` (the script version the operator customer page's install-command generator diff --git a/hub/internal/api/handler.go b/hub/internal/api/handler.go index 33ba7ff..95f14b0 100644 --- a/hub/internal/api/handler.go +++ b/hub/internal/api/handler.go @@ -1563,15 +1563,15 @@ var allowedEventTypes = map[string]bool{ // controller v0.134.1 — enlarged offsite push refused by the quota gate (warning; the controller's // dynamic Hungarian message is customer-grade — deliberately NO customerMessages entry, which would // discard the numbers (templates.go:129 priority)). - "offbox_enlarge_blocked": true, + "offbox_enlarge_blocked": true, // controller v0.142.0 — offsite-repo continuity: the remote repo is orphaned (reinstall shape) / // was reset (move-aside + re-init). Customer-grade messages below. - "offbox_repo_orphaned": true, - "offbox_repo_reset": true, - "storage_disconnected": true, - "storage_reconnected": true, - "disk_warning": true, - "disk_critical": true, + "offbox_repo_orphaned": true, + "offbox_repo_reset": true, + "storage_disconnected": true, + "storage_reconnected": true, + "disk_warning": true, + "disk_critical": true, // controller v0.169.0 — per-disk SMART degradation (Rendben→Figyelmeztetés/Hiba). The controller // sends a dynamic Hungarian message (disk label + the triggering attribute names), so — like // offbox_enlarge_blocked — there is deliberately NO customerMessages entry (which would discard the @@ -1594,6 +1594,10 @@ var allowedEventTypes = map[string]bool{ "agent_channel_construction_error": true, "agent_channel_unknown": true, "agent_channel_recovered": true, + // controller v0.173.0 (R-77): controller.yaml and bootstrap.json disagree on local_api.endpoint. + // Operator-only, deliberately NOT an agent_channel_* type — during the 2026-07-25 island-migration + // outage the generic "unreachable" alert was the only signal and it hid a specific config fault. + "local_api_endpoint_drift": true, // Hub-generated events "node_stale": true, "node_down": true, diff --git a/manifests/hub.yaml b/manifests/hub.yaml index 740643f..db0561c 100644 --- a/manifests/hub.yaml +++ b/manifests/hub.yaml @@ -125,7 +125,7 @@ spec: spec: containers: - name: hub - image: gitea.dooplex.hu/admin/felhom-hub:0.73.1 + image: gitea.dooplex.hu/admin/felhom-hub:0.74.0 ports: - containerPort: 8080 name: http