diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 0e4d7cc..d244add 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -76,7 +76,7 @@ | Scenario | Components | Status | Evidence | Gap / roadmap | |---|---|---|---|---| -| Drive wizard: scan/format/mount/enroll, incl. legacy-boot LVM-root hosts | controller, agent v0.87 | **PROVEN-LIVE** | `DISPOSITION-ia-finding2-systemdisks-2026-07-13` (legacy EFI+LVM host, root not offered, byte-identical); enroll/format live in `storage-lifecycle-acceptance-2026-06-15` (E10 re-enroll, data intact); agent fence self-test refuses `/dev/sda` | (Cited `CAMPAIGN-2` T-STG-ENROLL/SEC-FORMAT were auth-hollow CSRF-403.) **Fresh-USB wizard enroll+format through the customer UI PROVEN-LIVE (controller v0.141.0, 2026-07-17):** a 64 GB scratch USB driven through the real `/api/storage/init` endpoints (login+CSRF) → confirm → detached format (~27 s mkfs) → mount → register → mounted+registered at `/mnt/felhom-drives/scratch1`. **F6 (initialize-to-usable) now covered:** the wizard runs the chain as a detached, disconnect-safe, pollable job (3-step progress) with an agent format-status poll for a slow mkfs | +| Drive wizard: scan/format/mount/enroll, incl. legacy-boot LVM-root hosts | controller, agent v0.87 | **PROVEN-LIVE** | `DISPOSITION-ia-finding2-systemdisks-2026-07-13` (legacy EFI+LVM host, root not offered, byte-identical); enroll/format live in `storage-lifecycle-acceptance-2026-06-15` (E10 re-enroll, data intact); agent fence self-test refuses `/dev/sda` | (Cited `CAMPAIGN-2` T-STG-ENROLL/SEC-FORMAT were auth-hollow CSRF-403.) **Fresh-USB wizard enroll+format through the customer UI PROVEN-LIVE (controller v0.141.0, 2026-07-17):** a 64 GB scratch USB driven through the real `/api/storage/init` endpoints (login+CSRF) → confirm → detached format (~27 s mkfs) → mount → register → mounted+registered at `/mnt/felhom-drives/scratch1`. **F6 (initialize-to-usable) now covered:** the wizard runs the chain as a detached, disconnect-safe, pollable job (3-step progress) with an agent format-status poll for a slow mkfs **2026-07-26 — a SILENT failure class on the channel every agent-backed capability depends on (this row, data migration, USB enrollment, guest RAM, quiesce/PBS) is now DETECTED (controller v0.173.0, R-77). No row status flips.** `controller.yaml` and `bootstrap.json` could disagree on `local_api.endpoint` indefinitely with no signal: the R-50 island migration rewrote the latter, the fleet kept dialling the former, and for 17.5 h the only alert was a generic "agent unreachable" that read as an infrastructure blip. Drift now raises its own event type (`local_api_endpoint_drift`) naming both values. It is DETECTION ONLY — the authority ruling is R-78 — so the class is now loud, not prevented. Evidence: `audits/DIAG-agent-channel-2026-07-26.md`. | | Data migration between drives (all / per-app), crash-safe | controller | **PROVEN-LIVE** | `CAMPAIGN-6C` 4P-5 (scope=app round-trip, byte-identical); `storage-lifecycle-acceptance-2026-06-15` (two migrate-all runs via dashboard UI, sha256 byte-identical) | (Cited `CAMPAIGN-2` T-STG-MIGRATE-* were auth-hollow.) "crash-safe" is design-level (copy→verify→remove) — no clean live crash-during-migration PASS | | NAS (NFS/SMB-client) verify-before-commit, uid-1000 probe, categorized Hungarian errors, DSM-validated | controller v0.113–117, agent v0.81/84/85 | **PROVEN-LIVE** | `SPIKE-nas-verify-2026-07-11`, `SPIKE-nas-dsm-2026-07-11`, `CAMPAIGN-3-2026-07-11` (boot/reassert fixes) | | | USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | `storage-lifecycle-acceptance-2026-06-15` E4 (yanked-while-running → agent auto-rebind) + E10 (re-enroll, data intact); `CAMPAIGN-4`/`6A` (3 USB re-establish across device-letter reshuffle) | (Cited `RUNBOOK-usb` could NOT complete a wizard enrollment; `CAMPAIGN-2` legs were auth-hollow.) Fresh-USB **wizard enrollment** specifically still unproven | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index bdae190..34d0e52 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -105,6 +105,10 @@ | R-73 | **SMART history / trending (hub-side)** | M | idea (parked) | Parked follow-up to the v0.169.0 disk-health card + degradation alert (agent v0.94.0 serializes SMART into `/disks`; controller renders „Lemezek állapota" + fires `disk_health_degraded` on a degradation). Today the verdict is point-in-time and the baseline is in-memory (a controller restart re-baselines silently). Hub-side would let the operator SEE a disk's SMART counters trend over time (reallocated/pending sectors climbing, NVMe wear %), catch slow degradation the 6h transition check can miss between reboots, and persist the baseline across restarts. Needs a hub ingest surface for periodic SMART snapshots + a small time-series store + an operator trend view. **No demo hardware exposes real SMART** (virtualized), so this only earns evidence on a real box | | R-74 | **Island control plane on a CLUSTER (Peti's 2 nodes)** — bring R-50's island bridge to a multi-node PVE cluster. | M | idea (Phase C of R-50, parked) | R-50 shipped the island for the ONE-host fleet (demo-hp, demo-felhom). A cluster needs **bridge parity on every node**: either per-node identical `/etc/network/interfaces` `vmbr9` stanzas (simplest, drift-prone) or — preferred at ≥2 nodes — a Proxmox **SDN zone/vnet** defined cluster-wide (one definition, auto-applied per node). The guest island IP is per-guest + node-independent; the **agent-follows-guest** rule holds (each node's agent binds its own `vmbr9` `169.254.253.1`). Migration order per the spike: drill-proven → demo (done) → **Peti (this row)**. Its own supervised runbook, coordinated with Peti (a live customer). Completes the capability-map "site/network change" row for clustered installs. Source: `audits/SPIKE-island-bridge-2026-07-25.md` (cluster-parity finding) + `RUNBOOK-island-migration.md` (single-host procedure to generalise) | | R-75 | **Catalog-derived userdata skeleton + import surfaces** | M | **SHIPPED (controller v0.172.0 + catalog, 2026-07-26)** | `${IMPORT_PATH}` = `/userdata/import` — ONE canonical drop-zone per box on the system drive, with **no per-drive fallback** (unresolvable leaves the var unset so compose fails loudly). *Operator ruling overriding the spike's Fork-1 per-drive recommendation:* each drop-zone app has exactly one ingest bind, so a per-drive `import/` would put a dead lookalike on every other drive — unbacked too, since `import/*` is `class: excluded`. Third `BindRoot` (`RootImport`) threaded through the classifier AND every resolver (`resolveAbs` would otherwise have named a directory on the WRONG DRIVE); catalog-derived skeleton, **sorted** (the naive map-order derivation measured 20 distinct outputs from 20 identical runs — a fleet-wide FileBrowser restart loop via `fbNeedsRecreate`); one authoritative compose parser; FileBrowser „Beolvasás" source; app-page „Hova tegyem a fájlokat?" with `PathEscape` deep links and **class-driven** copy; `data_paths:` annotation (malformed path ⇒ whole-block reject, unknown role ⇒ fails OPEN); system-owned `beolvasas` SMB share refused server-side at handler AND store. **Live-proven on demo-hp:** consume bind resolves to the system drive, a file dropped through FileBrowser's mount was ingested and deleted in ~30 s, the share auto-appeared on enable and survived a direct delete POST, and the before/after directory diff on BOTH boxes shows **zero removals** (demo-hp gained `media/podcasts`, the one genuinely derived entry). Two defects found and fixed during the live legs: `EnsureImportRoot` left the parent `userdata` at 755, and the carry-list initially re-created the per-drive drop-zone forever. Capability-map row *File access via browser* **deliberately NOT upgraded** — nothing drove the FileBrowser HTTP UI. Docs: `controller/import-and-data-paths.md`, spike `audits/SPIKE-catalog-data-paths-2026-07-26.md` | +| R-77 | **Endpoint-drift detection, samba protected-set gate, channel log honesty** | S | **SHIPPED (controller v0.173.0 + hub v0.74.0, 2026-07-26)** | Fixes the two defects in `audits/DIAG-agent-channel-2026-07-26.md`. `bootstrap.DetectEndpointDrift` names a `controller.yaml` vs `bootstrap.json` `local_api.endpoint` divergence with its own event type `local_api_endpoint_drift` and its own banner — and **writes nothing** (see R-78). `EffectiveProtected` now gates samba on `Enabled && UserSet`, mirroring both of `reconcileSambaAt`'s early returns, plus the doc-comment correction. Channel log distinguishes born-down (`unseeded->down`) from a real transition, logging-only. **Part 0 repaired both production boxes** (endpoint → `169.254.253.1:8443`; fingerprint+token agreed). Tests 951→959; red-proofs A/E/F. | +| R-78 | **`local_api` authority ruling — auto-reconcile vs detect-only** | M | idea (deferred OUT of R-77 on purpose) | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | +| R-79 | **`report.Issues` / `report.Warnings` are English on customer-facing surfaces** | M | idea | **Whole-surface, not a one-off** (DIAG §6): every producer is English — `"SSD/HDD disk usage critical"`, `"Docker: %v"`, `"Protected container not running: %s"`, and all six `Warnings` strings. They render on the customer's Hungarian dashboard, and the `health_critical` path has reached the **customer** email channel three times historically. Deliberately NOT bundled into R-77: a copy sweep across every producer would have buried two safety fixes in string churn, and the seam is not obvious — translate at the producer, or at the render/notification boundary where operator-English and customer-Hungarian already diverge? Pick the seam in a spike; the strings are mechanical after. | +| R-80 | **`expected_backup_missed` firing nightly on all three customers — one reaching a CUSTOMER channel** | M | idea — **likely outranks R-77** | 2026-07-26 03:00 fired for demo-felhom, demo-hp and drill-r50; the demo-felhom one went to the **customer** channel reporting `newest backup is 176h0m0s old` — **7.3 days**. **The causal link the DIAG hedged on is unlikely to be the whole story: 7.3 days materially exceeds the ~1.5-day channel outage (2026-07-25 12:44 → 2026-07-26 07:00), so backups were already stale for ~6 days BEFORE the channel broke.** The channel outage may compound it but cannot explain it. This is a customer-visible claim about their data protection and needs its own diagnostic: is it a real backup gap, a stale-report artefact, or a threshold/derivation defect? Do not assume it resolves with R-77's repair — verify against PBS snapshot reality. | | R-76 | **FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable** | S | idea (surfaced by the R-75 spike, 2026-07-26) | Two related findings from `audits/SPIKE-catalog-data-paths-2026-07-26.md` P3/P5, both **pre-existing** and deliberately left alone by that spike. **(a)** FileBrowser Quantum 1.3.3 creates files `0644` and folders `0755` and does **not** propagate the setgid bit — even though the entrypoint wrapper's `umask 002` really is in effect (`/proc/1/status` `Umask: 0002`). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's *group* half holds and only its *mode* half is lost. The consequence is proven with a control: inside a UI-created `0755` folder a gid-1000 process's file landed group **1000**, while the identical write into the 2775 parent landed group **100**. So **any folder a customer creates through FileBrowser breaks the shared-group chain one level down.** Latent today — every userdata-touching catalog app that declares an identity declares uid/gid **1000**, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at `infra/infra.go:156` is right that the image ignores `-e UMASK` but does not say the wrapper fails to achieve the intended mode either. **(b)** `import/calibre` is live on demo-felhom at `755 1000:1000` where every sibling is `2775 root:1000` — with `media/books` on the same box, same app, same deploy, at `2775` as the control, and no parser asymmetry (checked: `ParseComposeUserdataMounts` picks up both calibre-web binds). Consistent with the consuming app rewriting the mode of its own ingest dir after the deploy belt sets it; **not confirmed causally**. Together they mean **a drop-zone directory's mode is not stable against either the customer or the consuming app** — verify before building anything that assumes `import/*` stays 2775. Flips no capability-map row today (latent); would become customer-visible the moment a non-1000 content app enters the catalog | | R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → universal secret-free / operator-bind (option C) | XL | **SHIPPED + PHYSICALLY CLOSED (slices A+B+C; rehearsal executed 2026-07-18)** | **PHYSICAL CLOSURE 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the generic pairing ISO v1.20.0 (`--loader mkimage`, SB off) **booted the very AMI board that F1 blocked**, installed unattended, and the box self-registered as an unclaimed appliance the same second it first booted (16:17:14) → self-bind → credential → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware and the rehearsal dependency on this item is discharged.** Two residual notes stay open, neither blocking: PXE/network-boot is still unbuilt (the third F1 option, for boards where even USB-mkimage fails), and mkimage remains unsigned → **SB must be OFF**. The installer's GRUB menu still offers interactive installers → new item **R-38**. — Prior: **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17):** the GENERIC secret-free universal ISO (`build-felhom-iso.sh --pairing`). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (`/api/v1/appliance/register` + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. **One unit, two modes** (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest `secret-bearing: no`). The **bind is operator-password-gated** → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — **which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0**. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). **Customer-facing self-bind page = R-27 (future).** Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* | **SECOND-HARDWARE PROOF 2026-07-21 (slice C, demo-hp): the pairing flow is no longer a one-board result.** The universal secret-free ISO was booted on a completely different machine — an **HP t740 (Ryzen V1756B, AMI M42 firmware)** versus the N100 it was proven on — and the whole chain ran on **virgin hardware in one pass**: armed install → self-registration as an unclaimed appliance → operator bind → day-0, ending with a running customer guest 9201 and agent 0.92.1 checking in as host `demo-hp-bb76ea`. **Two things generalise from the second board specifically:** the **shim** loader booted with **Secure Boot ENABLED** (`mokutil --sb-state` → `SecureBoot enabled`), confirming the mkimage/SB-off dance is an N100-firmware workaround and NOT a Felhom requirement; and the exact-serial disk filter selected the SanDisk system SSD while leaving the box's **1TB NVMe untouched and unenrolled** (its prior NTFS partition is still intact, unmounted, in no LVM/ZFS) — the destructive path stayed inside its filter on hardware it had never seen. **Not clean, and the failures are filed:** the install got no DHCP on the 4-port NIC and baked a static fallback rather than aborting (**R-59**), which cost a cable move and a hand-repair; and the console was unreachable because the baked root password is unknowable (**R-61**) | R-29 | **The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases.** `controller/scripts/docker_run_volume_path_gate.py` has failed continuously since **2026-07-14 (v0.129.0)** and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. **(a) The finding itself is benign and the fix is 3 lines.** The flagged call is `internal/appexport/estimate.go:179` `docker run --rm -v :/vol:ro alpine du` — a **NAMED-VOLUME** mount, i.e. daemon-side with no host path, which is the *safe* shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (`export.go` `volName+":/vol"`, `backup.go` `volName+":/vol:ro"`, `restore.go` `volName+":/vol"`). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, **not** a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. **(b) The systemic half is the real item:** the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note *'Windows green gate silently red (read-only fsync)'*. Decide where they run (pre-push hook, `build.sh` step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, `felhom-controller` REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller `template_id_gate` / `emoji_gate` / `native_confirm_gate` / `offbox_rename_gate` / `mojibake_gate` / `app_row_dedup_gate` / `docker_run_volume_path_gate`, hub `hub_confirm_gate`, manifests `manifest_bearer_gate`, website `site_gates`. **Do not bundle (a) into an unrelated feature commit** — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff. **2026-07-18 rehearsal note:** the run's finding list independently re-raised "assign the pre-existing `docker_run_volume_path_gate` failure its ID so red stops normalizing" — **that is this item; no second ID was minted.** | diff --git a/documentation/runbooks/RUNBOOK-local-api-endpoint-drift.md b/documentation/runbooks/RUNBOOK-local-api-endpoint-drift.md new file mode 100644 index 0000000..18da80d --- /dev/null +++ b/documentation/runbooks/RUNBOOK-local-api-endpoint-drift.md @@ -0,0 +1,108 @@ +# RUNBOOK — repairing a `local_api` endpoint drift + +**When to run:** the controller logs +`[ERROR] bootstrap: local_api endpoint DRIFT — …` at startup, or the operator receives a +`local_api_endpoint_drift` event, or the „A tárolókezelő ügynök címe elavult a beállításokban." +banner appears. Detection shipped in controller **v0.173.0** (R-77). + +**What drift means:** `controller.yaml` and `bootstrap.json` disagree on `local_api.endpoint`. The +controller dials **`controller.yaml`**. The agent listens wherever it was last configured — which, +after the R-50 island migration, is `169.254.253.1:8443`. + +**Why the controller does not fix this itself:** auto-reconcile would clobber a *correct* +`controller.yaml` from a *stale* `bootstrap.json` — a half-completed re-provision or a hand-repaired +guest would lose a working channel on its next restart, fleet-wide and silently. Which file is +authoritative is **R-78**, unresolved. Until it is ruled, repair is a deliberate operator action. + +> **Operator-present.** This edits a live guest's config and restarts its controller. The channel +> being down also degrades the drive gate, guest-reboot recovery and the quiesce/backup loop, so it +> is worth doing promptly — but not blind. + +--- + +## 1. Establish which value is correct — do NOT assume `bootstrap.json` + +`bootstrap.json` is right *when the drift came from a migration or re-provision* — the common case. +It is **wrong** if someone hand-repaired `controller.yaml` after a failed provision. Decide with +evidence, not by default: + +```bash +# What is the agent ACTUALLY listening on? This is the ground truth. +ssh "ss -lntH 'sport = :8443'" # post-island: 169.254.253.1:8443 +``` + +**The correct endpoint is whatever the agent is bound to.** Both files are just claims about it. + +## 2. Compare all three fields, not only the endpoint + +`mergeLocalAPI` replaces the **whole** block, so a migration may have moved more than the address. + +```bash +pct exec -- python3 - <<'PY' +import json, re, hashlib +CY="/var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml" +BJ="/etc/felhom-bootstrap/bootstrap.json" +def yget(path): + blk={}; inb=False + for line in open(path): + if re.match(r'^local_api:', line): inb=True; continue + if inb and re.match(r'^\S', line): break + if inb: + m=re.match(r'\s+(\w+):\s*(.*)\s*$', line) + if m: blk[m.group(1)]=m.group(2).strip().strip('"').strip("'") + return blk +cy=yget(CY); bj=(json.load(open(BJ)).get("local_api") or {}) +print("endpoint cy/bj :", repr(cy.get("endpoint")), repr(bj.get("endpoint"))) +print("fingerprint AGREE :", cy.get("fingerprint")==bj.get("fingerprint")) +print("token AGREE :", cy.get("token")==bj.get("token")) +PY +``` + +**Never print the fingerprint or token values** — booleans only, and never into a committed file. + +> **STOP if `fingerprint` or `token` disagree.** A moved pin or a rotated token is a *different* +> failure: correcting only the address would then fail closed on the pin (`agent_channel_pin_mismatch`) +> or 401 (`agent_channel_unauthorized`). Diagnose that separately before touching anything. + +## 3. Back up, edit the endpoint only, restart + +```bash +CY=/var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml +pct exec -- cp -a $CY $CY.pre-fix.bak +# edit ONLY local_api.endpoint to the value from step 1 +pct exec -- systemctl restart felhom-controller-bootstrap.service +``` + +## 4. Verify — and know what success looks like + +```bash +pct exec -- docker logs felhom-controller 2>&1 | grep -i "\[channel\]" +``` + +**Expect SILENCE, not a "recovered" line.** On a freshly restarted controller the first probe +succeeds and `Check` returns early (*healthy first-obs → no notify*), so nothing is logged. A +`recovered` line only appears when a *running* controller transitions down→up. + +Positive confirmation is therefore: + +- **zero** `[channel]` lines after ~90 s (a broken channel logs `transient down` within 60 s and + `DOWN` within 120 s); +- the dashboard banner cleared; +- **zero** new `agent_channel_*` events hub-side; +- no `local_api endpoint DRIFT` line at startup (the two files now agree — if they still disagree + because you corrected `controller.yaml` rather than `bootstrap.json`, the drift alert will keep + firing and that is **correct**: it is telling you the two sources still disagree. Align the other + file too, or accept the alert until R-78). + +--- + +## Executed instances + +| date | boxes | before → after | fingerprint/token | outcome | +|---|---|---|---|---| +| 2026-07-26 | demo-felhom 9201 | `192.168.0.162:8443` → `169.254.253.1:8443` | both AGREE | channel healthy; 0 `[channel]` lines in 90 s; banner cleared | +| 2026-07-26 | demo-hp 9201 | `192.168.0.87:8443` → `169.254.253.1:8443` | both AGREE | channel healthy; 0 `[channel]` lines in 90 s | + +Root cause of both: the R-50 island migration (2026-07-25 12:42–12:46) rewrote `bootstrap.json` and +restarted the controllers, which re-read the untouched `controller.yaml`. Full analysis: +`audits/DIAG-agent-channel-2026-07-26.md`. diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 6ef92d8..87fcc5b 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,5 +1,20 @@ # Felhom Hub — Changelog +## v0.74.0 — allow `local_api_endpoint_drift` (controller v0.173.0 / R-77) (2026-07-26) + +One line in `allowedEventTypes`. It is **not optional**: `handleEvent` 400s an unknown `event_type` +("Invalid event_type"), so the controller's new drift alert would have been **silently inert** without +it — the exact seam-wiring failure class this project has hit four times. Shipped with the controller +that emits it, not after. + +Operator-only, `error` severity (drift never self-heals), and deliberately **not** an `agent_channel_*` +type: during the 2026-07-25 island-migration outage the generic "agent unreachable" event was the only +signal for 17.5 h and it hid a specific, fixable config fault. Naming the cause separately from the +symptom is the whole point. No customer notification toggle, matching the other `agent_channel_*` and +`host_*` operator events. + +Source: `documentation/audits/DIAG-agent-channel-2026-07-26.md`. + ## v0.73.2 — sync `hostInstallVersion` → 1.19.0 (R-50 island host-install) (2026-07-25) `hostInstallVersion` (the script version the operator customer page's install-command generator diff --git a/hub/internal/api/handler.go b/hub/internal/api/handler.go index 33ba7ff..95f14b0 100644 --- a/hub/internal/api/handler.go +++ b/hub/internal/api/handler.go @@ -1563,15 +1563,15 @@ var allowedEventTypes = map[string]bool{ // controller v0.134.1 — enlarged offsite push refused by the quota gate (warning; the controller's // dynamic Hungarian message is customer-grade — deliberately NO customerMessages entry, which would // discard the numbers (templates.go:129 priority)). - "offbox_enlarge_blocked": true, + "offbox_enlarge_blocked": true, // controller v0.142.0 — offsite-repo continuity: the remote repo is orphaned (reinstall shape) / // was reset (move-aside + re-init). Customer-grade messages below. - "offbox_repo_orphaned": true, - "offbox_repo_reset": true, - "storage_disconnected": true, - "storage_reconnected": true, - "disk_warning": true, - "disk_critical": true, + "offbox_repo_orphaned": true, + "offbox_repo_reset": true, + "storage_disconnected": true, + "storage_reconnected": true, + "disk_warning": true, + "disk_critical": true, // controller v0.169.0 — per-disk SMART degradation (Rendben→Figyelmeztetés/Hiba). The controller // sends a dynamic Hungarian message (disk label + the triggering attribute names), so — like // offbox_enlarge_blocked — there is deliberately NO customerMessages entry (which would discard the @@ -1594,6 +1594,10 @@ var allowedEventTypes = map[string]bool{ "agent_channel_construction_error": true, "agent_channel_unknown": true, "agent_channel_recovered": true, + // controller v0.173.0 (R-77): controller.yaml and bootstrap.json disagree on local_api.endpoint. + // Operator-only, deliberately NOT an agent_channel_* type — during the 2026-07-25 island-migration + // outage the generic "unreachable" alert was the only signal and it hid a specific config fault. + "local_api_endpoint_drift": true, // Hub-generated events "node_stale": true, "node_down": true, diff --git a/manifests/hub.yaml b/manifests/hub.yaml index 740643f..db0561c 100644 --- a/manifests/hub.yaml +++ b/manifests/hub.yaml @@ -125,7 +125,7 @@ spec: spec: containers: - name: hub - image: gitea.dooplex.hu/admin/felhom-hub:0.73.1 + image: gitea.dooplex.hu/admin/felhom-hub:0.74.0 ports: - containerPort: 8080 name: http