hub v0.74.0: allow local_api_endpoint_drift; R-77 docs + R-78/79/80
The allowlist entry is REQUIRED, not cosmetic: handleEvent 400s an unknown event_type, so controller v0.173.0's new drift alert would be silently inert without it. Shipped with the controller that emits it. Docs: - RUNBOOK-local-api-endpoint-drift.md — how to repair a drift, including the step everyone will want to skip (establish which value is CORRECT from what the agent is actually bound to, rather than assuming bootstrap.json wins) and what success looks like (SILENCE, not a "recovered" line, because a fresh controller's healthy first observation is not logged). Records both 2026-07-26 repairs. - ROADMAP: R-77 shipped; R-78 the local_api authority ruling, with the clobber-a-working-channel risk spelled out in BOTH directions so it is not resolved opportunistically; R-79 the whole-surface English-strings sweep; R-80 expected_backup_missed, flagged as likely outranking R-77 because 7.3 days of stale backup materially exceeds the ~1.5-day channel outage, so the causal link the DIAG hedged on cannot be the whole story. - Capability map: note against the drive-wizard row (every agent-backed capability rides this channel) that a silent drift class is now detected. NO row status flips — detection is not prevention.
This commit is contained in:
@@ -76,7 +76,7 @@
|
||||
|
||||
| Scenario | Components | Status | Evidence | Gap / roadmap |
|
||||
|---|---|---|---|---|
|
||||
| Drive wizard: scan/format/mount/enroll, incl. legacy-boot LVM-root hosts | controller, agent v0.87 | **PROVEN-LIVE** | `DISPOSITION-ia-finding2-systemdisks-2026-07-13` (legacy EFI+LVM host, root not offered, byte-identical); enroll/format live in `storage-lifecycle-acceptance-2026-06-15` (E10 re-enroll, data intact); agent fence self-test refuses `/dev/sda` | (Cited `CAMPAIGN-2` T-STG-ENROLL/SEC-FORMAT were auth-hollow CSRF-403.) **Fresh-USB wizard enroll+format through the customer UI PROVEN-LIVE (controller v0.141.0, 2026-07-17):** a 64 GB scratch USB driven through the real `/api/storage/init` endpoints (login+CSRF) → confirm → detached format (~27 s mkfs) → mount → register → mounted+registered at `/mnt/felhom-drives/scratch1`. **F6 (initialize-to-usable) now covered:** the wizard runs the chain as a detached, disconnect-safe, pollable job (3-step progress) with an agent format-status poll for a slow mkfs |
|
||||
| Drive wizard: scan/format/mount/enroll, incl. legacy-boot LVM-root hosts | controller, agent v0.87 | **PROVEN-LIVE** | `DISPOSITION-ia-finding2-systemdisks-2026-07-13` (legacy EFI+LVM host, root not offered, byte-identical); enroll/format live in `storage-lifecycle-acceptance-2026-06-15` (E10 re-enroll, data intact); agent fence self-test refuses `/dev/sda` | (Cited `CAMPAIGN-2` T-STG-ENROLL/SEC-FORMAT were auth-hollow CSRF-403.) **Fresh-USB wizard enroll+format through the customer UI PROVEN-LIVE (controller v0.141.0, 2026-07-17):** a 64 GB scratch USB driven through the real `/api/storage/init` endpoints (login+CSRF) → confirm → detached format (~27 s mkfs) → mount → register → mounted+registered at `/mnt/felhom-drives/scratch1`. **F6 (initialize-to-usable) now covered:** the wizard runs the chain as a detached, disconnect-safe, pollable job (3-step progress) with an agent format-status poll for a slow mkfs **2026-07-26 — a SILENT failure class on the channel every agent-backed capability depends on (this row, data migration, USB enrollment, guest RAM, quiesce/PBS) is now DETECTED (controller v0.173.0, R-77). No row status flips.** `controller.yaml` and `bootstrap.json` could disagree on `local_api.endpoint` indefinitely with no signal: the R-50 island migration rewrote the latter, the fleet kept dialling the former, and for 17.5 h the only alert was a generic "agent unreachable" that read as an infrastructure blip. Drift now raises its own event type (`local_api_endpoint_drift`) naming both values. It is DETECTION ONLY — the authority ruling is R-78 — so the class is now loud, not prevented. Evidence: `audits/DIAG-agent-channel-2026-07-26.md`. |
|
||||
| Data migration between drives (all / per-app), crash-safe | controller | **PROVEN-LIVE** | `CAMPAIGN-6C` 4P-5 (scope=app round-trip, byte-identical); `storage-lifecycle-acceptance-2026-06-15` (two migrate-all runs via dashboard UI, sha256 byte-identical) | (Cited `CAMPAIGN-2` T-STG-MIGRATE-* were auth-hollow.) "crash-safe" is design-level (copy→verify→remove) — no clean live crash-during-migration PASS |
|
||||
| NAS (NFS/SMB-client) verify-before-commit, uid-1000 probe, categorized Hungarian errors, DSM-validated | controller v0.113–117, agent v0.81/84/85 | **PROVEN-LIVE** | `SPIKE-nas-verify-2026-07-11`, `SPIKE-nas-dsm-2026-07-11`, `CAMPAIGN-3-2026-07-11` (boot/reassert fixes) | |
|
||||
| USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | `storage-lifecycle-acceptance-2026-06-15` E4 (yanked-while-running → agent auto-rebind) + E10 (re-enroll, data intact); `CAMPAIGN-4`/`6A` (3 USB re-establish across device-letter reshuffle) | (Cited `RUNBOOK-usb` could NOT complete a wizard enrollment; `CAMPAIGN-2` legs were auth-hollow.) Fresh-USB **wizard enrollment** specifically still unproven |
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,108 @@
|
||||
# RUNBOOK — repairing a `local_api` endpoint drift
|
||||
|
||||
**When to run:** the controller logs
|
||||
`[ERROR] bootstrap: local_api endpoint DRIFT — …` at startup, or the operator receives a
|
||||
`local_api_endpoint_drift` event, or the „A tárolókezelő ügynök címe elavult a beállításokban."
|
||||
banner appears. Detection shipped in controller **v0.173.0** (R-77).
|
||||
|
||||
**What drift means:** `controller.yaml` and `bootstrap.json` disagree on `local_api.endpoint`. The
|
||||
controller dials **`controller.yaml`**. The agent listens wherever it was last configured — which,
|
||||
after the R-50 island migration, is `169.254.253.1:8443`.
|
||||
|
||||
**Why the controller does not fix this itself:** auto-reconcile would clobber a *correct*
|
||||
`controller.yaml` from a *stale* `bootstrap.json` — a half-completed re-provision or a hand-repaired
|
||||
guest would lose a working channel on its next restart, fleet-wide and silently. Which file is
|
||||
authoritative is **R-78**, unresolved. Until it is ruled, repair is a deliberate operator action.
|
||||
|
||||
> **Operator-present.** This edits a live guest's config and restarts its controller. The channel
|
||||
> being down also degrades the drive gate, guest-reboot recovery and the quiesce/backup loop, so it
|
||||
> is worth doing promptly — but not blind.
|
||||
|
||||
---
|
||||
|
||||
## 1. Establish which value is correct — do NOT assume `bootstrap.json`
|
||||
|
||||
`bootstrap.json` is right *when the drift came from a migration or re-provision* — the common case.
|
||||
It is **wrong** if someone hand-repaired `controller.yaml` after a failed provision. Decide with
|
||||
evidence, not by default:
|
||||
|
||||
```bash
|
||||
# What is the agent ACTUALLY listening on? This is the ground truth.
|
||||
ssh <pve-host> "ss -lntH 'sport = :8443'" # post-island: 169.254.253.1:8443
|
||||
```
|
||||
|
||||
**The correct endpoint is whatever the agent is bound to.** Both files are just claims about it.
|
||||
|
||||
## 2. Compare all three fields, not only the endpoint
|
||||
|
||||
`mergeLocalAPI` replaces the **whole** block, so a migration may have moved more than the address.
|
||||
|
||||
```bash
|
||||
pct exec <VMID> -- python3 - <<'PY'
|
||||
import json, re, hashlib
|
||||
CY="/var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml"
|
||||
BJ="/etc/felhom-bootstrap/bootstrap.json"
|
||||
def yget(path):
|
||||
blk={}; inb=False
|
||||
for line in open(path):
|
||||
if re.match(r'^local_api:', line): inb=True; continue
|
||||
if inb and re.match(r'^\S', line): break
|
||||
if inb:
|
||||
m=re.match(r'\s+(\w+):\s*(.*)\s*$', line)
|
||||
if m: blk[m.group(1)]=m.group(2).strip().strip('"').strip("'")
|
||||
return blk
|
||||
cy=yget(CY); bj=(json.load(open(BJ)).get("local_api") or {})
|
||||
print("endpoint cy/bj :", repr(cy.get("endpoint")), repr(bj.get("endpoint")))
|
||||
print("fingerprint AGREE :", cy.get("fingerprint")==bj.get("fingerprint"))
|
||||
print("token AGREE :", cy.get("token")==bj.get("token"))
|
||||
PY
|
||||
```
|
||||
|
||||
**Never print the fingerprint or token values** — booleans only, and never into a committed file.
|
||||
|
||||
> **STOP if `fingerprint` or `token` disagree.** A moved pin or a rotated token is a *different*
|
||||
> failure: correcting only the address would then fail closed on the pin (`agent_channel_pin_mismatch`)
|
||||
> or 401 (`agent_channel_unauthorized`). Diagnose that separately before touching anything.
|
||||
|
||||
## 3. Back up, edit the endpoint only, restart
|
||||
|
||||
```bash
|
||||
CY=/var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml
|
||||
pct exec <VMID> -- cp -a $CY $CY.pre-fix.bak
|
||||
# edit ONLY local_api.endpoint to the value from step 1
|
||||
pct exec <VMID> -- systemctl restart felhom-controller-bootstrap.service
|
||||
```
|
||||
|
||||
## 4. Verify — and know what success looks like
|
||||
|
||||
```bash
|
||||
pct exec <VMID> -- docker logs felhom-controller 2>&1 | grep -i "\[channel\]"
|
||||
```
|
||||
|
||||
**Expect SILENCE, not a "recovered" line.** On a freshly restarted controller the first probe
|
||||
succeeds and `Check` returns early (*healthy first-obs → no notify*), so nothing is logged. A
|
||||
`recovered` line only appears when a *running* controller transitions down→up.
|
||||
|
||||
Positive confirmation is therefore:
|
||||
|
||||
- **zero** `[channel]` lines after ~90 s (a broken channel logs `transient down` within 60 s and
|
||||
`DOWN` within 120 s);
|
||||
- the dashboard banner cleared;
|
||||
- **zero** new `agent_channel_*` events hub-side;
|
||||
- no `local_api endpoint DRIFT` line at startup (the two files now agree — if they still disagree
|
||||
because you corrected `controller.yaml` rather than `bootstrap.json`, the drift alert will keep
|
||||
firing and that is **correct**: it is telling you the two sources still disagree. Align the other
|
||||
file too, or accept the alert until R-78).
|
||||
|
||||
---
|
||||
|
||||
## Executed instances
|
||||
|
||||
| date | boxes | before → after | fingerprint/token | outcome |
|
||||
|---|---|---|---|---|
|
||||
| 2026-07-26 | demo-felhom 9201 | `192.168.0.162:8443` → `169.254.253.1:8443` | both AGREE | channel healthy; 0 `[channel]` lines in 90 s; banner cleared |
|
||||
| 2026-07-26 | demo-hp 9201 | `192.168.0.87:8443` → `169.254.253.1:8443` | both AGREE | channel healthy; 0 `[channel]` lines in 90 s |
|
||||
|
||||
Root cause of both: the R-50 island migration (2026-07-25 12:42–12:46) rewrote `bootstrap.json` and
|
||||
restarted the controllers, which re-read the untouched `controller.yaml`. Full analysis:
|
||||
`audits/DIAG-agent-channel-2026-07-26.md`.
|
||||
@@ -1,5 +1,20 @@
|
||||
# Felhom Hub — Changelog
|
||||
|
||||
## v0.74.0 — allow `local_api_endpoint_drift` (controller v0.173.0 / R-77) (2026-07-26)
|
||||
|
||||
One line in `allowedEventTypes`. It is **not optional**: `handleEvent` 400s an unknown `event_type`
|
||||
("Invalid event_type"), so the controller's new drift alert would have been **silently inert** without
|
||||
it — the exact seam-wiring failure class this project has hit four times. Shipped with the controller
|
||||
that emits it, not after.
|
||||
|
||||
Operator-only, `error` severity (drift never self-heals), and deliberately **not** an `agent_channel_*`
|
||||
type: during the 2026-07-25 island-migration outage the generic "agent unreachable" event was the only
|
||||
signal for 17.5 h and it hid a specific, fixable config fault. Naming the cause separately from the
|
||||
symptom is the whole point. No customer notification toggle, matching the other `agent_channel_*` and
|
||||
`host_*` operator events.
|
||||
|
||||
Source: `documentation/audits/DIAG-agent-channel-2026-07-26.md`.
|
||||
|
||||
## v0.73.2 — sync `hostInstallVersion` → 1.19.0 (R-50 island host-install) (2026-07-25)
|
||||
|
||||
`hostInstallVersion` (the script version the operator customer page's install-command generator
|
||||
|
||||
@@ -1563,15 +1563,15 @@ var allowedEventTypes = map[string]bool{
|
||||
// controller v0.134.1 — enlarged offsite push refused by the quota gate (warning; the controller's
|
||||
// dynamic Hungarian message is customer-grade — deliberately NO customerMessages entry, which would
|
||||
// discard the numbers (templates.go:129 priority)).
|
||||
"offbox_enlarge_blocked": true,
|
||||
"offbox_enlarge_blocked": true,
|
||||
// controller v0.142.0 — offsite-repo continuity: the remote repo is orphaned (reinstall shape) /
|
||||
// was reset (move-aside + re-init). Customer-grade messages below.
|
||||
"offbox_repo_orphaned": true,
|
||||
"offbox_repo_reset": true,
|
||||
"storage_disconnected": true,
|
||||
"storage_reconnected": true,
|
||||
"disk_warning": true,
|
||||
"disk_critical": true,
|
||||
"offbox_repo_orphaned": true,
|
||||
"offbox_repo_reset": true,
|
||||
"storage_disconnected": true,
|
||||
"storage_reconnected": true,
|
||||
"disk_warning": true,
|
||||
"disk_critical": true,
|
||||
// controller v0.169.0 — per-disk SMART degradation (Rendben→Figyelmeztetés/Hiba). The controller
|
||||
// sends a dynamic Hungarian message (disk label + the triggering attribute names), so — like
|
||||
// offbox_enlarge_blocked — there is deliberately NO customerMessages entry (which would discard the
|
||||
@@ -1594,6 +1594,10 @@ var allowedEventTypes = map[string]bool{
|
||||
"agent_channel_construction_error": true,
|
||||
"agent_channel_unknown": true,
|
||||
"agent_channel_recovered": true,
|
||||
// controller v0.173.0 (R-77): controller.yaml and bootstrap.json disagree on local_api.endpoint.
|
||||
// Operator-only, deliberately NOT an agent_channel_* type — during the 2026-07-25 island-migration
|
||||
// outage the generic "unreachable" alert was the only signal and it hid a specific config fault.
|
||||
"local_api_endpoint_drift": true,
|
||||
// Hub-generated events
|
||||
"node_stale": true,
|
||||
"node_down": true,
|
||||
|
||||
+1
-1
@@ -125,7 +125,7 @@ spec:
|
||||
spec:
|
||||
containers:
|
||||
- name: hub
|
||||
image: gitea.dooplex.hu/admin/felhom-hub:0.73.1
|
||||
image: gitea.dooplex.hu/admin/felhom-hub:0.74.0
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
name: http
|
||||
|
||||
Reference in New Issue
Block a user