docs+scripts: TASK-E parts 4-7 - operator one-pager, rulings, hub build.sh adoption
PROMPT-TEMPLATE: standard 'For the operator' plain-language section, mandatory for M+ tasks and anything with a STOP. ROADMAP rulings (operator, 2026-07-21): R-25b full-teardown cascade with three acks + typed name (M-sized, spec to follow, no longer blocks R-3); R-11 channel = direct Messenger, doc is the architect's; R-42 option (a), sidecars follow the app; R-17 delete the archive - spike-lite found NO tooling verb targets it, so it is an operator console action; R-4 complete (freemail.hu verified). R-55 + R-41 slice 1 marked shipped; new R-56 (app difficulty classification - the constructive half of the glance ruling). scripts/build-hub.sh v1.23.0: the hub build script was outside any repo. Adopted verbatim + versioned; the build-dir path is now a symlink to it. felhom-testing skill: the ~1/5 recovery-code 'known flake' is retired - it was a real defect the test was correctly detecting.
This commit is contained in:
@@ -24,6 +24,38 @@
|
||||
|
||||
---
|
||||
|
||||
## For the operator — what this fixes, in one page (plain language)
|
||||
|
||||
<!--
|
||||
MANDATORY for any M+ task and for any task carrying a STOP. Optional but encouraged below that.
|
||||
Precedent: TASK-B, TASK-C and TASK-D shipped with this section and it is why those runs needed no
|
||||
mid-flight explanation; TASK-E made it a standard block.
|
||||
|
||||
WHO IT IS FOR: a technical-but-not-in-the-code reader — someone who runs the business and the
|
||||
hardware, knows what the components are, and has not read the file you are about to change.
|
||||
|
||||
IT MUST SAY, in this order:
|
||||
1. WHAT BREAKS TODAY — the observable symptom, not the code defect. "Your Stop is silently
|
||||
undone on a reboot", not "shouldRecreateOnBoot ignores container state".
|
||||
2. WHAT CHANGES — the new behaviour, in the same observable terms.
|
||||
3. WHAT THE OPERATOR WILL BE ASKED TO DO — every STOP, every ruling needed, every physical
|
||||
action (flash this, press that, confirm this), and anything that is HUMAN-only. If the answer
|
||||
is "nothing", say "nothing".
|
||||
4. WHAT IS NOT AFFECTED — especially for anything touching customer data, credentials or
|
||||
recovery. "Existing codes still work" belongs here.
|
||||
|
||||
RULES:
|
||||
- No jargon the ROADMAP row doesn't already explain. If a term must be introduced, define it in
|
||||
the sentence that uses it.
|
||||
- No file paths, no function names, no version numbers except the ones the operator acts on.
|
||||
- One page. If it does not fit, the task is probably two tasks.
|
||||
- Write it LAST, after the parts are settled, but place it HERE — it is what gets read first.
|
||||
- It is a summary, never the spec: it must not be the only place a requirement appears, or CC
|
||||
will implement from prose. Everything here is restated precisely in §0 onward.
|
||||
-->
|
||||
|
||||
---
|
||||
|
||||
## 0. Task class & scope
|
||||
|
||||
<!-- Pick ONE primary class. It sets which sections are mandatory. -->
|
||||
|
||||
@@ -34,7 +34,7 @@
|
||||
| Bare-metal Felhom ISO (blank hardware → zero-touch auto-install → first-boot `host-install`); selectable UEFI loader; **universal secret-free / operator-bind** mode | scripts v1.19.0 (`scripts/iso/`) + hub v0.62.0 + assistant container | **PROVEN-LIVE** (physical N100, one pass, 2026-07-18) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — the full chain on real metal in a single pass:** the generic reusable pairing ISO (v1.20.0, `--loader mkimage`, SB off) booted the cheap AMI board that F1 had blocked, installed unattended, and the box **self-registered as an unclaimed appliance at 16:17:14 — the same second it first booted** (`appliance_registrations` id=3), then bound → credential-delivered → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware.** Prior nested legs: slice A `SPIKE-baremetal-iso-2026-07-16` (build gate, disk-filter fail-safe, stub→host-install fetch); slice B RUNBOOK-B (shim boots+installs OVMF SB-enforcing + SeaBIOS; `--loader mkimage` boots+installs SB-off; mkimage SB-enforcing **FAILS** `Access Denied`; surgery byte-identical); **slice C (2026-07-17): the GENERIC secret-free ISO** — box self-registers as an unclaimed appliance (`POST /api/v1/appliance/register`, one-shot poll delivery, 404-no-oracle — all live-verified through the public ingress), operator binds on the Hosts page, hub delivers credentials once; bootstrap harness proves direct(zero-appliance-calls)/pairing/delivery; artifact proven secret-free (baked env = hub URL only) | **F1 loader caveat:** `--loader mkimage` fixes cheap AMI firmware that can't USB-boot the stock GRUB — UNSIGNED → **Secure Boot must be OFF**; default `shim` keeps SB. **Slice C bind is operator-password-gated** (CC stages, Viktor binds) → the live boot→register→bind→day-0 composition + physical N100 boot fold into the supervised rehearsal (R-1). Customer-facing **self-bind page = R-27 slice 1 SHIPPED (hub v0.66.0, 2026-07-17)** — see the dedicated self-bind row |
|
||||
| Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** (drill VM) | `DRILL-day0-vm-2026-07-12` §10/F-4 (gate ON via real edge; claimed, code consumed) | Never executed by a non-Viktor human → R-3. **Deliverability (R-4), gmail half DONE 2026-07-18:** the rehearsal's claim email was the first sent under the tightened DMARC `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu remains Viktor's open half.** (Dropped mis-cited `CAMPAIGN-4` F-C — that is the escrow-claim 502, not password claim) |
|
||||
| Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** (real customer-zero bind on metal, 2026-07-18) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md`:** operator minted + emailed the link 16:28:55 (7-day TTL, expiry 2026-07-25 recorded); **the customer bound their own box at 16:29:55 with `attempts=0`, `locked=0`** — `appliance_bound` carries source **`customer_selfbind`**, and the credential was delivered **26 s later** with no operator action. Hub-side lifecycle in `hub-state.txt` (`selfbind_tokens` mint→email→consume). Prior unit evidence: hub v0.66.0 (`web/selfbind.go`, `store/selfbind.go`; Scenarios A–F + F1/F2; 4 red-proofs verified red — THE TRAP `/bind/` exemption, no-oracle, lockout, single-active); GC verdict §3 (no appliance GC → TTL stands alone) | R-27 **slice 1**. No appliance list ever rendered; wrong code == wrong passphrase (one generic failure); 5-attempt lockout → call support; expiry falls back to operator-bind. **Live first-run DONE 2026-07-18** (rehearsal; the console banner rendered on the real ISO). **R-27b** (controller second-box dismissable prompt) deferred; **multi-box-per-link** = repeated operator sends |
|
||||
| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody` |
|
||||
| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test |
|
||||
| DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction); and `pbs.ProbeAuth` turns a 401 into a loud `auth_failed` that the existing `pbsdrheal` damper escalates to a fresh mint. Plus a consumed_at honesty gauge for the disagreement no single tier can see (box says `applied`, hub's staged secret never consumed). Proven live on felhom-pve: the agent read its credential through the wrapper (`rc=0`) and probed successfully (`credential probe OK storage=felhom-pbs`). **STOP-2 RAN 2026-07-21 AND THE CHAIN CLOSED — 13 SECONDS, operator click to converged.** The operator pressed **Re-issue PBS credentials**; the identical click on 2026-07-18 did nothing at all. Full chain (hub UTC / host CEST = UTC+2): `08:39:31Z` hub mints a fresh secret, **generation 0 → 1**, and the descriptor gains `"secret_generation": 1` — with `token_id` and `fingerprint` **byte-identical**, i.e. exactly the re-key shape that used to be invisible → `10:39:34` the agent READS its credential through the wrapper (leg b — the read that was impossible until v0.91.0) → `10:39:38` **`ERROR pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD` `previous_state=applied`** (leg c: the exact R-39 failure state, detected out loud for the first time ever) → `10:39:45` **`one-time token secret consumed`** `secret_len=36` (leg a: **NO short-circuit** — this is the line that never appeared on 2026-07-18) → `10:39:45` `felhom-pbs-apply reconcile` (the set-only wrapper, no `--server`) → `10:39:47` **`pbsdr: converged state=applied`**. Corroboration: the agent marker hash moved to `afbb3b41…` (it was byte-identical to the pre-reissue marker in the failure); `consumed_at` stamped `08:39:45Z`; the on-disk secret's mtime moved `2026-07-18 20:28:52` → `2026-07-21 10:39:45`; a live probe with the NEW credential returns **200**; three consecutive hub reports trace the whole state machine `applied → auth_failed → applied`; and **zero** `pbsdr_selfheal` escalations fired — the box healed through the descriptor path before the damper was ever needed, with exactly ONE mint and ONE consume and no `consumed-failed.json`. **Row upgraded to PROVEN-LIVE (2026-07-21).** Evidence: `felhom-agent/REPORT.md` (2026-07-21). |
|
||||
| Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown, incl. two real firings)** | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **Consistency gap → R-25b:** the Danger-zone DELETE leaves host rows and doesn't run this teardown |
|
||||
| Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | `DRILL-GL6-2026-07-08` Phase 1/5 (KEPT-vs-WIPED printed verbatim; drive data intact ×3); GL-4 code | Secret purge (GL6-F1 `.bak` residue) fixed v1.12.0; enrolled-drive `mnt-*.mount` units survive (GL6-F2, open); cluster-aware `felhom_guests` guard + saferemove cost warning missing → R-9 |
|
||||
@@ -62,7 +62,7 @@
|
||||
| Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | `CAMPAIGN-6D` P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking `CAMPAIGN-6B` P2 (100 MiB via real CF edge, 120 MiB→413) | Chunking proven at the real CF edge via `curl --resolve`; the **rendered browser file-picker** upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B *finding*; fix verified in 6D |
|
||||
| Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill |
|
||||
| PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | hub v0.56.0 (`pbsdrheal/reconciler.go`, `RestageHostPBSSecret`, all §10 red-proofs); `SPIKE-pbsdr-selfheal-2026-07-15` (root cause) | Reconciler is **scoped to one host** (`PBSDRHEAL_ONLY_HOST`), not fleet-wide; already fired live hands-free on drill qm300 (07-15) — real-customer firing + fleet-wide widening pending |
|
||||
| Box survives an **unattended app or guest-network failure** (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired | controller v0.156.0, agent v0.92.1 | **PROVEN-LIVE** (2026-07-21) | All three legs exercised on the live demo box, operator-present, in one session — `felhom-controller/REPORT.md` + `felhom-agent/REPORT.md` (2026-07-21). **Dead primary:** `docker stop immich-server` 12:50:40 CEST → `degraded` 13 s later → **exactly one** `app_start_failed` + dashboard banner → restart → banner self-cleared (the 2026-07-20 shape that was silent for 18 h). **Boot orphan:** `pct reboot 9201` → `[bootrecon] 1 boot-orphaned app(s) found: [bookstack]` → started in 1 attempt of 2, **zero alerts** (success inside the boot grace is silent); `StartedAt` proves Docker's `unless-stopped` did NOT resurrect it — only the sweep did, which also answers P1 and confirms the F5 hypothesis. **Dead DHCP client:** deliberate replay of the incident — `kill -9` 12:43:18 → detected on process liveness 57 s later while the lease was still live → healed 12:45:18 with the incident's verbatim invocation; **the tunnel never dropped (`cloudflared Up 29 hours`)**, i.e. the outage was prevented rather than merely observed | **A real gap this validation surfaced → R-55:** for a *drive-backed* app a customer's deliberate Stop does NOT survive a reboot — the boot bind gate recreates and starts every deployed drive-backed app unconditionally. Pre-existing, not introduced by R-52 (whose own gate was observed correct), but it means "a stopped app stays stopped" is currently true only for non-drive-backed apps. The **static-guest** half of the network leg stays deliberately out of scope → **R-50** |
|
||||
| Box survives an **unattended app or guest-network failure** (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired | controller v0.156.0, agent v0.92.1 | **PROVEN-LIVE** (2026-07-21) | All three legs exercised on the live demo box, operator-present, in one session — `felhom-controller/REPORT.md` + `felhom-agent/REPORT.md` (2026-07-21). **Dead primary:** `docker stop immich-server` 12:50:40 CEST → `degraded` 13 s later → **exactly one** `app_start_failed` + dashboard banner → restart → banner self-cleared (the 2026-07-20 shape that was silent for 18 h). **Boot orphan:** `pct reboot 9201` → `[bootrecon] 1 boot-orphaned app(s) found: [bookstack]` → started in 1 attempt of 2, **zero alerts** (success inside the boot grace is silent); `StartedAt` proves Docker's `unless-stopped` did NOT resurrect it — only the sweep did, which also answers P1 and confirms the F5 hypothesis. **Dead DHCP client:** deliberate replay of the incident — `kill -9` 12:43:18 → detected on process liveness 57 s later while the lease was still live → healed 12:45:18 with the incident's verbatim invocation; **the tunnel never dropped (`cloudflared Up 29 hours`)**, i.e. the outage was prevented rather than merely observed | **The gap this validation surfaced → R-55, now FIXED (controller v0.157.0, 2026-07-21).** For a *drive-backed* app a customer's deliberate Stop did NOT survive a reboot — the boot bind gate recreated and started every deployed drive-backed app unconditionally. Pre-existing, not introduced by R-52 (whose own gate was observed correct). The gate now also requires the app to still HAVE containers, which is R-52's own `existing-Exited vs absent` predicate: a UI Stop is `compose down` and removes them. So **"a stopped app stays stopped" now holds for drive-backed apps too** — with one honest caveat on the evidence: the fix is deployed and verified present in the running binary, but the reboot leg proving it end to end has NOT been run yet (it needs an operator-present `pct reboot 9201`), so this half is **shipped-not-yet-proven-live**. The **static-guest** half of the network leg stays deliberately out of scope → **R-50** |
|
||||
| Crash/power-loss mid-backup/mid-migration → self-heal on next run | controller, agent | **PROVEN-LIVE** | `CAMPAIGN-6D` P5-REST (SIGKILL mid-offbox → auto-restart ~15s, run marked failed not false-success, no stale lock); `CAMPAIGN-6E` B1-B3 | (Cited `CAMPAIGN-2` T-RBT-* legs were empty / auth-hollow — corrected.) Live mid-**migration** crash→self-heal is the weakest sub-claim (P5-REST is mid-backup) |
|
||||
| Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent, controller, bootstrap | **PARTIAL** | `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — a real relocation of the demo box: guest + hub telemetry + WG/PBS + Cloudflare tunnel all survived untouched, but the **controller↔agent control plane did not** (agent binds a LAN literal → `bind: cannot assign requested address` → storage/PBS-backup/quiesce/restore-test/DR down until fixed). Mitigated for the window by pinning `vmbr0` static | **R-50** (island-bridge control plane, spike-first) is the durable fix. Related: **R-51** (dead-primary alerting) and **R-52** (boot desired-state reconciliation) — the same event left two apps `Exited` with no alarm and no recovery |
|
||||
| Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | |
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -1,5 +1,21 @@
|
||||
# Felhom scripts — Changelog
|
||||
|
||||
## build-hub.sh v1.23.0 — the hub build script enters the repo (2026-07-21)
|
||||
|
||||
It never was in one. `/mnt/5_hdd/felhom.eu/build/felhom-hub/build.sh` lived only in a DooPlex build
|
||||
directory — unversioned, unreviewed, un-diffable and gone with the box, while being the only way the
|
||||
hub image gets built. Recorded as a follow-up in `felhom.eu/REPORT.md` §6; adopted here.
|
||||
|
||||
Adopted verbatim — the build logic is byte-identical (verified by diff), and the only additions are
|
||||
a `SCRIPT_VERSION` constant echoed in the banner and a header naming the canonical/operative split.
|
||||
The build-dir path is now a **symlink** to this file, so there is one source of truth and a repo edit
|
||||
is live immediately (the `install_skills.py` pattern). The previous copy is retained beside it as
|
||||
`build.sh.pre-adopt-2026-07-21`.
|
||||
|
||||
It already carried the GitOps deploy hint fixed earlier in the week (manifest bump → hard-refresh →
|
||||
deliberate sync, and the named trap that `kubectl set image` / `kubectl apply` are silently reverted
|
||||
by the next ArgoCD sync) — that wording now lives in git along with the rest of it.
|
||||
|
||||
## install_skills.py — cross-platform (POSIX symlink / Windows junction) (2026-07-19)
|
||||
|
||||
Claude Code now runs on DooPlex (Debian 13), where `mklink /J` does not exist — the script would
|
||||
|
||||
Executable
+218
@@ -0,0 +1,218 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# felhom-hub — Docker image build script
|
||||
# =============================================================================
|
||||
# CANONICAL LOCATION: felhom.eu/scripts/build-hub.sh (this file, in git).
|
||||
# OPERATIVE LOCATION: /mnt/5_hdd/felhom.eu/build/felhom-hub/build.sh — a SYMLINK to this file, so
|
||||
# there is exactly one source of truth and a repo edit is live immediately (the install_skills.py
|
||||
# pattern). It lived ONLY in that build dir until v1.23.0, outside any repo and therefore
|
||||
# unversioned, unreviewed and unrecoverable — noted as a follow-up in felhom.eu/REPORT.md §6.
|
||||
#
|
||||
# Copies hub/ source from the felhom.eu git repo, builds the Docker image. Build artifacts stay in
|
||||
# the build dir's workspace/ — the git repo stays clean.
|
||||
#
|
||||
# Usage:
|
||||
# ./build.sh # Build for current platform, tag as :dev
|
||||
# ./build.sh 0.1.0 # Build with version tag
|
||||
# ./build.sh 0.1.0 --push # Build + push to Gitea registry
|
||||
# ./build.sh 0.1.0 --multiarch # Build amd64+arm64 + push
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_VERSION="1.23.0"
|
||||
|
||||
# --- Configuration (edit these if your paths differ) ---
|
||||
REPO_DIR="/mnt/5_hdd/felhom.eu/git/felhom.eu"
|
||||
HUB_SRC="${REPO_DIR}/hub"
|
||||
REGISTRY="gitea.dooplex.hu/admin"
|
||||
IMAGE="${REGISTRY}/felhom-hub"
|
||||
|
||||
# Build workspace — a temp directory next to this script, NOT in the git repo
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
BUILD_DIR="${SCRIPT_DIR}/workspace"
|
||||
|
||||
# --- Parse arguments ---
|
||||
VERSION="${1:-dev}"
|
||||
ACTION="${2:-}" # --push or --multiarch
|
||||
|
||||
# --- Colors ---
|
||||
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; CYAN='\033[0;36m'; NC='\033[0m'
|
||||
info() { echo -e "${GREEN}[INFO]${NC} $*"; }
|
||||
warn() { echo -e "${YELLOW}[WARN]${NC} $*"; }
|
||||
error() { echo -e "${RED}[ERROR]${NC} $*"; }
|
||||
step() { echo -e "${CYAN}[STEP]${NC} $*"; }
|
||||
|
||||
# --- Pre-flight checks ---
|
||||
if [[ ! -d "${HUB_SRC}" ]]; then
|
||||
error "Hub source not found: ${HUB_SRC}"
|
||||
error "Clone the repo first: git clone https://gitea.dooplex.hu/admin/felhom.eu.git ${REPO_DIR}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v docker &>/dev/null; then
|
||||
error "Docker not found."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Get git commit for build metadata
|
||||
GIT_COMMIT=$(cd "${REPO_DIR}" && git rev-parse --short HEAD 2>/dev/null || echo "unknown")
|
||||
|
||||
echo ""
|
||||
info "╔══════════════════════════════════════╗"
|
||||
info "║ felhom-hub build ║"
|
||||
info "╚══════════════════════════════════════╝"
|
||||
info "Version: ${VERSION} (build-hub.sh v${SCRIPT_VERSION})"
|
||||
info "Commit: ${GIT_COMMIT}"
|
||||
info "Source: ${HUB_SRC}"
|
||||
info "Build: ${BUILD_DIR}"
|
||||
info "Image: ${IMAGE}:${VERSION}"
|
||||
echo ""
|
||||
|
||||
# =========================================================================
|
||||
# Step 1: Pull latest code
|
||||
# =========================================================================
|
||||
step "1/3 — Pulling latest code..."
|
||||
|
||||
cd "${REPO_DIR}"
|
||||
git pull --ff-only 2>/dev/null || warn "git pull failed (offline or dirty tree?)"
|
||||
|
||||
# =========================================================================
|
||||
# Step 2: Sync source to build workspace
|
||||
# =========================================================================
|
||||
step "2/3 — Syncing source to workspace..."
|
||||
|
||||
mkdir -p "${BUILD_DIR}"
|
||||
|
||||
rsync -a --delete \
|
||||
--exclude '.git' \
|
||||
--exclude 'bin/' \
|
||||
--exclude '*.exe' \
|
||||
"${HUB_SRC}/" "${BUILD_DIR}/"
|
||||
|
||||
info "Source synced to ${BUILD_DIR}"
|
||||
|
||||
# Sync app assets from website directory into build workspace
|
||||
WEBSITE_ASSETS_DIR="${REPO_DIR}/website/assets"
|
||||
if [[ -d "${WEBSITE_ASSETS_DIR}" ]]; then
|
||||
mkdir -p "${BUILD_DIR}/assets"
|
||||
cp "${WEBSITE_ASSETS_DIR}"/*-logo.svg "${BUILD_DIR}/assets/" 2>/dev/null || true
|
||||
cp "${WEBSITE_ASSETS_DIR}"/*-logo.png "${BUILD_DIR}/assets/" 2>/dev/null || true
|
||||
cp "${WEBSITE_ASSETS_DIR}"/*-screenshot-*.webp "${BUILD_DIR}/assets/" 2>/dev/null || true
|
||||
ASSET_COUNT=$(ls "${BUILD_DIR}/assets/" 2>/dev/null | wc -l)
|
||||
info "App assets synced (${ASSET_COUNT} files)"
|
||||
else
|
||||
warn "Website assets dir not found: ${WEBSITE_ASSETS_DIR}"
|
||||
mkdir -p "${BUILD_DIR}/assets"
|
||||
fi
|
||||
|
||||
# Verify structure
|
||||
cd "${BUILD_DIR}"
|
||||
|
||||
for dir in cmd/hub internal/api internal/store internal/web; do
|
||||
if [[ ! -d "${dir}" ]]; then
|
||||
error "Missing expected directory: ${dir}"
|
||||
error "Hub source structure may be incomplete."
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
info "Package structure verified ✓"
|
||||
|
||||
if [[ ! -f "go.mod" ]]; then
|
||||
error "go.mod not found in build workspace!"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Ensure go.sum exists (Dockerfile COPY will fail without it)
|
||||
if command -v go &>/dev/null; then
|
||||
info "Running go mod tidy..."
|
||||
go mod tidy 2>&1 || warn "go mod tidy had issues (Docker build may still work)"
|
||||
elif [[ ! -f "go.sum" ]]; then
|
||||
warn "go.sum missing and Go not installed locally — creating empty go.sum"
|
||||
warn "(Docker build stage will resolve dependencies via 'go mod download')"
|
||||
touch go.sum
|
||||
fi
|
||||
|
||||
# =========================================================================
|
||||
# Step 3: Docker build
|
||||
# =========================================================================
|
||||
step "3/3 — Building Docker image..."
|
||||
|
||||
BUILD_ARGS=(
|
||||
--build-arg "VERSION=${VERSION}"
|
||||
--build-arg "BUILD_TIME=$(date -u +"%Y-%m-%dT%H:%M:%SZ")"
|
||||
)
|
||||
|
||||
case "${ACTION}" in
|
||||
--push)
|
||||
info "Building for current platform + pushing..."
|
||||
docker build "${BUILD_ARGS[@]}" \
|
||||
-t "${IMAGE}:${VERSION}" \
|
||||
-t "${IMAGE}:latest" \
|
||||
.
|
||||
|
||||
info "Pushing..."
|
||||
docker push "${IMAGE}:${VERSION}"
|
||||
docker push "${IMAGE}:latest"
|
||||
;;
|
||||
|
||||
--multiarch)
|
||||
info "Building multi-arch (amd64 + arm64) + pushing..."
|
||||
|
||||
# Ensure buildx builder exists
|
||||
if ! docker buildx inspect felhom-builder &>/dev/null; then
|
||||
info "Creating buildx builder (one-time setup)..."
|
||||
docker buildx create --name felhom-builder --use --bootstrap
|
||||
else
|
||||
docker buildx use felhom-builder
|
||||
fi
|
||||
|
||||
docker buildx build "${BUILD_ARGS[@]}" \
|
||||
--platform linux/amd64,linux/arm64 \
|
||||
-t "${IMAGE}:${VERSION}" \
|
||||
-t "${IMAGE}:latest" \
|
||||
--push \
|
||||
.
|
||||
;;
|
||||
|
||||
*)
|
||||
info "Building for current platform (local only)..."
|
||||
docker build "${BUILD_ARGS[@]}" \
|
||||
-t "${IMAGE}:${VERSION}" \
|
||||
-t "${IMAGE}:latest" \
|
||||
.
|
||||
;;
|
||||
esac
|
||||
|
||||
# =========================================================================
|
||||
# Summary
|
||||
# =========================================================================
|
||||
echo ""
|
||||
info "╔══════════════════════════════════════╗"
|
||||
info "║ Build complete ✓ ║"
|
||||
info "╚══════════════════════════════════════╝"
|
||||
info "Image: ${IMAGE}:${VERSION}"
|
||||
|
||||
# Show image size if available locally
|
||||
SIZE=$(docker image inspect "${IMAGE}:${VERSION}" --format='{{.Size}}' 2>/dev/null || echo "")
|
||||
if [[ -n "${SIZE}" ]]; then
|
||||
SIZE_HUMAN=$(numfmt --to=iec "${SIZE}" 2>/dev/null || echo "${SIZE} bytes")
|
||||
info "Size: ${SIZE_HUMAN}"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
if [[ "${ACTION}" == "" ]]; then
|
||||
info "Image is local only. To push: ./build.sh ${VERSION} --push"
|
||||
else
|
||||
info "Image pushed to registry."
|
||||
echo ""
|
||||
info "Deploy to k3s — GitOps ONLY. The manifest in git is the source of truth:"
|
||||
info " 1. bump the image tag in ${REPO_DIR}/manifests/hub.yaml to ${VERSION}, commit, push"
|
||||
info " 2. hard-refresh + sync the ArgoCD app (auto-sync is OFF — the sync is deliberate):"
|
||||
info " sudo kubectl -n argocd annotate application felhom argocd.argoproj.io/refresh=hard --overwrite"
|
||||
info " sudo kubectl -n argocd patch application felhom --type merge \\"
|
||||
info " -p '"'"'{\"operation\":{\"initiatedBy\":{\"username\":\"cc\"},\"sync\":{\"syncStrategy\":{\"apply\":{}}}}}'"'"'"
|
||||
info ""
|
||||
info " NEVER \`kubectl set image\` or \`kubectl apply\` by hand — ArgoCD reverts it on the next"
|
||||
info " sync, so the change looks applied, works for a while, and then silently disappears."
|
||||
fi
|
||||
echo ""
|
||||
@@ -57,8 +57,13 @@ classifier functions get fixture tables (agent `classifyClaim` style).
|
||||
go build ./... && go vet ./... && go test ./...
|
||||
```
|
||||
Run it in the module dir: `felhom.eu/hub/`, `felhom-controller/controller/`, `felhom-agent/` root.
|
||||
Known flake: agent `TestGenerateRecoveryCode_EntropyAndFormat` fails ~1/5 (hyphenated wordlist word) —
|
||||
re-run before diagnosing; it is not a regression.
|
||||
**No known flakes.** The long-standing agent `TestGenerateRecoveryCode_EntropyAndFormat` ~1/5 failure
|
||||
was **fixed in agent v0.93.0 (2026-07-21)** — and it was never a flaky test. It was a real defect the
|
||||
test was correctly detecting and everyone had been told to re-run past: the EFF wordlist contains four
|
||||
hyphenated entries, so a recovery code could come out reading as 11 words instead of 10. If it fails
|
||||
now, it is a regression. **The lesson generalises: "known flake, just re-run it" is a diagnosis, and
|
||||
it needs the same evidence as any other one.** A test that fails at a stable, explainable rate is
|
||||
usually telling the truth about a rare input, not misbehaving.
|
||||
|
||||
## Live validation doctrine (after unit-land)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user