admin
3a9d744360
The offsite restore list rendered up to five inline forms per app row. Two of them —
„Helyreállítás az élő adatok közé (csak a hiányzó fájlok)" and „Teljes visszaállítás
(fájlok + adatbázis)" — were sibling buttons whose difference is whether the customer's
data comes back at all. That mis-selection CAUSED the round-2 incident: an operator who
had read the source pressed the missing-only button and /backup/offbox/reconstitute was
never hit (DIAG-immich-restore-round2-2026-07-19, finding 1).
The rule this establishes: two adjacent controls whose difference is "your data comes
back" vs "your data cannot come back" must never be distinguishable only by layout.
Each row now carries ONE „Visszaállítás…" entry linking to a per-app wizard at
GET /backups/restore/app?name=<app>: three intent CARDS with consequence sentences, a
visible phase strip, danger styling plus the R-43 double-confirm carried over verbatim on
the destructive one, and the pair-honesty panel bound to real OffsiteScratchPair facts.
deriveWizardStep is pure over (op running, size-gate flash, scratch ready); the step is
never accepted from the request, and a running op outranks a stale ?full_prep= so no
commit button survives into a restore. While ANY op runs every mutation form is suppressed
server-side instead of being offered and refused.
No new mutation endpoint: every card posts to the pre-existing /backup/offbox/* handler
with the same field names and gates, and the page works with JavaScript disabled.
internal/{backup,appbackup,selfupdate} untouched. R-45's job registry stays its own item.
Fixes a latent bug found on the way: offboxRedirectTo hardcoded "?" when appending its
flash, which against the wizard's ?name=<app> target would have buried the flash inside
the name value.
No new agent coupling — MinAgent stays 0.90.0.
1363 lines
119 KiB
Markdown
1363 lines
119 KiB
Markdown
# CONTEXT.md — Project Memory
|
||
|
||
> This file serves as persistent project memory across Claude Code sessions.
|
||
> It replaces the auto-generated "Memory" from the claude.ai Project.
|
||
> **Update this file at the end of each working session** with current state,
|
||
> recent decisions, and anything the next session needs to know.
|
||
>
|
||
> Ask Claude Code: "Please update CONTEXT.md with what we did today"
|
||
|
||
Last updated: 2026-07-21 (v0.154.0 — R-48: one restore entry per app, intent as a described choice)
|
||
|
||
> **2026-07-21 — v0.154.0 (R-48).** Collapses the offsite restore controls to a single
|
||
> „Visszaállítás…" entry per app row plus a per-app wizard at `GET /backups/restore/app?name=<app>`.
|
||
> The defect it closes is the CAUSE of the round-2 incident: the list rendered up to five inline
|
||
> forms per row, two of which — the missing-only merge and the true reconstitution — were sibling
|
||
> buttons whose difference is whether the data comes back. The rule it establishes: *two adjacent
|
||
> controls whose difference is "your data comes back" vs "your data cannot come back" must never be
|
||
> distinguishable only by layout.*
|
||
>
|
||
> **DECISION: the wizard is server-rendered on the EXISTING endpoints.** No new mutation endpoint,
|
||
> no JSON state API, no client router. Every card is a real form POST to
|
||
> `/backup/offbox/{restore,place,reconstitute}` with the same field names and gates, and the server
|
||
> renders the next step — so it works with JavaScript disabled. `TestRestoreWizard_NoNewMutationEndpoints`
|
||
> makes that structural: adding a form that posts somewhere new fails the suite by design.
|
||
>
|
||
> **DECISION: R-45 stays its own item.** The wizard polls the two existing status surfaces as-is; the
|
||
> generalized job registry (and with it a real per-phase progress feed) is not built here.
|
||
>
|
||
> **DECISION: the step is derived, never requested.** `deriveWizardStep` is pure over (op running,
|
||
> size-gate flash, scratch ready). Precedence is load-bearing — a running op outranks a stale
|
||
> `?full_prep=` in the URL, or a commit button reappears mid-restore. While ANY op runs every
|
||
> mutation form is suppressed server-side rather than offered and then refused with a 409.
|
||
>
|
||
> Latent bug found and fixed on the way: `offboxRedirectTo` hardcoded `"?"` when appending its flash,
|
||
> which would have buried the flash inside `?name=<app>`. **No agent coupling — MinAgent stays
|
||
> 0.90.0.** 9 new tests + the Group-B red-proof; full suite green.
|
||
>
|
||
> **NOT live-validated at commit time by design:** v0.154.0 is published but deliberately NOT
|
||
> hand-deployed — the operator's hub floor save (0.153.0 → 0.154.0) pulls it via the self-update
|
||
> path, and that swap IS the R-23(a) single-fire validation (STOP-1).
|
||
|
||
|
||
> **2026-07-20 — v0.153.0 (R-47).** Closes the H4 race on **BOTH** restore paths. The replay needs a
|
||
> running DB container, so both paths started the WHOLE stack first — giving the application a window
|
||
> to rebuild the schema objects the dump was about to create. Measured at 8 s on 2026-07-19
|
||
> (`DIAG-immich-restore-round2-2026-07-19`): immich-server rebuilt `clip_index` two seconds before
|
||
> the dump's `CREATE INDEX`, the replay aborted `already exists` under `ON_ERROR_STOP=1`, and immich
|
||
> then reported schema drift. The photos came back **by accident** — `pg_dump` emits COPY before
|
||
> CREATE INDEX, so the abort landed after the rows; a collision earlier in the script would have left
|
||
> a genuinely half-restored database, reported identically.
|
||
>
|
||
> **DECISION: the DB-only bring-up is done by compose SERVICE scoping**, not by container tricks —
|
||
> `StartStackServices(name, []string{svc})` → `compose up -d <svc>`. Every catalog template's
|
||
> dependency direction is app→db, so naming the DB starts the DB and nothing else. `docker start
|
||
> <ctr>` was never an option: `StopStack` is `compose down`, so the containers no longer exist.
|
||
> `RestartStack`/`RedeployFromEnv` are traps here — both end in a full `up -d`.
|
||
>
|
||
> **DECISION: fail-closed.** A `.sql` dump with no identifiable DB service refuses BEFORE the first
|
||
> mutation, on both paths (one Hungarian string, shared). The alternative would be to start everything
|
||
> and replay into the race. It should be structurally unreachable — `dbTypeForImage` is now shared by
|
||
> `DiscoverDatabases` and `DBServiceNames`, and a dump can only exist because discovery matched the
|
||
> container's image, which IS the compose `image:` value — so this is the belt for template drift.
|
||
>
|
||
> Enablers: `RedeployFromEnv` split into `PersistUnitRedeployConfig` (persist, starts nothing) + the
|
||
> unchanged tail; `StackDataProvider.RecreateStackFromUnit` renamed to
|
||
> `RecreateStackDefinitionFromUnit` because the old name promised less than the method did — the
|
||
> hidden `up -d` inside it is what carried the defect on the local path. `StartStackServices` REFUSES
|
||
> an empty list (argument-less `up -d` is a full start). **No agent coupling — MinAgent stays 0.90.0.**
|
||
> 19 new tests, 3 red-proofs, 23/23 green. **NOT live-validated yet:** STOP-1 supervised reconstitute,
|
||
> golden 0.153.0 bake (P3 registry-reachability probe from the vacation site is load-bearing), Viktor's
|
||
> two hub saves, and his C6 customer-restore UI run.
|
||
|
||
|
||
> **2026-07-20 — v0.152.0 + felhom-samba 1.1.0 (Megosztás on a Mac).** Closes **S-3**. **A capture
|
||
> on the box overturned the earlier guess:** macOS DOES send a correct NBNS query for `<NÉV><20>` and
|
||
> nmbd DOES answer it correctly in 140 µs (flags `0x8580`, RCODE=0, right address) — macOS simply
|
||
> never acts on it. NetBIOS there feeds legacy browsing, not `smb://` URL resolution, so **the bare
|
||
> `smb://<NÉV>` can never work from a Mac** and nmbd was never the broken part (it is what serves
|
||
> Windows). felhom-samba 1.1.0 adds **avahi + dbus**, templating `avahi-daemon.conf` and the
|
||
> `_smb._tcp` service file from `FELHOM_SERVER_NAME` so a rename re-advertises; both daemons are
|
||
> non-fatal on failure. v0.151.0's card had offered `smb://<NÉV>` for Mac — the one dead form — now
|
||
> `smb://<NÉV>.local`; Windows keeps flat `\\<NÉV>`. Spiked live by hand and confirmed from the
|
||
> operator's Mac BEFORE publishing the image (the operator's call, and it chose the design too).
|
||
> **STILL OPEN: Finder-sidebar discovery is NOT shipped** — the record is published and answers
|
||
> browse queries, but was never observed working; likely a Finder Settings → Sidebar toggle, but
|
||
> unverified. **Windows was not retested.** Two test bugs fixed en route, neither a production
|
||
> defect: `TestRenderSambaCompose` pinned a literal image tag, and `TestFabUpload_GCAndIdleTimeout`
|
||
> asserted an async unlink synchronously (it passed alone, failed in the full package once the new
|
||
> render tests made `web` heavier). 23/23 green twice; 2 red-proofs.
|
||
|
||
> **2026-07-20 — v0.151.0 (Megosztás).** Closes **S-1/S-2/S-4-core/S-5** of
|
||
> `felhom.eu/documentation/audits/DIAG-sharing-2026-07-20.md`; **S-3 (no mDNS/Bonjour) stays OPEN**,
|
||
> awaiting Viktor's `smbutil lookup FELHOM` + `dns-sd -B _smb._tcp` from the Mac. **The `/sharing`
|
||
> page had been reload-looping at ~1.2 s for every customer with sharing enabled since v0.147.0** —
|
||
> `/sharing/status` coerced `idle`→`running` on the JOB phase channel, and the client answers a
|
||
> terminal `running` with a one-shot `location.reload()`, so the first poll of every steady-state
|
||
> page load re-armed it. The rule this leaves behind, now recorded against R-45 too: **a phase a
|
||
> client answers with a one-shot action is an EDGE — never synthesise it from a level, and serve it
|
||
> exactly once.** Both halves are server-side; `sharing.html`'s `<script>` is byte-identical to
|
||
> v0.150.0. `consumeIfRunning` is the serve-once half (`failed`/`needs_password`/in-flight are never
|
||
> consumed). Also new: the „Csatlakozás a megosztáshoz" card — Windows + Mac forms plus the direct
|
||
> `smb://<IP>`, read from the SAMBA container's netns because the controller is on a docker bridge
|
||
> and would answer `172.x`; **derived per render, cached nowhere** (the guest holds it by DHCP).
|
||
> Live-verified endpoint-level on 9201: `phase:"idle"` on 3/3 polls, and the card renders the real
|
||
> `192.168.0.104`. 23/23 green twice; 3 red-proofs. **The human "does the page sit still" check is
|
||
> still Viktor's** — no browser here.
|
||
|
||
> **2026-07-20 — v0.150.0.** **`go test ./...` on DooPlex is fully green again (23/23 packages, run
|
||
> twice) — no surviving `t.Skip`s, no weakened assertions, no deleted tests.** The 7 red
|
||
> `TestTier2V2_*` / `TestSharesTier2*` were all one environmental class: Tier-2's off-drive guard
|
||
> uses `system.SamePhysicalDevice` (st_dev equality), and every `t.TempDir()` here shares one
|
||
> filesystem, so the fixture's "two drives" looked identical and the guard correctly refused the
|
||
> target (`nincs másik fizikai meghajtó`). Fixed with a nil-defaulted `Manager.samePhysicalDevice`
|
||
> seam + `sameDevice` wrapper — **production behaviour is byte-for-byte unchanged** (nil → the real
|
||
> check); only the two fixtures inject a fake. All 7 mutation-proved. **F7/R-53 shipped:**
|
||
> `app_export.html` built the app URL from `{{$.CSRFToken}}`; now `{{$.Domain}}`, with
|
||
> `exportPageHandler` supplying the key (it bypasses `baseData`). Host: the orphaned `dhclient` on
|
||
> the non-existent `eth0` was killed and did not respawn. Still open from the arc: R-50 (durable F1,
|
||
> spike-first — its ROADMAP note about needing a new cert SAN was **corrected**: the pin is a raw
|
||
> leaf-DER SHA-256 with `InsecureSkipVerify`, so SAN never enters it), R-51, R-52, R-39(b)/F6.
|
||
|
||
> **2026-07-20 — remote-site remediation + v0.149.0.** **F1 is MITIGATED FOR THE WINDOW, not durably
|
||
> fixed:** `vmbr0` on the demo host is now **static `192.168.0.162/24`** (was DHCP; the remote router
|
||
> had handed it `.147`, and the agent binds that literal), applied with `ifreload -a`; the agent came
|
||
> up clean and the red „a tárolókezelő ügynök nem elérhető" banner is gone. The control plane is
|
||
> still pinned to a LAN literal — the durable fix (host-internal island bridge) is a separate
|
||
> spike-first arc, **R-50**. Restoring the agent immediately let the quiesce loop run the overdue
|
||
> whole-guest backup by itself (**F2 closed**), a manual app-data run followed (2 DBs, 3 volume dumps,
|
||
> 43 s), and **Immich is back** (`photos.demo-felhom.eu` → 200; it had been left `Exited` by the
|
||
> pre-transport shutdown, not by the offsite-restore test). **v0.149.0 fixes F3** — the dashboard card
|
||
> said „Utolsó mentés: Még nem futott" on every box because `dashboardHandler` never passed the
|
||
> `BackupStatus` key the template branches on. F4/F5/F6/F7 are roadmap-only (**R-51/R-52/R-53/R-54**).
|
||
> Evidence: `felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md` §Remediation.
|
||
|
||
> **2026-07-20 — demo box moved to a remote site until ~2026-08-02; `ssh felhom-pve` = tailnet
|
||
> 100.70.170.35 (direct, ~37 ms). THE HOST AGENT IS DOWN THERE:** its `localapi` binds the literal
|
||
> `192.168.0.162`, the host now DHCPs `192.168.0.147` → `bind: cannot assign requested address`, so
|
||
> the service has never started at the remote site and the controller's `agentapi` dials of the same
|
||
> literal get `no route to host` — that is the whole "A tárolókezelő ügynök nem elérhető" banner, and
|
||
> it kills storage/PBS-backup/quiesce/restore-test/DR until fixed (Viktor GO: config **and** guest
|
||
> bootstrap state). Calibre-Web and `immich-server` were left `Exited` by the pre-transport shutdown
|
||
> and never came back despite `unless-stopped`; Calibre-Web was restarted via the real UI endpoint,
|
||
> Immich deliberately left down. Two code defects found and NOT fixed: **the dashboard's "Utolsó
|
||
> mentés: Még nem futott" is a display bug** (`dashboardHandler` never sets `BackupStatus`, so
|
||
> `dashboard.html:116`'s `{{if}}` branch is unreachable — it renders on every box regardless of
|
||
> history), and **multi-container apps under-alert** (`IsDownState` excludes `unhealthy`, so Immich's
|
||
> dead primary container produced no banner and no `app_start_failed` event for 18 h).
|
||
> Full evidence + ranked findings: `felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md`
|
||
|
||
> **2026-07-19 — v0.148.0: R-43 + R-44.** Viktor deleted 11 immich photos to test offsite restore;
|
||
> both runs flashed success and the photos stayed gone (`DIAG-immich-restore-2026-07-19`). Two
|
||
> defects. **R-43:** no offsite path could restore a DATABASE — all three buttons were file-only, so
|
||
> a DB-indexed app got its bytes back and still could not see them. **R-44:** a manual push shipped
|
||
> whatever dump the 02:30 local run left; that day's predated the customer's account by four hours
|
||
> and held zero users and zero assets inside 52MB of shipped geodata.
|
||
>
|
||
> Now: every run (manual AND nightly) dumps FIRST, then captures, so each snapshot is a coherent
|
||
> `{DB@T, files@T}` pair stamped with `offsite_run_id`; and „Teljes visszaállítás (fájlok +
|
||
> adatbázis)" does safety-dump → stop → overwrite files → start → replay the snapshot's dump.
|
||
> **Two invariants: nothing is ever deleted, and the undo is verified on disk before the act.**
|
||
> Warn-level honesty surfaces for stale/empty-looking pairs — never gates.
|
||
>
|
||
> **The live acceptance has NOT run yet** (§9: upload → push → empty trash for real → one button →
|
||
> photos back). Until it does, the capability-map offsite row stays PARTIAL/scope-contested, the
|
||
> customer-restore row stays MISSING, and R-3 stays DRAFT. Floor raise = Viktor's click, BEFORE the
|
||
> acceptance run. Still open: the `00-capability-map.md:61` ruling — did CAMPAIGN-6D's "immich
|
||
> end-to-end from offsite alone" exercise the DB half, or only the file half?
|
||
|
||
> **2026-07-19 — v0.147.0 → v0.147.3: feedback slice 1.** The systemic complaint, twice in one
|
||
> evening: you press a button and nothing happens. Three worst offenders fixed on the two patterns
|
||
> that already existed (deploy 3-step panel; storage-init status poll). **Deliberately NOT a
|
||
> framework** — that is ROADMAP **R-45**, and the two lessons it must encode are already written
|
||
> down there: a terminal state must be **probed, not inferred** (`compose up -d` exits 0 on a
|
||
> crash-loop), and a progress source reporting nothing is **normal, not broken** (restic reports 0
|
||
> bytes for a whole incremental run).
|
||
>
|
||
> **4a** — the offsite verification restore names its **full path** in the flash, and
|
||
> `/backups/restore` lists existing verification copies (app · size · date · path) with a
|
||
> double-confirmed per-copy delete. That delete takes a **stack name, never a path**; red-proofed
|
||
> (neutralise the name guard and `stack:""` resolves to the offsite-restore ROOT and takes every copy
|
||
> with it). `backups/offsite-restore` now has ONE home, `offsiteRestoreRootFor`.
|
||
> **4b** — Megosztás enable/password no longer reconcile inside the POST; detached job + poll, with
|
||
> „képfájl letöltése" vs „indítás" decided BEFORE the work starts (afterwards the image is always
|
||
> present and the distinction is unrecoverable).
|
||
> **4c** — „Távoli mentés most" streams restic `--json`. **Manual only**; the nightly stays silent,
|
||
> pinned by a test.
|
||
>
|
||
> **Three of the four versions exist because the cards were watched against real runs on the demo
|
||
> box** — none of these would have surfaced from unit tests: (.1) an incremental run reports 0 bytes
|
||
> for its whole duration, so a byte-only bar looks hung in the COMMON case; (.2) restic 0.14 counts a
|
||
> file only when it completes, so one big archive freezes the file counters too — fall back to
|
||
> current file + elapsed; (.3) the run does not end with the last app — the shares leg and
|
||
> `forget --prune` took 40 of a 57-second run, and the card froze on the last app until phases were
|
||
> added.
|
||
>
|
||
> Also: `infra.Images()` + `--print-infra-images` close the golden/controller infra-image drift at
|
||
> the source. The golden's own copy had already drifted (felhom-samba missing → 3 of 4 baked), which
|
||
> is **why** enabling Megosztás pulled at runtime in the first place. Effective at the next bake
|
||
> (`felhom-agent` build-golden v2.1.0); no golden rebuilt. **Floor NOT raised — Viktor decides.**
|
||
>
|
||
> Earlier: 2026-07-18 (v0.145.0 — R-7b: share data enters the live backup runs, Model B′)
|
||
|
||
> **2026-07-18 — v0.145.0: R-7b — the „Felhőmentés" toggle is now TRUE (Model B′), + samba liveness.**
|
||
> Until v0.144.0 a share could be marked „Felhőmentés: bekapcsolva" while its files were in NO backup:
|
||
> both engines are recovery-unit shaped (`RunTier2` short-circuits on `os.Stat(unitDir)`; the offsite
|
||
> runner enumerates `GetOffboxApps()`) and a share-only infra stack has neither. **Viktor's ruling was
|
||
> Model B′: a SIBLING shares source** — additive job/leg code reusing the proven primitives (tier-2
|
||
> mirror seam, restic wrappers, quota gate, status recorders) while every per-app engine path stays
|
||
> **byte-identical**. That invariant is enforced by test in both tiers, red-proofed.
|
||
>
|
||
> Tier 2 → `RunSharesTier2` (legs grouped by SOURCE drive → `backups/secondary/_shares/<driveKey>/…`,
|
||
> payload at `_payload/`, marker LAST). Tier 3 → `runOffboxSharesLeg`, ONE extra restic call tagged
|
||
> `_shares`, placed after the app loop and BEFORE retention so `--group-by host,tags` covers it for
|
||
> free. Restore → „Megosztások" on /backups/restore: scratch, then a missing-only merge whose every
|
||
> destination is PREFIX-ASSERTED against live storage roots; definitions merge existing-wins; then
|
||
> `ReconcileSamba`; then the credential, best-effort.
|
||
>
|
||
> The **payload** is the point: a byte-deterministic `_shares-manifest.json` + a best-effort
|
||
> secret-bearing `passdb.tar`, so DR returns files + configuration + password, not loose bytes. A
|
||
> quota-blocked offsite push degrades to the MANIFEST ONLY — never to nothing.
|
||
>
|
||
> **Findings:** (a) the reserved-name assumption was FALSE — `nbNameRe` accepted „_shares" as a share
|
||
> name; `ValidateSMBShareName` now refuses a leading underscore and both run loops skip a `_shares`
|
||
> stack loudly. (b) The samba-liveness fold-in needed NO alert/e-mail pipeline change and adds no new
|
||
> event type (so the `allowedEventTypes` gotcha does not apply) — `EffectiveProtected` just gains a
|
||
> settings-backed dynamic extra watching the CONTAINER name (`felhom-samba` ≠ stack name `samba`).
|
||
> (c) A real bug surfaced in review-by-test: `shareSourceDrive` returned a slash-normalised path,
|
||
> making the target selector's equality check miss so a share group could target its own source drive.
|
||
>
|
||
> **OPEN:** Viktor raises the managed-update floor to **v0.145.0** (supersedes the 0.144 note) so the
|
||
> N100 rehearsal's day-0 box converges onto the honest version. Pre-existing, untouched:
|
||
> `docker_run_volume_path_gate` fails on `internal/appexport/estimate.go` (predates this work).
|
||
|
||
> **2026-07-18 — v0.144.0: „Megosztás" LAN SMB sharing (R-7 slice 1), LIVE on demo.** SMB ships as an
|
||
> **embedded controller feature** — the FOURTH protected infra stack (traefik/cloudflared/filebrowser/
|
||
> **samba**), NOT a catalog app (it needs `network_mode: host` per the R-6 spike, its config is a
|
||
> generated share list, and its roots ride the backup classification). New own image
|
||
> **`felhom-samba:1.0.0`** (pinned alpine + smbd + **nmbd** + wsdd + tini; smb.conf bind-mounted
|
||
> read-only, nothing templated inside, passdb on a volume). nmbd is REQUIRED alongside wsdd — the R-6
|
||
> spike proved wsdd-only leaves the box visible but the Explorer double-click fails `0x80070035`.
|
||
> New top-nav category „Megosztás" → „Hálózati megosztás": enable + one household SMB password
|
||
> (STDIN→smbpasswd, NEVER persisted — only `user_set`), shares table, and a create flow (new folder
|
||
> under `<storage>/shares/` or an existing folder via a guarded browse modal). Every customer path goes
|
||
> through `sharingResolvePath` (absolute → EvalSymlinks → containment in a registered LIVE storage root
|
||
> → deny-listed system subtree → is-a-dir) with **uniform** refusals so the picker is never a
|
||
> filesystem oracle; the deny-list is DERIVED from `ProtectedHDDPaths` (provably a subset).
|
||
> `ClassifiedBinds("samba")` resolves from the shares registry: Felhőmentés ON → mandatory
|
||
> (offsite+tier-2), OFF → optional (tier-2 only), with ZERO backup-engine edits.
|
||
> **OPEN / needs a Viktor ruling (suggested R-7b):** the classification seam is correct but share data
|
||
> is **not in any live backup run** — `RunTier2` short-circuits on the missing recovery unit before it
|
||
> reaches the seam, and the offsite runner enumerates `GetOffboxApps()`. Both engines are recovery-unit
|
||
> shaped; teaching them about a share-only stack is a structural change, so it was reported as a design
|
||
> fork rather than improvised (task STOP clause). Live-validated end-to-end through the real endpoints
|
||
> + a Windows 11 workstation (445, NetBIOS `FELHOM` resolves, write/read byte-compare PASS, write to a
|
||
> read-only share REFUSED with no effect, SMB writes land as uid 1000). **Explorer leg PASSED (Viktor): both
|
||
> shares open, an interactive Explorer save landed as uid 1000, a write into the read-only share was
|
||
> refused. Slice 1 fully PROVEN-LIVE.** Also: samba is protected in CODE (`config.alwaysProtectedStacks`) because
|
||
> controller.yaml is golden-generated — a side effect is that `monitor.EffectiveProtected` does NOT
|
||
> monitor samba liveness (deliberate: no false alarms while off; see REPORT §9).
|
||
|
||
|
||
|
||
> **2026-07-17 — v0.143.0: guest RAM resize UI (R-24), LIVE on demo. MinAgent: 0.90.0.** The customer
|
||
> sees the guest's current memory + allowed range on the **Rendszer** settings page and resizes it
|
||
> ("Szerver memória (RAM)" card). The controller proxies + maps the agent's machine `code` to Hungarian;
|
||
> the AGENT (felhom-agent v0.90.0) enforces every bound and applies live (no reboot — R-24's old
|
||
> hub-desired-state framing is SUPERSEDED by Viktor's controller-direct ruling). agentapi
|
||
> `GuestMemory`/`ResizeMemory` (`*MemoryRefusedError` carries the code); capability
|
||
> `FeatureGuestMemoryResize` (featureMinAgent 0.90.0, probe type-asserts GuestMemory so the shared
|
||
> SupportProber/netAgent stay untouched); `POST /api/system/memory/resize`; JS confirm only on a shrink;
|
||
> agent-outdated hides the control, agent-unreachable falls back to the guest's `/proc/meminfo`. Deployed
|
||
> to BOTH demo guests (felhom-pve 9201 + nested demo-vm-felhom-4846bc 9201). **LIVE-validated** through the
|
||
> real endpoint on the nested demo (above_max + below_min refusals render the Hungarian message; the agent
|
||
> English never leaks; the capability gate resolves SupportYes via the 0.90.0 version header). A successful
|
||
> grow couldn't be shown on the tiny 4 GB nested host (max<current, correctly refused); the apply is
|
||
> Phase-0-proven at the agent layer. See REPORT.md.
|
||
|
||
> **2026-07-17 — v0.142.0: offsite repo continuity (Parts A + C), LIVE on demo.** Closes the
|
||
> reinstall-orphaned-repo incident (a recreated data volume mints a new repo passphrase → the offsite
|
||
> repo, keyed under the old one, errors nightly with `wrong password or no key found`). Part A:
|
||
> `ensureOffboxRepo` classifies the `cat config` failure → ORPHANED state + calm Hungarian card
|
||
> (exception color) + `offbox_repo_orphaned` event (once, not nightly); reset = move-aside (never
|
||
> delete, `mv <repo> <repo>.orphaned-<date>`) + init — UNCLAIMED auto, CLAIMED reveal-then-confirm.
|
||
> Part C: `GET /backup/offbox/status` + poll on backups_remote flips Fut→Rendben/Hiba without a manual
|
||
> reload. Pairs with hub v0.60.0 (superseded-escrow retention). Live leg staged for the rehearsal
|
||
> (scratch-target swap would disturb the live escrow state; live repo untouchable) — mechanism covered
|
||
> by 3 fake-based scenarios + 2 red-proofs. Details: REPORT.md.
|
||
|
||
> **2026-07-17 — v0.141.0: N100 polish (F6 + F7), LIVE on demo.** F6 (MEDIUM): drive "initialize"
|
||
> now ends in a mounted+registered drive even on a client disconnect. `POST /api/storage/init` runs
|
||
> the format→mount→register chain as a DETACHED single-flight job (`web/storage_init_job.go`,
|
||
> `context.Background()`, netAddState shape) the wizard polls via `GET /api/storage/init/status`
|
||
> (3-step Hungarian progress); register is the last step (marker-last crash-safety). Fork verdict:
|
||
> controller-side, NO agent change (the chain must reach FileBrowser sync = controller-only).
|
||
> **Deeper half found on the live leg:** a slow mkfs (64 GB USB, ~27 s) outruns the agentapi client's
|
||
> 15 s timeout → the controller now polls the agent's `GET /disks/format/status`
|
||
> (`agentapi.FormatStatus` → `awaitAgentFormat`) then continues. Live-validated on `/dev/sdd` →
|
||
> `/mnt/felhom-drives/scratch1` (mounted+registered). F7 (LOW): storage init/attach Vissza → `/storage`.
|
||
> Red-proofs for both F6 halves + F7. Security review of the commit flagged the pre-existing
|
||
> format→resolve→assign device-node TOCTOU (agent-guarded destructive step, benign fs-UUID mount) —
|
||
> acknowledged as an Observation, not expanded. Fork/landmarks/live evidence: REPORT.md.
|
||
|
||
> **2026-07-16 — v0.140.0: Direction-2 immediate-sync (hub→box) SHIPPED.** The reverse of v0.139.0:
|
||
> an OPERATOR action on the hub now reaches the box in seconds. `report.Waiter`
|
||
> (`internal/report/waiter.go`) holds a hanging `GET {hub}/api/v1/wait?gen=N` (same hub URL+key as
|
||
> the pusher — no new config keys) against hub ≥ v0.58.0's in-memory operator-intent generation
|
||
> counter; on a generation CHANGE it fires the v0.139.0 `report.Trigger` and nothing else, so the
|
||
> report ACK delivers everything through the UNCHANGED machinery (the box pulls even the wake-up).
|
||
> No overall client timeout (held GET); first-observation records-not-fires (no restart echo);
|
||
> same-gen timeout fires nothing; errors incl. a 404 from a pre-v0.58.0 hub back off 5s→5min while
|
||
> the 15-min cycle reconciles. Gated on the SAME `hubPusher!=nil && Hub.Enabled` as the trigger.
|
||
> Copy soften: backups_remote/escrow "néhány **másodperc**, legfeljebb 15 perc" (15-min bound stays
|
||
> as the honest worst case). Red-proof: baseline-branch-off → first-obs fires (reverted). Agent-plane
|
||
> poke is PARKED in the OOB arc (spike P4). Deployed to 9201; live-validated (hold + immediacy).
|
||
> Detail: `CHANGELOG.md` v0.140.0, `controller/README.md` §9, `REPORT.md`.
|
||
|
||
Last updated (prior): 2026-07-16 (v0.139.0 — immediate out-of-cycle hub report, Direction 1)
|
||
|
||
> **2026-07-16 — v0.139.0: immediate out-of-cycle hub report on user actions (Direction 1
|
||
> SHIPPED; Direction 2 pending SPIKE-immediate-sync-transport).** Viktor's ruling: user actions
|
||
> with hub-side effects round-trip in seconds. New `report.Trigger` (`internal/report/trigger.go`):
|
||
> buffered-1 chan + worker, quiet 2 s → drain → min-interval 15 s → ONE full BuildReport+Claimed+
|
||
> Push; trailing-edge coalescing (burst ≤ 1+ceil(burst/15 s) pushes, last state always lands),
|
||
> no own retries, failures degrade to the UNTOUCHED 15-min cycle. Generalizes the v0.70.0 geo
|
||
> `reportPushNow` seam (raw goroutine in main.go replaced by the debounced trigger). Wired: geo
|
||
> save/sync + app deploy/remove/delete (api), and via `web.SetReportTrigger`/`reportTriggerNow`
|
||
> (nil-safe, AFTER successful local commit only): escrow recovery-code claim (headline — the
|
||
> v0.138.0 "megerősítésre vár" card now collapses in seconds via the unchanged EscrowAutoConfirmer
|
||
> ACK hash-match), notification-prefs save, app-email toggle, offsite config + per-app toggle,
|
||
> customer claim. `hub.enabled:false` → seams nil → strict no-op. NO hub change, NO UI copy change
|
||
> ("legfeljebb 15 perc" stays the honest worst case; post-live-proof a soften to "általában néhány
|
||
> másodperc" is a later one-liner). Tests: trigger_test.go (2 red-proofs recorded),
|
||
> report_trigger_seam_test.go, report_trigger_nilsafe_test.go. Known pre-existing Windows-only
|
||
> test failures (appexport df=0, stacks paperless, web fab pipelines) verified failing on base
|
||
> 8f3564c too.
|
||
|
||
> **2026-07-16 — v0.138.0: escrow "awaiting hub confirmation" waiting state.** Fixes the customer-zero
|
||
> (N100) UX gap: after a completed escrow ceremony the Távoli mentés page kept showing the yellow
|
||
> "Helyreállítási kód szükséges" card for ~15 min until the next hub-report ACK flipped
|
||
> pending→escrowed. **Phase-0 diagnosis (read-only) = verdict A (report-cycle lag), already resolved on
|
||
> the box:** demo logs show the ceremony claimed `16:13:39`, the next `hub-report` ACK at `16:27:58`
|
||
> auto-confirmed via hash-match (`d517ce7f…`); `settings.json` = `escrow_state:"escrowed"`. **Hub
|
||
> Hypothesis B verified FALSE → no hub change:** `SaveHostEscrow`'s `ON CONFLICT … stale_at = NULL`
|
||
> already clears stale on upload (store.go:2084); withhold only fires while `stale_at != ""`
|
||
> (store.go:2150). **Part 1 (code):** new persisted `OffboxTarget.CeremonyCompletedAt` (stamped on the
|
||
> recovery-code claim, zeroed on the flip + manual confirm); `offboxCeremonyWaitState` +
|
||
> `escrowCeremonyGraceWindow`=35m; `backups_remote.html` gains an info "megerősítésre vár, legfeljebb 15
|
||
> perc" card → warn "a megerősítés nem érkezett meg" past the window; `backups_escrow.html` final step
|
||
> gains a "Mi történik ezután?" note. Test `web/escrow_wait_state_test.go` + red-proof. No
|
||
> scheduler/agent/hub/endpoint changes. Deploy 0.136.0→0.138.0 to 9201. First "Távoli mentés most" =
|
||
> Viktor's click (NOT done). Note: demo host key changed (box reprovisioned for N100) → known_hosts
|
||
> refreshed.
|
||
|
||
|
||
> **2026-07-15 — v0.137.0: cleanup bundle (email-wipe guard + carried hygiene).** Closes the arc's
|
||
> carried micro-queue. **Part 1 (code):** `settingsNotificationsHandler` now REFUSES a save with a
|
||
> blank email box while events are enabled (it would push empty to the hub → wipe the customer's
|
||
> provisioning-seeded alert address — the 2026-07-15 demo incident). Returns before
|
||
> SetNotificationPrefs + sync, Hungarian error, repaints submitted checkboxes; empty+zero-events
|
||
> clear-all still allowed. No HTML `required` (it would block the legit clear-all). Tests + red-proof
|
||
> in `web/notifications_guard_test.go`. Deployed 0.137.0 to 9201 (healthy). **Part 2 (hygiene):**
|
||
> removed the confirmed-older `felhom-flash/backups/primary/immich` recovery unit (CreatedAt 06-23 <
|
||
> live usb 07-15, 44M); STOPPED audiobookshelf/komga/romm on flash (CreatedAt TIED with usb →
|
||
> tie-break is drive-order-dependent, not confirmable — manual disposition pending). **Part 3:**
|
||
> campaign6 is a bare empty leftover dir (not a live mount); safe `rmdir` refused (Permission
|
||
> denied — autofs-ghost/immutable); no mount disturbed → left for Viktor's reboot window. **Part 4:**
|
||
> tagged the campaign6 6D-audit finding track-only (felhom.eu `dee72cd`).
|
||
|
||
> **2026-07-15 — v0.136.0: `.fab` exclusion scoping (Task 4).** Architecture §2 `.fab` row + SQ5
|
||
> verdict + R1-C. SQ6 over-capture FIXED for classified apps: the userdata root tar is exclude-scoped
|
||
> (keeps only ancestors/descendants of a SELECTED bind relpath — the tier2Reconcile keep-rule); no
|
||
> selected userdata bind ⇒ no root tar (radarr state-only). New `appbackup.ComputeFabBuckets` (shared
|
||
> `resolveGuardCollapse` pipeline; class buckets; guards over ALL classes; no cross-bucket containment).
|
||
> New `appexport/fabplan.go`: `computeFabPlan` (SkipMounts/SkipUserdataTar/UserdataExcludeRels) +
|
||
> `tarDirectoryExcluding` + `fabEstimateSplit`. `ExportRequest` += DeselectOptional/OptInExcluded (both
|
||
> start handlers — two-call-site); mandatory is a SERVER-SIDE floor. Manifest v1 + import UNTOUCHED;
|
||
> legacy apps byte-identical to v0.130.0. Estimate gains an additive class split; export UI shows
|
||
> locked-mandatory / optional-checkboxes / excluded-opt-in + the two-number warning. All 6 §10
|
||
> red-proofs verified. CAMPAIGN-6D Accept #1 (≥1 GiB .fab full circle) now runs against this shape.
|
||
|
||
> **2026-07-15 — v0.135.0: tier-2 engine rework (Task 3b).** Architecture §2/§8. New
|
||
> `tier2_capture.go`: classified apps get the `TierSecondary` per-bind legs (paperless copy shrinks —
|
||
> export drops); legacy apps keep the byte-identical resolver set. v2 relpath-mirroring layout
|
||
> (`backups/secondary/<stack>/{.felhom-tier2-layout marker LAST, recovery-unit/, hdd/<rel>/,
|
||
> userdata/<rel>/}`) — N>1 native (flat-appdata refusal + `errTier2MultiDir`/`tier2AppDataName`
|
||
> deleted). Migration=delete-and-rebuild + reconcile (prunes dest dirs a bind no longer covers); all
|
||
> `os.RemoveAll` via `tier2SafeRemove` (refuses outside backups/secondary/). SSD=state-only tier
|
||
> (unit+mandatory). `selectTier2Target` never picks NETWORK storage (pinned+auto, F-6C-1). Restore
|
||
> reads v2 behind a marker gate (pre-v2 refused); two-subtree missing-only merge. **Part 0:**
|
||
> offbox_enlarge_blocked is now a persisted one-time Load seed (`OffboxEnlargeNoticeSeeded`), NOT a
|
||
> getter append — opt-out STICKS (fixes the 3a-fix un-disableable checkbox). **Part 0.5:** offsite
|
||
> restore scratch prefers a local (non-network) path. Full v2 test suite + all 10 §10 red-proofs
|
||
> verified. Every destructive write bounded to backups/secondary/.
|
||
|
||
> **2026-07-15 — v0.134.1 (+ hub v0.55.0): Task 3a-fix.** Placement hardening in
|
||
> `offbox_restore.go`: F-3a-1a live target uses raw `GetStackHDDPath` (not `AppNamespaceRoot` — its
|
||
> systemDataPath fallback would merge userdata onto the SSD; empty ⇒ undeployed ⇒ refuse); F-3a-1b
|
||
> placement headroom gate; F-3a-4 stat PRE-PASS over all placements before any copy (no partial
|
||
> writes); F-3a-3 `mapOffsiteRestorePaths` refuses the namespace root itself; F-3a-2 scratch removed
|
||
> on success (place button then gone), kept on failure. Enlarge-blocked notification delivery chain:
|
||
> `DefaultEnabledEvents` + `GetNotificationPrefs` append-if-absent migration + settings checkbox +
|
||
> handler slice (controller), and hub v0.55.0 allowlists `offbox_enlarge_blocked` — NO
|
||
> customerMessages entry (raw dynamic message must survive). +8 controller tests, +2 hub; all §10
|
||
> red-proofs verified. Hub LIVE (ArgoCD synced, :0.55.0). Migration trade-off noted (getter re-enables
|
||
> on opt-out — future persisted marker). 3a deferred list shrinks after §13 live legs.
|
||
|
||
> **2026-07-14 — v0.134.0: offsite tier policy engine (Task 3a — FIRST behavior change).**
|
||
> Implements architecture §2/§6/§7/§9. Each toggled app's offsite push = ONE multi-path restic
|
||
> snapshot (recovery unit + TierOffsite mandatory userdata via `ComputeCaptureSet`); legacy/undeployed
|
||
> stay unit-only. New `offbox_capture.go` (`offboxCaptureSet` + loud gaps: restic 0.14.0 silently
|
||
> skips missing paths, SP-3.4) + `offbox_restore.go` (ID-first snapshot introspection, scratch off the
|
||
> rootfs + headroom gate F-A1, `RestoreOffboxScratch(full)` unit-only default via `--include`,
|
||
> `PlaceOffsiteRestore` missing-only merge, pure `mapOffsiteRestorePaths`). Quota → `stats --mode
|
||
> raw-data` (SP-1; **displayed size drops once after deploy**). Pre-push enlargement gate blocks the
|
||
> userdata enlargement over-quota (unit-only push continues; `OffboxTarget.EnlargedBlocked`;
|
||
> edge-triggered notify). `forget --group-by host,tags` on both sites (SP-2). UI: /backups/restore
|
||
> three actions (unit / full two-step / place-to-live); /backups/remote per-app blocked note. New
|
||
> route `POST /backup/offbox/place`. **HUB FLAG:** `offbox_enlarge_blocked` event needs hub
|
||
> allowlist+customerMessages for push delivery (in-dashboard LastWarning works now). +13 tests, all 10
|
||
> §10 red-proofs verified. NOT-live-yet (6D): PlaceOffsiteRestore, large full restore, live
|
||
> enlarge-block, notification delivery, SQ3 immich full-circle. Tier-2 (3b) + .fab (Task 4) untouched.
|
||
|
||
> **2026-07-14 — v0.133.0: capture-set computation (Task 3-core, INERT).** Task 3-core of the
|
||
> backup-classification-redesign arc (architecture `felhom.eu/documentation/architecture/07-backup-architecture.md`
|
||
> §3; spike verdicts `SPIKE-restic-snapshot-shape-2026-07-14.md`). New
|
||
> `appbackup/captureset.go`: pure `ComputeCaptureSet(binds, hasClassification, tier, hddPath)` →
|
||
> `CaptureSet{HasClassification, Paths []CapturePath, Skipped []SkippedPath}`. Pipeline: legacy
|
||
> short-circuit → tier filter (`TierOffsite`=mandatory only, `TierSecondary`=mandatory+optional,
|
||
> excluded dropped) → structural guards (traversal / bare HDD drive-root / reserved `backups/` →
|
||
> `Skipped` with English reasons; bare userdata allowed) → equal-Abs collapse (mandatory>optional) →
|
||
> containment dedup (keep ancestor) → sort by Abs. Slash algebra only (no `filepath`). Pure
|
||
> `CrossAppOverlaps` advisory (WARN wiring deferred to 3a/3b). **Deliberately INERT — no engine
|
||
> consumes it yet; 3a (offsite policy) and 3b (tier-2 rework) are the consumers.** ARCHITECTURE
|
||
> IMPACT from the spike (SP-3.4, already in §2.5): restic 0.14.0 does NOT error on a missing source
|
||
> path (exit 0, silent partial snapshot) → the stat-filter in 3a/3b is load-bearing. Wiring test
|
||
> through a real Manager (F-S3 no-seam); all 6 §10 red-proofs verified. felhom.eu §3 docs aligned
|
||
> (`8d85da7`).
|
||
|
||
> **2026-07-14 — v0.132.0: backup classification (Task 2, INERT).** Task 2 of the
|
||
> backup-classification-redesign arc (spike: `felhom.eu/documentation/audits/SPIKE-backup-classification-2026-07-14.md`).
|
||
> Ships the referential-coupling classification as DATA + PARSER + PURE CLASSIFIER, **deliberately
|
||
> inert** — no backup tier changes behavior. New `appbackup/classify.go`: `BackupSpec`/`BindSpec`
|
||
> (the `.felhom.yml` `backup:` block), `ComposeBind` (`${VAR}`-relative + `:ro`), `ClassifyBinds`
|
||
> (SQ5 two-level default: explicit beats `:ro`; unlisted writable→mandatory, unlisted `:ro`→excluded;
|
||
> **no block → legacy/false**), `ValidateBackupSpec` (whole-block-reject on any defect). New
|
||
> `stacks/classify_binds.go` `ParseComposeClassifiableBinds` (relative-space, keeps `:ro` — NOT
|
||
> `ParseComposeHDDMounts`/`ExportDataMounts`, the classifier-input traps). `LoadMetadata` is the
|
||
> SINGLE validation choke point (bad catalog block → nil + one `[ERROR]` within one sync cycle →
|
||
> legacy). Wired seam `Manager.ClassifiedBinds` + `StackDataProvider.GetStackClassifiedBinds`
|
||
> (nil-stubbed in every fake) so **Task 3 (tier policy engine)** consumes a tested seam, not a fresh
|
||
> one. Inertness proven: full pre-existing suite green with ZERO test-logic edits. The 13 catalog
|
||
> `backup:` blocks ship in the same `app-catalog-felhom.eu` change (controller deployed FIRST so the
|
||
> parser validates on first sync). audiobookshelf PENDING-VETO: media/audiobooks ruled **optional**
|
||
> (consistency with komga/romm) pending a Viktor veto to excluded. +14 tests, RP-1..RP-4 confirmed.
|
||
> **Next: Task 3** consumes `ClassifiedBinds` to scope offsite/tier-2/`.fab` capture by class.
|
||
|
||
> **2026-07-14 — v0.131.0: F-S2 + F-S3 (compose-derived appdata dir resolution).** Task 1 of the
|
||
> backup-classification-redesign arc (spike: `felhom.eu/documentation/audits/SPIKE-backup-classification-2026-07-14.md`).
|
||
> The controller assumed `appdata/<stackName>`; paperless-ngx writes `appdata/paperless` (stack
|
||
> `paperless-ngx`). ONE canonical resolver `appbackup.AppDataDirNames(hddPath, stackName, mounts)`
|
||
> derives the real dir name(s) from compose `${HDD_PATH}` binds (deduped/sorted; fallback `[stackName]`);
|
||
> all consumers use it. **F-S2** (spike-proven): `RunTier2`/`Tier2Info`/`RestoreTier2Files` now hit the
|
||
> resolved dir (paperless documents got NO tier-2 copy before — the appdata leg stat-skipped a dir that
|
||
> never existed). **F-S3 (NEW, found this session):** `migrate.go` keyed all six per-app appdata legs by
|
||
> stack name; **scope="app"** has no merge walk, so migrating paperless-ngx copied nothing, verified
|
||
> vacuously, flipped HDD_PATH → **empty media dir** (scope="all" was saved by the merge walk — data safe,
|
||
> accounting off). All six legs now loop resolved names. **Multi-dir (N>1) refusal** is defensive (no
|
||
> catalog app hits it today: immich/nextcloud/romm match, paperless mismatches, each app = exactly ONE
|
||
> dir): tier-2 backup/info/restore refuse loudly (Hungarian); **migrate supports N naturally**. This
|
||
> limitation is **deferred to Task 3 (tier-policy engine)**, which owns the destination layout. Storage
|
||
> page sums resolved dirs. Truth repairs: the v0.130.0 CHANGELOG/CONTEXT "tier-2 copies the namespace
|
||
> wholesale" claim is FALSE — corrected in the v0.131.0 CHANGELOG entry + `main.go` export-adapter
|
||
> comment; tier-2 copies the recovery unit + resolved `appdata/<name>` ONLY (NOT userdata — F-S1,
|
||
> unaddressed here). New seam `tier2Mirror`; `migSeams.resolveNames`. +9 tests, RP-1..RP-5 all
|
||
> confirmed. Controller-only, no agent/hub coupling. **NOT live-validated here:** scope="app" migration
|
||
> of a real app between drives (F-S3 live proof — supervised leg, Viktor's session).
|
||
|
||
> **2026-07-14 — v0.130.0: CRITICAL C6B-F1 (hollow .fab export) + C6B-F2 (share-removal guard).**
|
||
> CAMPAIGN-6B proved `.fab` export shipped **config-only, data-free bundles** for 12/13 `needs_hdd`
|
||
> catalog apps (sonarr 4.17 GB → 2308 B, success, past the v0.125.0 guard). Three compounding fixes
|
||
> (all red-proofed run→fail→revert): (1) `stacks.ExportDataMounts` — export mount discovery unions
|
||
> `${HDD_PATH}` binds + the `${USERDATA_PATH}` **ROOT** (single `userdata` entry; root-not-per-bind
|
||
> is LOAD-BEARING: the manifest keys tars by basename and the untouched import maps basename →
|
||
> `<HDD_PATH>/<subdir>` — per-bind subpaths would restore to wrong places; this deviates from the
|
||
> task's literal per-bind+namespaced-names instruction, which could not round-trip without import
|
||
> changes the task forbade); (2) export + estimate are ADDITIVE for `needs_hdd` apps (HDD data AND
|
||
> named volumes — sonarr_config was silently dropped); (3) anti-hollow guard: `needs_hdd` manifest
|
||
> with zero data fails loudly. Plus §8: basename collision between mounts = loud Hungarian failure
|
||
> (was silent overwrite). **C6B-F2:** `netstorage/remove` refuses 409 while a DEPLOYED app's
|
||
> HDD_PATH is on the share (the orphaned-autofs trigger); resolves via the netAgent seam.
|
||
> **Residual flagged for a felhom-agent task:** RemoveNetworkMount's tolerate-and-continue stop
|
||
> (felhom-agent netmount.go:434-443) still deletes unit files under a busy mount if some non-product
|
||
> path calls it. Scheduled/tier-2 backup path was NOT affected and is untouched (`stackAdapter`
|
||
> deliberately unchanged). CAMPAIGN-6C's first acceptance test = the full-circle byte-compare this
|
||
> unblocks.
|
||
|
||
> **2026-07-13 night — v0.128.1 + demo storage hygiene (ruling F5).** `classTag` suppresses the
|
||
> rotational class hint for `type==='usb'` (card already carries the USB tag; hub `ClassHint`
|
||
> UNCHANGED; pinned by `TestStorageTemplate_USBClassBadgeSuppressed` + red-proof). Host op on
|
||
> demo-felhom: the two pre-intermediary legacy `dir:` storages (`felhom-usb`, `felhom-flash`,
|
||
> content=Backup, is_mountpoint) RETIRED via `pvesm remove` after G1/G2/G3 gates all PASSED
|
||
> (agent-owned UUID .mount units; both `enrolled` in drive-intents.json; zero /etc/pve refs, empty
|
||
> dump/, no customer app on either drive). Post-removal: mounts+binds intact (marker round-trip
|
||
> through the guest), `GET /disks` shows both registry-sourced (role+durable-id intact, class
|
||
> absent), `pvesm status` clean. **The demo node now matches the fresh-install storage shape** —
|
||
> drives are registry+units-sourced only, no legacy PVE dir: storages. v0.128.1 LIVE on demo 9201
|
||
> (drill guest skipped — optional, no behavioral dependency; it runs 0.128.0).
|
||
|
||
> **2026-07-13 night — v0.128.0: CHUNKED BROWSER .FAB UPLOAD on /import (ruling F3: chunked).**
|
||
> Step-0 probe on the real tunnel PROVED the Cloudflare edge cap (120 MiB POST → edge 413 with
|
||
> `Server: cloudflare` on Content-Length alone; 80 MiB → origin 302 /login; local DNS overrides
|
||
> the hostname to the LAN guest, probe needed `--resolve` onto CF's public IP). Design: JS
|
||
> `File.slice` 64 MiB strictly-sequential chunks → `POST /api/export/upload/{init,chunk,finalize,
|
||
> abort}` inside `ServeExportAPI` (inherits RequireAuth+CsrfProtect; single-flight; offset must
|
||
> equal received else 409+echo; 96 MiB request cap; free-space gate size+1 GiB; finalize =
|
||
> exact-size + fsync + atomic rename, collision → lowest-free `"name (N).fab"`). Lands in the
|
||
> DEFAULT drive's exports dir — scan/validate/import pipeline untouched. No client hash
|
||
> (deliberate: .fab self-validates). In-memory state: startup GC of `*.part-*`, 15-min idle
|
||
> abort. §7 A–F tested + 3 red-proofs. `appexport.DiskFree` exported (REUSE.md row).
|
||
> **NOT live-validated: the end-to-end multi-GB browser upload through the real tunnel needs a
|
||
> dashboard login → Viktor's 5-minute leg (export an app to .fab, download, re-upload, import —
|
||
> full circle).** Possible follow-up if it itches: per-drive target picker (v1 = default drive only).
|
||
|
||
> **2026-07-13 eve — v0.127.0: CUSTOMER-FACING ESCROW CEREMONY WIZARD (/backup/escrow) +
|
||
> Scenario-F stale-blob re-check. MinAgent 0.88.0 (wizard only). LIVE on demo 9201 + drill guest
|
||
> (both healthy).** The friend-alpha missing piece: preflight → warnings → password re-auth
|
||
> (login rate limiter) → **re-stage-first** (abort on failure — the UI can never mint a hash-less
|
||
> blob) → agent job (poll 2 s) → ONE-SHOT R reveal (no-store; R only in the claim XHR + page JS;
|
||
> 10-min TTL → void) → typed-back (two random words) → finish. Ruling F1: R over the CF tunnel
|
||
> once = accepted (threat model in felhom.eu RUNBOOK-escrow-ceremony.md). Scenario F: an ESCROWED
|
||
> box re-checks the ACK hash — mismatch/hash-less ⇒ stale flag (card warning + CTA) + one WARN
|
||
> per hash; never flips, never blocks; **fired LIVE on both boxes' hash-less blobs at first ACK**
|
||
> (drill = the spike's superseded blob, since REPAIRED via a real ceremony —
|
||
> `restic_pw_sha256` now covers; demo = its legacy blob, warning stays until a wizard run).
|
||
> Manual-confirm BUTTON removed (endpoint stays, deprecated). **OPEN: one supervised full-browser
|
||
> wizard pass with Viktor's login (re-auth needs the customer-owned password — CC validated
|
||
> everything beneath it endpoint-exact); demo wizard run to clear its stale warning.**
|
||
|
||
> **2026-07-13 — v0.126.0: UI UNIFORMITY BUNDLE (shared app-list rows + infra identity +
|
||
> restore-form polish + mojibake gate + honest stale line). Presentation-layer only — NO
|
||
> backup/toggle/engine behavior change. MinAgent 0.81 + floor unchanged.**
|
||
> (A) `templates/app_row.html` `app_list_row`/`app_list_row_end` is THE canonical list row
|
||
> (icon+name left, caller action right, compact 44px) — dashboard installed-apps, Távoli mentés
|
||
> toggles, Visszaállítás restore-to-verify + .fab lists render through it; the backups-apps
|
||
> expander header is ALIGNED (own markup, allowlisted); gate `scripts/app_row_dedup_gate.py`
|
||
> (red-proven). funcmap: `dict`/`appHref`/`infraMeta`. (B) `inframeta.go`: cloudflared →
|
||
> „Cloudflare Tunnel", traefik → „Traefik", filebrowser → „FileBrowser" + Hungarian descriptions
|
||
> + generic `/static/infra-logo.svg` fallback; filebrowser = the ONLY Linked infra
|
||
> (files.<domain>); render test counts exactly one customer link (red-proven). (C) .fab password
|
||
> field standard („Opcionális jelszó" + helper; import-page input got `.form-input`).
|
||
> (D) `scripts/mojibake_gate.py` — templates+Go strict UTF-8, zero Ã/Â/Ă-signature chars,
|
||
> allowlist ZERO (red-proven); source had NO mojibake — the live „Tárhely" is the felhom-usb
|
||
> drive-label DATA, repaired via the label-edit UI (live step). (E) `offboxWarningDisplay`
|
||
> display pick — stale „nincs mentésre jelölt alkalmazás" run-warning → „A kijelölés módosult…"
|
||
> note once ≥1 app toggled (neutral color); 0 toggled unchanged (red-proven).
|
||
> Housekeeping: one-shot `backups_split_move_check.py` RETIRED (served its purpose).
|
||
> **Live QA fixes:** v0.126.1 — `.form-input`/`.form-row` had NO CSS rule at all (root cause of
|
||
> the unstyled .fab password field; styled as the `.form-control` twin). v0.126.2 — CF edge
|
||
> caches /static/style.css 4h → stylesheet link now `?v={{.Version}}` (auto-bust per release).
|
||
> **0.126.2 LIVE drill+demo (both healthy).** §13 visual QA ran on the DRILL box via a
|
||
> reversible SSH gate-lift (hash restored byte-identical, gate verified back ON) — the demo is
|
||
> customer-claimed and CC does not enter credentials. OPEN human step: demo login → felhom-usb
|
||
> label repair via the label-edit UI (data = 'Tárhely (felhom-usb)' in storage_paths, documented
|
||
> read-only; the Part D gate closed the code side).
|
||
|
||
> **2026-07-13 — v0.125.0: .FAB VOLUME PATH-STRAND DATA LOSS FIXED (IA finding 1, HIGH).
|
||
> MinAgent 0.81 unchanged; floor may advance to 0.125.0 next train (must NOT halt above 0.124.0
|
||
> without this).** Both volume legs stream via docker cp (helper container + `dockerExec` seam —
|
||
> zero shared paths, correct bare-metal AND containerized; §3 live probe first). Export FAILS
|
||
> LOUD on any missing/empty claimed tar (`assertBundleDataComplete`); import VALIDATES BEFORE it
|
||
> destroys (`validateBundleData` in step 0 — hollow bundle → refusal, app untouched). Class
|
||
> extinguished by `scripts/docker_run_volume_path_gate.py` (every `"-v"` allowlisted with WHY;
|
||
> Tier-1/2 mounts documented host-visible). Live: the exact failed ActualBudget leg round-trips
|
||
> byte-identically (`ec8ea6cb…` before==after); engine-invalid volume → loud export failure.
|
||
> **ASYMMETRY (needs a customer-docs line):** .fab bundles exported by containerized ≤0.124.0
|
||
> controllers are hollow — re-export; the import guard refuses them loudly.
|
||
|
||
> **2026-07-13 — v0.124.0: BACKUPS IA RESTRUCTURE. MinAgent 0.81.0 + floor unchanged.
|
||
> Operator decisions (2026-07-13, treat as settled):** (1) single active offsite destination per
|
||
> box STANDS — the dual-destination `managed_by` model is the separate queued Task B;
|
||
> (2) the Felhom-offsite status card NEVER changes anything — display + opt-in pointers only;
|
||
> (3) .fab export/download is PORTABILITY, not a backup tier — no scheduling, no status surface,
|
||
> point-in-time framing. Mechanics: four sub-pages (`/backups{,/remote,/apps,/restore}`, sections
|
||
> moved VERBATIM — `scripts/backups_split_move_check.py` gates vs df7ad37), status card (3 states,
|
||
> display-only), .fab download exit (existing exporter + staging dir + guarded stream + 1h TTL;
|
||
> traversal guard red-proven). Live-validated on drill+demo incl. a supervised import round-trip.
|
||
> **NEW FINDINGS:** **(HIGH)** containerized .fab export strands Docker-VOLUME tars on the guest
|
||
> host (`docker run -v <container-tmp>` → host path) — bundle ships empty volumes, import brings
|
||
> the app up EMPTY; fix = host-visible staging + fail-loud post-export assertion. **(MEDIUM,
|
||
> agent)** legacy-boot PVE (LVM root, no ESP mount) → SystemDisks empty → sysKnown=false → drive
|
||
> wizard offers ZERO candidates ever. Follow-ups: .fab browser-upload; mega-zip parked.
|
||
|
||
> **2026-07-13 — v0.123.0: POLISH BATCH (take-two F-15/F-11 + rename + zero-toggle). MinAgent
|
||
> 0.81.0 unchanged; floor 0.122 unchanged. Requires hub v0.52.0 for F-15 (old hub = clean no-op).**
|
||
> (1) F-15: the reset-request RESPONSE carries the rotated code hash, applied via the ACK's
|
||
> generation-guarded ClaimSync — emailed codes work immediately (live: 1 s, first-try accept).
|
||
> (2) F-11: zero native `confirm()` — `felhomConfirm`/`data-confirm` inline Igen/Mégse (layout.html);
|
||
> gate `scripts/native_confirm_gate.py`. (3) Tier-3 customer branding is **"Távoli mentés"**
|
||
> (NAS-mentés gone; manual form generalized to any SFTP target; gate
|
||
> `scripts/offbox_rename_gate.py`; "Hálózati tárhely" feature untouched). (4) Zero-toggle honesty:
|
||
> hint + "Sikeres — nincs mentésre jelölt alkalmazás" run copy. Also: `atomicPromoteTar` O_RDWR
|
||
> fsync (Windows dev-box green gate was permanently red). Deployed drill qm300 + demo 9201.
|
||
> NOT started (separate queued task, operator fork pending): offbox `managed_by` coexistence.
|
||
|
||
> **2026-07-12 — v0.122.0: CUSTOMER-CLAIM PASSWORD GATE (closes DRILL-day0-vm F-4/F-5). MinAgent
|
||
> 0.81.0 unchanged. Requires hub v0.50.0.** The dashboard password is CUSTOMER-OWNED via a one-time
|
||
> claim code the hub emails to the registered address — the "no password → open dashboard" default
|
||
> is GONE. Unclaimed box (code hash delivered, no password) → serves ONLY `/claim`; every other route
|
||
> → claim page (302) or 401 (API). A set password disables the gate (auth wins). Reset rides the same
|
||
> code engine (login "Elfelejtett jelszó"). Legacy-open (no password + no hash) → red transition
|
||
> banner until the hub delivers a hash. `internal/web/claim.go` (gate + pages + HMAC pre-auth CSRF +
|
||
> 5-try/15-min lockout → `claim_lockout` event), `report/claim_sync.go` (ACK cache, idempotent by
|
||
> generation), settings `Claimed`/`ClaimCode*`/`ClaimConsumedGeneration`, `config.web.claim_code_*`
|
||
> (hub-baked), `--print-reset-code` root hatch. Gate-coverage signature test + 4 red-proofs.
|
||
> **LIVE-PROVEN on drill guest 9201 (0.122.0): gate ON via the real edge (/ → 302 claim page, /api →
|
||
> 401), code emailed to demo-vm-felhom's registered address.** Floor raise 0.120→0.122 = operator's
|
||
> LAST step (supervised). Details: felhom.eu/documentation/audits/DRILL-day0-vm-2026-07-12.md (F-4/F-5
|
||
> RESOLVED).
|
||
|
||
> **2026-07-12 — v0.121.0: BACKUPS PAGE TRUTH PASS. MinAgent 0.81.0 unchanged. Controller-only, no
|
||
> agent-API change.** Pure UI/data-plumbing on `/backups`; no backup-engine behavior change. Fixes the
|
||
> self-contradicting live page: (1) **removed the dead "Részletek" card** (operator decision — redundant;
|
||
> per-app rows + Adatbázisok section already carry the truth) — kills the last uses of the never-set
|
||
> template fields `Tier2DriveGroups`/`ResticPassword`, the `restic-pw` element, and the `toggleTier`/
|
||
> `toggleResticPw`/`copyResticPw` JS. (2) **per-app "3. mentés" row now shows real off-box state** via a
|
||
> new pure `tier3State` (configured→toggle→escrow precedence): unconfigured/off/escrow_pending/active —
|
||
> the "Hamarosan — B2/S3/SFTP" placeholder is gone. (3) **SQLite-honest DB messaging** via pure
|
||
> `dbSectionState(discovered,dumps)` → dumps/pending/embedded (embedded-only box shows "–" + "beágyazott
|
||
> DB-k a kötetmentésben", not a bare "0"). (4) **dead/raw fields fixed** — `Tier1LastRun`/`Tier1LastStatus`
|
||
> now populated from `ListRestorePoints`; Tier-1/Tier-2 labels via `timeAgoStr` (relative), confirm()
|
||
> dialog keeps raw. (5) **terminology split** — off-box section = "Távoli mentés (3. mentés)" (+
|
||
> `#offbox-section` anchor); whole-guest PBS card = "Távoli rendszermentés" (was both "Távoli mentés").
|
||
> (6) deploy page gains a "Mentési beállítások →" link. DECISIONS: Részletek removed as redundant
|
||
> (operator-approved); "Távoli mentés (3. mentés)" (app off-box) vs "Távoli rendszermentés" (PBS whole-CT)
|
||
> are two distinct customer-facing names. Pure helpers in `internal/web/backup_page_state.go`. +9 web
|
||
> tests, 4 red-proofs. Observations: orphaned style.css classes from the Részletek removal left in place
|
||
> (details-tier*, repo-encryption*, restic-pw-field, drive-detail-*, tier-empty-state, repo-info-row*,
|
||
> repo-tier-title) — noted, not cleaned. Backlog: felhom.eu backup-architecture.md offbox refresh (separate task).
|
||
|
||
> **2026-07-12 — v0.120.0: fix-3 + fix-6 → CAMPAIGN-3 CLOSED (LIVE on 9201 + hub 0.48.0).
|
||
> MinAgent 0.81.0 unchanged.** **fix-3:** a `deadapp-check` job (30s, 90s boot grace) flags a DEPLOYED
|
||
> app in stopped/exited state (`stacks.IsDownState`) → self-clearing WARN dashboard banner + one
|
||
> `app_start_failed` hub event per running→down transition (`Notifier.NotifyAppStartFailures`, in-memory
|
||
> tracker, hub owns cooldown). **fix-6:** ring cap 1000→5000 (display cap raised too); periodic
|
||
> scheduler/refresh success lines → `[TRACE]` (ring drops at write-time, failures never TRACE); atomic
|
||
> JSON-lines spill to `<DataDir>/debug-ring.log` (SSD, survives recreate) every 30s + shutdown, loaded
|
||
> on boot. **hub v0.48.0** accepts `app_start_failed` (allowlist + customerMessages). LIVE: docker stop
|
||
> seerr → banner + ONE hub event across 3 cycles (anti-spam) → docker start → banner self-cleared; ring
|
||
> 0 spam lines + restart PRESERVED the pre-restart window (oldest unchanged, 63KB spill on SSD volume).
|
||
> **CAMPAIGN-3 CLOSED** (F12/F11/F10/F9/F2/F1→agent 0.85; F7/F6/F5→0.118; F8/F4→0.119; fix-3/6→0.120).
|
||
> Follow-ups: agent-ring persistence; F13 (active-nfs-mp8 rc255); publish train (agent 0.85 + ctrl
|
||
> 0.118/0.119/0.120 + hub 0.48) to Peti. Seams: deadapp scanDeployedAppRunStates, notify.pushFn.
|
||
|
||
> **2026-07-12 — v0.119.0: STORAGE-HEALTH COHERENCE (LIVE on 9201). MinAgent 0.81.0 unchanged.** Fixes
|
||
> CAMPAIGN-3 F8+F4. **F8 (MED):** the share row's health came only from the agent's server-level TCP
|
||
> dial (blind to a single unexported share) → it showed benign "Készenlét" while the stacks cards
|
||
> showed the stub — a contradictory UI. `networkStorageItems`→`fuseNetHealth` now reuses the SAME
|
||
> `system.ClassifyPathFS` the stacks stub badge reads (§3 fork = option B, controller-only): a new
|
||
> `stub` health (badge "Hibás — az alkalmazások nem a NAS-t látják") overrides idle/ok when the
|
||
> namespace sees local disk; `unreachable` still wins over stub; autofs/network/unknown leave agent
|
||
> health intact (never force-mount). Row + stacks badge now share ONE classifier → can't contradict.
|
||
> **F4 (LOW):** `handleNetStorageAdd` range-checks container uid/gid 1..65533 (`validMappedID`) →
|
||
> friendly 400, nothing installed (was raw agent_error on 101000). LIVE: F8 row=stub matching stacks
|
||
> badge through an exportfs cut, cleared to ok on re-export; F4 uid 101000→400, uid 1000 passes.
|
||
> Seam: `s.classifyFSPath`. Task D (fix-3 alerting + ring revision) still queued.
|
||
|
||
> **2026-07-12 — v0.118.0: BACKUP INTEGRITY (LIVE on 9201). MinAgent 0.81.0 unchanged.** Fixes
|
||
> CAMPAIGN-3 backup findings (`documentation/audits/CAMPAIGN-3-2026-07-11.md`). **F7 (HIGH) atomic
|
||
> volume dumps:** `DumpAppVolumes` writes `<vol>.tar.tmp` → fsync → `os.Rename` over the `.tar` only on
|
||
> success (`atomicPromoteTar`), mirroring dbdump.go DumpOne; a mid-write NFS cut can no longer
|
||
> truncate the last good tar to 0 bytes. **F6 (LOW) no single-copy:** `RunAllTier2` no longer skips
|
||
> volume-only apps (they now get a cross-drive tier-2 copy); sys_drive restore-point label is clear
|
||
> ("Belső SSD (rendszer)"); single-drive box shows an honest `SingleCopyWarning` banner. **F5 (LOW)
|
||
> stale-primary sweep:** `pruneStalePrimaryDirs` removes an orphaned `backups/primary/<app>` dir on an
|
||
> OLD drive after an HDD_PATH move (guarded: deployed + different-current-drive only, never a restore
|
||
> point). **Part 4 locality fork → operator chose (A) keep locality, doc-only** (NAS tier-1 stays on
|
||
> the NAS; tier-2 is the off-NAS leg). LIVE: F7 money-shot (all NAS tars byte-identical through a
|
||
> mid-write cut, no 0-byte, success:false); F6 (actualbudget/seerr on felhom-usb/secondary); F5
|
||
> (seeded stale dir swept, current kept); restore round-trip byte-identical. Seams: `tarVolume`,
|
||
> `perAppTier2`. Task C (F8/F4) + Task D (ring/alerting) still queued; Peti reaches 0.118 + agent 0.85
|
||
> at his next train (agentless-on-proxmox2 gap noted).
|
||
|
||
> **2026-07-11 — v0.116.0/0.116.1: OBSERVABILITY PASS (LIVE on 9201; agent v0.83.0 + hub v0.46.0).
|
||
> MinAgent: 0.81.0 unchanged.** The debug ring (`LogBuffer`) now ALWAYS exists — logger =
|
||
> `MultiWriter(LevelFilterWriter(stdout, logging.level), ring)`, so DEBUG detail is remotely
|
||
> readable on an `info` box while docker logs keep the configured level. New `internal/logx`
|
||
> leveled helpers = the sweep standard (netstorage_job phases/verdicts/durations, netprobe,
|
||
> validation refusals, orphan WARN, `SupportsWithSource` gate line, agentapi per-call DEBUG,
|
||
> migrate phases, tier2/offbox unswallowed persists). Report ACK gains `controller_log_requested` →
|
||
> next report ships `controller_log_tail` (selftail.go, consume-once, 128 KB; the customer-visible
|
||
> `operator log pull served` INFO rides in the tail; app-tail wire byte-compatible). Debug page:
|
||
> `Vezérlő | Ügynök` tabs — the agent tab proxies agent `GET /debug/logs` (`Client.DebugLogs`;
|
||
> typed-404 → the "after the agent's next update" notice). **v0.116.1 (found by live validation):
|
||
> `/debug` + `/api/debug/*` + the nav item were STILL gated on logging.level=debug — ungated (auth
|
||
> unchanged), the incident's actual blind spot.** Live-proven at info: a real refused NAS add is
|
||
> fully reconstructable on both tabs (capability gate w/ source=version, phase lines, 502+duration,
|
||
> category). Conventions: `felhom.eu/documentation/runbooks/logging-conventions.md`. OPEN: operator
|
||
> clicks the hub's two "Request logs" buttons (hub UI password-gated) to close the live bundle
|
||
> round-trip; legacy `isDebug()` emission sites left for incremental migration. NOT published —
|
||
> Peti stays 0.113/0.81.
|
||
|
||
> **2026-07-11 — v0.115.0: version-aware Supports + DSM-validated NAS guidance (LIVE on 9201, pairs
|
||
> with agent v0.82.0 + hub v0.45.0). MinAgent: 0.81.0.** Capability detection now compares the agent
|
||
> version from agent v0.82.0's `X-Felhom-Agent-Version` header (`Client.noteAgentVersion` captures it
|
||
> on every response, strict semver; `features.go featureMinAgent` table + version-first `Supports`)
|
||
> instead of route-probing — the probe stays as the fallback for header-less (≤0.81) agents, so
|
||
> nothing changed for Peti's box. THE one comparator moved to `internal/util/version.go` (selfupdate
|
||
> aliases it). Part A DSM spike (real DSM 7.2 via virtual-dsm) validated the consumer recipes E2E; the
|
||
> NAS-page NFS guidance gained the verified Synology steps (File Services → NFS → **NFSv4.1**; "Map
|
||
> all users to admin"; `/volume1/<share>`); caveat narrowed to QNAP-only. Live-checked on demo: a real
|
||
> add shows `capability gate: netstorage_verify=yes` via the version compare, zero probes. **Q1c
|
||
> (Part E, supervised) FAILED**: a NAS automount trigger does NOT survive a guest reboot (guest sees
|
||
> an empty dir; agent has no network-mount reassert) — fix is felhom-agent's, spec'd at
|
||
> felhom.eu/documentation/backlog/FOLLOWUP-nas-automount-guest-reboot-reassert.md (controller
|
||
> health-cross-check follow-on noted there). NOT published (agent 0.82 demo-only; Peti 0.81).
|
||
|
||
> **2026-07-11 — v0.114.0: agent-capability gate (option-1) + publish-train rules (option-2).**
|
||
> Answer to the 0.81/0.113 train's 9-minute controller-before-agent skew (Peti's box): the box now
|
||
> protects itself. `internal/agentapi/features.go` — `Supports(Feature)` route-probes the agent
|
||
> (`GET /netstorage/verify-status` = the v0.81.0 coupling signal; typed `StatusError` 404 ⇒ No, 2xx
|
||
> ⇒ Yes, transport/5xx ⇒ Unknown NEVER refused; `SupportCache` TTL 5m both polarities, Unknown
|
||
> uncached). `handleNetStorageAdd` refuses on No BEFORE the single-flight claim (412 +
|
||
> `agent_outdated` + honest Hungarian message); settings page swaps the add form for a banner
|
||
> (list/remove untouched in every state). remove/list NOT gated. NO agent/hub changes; NOT
|
||
> published, floor untouched, Peti stays 0.113.0 — the gate is inert protection until the next
|
||
> train. Rules codified: `felhom.eu/documentation/runbooks/publish-train-rules.md` (manifest before
|
||
> floor; floor field LAST — hub_settings DB row overrides env + acts immediately; MinAgent fleet
|
||
> gate — CHANGELOG header convention starts with this release; the gate as box-level backstop).
|
||
> Tests T1–T6 + wire-level 404-typing; red-proofs RP1–RP5 in REPORT.md. Roadmap: agent
|
||
> version-in-envelope upgrade of `Supports`; hub floor-UI separation = its own task. The
|
||
> `agent_outdated` branch is test-proven only (demo agent is current — downgrade not justified).
|
||
|
||
> **2026-07-11 — v0.113.0: NAS verify-before-commit + page redesign (LIVE on 9201, pairs with agent
|
||
> v0.81.0 + host-install v1.13.0).** `POST /api/storage/netstorage/add` no longer registers blind
|
||
> (the bogus-share-at-Készenlét bug is dead): detached single-flight orchestration
|
||
> (`internal/web/netstorage_job.go`, migrate shape; poll `GET .../add/status`) = agent add (units +
|
||
> agent-side detached verify with journal classification + auto-rollback) → controller **uid-1000
|
||
> re-exec write probe** (`--netprobe`, SysProcAttr.Credential — catches the squash trap) → register
|
||
> LAST. Any failure = full rollback; verify-lost after agent restart ⇒ controller rollback; unregistered
|
||
> agent shares surface as remove-only "Árva megosztás" rows. §3.2 Hungarian error map server-side
|
||
> (`netAddMessage`; `nfs_export` MERGES not-found/not-permitted — NFSv4 identical strings).
|
||
> storage_network.html rebuilt on the storage_attach pattern (form-row/form-input killed), SMB listed
|
||
> first, NFS two-recipe guidance with live computed uid+100000. Agent v0.81.0: NFS `retry=0`
|
||
> (dead-NAS access 91 s→3.8 s), `ClassifyNetVerifyFailure` (Q4-verbatim), unprivileged journal read
|
||
> (systemd-journal group — host-install v1.13.0 adds it; NO new sudoers). Live-validated A–E on 9201
|
||
> vs an isolated sim NAS (all transcripts + red-proofs in REPORT.md); Route A proven in production
|
||
> (alien-uid squash → server-side 1060:1060). Authoritative doc:
|
||
> felhom.eu/documentation/controller/network-storage-nas.md. NOT published (0.81.0 not in Gitea /
|
||
> Day-0 manifest; Peti untouched — his rollout incl. the usermod one-liner comes with the floor bump).
|
||
> Gotcha for future sessions: the controller container is bridge-only — in-guest API tests need the
|
||
> CONTAINER IP + `Host: felhom.demo-felhom.eu` (127.0.0.1:8080 is stale advice).
|
||
|
||
> **2026-07-10 — v0.112.0: self-update without credentials (LIVE on 9201, pairs with hub v0.43.1).**
|
||
> Root cause on Peti's box: the updater refused without Git Sync creds, but the public package is
|
||
> anonymously pullable. `queryRegistry` now does the Docker v2 anonymous token dance when both creds are
|
||
> empty (realm/service parsed FROM the WWW-Authenticate header — never hardcoded); `pullImage` skips
|
||
> `docker login` credential-less; creds path byte-unchanged (private catalogs); half-configured pair =
|
||
> loud misconfig; denial = "registry denied anonymous access — a private registry requires Git Sync
|
||
> credentials". Settings panel gains the mode line "Registry: nyilvános (hitelesítés nélkül) /
|
||
> hitelesített" — credential-less is a supported mode, not an error. Red-proof green (old guard restored
|
||
> → anonymous tests fail with the old message). LIVE-PROVEN on the credential-less demo (git creds are
|
||
> quoted-empty): /api/selfupdate/check → ok, latest=0.112.0, no error; settings shows "nyilvános".
|
||
> PENDING OPERATOR: floor-bump Peti to 0.112.0, then delete his temp Git Sync creds → clean "nyilvános"
|
||
> check. runCommand/runCommandStdin are now package VARS (test seam).
|
||
|
||
> **2026-07-10 — v0.111.0: remote app-log diagnostics (LIVE on 9201, pairs with hub v0.43.0).** The
|
||
> telemetry scraper now attaches `LogIssue.Context` (±5 raw lines around the FIRST occurrence of each
|
||
> error-severity issue; ≤11 lines, ≤400 chars/line, 16KB/report budget dropping lowest-count first; warns
|
||
> carry none) and `metrics.RedactLine` sanitizes EVERY off-box context/tail line (password/token/api-key/
|
||
> authorization/bearer → `[REDACTED]`, 64-hex → `[REDACTED-HEX64]`). On-demand log tails ride the ACK pull
|
||
> pattern: hub ACK `log_tail_requests` → next report `log_tails` (200 lines via stacks.GetLogs /
|
||
> FetchContainerLogTail, ordered, ≤64KB/app newest-kept, redacted, consume-once drain). Hub v0.43.0 stores
|
||
> context (first-capture-wins + `context_customer` provenance), renders click-to-expand copyable issues,
|
||
> fixes the period filter on Known Issues, replaces issue deletion with DISMISSAL (`dismissed_at`,
|
||
> resurface only on `last_seen > dismissed_at`), adds `?customer=` filtered drill-down, and keeps the last
|
||
> 2 tails per app with an ordered viewer + .log download. All red-proofs green (capture, redaction,
|
||
> consume-once ×2, dismissal guard, range filter, context clobber). Live-proven on demo: synthetic error →
|
||
> hub row with ordered 11-line context and `password=[REDACTED]`. OPERATOR: one click ("Request log tail"
|
||
> on demo felhom-controller) completes the live tail round-trip — hub UI is password-gated, CC cannot.
|
||
|
||
> **2026-07-09 — v0.106.0: offsite provisioning SLICE 2 — the apply-bridge (pairs with hub v0.38.0).** On
|
||
> startup the controller reconciles the hub-served `offsite:` descriptor into a key-only offbox target:
|
||
> `internal/offsiteapply.Bridge.Reconcile` — verify-pin the box host key against `host_fingerprint` (NO blind
|
||
> TOFU) → generate keypair → consume the one-time password (`POST /api/v1/offsite/consume-password/{id}`,
|
||
> single-use, never logged) → `sshpass ssh-copy-id -s -f` install + verify → `Manager.ApplyOffsiteTarget`
|
||
> (fork-4 enable → `EscrowState="pending"`) → persist a descriptor-hash marker LAST. **Idempotent** (no
|
||
> re-consume of a spent password) + **fail-safe** (any step fails → nothing persisted, retry next restart;
|
||
> consumed-but-failed install = loud "reset on the hub"). Seams faked in tests; both red-proofs (no-TOFU,
|
||
> marker-after-success) green. `Dockerfile` + `sshpass`. **NOT yet live-applied** — supervised end-to-end
|
||
> (hub provision → controller apply) is the next runbook, gated on the hub's new scoped `HETZNER_TOKEN`.
|
||
> NEXT slices: SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota).
|
||
|
||
> **2026-07-09 — v0.105.0: fork-4 offsite password custody (pairs with agent v0.77.0).** The restic-offsite
|
||
> repo password now rides the **customer-R escrow** (age-under-R in the agent `IdentityBundle`; custody spike
|
||
> `febdc56`). Enable → controller pushes the password (`StageEscrowSecret` → agent `POST /escrow/stage-secret`)
|
||
> → `EscrowState="pending"`. **Atomicity gate:** no offsite RUN until `EscrowState="escrowed"` (operator
|
||
> `POST /backup/offbox/confirm-escrow` after the escrow ceremony) — so no un-recoverable offsite ciphertext
|
||
> exists. **DR:** `POST /backup/offbox/inject-password` pre-places the recovered password (honored by
|
||
> `WriteOffboxSecrets`). DR recipe gains non-secret `offsite_restic` coords (`DRResticCoord`); the SFTP key is
|
||
> regenerated at DR (not escrowed). Atomicity + inject companion red-proofs green. **NOT yet live-validated**
|
||
> — the supervised escrow ceremony (enable→stage→escrow-create→confirm→gated run) is operator-run; NEXT =
|
||
> hub-verified auto-confirm + customer-self-serve enable (provisioning task). Deployed to 9201; see REPORT.
|
||
|
||
> **2026-07-09 — v0.104.0: off-box discovery over inference + no-silent-success.** The Storage-Box spike
|
||
> found offbox reporting `ok`/0 snapshots while backing up nothing; DIAG pinned it: offbox resolved each
|
||
> toggled app's recovery unit via `AppNamespaceRoot`→`GetAppDrivePath`, which reads the app's *live*
|
||
> `app.yaml` `HDD_PATH` and **silently falls back to `systemDataPath`** when the app isn't deployed → it
|
||
> looked on the wrong drive. **Decision: DISCOVER, don't infer** — scan the durable storage registry
|
||
> (schedulable, non-decommissioned paths ∪ systemDataPath) for `backups/primary/<app>`, deployment-state
|
||
> independent; newest-by-manifest wins on drive churn. **Silent-success closed:** 0-of-N toggled → hard
|
||
> error + operator alert; partial → `ok` + customer `LastWarning`. Write paths + `AppNamespaceRoot`
|
||
> untouched. Unit suite + both companion red-proofs green. **NEXT:** supervised box re-provision + a real
|
||
> offbox→Storage-Box endpoint round-trip (this task did NOT re-point at the live box — spike creds were
|
||
> torn down). Deployed to 9201; see REPORT.md.
|
||
|
||
> **2026-07-07 — v0.103.0: F-C2-1 (LIVE on 9201).** The config loader ran `os.ExpandEnv` over the
|
||
> whole YAML before parse, silently corrupting a bcrypt `web.password_hash` (`$2a$10$…` → `"a0"`) — a
|
||
> silent auth-integrity bug. Removed both `ExpandEnv` calls (parse raw bytes); typed
|
||
> `FELHOM_WEB_PASSWORD_HASH` override unchanged. Live-proven: a bcrypt hash in controller.yaml now
|
||
> loads intact and login succeeds (pre-fix it corrupted → login fail). Behavior change: literal
|
||
> `${VAR}` in a value is now preserved verbatim (no repo config depends on the old expansion).
|
||
|
||
Last updated: 2026-07-06 (v0.102.0 — async restore family; F4 re-adjudicated + fixed)
|
||
|
||
> **2026-07-06 — v0.102.0: async restore family (F4 UX fix, LIVE on 9201).** All three restore surfaces
|
||
> (`/backup/restore`, `/backup/tier2/restore`, `/backup/offbox/restore`) blocked the HTTP request until
|
||
> completion → through cloudflared's 100s cap a customer got an error page while the restore succeeded
|
||
> (offbox worse: bounded on `r.Context()`, canceling the SFTP restore mid-flight). Now async (offboxRun
|
||
> shape): fast-path IsRunning refuse → background goroutine (offbox ctx off r.Context()→Background+30m) →
|
||
> instant redirect. New `GET /api/backup/restore-status` + mutex op-status (`opstatus.go`) + 3s-polling
|
||
> `backups.html` banner. Live-proven: restore POST 0.018s internal / **0.235s external (F4 tunnel)**, canary
|
||
> bit-identical, status transitions. Restore single-flight unchanged. OPEN: op-status is in-memory (no
|
||
> persistence, by design).
|
||
|
||
> **2026-07-06 — v0.101.0: no-mercy campaign findings.** F3: git subprocess deadline in
|
||
> `internal/sync/sync.go` (`gitCmdTimeout=120s`, `exec.CommandContext`) — a hung remote no longer
|
||
> wedges `syncing=true` until restart. F2 evidence gap: `agentapi.EjectDisk`/`Decommission` now use
|
||
> `postWithStatus` + `refusalError` so the agent's `"…refused (role: X)"` reaches the operator
|
||
> instead of a bare `HTTP 403`. Companion: catalog `d86e256` (F1 vaultwarden `_ENABLE_SMTP` boot-gate
|
||
> — fresh email-off deploys crash-looped; live-validated Scenarios A/B on 9201). F2 diagnosed to a
|
||
> verdict (REAL finding — `roleForMountPath` over-refuses an enrolled user-data drive that isn't a
|
||
> PVE storage; fail-safe direction; agent fix DEFERRED). Full triage:
|
||
> `felhom.eu/documentation/audits/CAMPAIGN-nomercy-2026-07-06.md` addendum. OPEN follow-ups: agent
|
||
> `roleForMountPath` fallback; the targeted P1–P3 campaign re-run for clean backup/restore coverage.
|
||
|
||
> **2026-07-05 — v0.100.0 (TASK C2): drill finding F2 CLOSED — one-click class-C file restore.**
|
||
> `POST /backup/tier2/restore` + "Fájlok visszaállítása" on the Tier-2 row: in-place, ADDITIVE-ONLY
|
||
> (`rsync -a --ignore-existing` from the recorded Tier-2 copy — never overwrites, never deletes).
|
||
> Serves "I deleted my files"; corruption/point-in-time stays offbox/operator. **The C-series
|
||
> (drill findings F1/F2/F3/O4) is now fully closed.** Reindex caveat (e.g. Nextcloud occ files:scan)
|
||
> documented in backup-architecture.md.
|
||
|
||
> **2026-07-05 — v0.99.0 restore-path fixes (TASK C1): drill findings F1/F3/O4 RESOLVED.**
|
||
> F1: `GET /api/backup/snapshots` implemented (`backup.ListRestorePoints`) — the restore panel
|
||
> populates and the restore button enables. F3: `runVolumeDumps` wired into the nightly/manual
|
||
> backup run (volume gate BEFORE DumpAppVolumesSafe; before unit capture). O4: unrecoverable
|
||
> resettable secrets get a generated replacement (`GenerateSecretForField` + `SetSecretGenerator`
|
||
> seam); data-key gate untouched. **F2 (one-click in-place class-C restore) remains OPEN → TASK C2.**
|
||
> O4 residual: restored volume tar with an OLD credential hash may still need a manual in-DB reset.
|
||
|
||
> **2026-07-04 — app-data restore drill** → see `felhom.eu/documentation/audits/DRILL-appdata-restore-2026-07-04.md`.
|
||
> Keep-side restore proven live on 9201 (class-A DB replay + fail-closed data-key gate + non-destruction). **F1 (HIGH): UI restore is dead — `/api/backup/snapshots` has no handler, so the restore button never enables.** F2: no one-click in-place class-C (HDD bind-mount) restore. F3: named-volume data never backed up (`DumpAppVolumes*` has no caller). *(F1/F3/O4 resolved in v0.99.0, see above.)*
|
||
|
||
> **2026-07-03 — CLAUDE.md slimmed to stable orientation** (full package map, verified 9201 deploy
|
||
> summary, no version-pinned state). Deep runbooks/design/testing doctrine now in the personal
|
||
> skills `felhom-build-deploy` / `felhom-ui-design` / `felhom-testing` (source: `felhom.eu/skills/`).
|
||
|
||
> **2026-07-03 — `REUSE.md` exists at the repo root** (canonical helpers / patterns / traps / seams, code-verified). Check it before writing new code; update it in the same commit that adds/changes a shared helper (maintenance rule now in CLAUDE.md).
|
||
|
||
> **2026-07-02 — v0.98.0 (deployed on 9201): Tárhely IA follow-up.** User feedback on D1: NAS-add and
|
||
> local-drive enrollment buttons sat side by side — confusing. `/storage` split into two subpages
|
||
> under the Tárhely nav item (nested sub-links, `.nav-links-nested`): **/storage = Meghajtók**
|
||
> (drives + agent view + wizards + manual add) and **/storage/network = Hálózati tárhely (NAS)**
|
||
> (NAS-megosztások list + add form; own openDialog copy). `networkStoragePageData` (page key
|
||
> `storage-network`) split out of `storagePageData`. No API/storage-semantics change.
|
||
|
||
> **2026-07-02 — v0.97.0 (deployed on 9201): TASK-D1 — settings split + Tárhely page (IA only).**
|
||
> The 1451-line `settings.html` monolith is split into four pages: `/storage` (new main-nav
|
||
> **Tárhely**: Adattárolók + NAS + unified agent drive view + wizard entry), `/settings` (Rendszer:
|
||
> konfig, verzió/frissítés, vezérlő+kiszolgáló újraindítás), `/settings/notifications` (Értesítések
|
||
> + Alkalmazás-email), `/settings/security` (Jelszó, Földrajzi korlátozás, Vészhelyzeti információk).
|
||
> Sidebar gains the Tárhely item + a "Beállítások" group with three sub-links. **No storage/agent/API
|
||
> behavior change** — pages moved, logic didn't; every `/api/storage/*` endpoint + POST action
|
||
> unchanged. **Decisions:** (a) storage flashes + wizards moved to `/storage`; old
|
||
> `/settings/storage/{init,attach}` **301** to `/storage/{init,attach}`. (b) The two duplicated drive
|
||
> views MERGED: registry cards render server-side, then JS enriches each connected user-data card in
|
||
> place from the agent `/api/disks` (role tag, durable-id, agent actions) joined on mount path, plus
|
||
> two groups — **Rendszermeghajtók** (system/backup, read-only, lock tag, NO actions) and **Nem
|
||
> regisztrált** (register only); agent-down → one warn note, registry cards still render. (c) Every
|
||
> native `confirm()`/`prompt()` on the four pages now routes through a light `.confirm-overlay`
|
||
> (`openDialog`; texts verbatim; type-to-confirm kept only where it existed) — the D0 tab-freeze is
|
||
> gone. **Contract for future UI:** new templates must pass `scripts/template_id_gate.py` (every JS
|
||
> element-ID resolves in its own template) and `scripts/emoji_gate.py` (0 emoji — the D0 grep gate
|
||
> false-negatived multibyte emoji on Windows; 8 survivors removed). `.badge-lock`/`.lock-ico` CSS
|
||
> deleted; other `.badge*` still used on secondary pages (D2 to finish). **Next: D2** (badge→tag on
|
||
> backups/secondary pages; storage.html is still ~700 lines — NAS-add partial candidate). NOT
|
||
> live-validated: agent-down degradation (don't stop the live agent — static/unit only); destructive
|
||
> storage ops via the moved overlay paths (endpoints unchanged; supervised session).
|
||
|
||
> **2026-07-02 — v0.96.0 (deployed on 9201): TASK-D0 — design system v2 re-skin (appearance only).**
|
||
> The whole customer UI (dashboard AND setup wizard) now renders in the approved v2 language: navy
|
||
> token palette (`--bg-0/1/2, --line, --text-1/2/3, --blue, --warn, --crit`), single 2px radius, no
|
||
> shadows; **exception-based status color** — nominal is blue/neutral, amber/red only on deviation;
|
||
> zone bars → the hairline `.meter` (neutral 70/85 ticks, warn/crit flag „Fogyóban a hely" /
|
||
> „Kritikusan kevés hely"); pills → `.tag` / `.metarow`. **Decisions:** (a) `stateColor` semantics
|
||
> changed — **stopped/exited = neutral, NOT red** (operator-approved; failures surface via unhealthy
|
||
> + alerts); restarting = warn; outputs are now `run/progress/warn/neutral/off` and
|
||
> `nominal/warn/crit` — any new template MUST use these suffixes (truth tables guarded by unit tests,
|
||
> `stateLabel` copy frozen byte-identical). (b) Fonts (PJS + JBM variable woff2, latin+latin-ext) and
|
||
> a 30-icon Lucide sprite are **vendored in the binary** (`/static/fonts/`, `templates/icons.html`)
|
||
> — no CDN; `fonts.googleapis.com`, `box-shadow`, 999px and the emoji set are banned strings (grep
|
||
> gate in REPORT §5, 143→0). (c) Setup wizard `handleCSS` bug fixed: it read a dataDir-derived path
|
||
> that never exists in the container → always served minimalCSS in production; now serves embedded
|
||
> `web.StyleCSS()`. (d) Found+fixed pre-existing v0.93.0 bug: `/backups` 500'd once an off-box backup
|
||
> had run (`OffboxTarget.LastRun` is an RFC3339 string fed to `timeAgo`; new `timeAgoStr`).
|
||
> Canonical reference: `felhom.eu/documentation/design/design-system.md`. **Next: D1** (settings IA
|
||
> split + Tárhely main-nav promotion; also migrate the remaining native `confirm()` dialogs to the
|
||
> type-to-confirm overlay and finish badge→tag markup on secondary pages). NOT live-validated: setup
|
||
> wizard visuals (unit-only), warn/crit meters on real hardware (demo healthy).
|
||
|
||
> **2026-07-01 — v0.95.0 (deployed on 9201): Impl-2b — raw-drive enrollment wizards.** Both enrollment
|
||
> wizards (`storage_init.html` / `storage_attach.html`) now fetch candidates from the agent's raw-device
|
||
> scan `GET /api/disks/candidates` (proxy `agentDiskCandidatesHandler` → agent Impl-2a) instead of the
|
||
> `Observe()`-based `/api/disks`, so a brand-new (non-PVE-storage) drive is discoverable + enrollable.
|
||
> `agentapi.ListCandidates` + `resolveEnrollUUID` (a raw candidate isn't in `/disks` → resolve its
|
||
> fs-UUID from the scan). Live end-to-end on felhom-pve: raw `/dev/sdd` (SD card) → wizard → format (Impl-1
|
||
> guard) / attach → mount → bind → intent → guest-bind → **"Aktív"** in the UI (3rd drive, no false
|
||
> detach). Needed a chain of AGENT fixes (v0.56–0.58: durableIDForMount / ReassertGuestBinds / HostReader
|
||
> wiring / /disks GuestPath+BoundUnderParent for registry rows) — the whole enroll+tracking machinery had
|
||
> assumed a PVE-storage drive. **Follow-ups:** `runStorageInit` should poll the agent's detached-format
|
||
> status (`/disks/format/status`) so a SLOW-device format completes in one flow (pre-existing); Impl-3
|
||
> shared-box operator format gate.
|
||
|
||
> **2026-06-30 — v0.94.0 (deployed on 9201): pull-based config-refresh.** The hub report ACK now also
|
||
> carries a per-customer **`config_version`** (hub v0.26.0; a stored counter bumped on every config save).
|
||
> `OnPushResponse` → `ConfigRefresher.Reconcile` (`internal/report/config_refresh.go`): on a change vs.
|
||
> `settings.applied_config_version`, `bootstrap.RefreshConfig` re-pulls `controller.yaml` (re-merging
|
||
> `local_api` from bootstrap.json; overwrites controller.yaml, never settings.json) → record → graceful
|
||
> self-restart (`api.GracefulSelfRestart`). First-run records baseline (no restart); unchanged = no-op
|
||
> (no storm); failed pull keeps config + retries. This is the box-pulls-config replacement for the hub's
|
||
> retired inbound "Push Config" — the hub never connects into the box. **Companion hub v0.26.0** also
|
||
> retired Trigger Update / Pull Config / Show Diff and dropped geo-disable's inbound notify (kept the
|
||
> hub→Cloudflare WAF removal), and fixed the stale `docker-setup.sh` setup command → host-install.
|
||
> Live-validated on 9201: first-run baseline (applied=1, no restart) → DB bump 1→2 → re-pull+self-restart
|
||
> (session_secret rotated, local_api preserved, RestartCount 0→1) → 4 cycles no-loop → apps stayed up.
|
||
|
||
> **2026-06-29 — Self-health arc complete (agent v0.48.0 + hub v0.22.1).** The hub now proactively
|
||
> watches every agent's served local-API **leaf fingerprint** fleet-wide: the agent reports
|
||
> `leaf_fingerprint` on its host report (v0.48.0), and the hub `HostLeafChecker` (`monitor/host_leaf.go`)
|
||
> raises `host_leaf_changed` when it changes (trust-on-first-report baseline; empty fp = unknown, never
|
||
> alerts; reads from report_json, no migration; hub-generated so no allowlist change). Independent of the
|
||
> controller channel-check. Live: a leaf regen raised `host_leaf_changed` (`82078fab…→60b5974d…`) AND the
|
||
> controller's `agent_channel_pin_mismatch` — complementary. NOTE: v0.22.0 shipped the checker but the
|
||
> main.go wiring never applied (concurrent file-touch); the LIVE TEST caught it (no event on regen) →
|
||
> fixed in v0.22.1. The original silent-multi-day-outage incident class is now caught from THREE angles
|
||
> (agent capabilities v0.44.0 / controller channel-check v0.90.0+F2 / hub leaf-fp v0.22.x) AND prevented
|
||
> (agent v0.46.0 loud-regenerate + host-install --preserve-state-from). Backlog: the served-fp-vs-pinned-fp
|
||
> authoritative cross-check; the test-run coverage gaps (capability→hub live, host-reboot doubling).
|
||
|
||
> **2026-06-29 — v0.91.0: F2 (channel-health born-down alerting).** The v0.90.0 channel-health check
|
||
> alerted only on a live up→down transition; a channel broken at **startup/reseed** (e.g. the
|
||
> controller boots right after a leaf regen → first observation is `pin_mismatch`) was dashboard-only,
|
||
> no operator email — forever. Fixed with an `alerted` flag in `channelhealth/checker.go`: a confirmed
|
||
> down coming from up/unseeded OR a reason-change re-arms then alerts once; steady-down no dup; recovery
|
||
> re-arms; debounce intact (transient born-down still N≥2; healthy first-obs silent). Live-validated:
|
||
> controller restarted into a regenerated-leaf channel → `[channel] DOWN (unseeded->down:pin_mismatch)`
|
||
> + `Event pushed: agent_channel_pin_mismatch` (~60s) — the capstone's missing half. Companions: hub
|
||
> v0.21.0 (same fix for HostCapabilityChecker/HostStalenessChecker — seed only healthy, born-degraded/
|
||
> stale emits on first Check, cooldown dedups), agent v0.46.0 (EnsureLeaf loud-WARNs a REGENERATED leaf
|
||
> + host-install `--preserve-state-from`/populated-host guard = the prevention for the original incident
|
||
> class). Self-health story now complete: agent watches its capabilities, controller watches its link,
|
||
> detection works at startup, and a reinstall can preserve the leaf. Backlog: F1 swap-rollback
|
||
> RestartCount hardening; full live reinstall (supervised); hub-side leaf-fp fleet comparison.
|
||
|
||
> **2026-06-29 — v0.90.0: controller→agent channel health-check (self-health slice).** A ~60s
|
||
> scheduler job (`internal/channelhealth`) probes the local-API channel via the PRODUCTION memoized
|
||
> client (`Server.ProbeAgentChannel` → `s.agentClient()` + GET /storage — NOT a fresh client, per the
|
||
> spike: self-heals, mirrors the disk UI, no transport leak). Classifies failures (spike Q1 map:
|
||
> pin_mismatch/unauthorized/unreachable/timeout/misconfigured/construction_error/unknown), **debounces**
|
||
> transient reasons (refused/timeout need N≥2 consecutive so the ~1s agent-restart blip doesn't page;
|
||
> pin/401/DNS/construction alert first-obs), seeds the first observation silently, and on a transition
|
||
> emits an **English operator-only** event (`Notifier.NotifyAgentChannelDown/Recovered`) + a Hungarian
|
||
> dashboard banner (`AlertManager.SetAgentChannelAlert`). Closes the R1-incident gap (channel was only
|
||
> checked once at startup). **Required a hub change** (felhom-hub v0.20.0): the `agent_channel_*` event
|
||
> types were rejected by the hub's `allowedEventTypes` allowlist (HTTP 400) — controller-pushed events
|
||
> are gated, unlike the hub-generated host_* ones. Live-validated: transient restart → no alert;
|
||
> sustained stop → debounce(1/2) then DOWN transition + dashboard banner; start → recovered. No agent
|
||
> change. Self-health story now: agent watches its own capabilities (v0.44.0) + controller watches its
|
||
> link to the agent (this). Backlog: hub-side leaf-fp comparison.
|
||
|
||
> **2026-06-29 — OPS (no code change): controller↔agent TLS leaf-pin mismatch RESOLVED via R1.**
|
||
> The 2026-06-28 root→non-root agent migration regenerated the agent's local-API leaf
|
||
> (`60b5974d…`→`ced34036…`) and truncated its token store, so the in-guest controller's pinned
|
||
> `agentapi` client failed every call (`TLS pin mismatch`) — disk UI dead AND the quiesce
|
||
> app-consistent-backup loop dead. Fix: restored the pre-migration `local-api.{crt,key}` **+**
|
||
> `local-tokens.log` from `/root/agent-backup-bundle-test/aside-var-lib/` on felhom-pve (R1 RUNBOOK,
|
||
> supervised). Agent now serves `60b5974d…` again (== controller pin + bootstrap.json seed) and the
|
||
> restored token store's 9201 entry matches the controller's current `local_api.token` — **zero
|
||
> in-guest change.** Gates passed: leaf loaded (not regenerated), wire fp `60b5974d…`, authenticated
|
||
> `GET /disks` 200/5-disks (bogus token 401), and the next quiesce cycle ran a real backup job.
|
||
> Rollback point kept at `/root/agent-rollback-20260629-150137/`. **OPEN (carried forward):** the
|
||
> migration/Day-0 path must carry these three files into the new `felhom-agent`-owned data dir (or
|
||
> treat their loss as a re-bootstrap trigger), else any re-migration reintroduces this. Full record:
|
||
> `felhom.eu/documentation/audits/SPIKE-agent-pin-mismatch-2026-06-29.md` (§9 Outcome).
|
||
|
||
> **2026-06-27 — v0.86.0 (deployed on 9201): Phase 2 managed updates — controller-version FLOOR.**
|
||
> On top of the Phase 1 opt-in "update to latest" button, the controller now honors an operator-enforced
|
||
> **minimum version (FLOOR)** delivered on the hub **report ACK** (`min_controller_version` +
|
||
> `latest_version`; `internal/report/pusher.go` `PushResponse`). `OnPushResponse` →
|
||
> `updater.SetFloor()` + `updater.MaybeAutoUpdate()` (rides the report cycle — no new timer). Below the
|
||
> floor → **auto-update to the floor** (reuses Phase 1 `performUpdate`: in-guest pull → agent swap →
|
||
> rollback; `initiatedBy="auto-floor"`). At/above floor → nothing (does NOT chase latest — that's the
|
||
> button). Guards: dev/no-agent/backup → skip; floor must be pullable (floor ≤ latest; floor>latest →
|
||
> warn+noop); one attempt per below-floor condition (in-mem + persisted state) → no flapping. Floor source
|
||
> + operator UI are hub-side (felhom-hub v0.15.0: per-customer override + global default + report ACK).
|
||
> **No agent change** (reuses Phase 1 `POST /controller/swap`). Day-0 now ships current (golden rebuilt at
|
||
> 0.85.1) AND stays current (floor) — the fleet-currency story is closed. Live: dogfood 0.85.1→0.86.0 via
|
||
> the button, then floor 0.87.0 → auto 0.86.0→0.87.0 (no click).
|
||
|
||
|
||
> **2026-06-26 — v0.84.0 (deployed on 9201): show an app's auto-generated first-login on its page.**
|
||
> General, catalog-driven mechanism: `.felhom.yml initial_credentials: {file, format json|regex|plain,
|
||
> username_key/password_key | username_pattern/password_pattern, note}`. The controller reads the file
|
||
> **live** from the container (`internal/stacks/initialcreds.go` `ReadInitialCredentials` + pure
|
||
> `parseInitialCreds`, unit-tested), never persists it, and renders a "Kezdeti belépési adatok" card on
|
||
> `/apps/{slug}` (masked password + reveal/copy). First consumer: crafty-controller (Crafty writes a
|
||
> random admin password to `/crafty/app/config/default-creds.txt`). Live-verified on 9201: card shows
|
||
> username `admin` + the real extracted password. Reuse for any future self-seeding app. Security: same
|
||
> exposure class as the existing post-deploy reveal / default_creds card — relies on the prod dashboard
|
||
> being auth-gated (demo public-unauth is the separate tracked issue). Backlog idea unchanged: a
|
||
> `backend_scheme` hint for TLS backends (v0.83.0 line).
|
||
|
||
> **2026-06-26 — v0.83.0 (deployed on 9201): scoped Traefik backend transport for self-signed HTTPS apps.**
|
||
> Crafty is the **first/only catalog app with an HTTPS backend** (self-signed TLS on `:8443`, no plain-HTTP
|
||
> port). The crafty healthcheck fix un-withheld its route, exposing a pre-existing 502 (Traefik proxied
|
||
> HTTP to the HTTPS backend). Since traefik v3 forbids `insecureSkipVerify` via Docker labels, the
|
||
> controller now renders a file-provider dynamic file `dynamic/serverstransports.yml` defining a **named**
|
||
> `insecure-skip-verify` transport (`infra.RenderServersTransports` + `stacks.ensureServersTransports`,
|
||
> written from `EnsureBaseStack` outside the traefik-running guard, write-if-changed, hot-loaded). Apps opt
|
||
> in per-service via catalog labels `scheme=https` + `serverstransport=insecure-skip-verify@file`.
|
||
> **No global `insecureSkipVerify`** — verification stays ON for every other backend (scoped Option B).
|
||
> Pattern to reuse for any future self-signed HTTPS backend. Live-verified: `minecraft.demo-felhom.eu`
|
||
> 502→302; filebrowser (HTTP) unaffected. Backlog: a `.felhom.yml backend_scheme` hint so the catalog
|
||
> convention sets these labels instead of hand-adding.
|
||
|
||
> **2026-06-22 — v0.76.0 (deployed on 9201): three campaign-#3 hardening fixes.**
|
||
> - **S1**: corrupt `settings.json` no longer crash-loops — `save()` writes a last-known-good `.bak`
|
||
> (after the primary rename); `Load()` recovers from `.bak`, else preserves the corrupt file as
|
||
> `*.corrupt-<ts>` + safe defaults. New `Settings.LoadWarning` → dashboard banner. (`main.go` Fatalf
|
||
> now only on the IO-unreadable path.)
|
||
> - **F2**: `web.validStackName` gates `backupRestoreHandler` + `apiExportStart` (reject `/ \ .. NUL`)
|
||
> before any restore/export — closes the traversal defense-in-depth gap (no escape had occurred, but
|
||
> it relied on downstream map-lookups).
|
||
> - **S3**: `quiesce.readMarker` logs `[WARN]` + quarantines a corrupt marker to `*.corrupt-<ts>`
|
||
> instead of silently dropping it.
|
||
> All three live-validated on 9201 (truncate settings → recover from .bak no crash-loop; traversal
|
||
> restore → rejected, /etc intact; corrupt quiesce marker → quarantined). Tests + red-proofs.
|
||
> Still-open campaign-#3 deferrals (NOT done): time-chaos on a dedicated VM, host reboot (supervised),
|
||
> `.fab` import compose-fuzzing. See `felhom.eu/documentation/tests/test-campaign-3-2026-06-22-findings.md`.
|
||
|
||
Last updated: 2026-06-22 (v0.75.0 — gate userdata MkdirAll on a live mountpoint)
|
||
|
||
> **2026-06-22 — v0.75.0 (deployed on 9201): userdata MkdirAll gated on a live mountpoint.**
|
||
> Both `MkdirAll`-into-`<drive>/userdata` sites — the deploy belt (`stacks.ensureUserdataMounts`) and
|
||
> the FileBrowser sync (`web.syncFileBrowserMounts`) — now skip when an **external** drive root (under
|
||
> `StableParentDir=/mnt/felhom-drives`, `!= sysDataPath`) is **not a live mountpoint** (`system.IsMountPoint`).
|
||
> Closes the campaign-#2 hazard where a drive-absent window produced `mkdir …/userdata: permission denied`
|
||
> + transient `Created` flapping AND could write app data onto the guest **rootfs** (shadowed when the
|
||
> drive returns). The app is held by `planDriveGates` instead. System/local path is never gated. New
|
||
> `Manager.isMountPoint` seam + pure `web.skipFileBrowserPath` helper for tests. Live-proven via a C6
|
||
> re-run: 0 permission-denied, 0 shadow dirs on the rootfs, apps recover on reconnect.
|
||
>
|
||
> **Boot-ordering design note (NOT implemented — decide separately):** the *boot-time* `mkdir …
|
||
> permission denied` is a different cause — **docker's** boot-restore auto-starts drive-backed
|
||
> containers (`restart: unless-stopped`) before the agent mounts the drives, so docker (not the belt)
|
||
> tries to create the bind source; `planDriveGates` recovers them after mount convergence (why a reboot
|
||
> ends healthy). Options: (A) accept + suppress (apps self-heal; lowest risk; maybe downgrade the
|
||
> boot-window log level) — recommended now; (B) drive-backed app containers don't docker-auto-start at
|
||
> boot (restart policy `no`/`on-failure`) so the controller's gate is the sole starter after mounts
|
||
> converge — cleaner but bigger (must cover crash-restart too). See
|
||
> `felhom.eu/documentation/tests/test-campaign-2-finding1-recovery-diagnosis.md` (boot-ordering observation).
|
||
|
||
Last updated: 2026-06-22 (v0.74.0 — controller→agent connection-leak fix)
|
||
|
||
> **2026-06-22 — v0.74.0 (deployed on demo guest 9201): fixed the controller→agent connection leak.**
|
||
> `Server.agentClient()` built a new `agentapi.Client` (new bare `http.Transport`, `IdleConnTimeout:0`)
|
||
> per call → one leaked idle ESTABLISHED socket per agent call to `192.168.0.162:8443`, exhausting the
|
||
> ephemeral source-port range after ~5 days of controller uptime → EADDRNOTAVAIL, which had taken down
|
||
> storage UI + host-metrics + whole-guest backup (found in the 2026-06-22 unattended campaign). Fix:
|
||
> memoize ONE shared client via `sync.Once` + harden the Transport (`MaxIdleConnsPerHost:2`,
|
||
> `IdleConnTimeout:90s`). Live-proven: idle sockets to `:8443` stay flat at 2 across a 120-call burst
|
||
> (pre-fix grew ~1/call → 132). Agent/firewall untouched. Diagnosis + campaign findings live in
|
||
> `felhom.eu/documentation/tests/unattended-test-campaign-2026-06-22-*.md`. **Separate open item:** the
|
||
> defense-in-depth host firewall rule scoping `:8443` to the guest bridge subnet is still absent
|
||
> (pve-firewall disabled) — to be closed independently.
|
||
|
||
Last updated: 2026-06-13 (v0.60.0 backlog-Medium cleanup)
|
||
|
||
> **Live version: controller v0.60.0** (deployed on demo guest 9201), agent **v0.30.0** (AGENT-001 deployed), hub **v0.11.0**.
|
||
> **NOTE:** the long-form sections far below this banner are stale (last full pass ~v0.16.1). Current
|
||
> state is the dated entries immediately below + `CHANGELOG.md` + the now-authoritative **central docs at
|
||
> `felhom.eu/documentation/controller/`** (code-verified) + auto-memory `MEMORY.md`.
|
||
>
|
||
> **2026-06-13 — v0.60.0 backlog-Medium cleanup (BUGHUNT reconcile):**
|
||
> - **M25** (Server.integrationMgr data race — constructor goroutine vs post-construction `SetIntegrationManager`):
|
||
> **FIXED** via `atomic.Pointer`; `-race`-verified. **M4/M5/M6** verified already FIXED (no action).
|
||
> - **M18** (dump re-validation every 5 min — perf) and **M19** (`deriveStackName` misattribution — low
|
||
> incidence) verified LIVE but cross-package-entangled → branches `fix/m18-dump-validation-cache` /
|
||
> `fix/m19-stackname-crossref` (notes + fix plan, PENDING REVIEW, not deployed).
|
||
>
|
||
> **2026-06-13 — v0.59.0 audit fixes + documentation centralization:**
|
||
> - Fixed the validated 2026-06-13 audit findings (records: `felhom.eu/documentation/audits/`):
|
||
> **CTRL-001** (`.fab` import path traversal — manifest segment validator, fail the parse);
|
||
> **CTRL-T2-1** (ghost-deployed on crash — `app.yaml` now persists `deployed:false` until `compose up -d`
|
||
> succeeds, flipped true only on success; in-memory flag still true during pull for UX);
|
||
> **H10** (plaintext secret on encrypt failure — `SaveAppConfig` now fail-closed, returns an error);
|
||
> **M2** (misleading lock on init-only `SetStackProvider` removed). All with regression tests.
|
||
> - **AGENT-001** (wrong-disk wipe TOCTOU) fixed on agent branch `fix/agent-001-wipe-durable-reresolve`
|
||
> — PENDING REVIEW, NOT deployed (supervised merge+golden-rebake reserved; see `AGENT-001-FIX-NOTES.md`).
|
||
> - Built + deployed v0.59.0 to demo guest 9201 (bootstrap mechanism); verified healthy, dashboard 200.
|
||
> - Documentation centralized under `felhom.eu/documentation/` (controller subtree + audits + top index),
|
||
> code-verified; `controller/README.md` banner points there; this CONTEXT banner refreshed.
|
||
>
|
||
> **2026-06-13 — v0.58.0 OS/Docker-data split prevention layer (Phase 2; Phase 1 = agent v0.29.0):**
|
||
> - OS rootfs + Docker data split onto separate local-lvm volumes (golden bakes 32G rootfs + 256G
|
||
> /var/lib/docker, backup=1, **overlay2** so images live on the data vol). Infra protected by
|
||
> PREVENTION not placement: `system.GetDockerVolumeHeadroom()` (statfs "/" = data vol) reserves
|
||
> max(5GB,10%); `deployStack` refuses HTTP 507 below buffer; deploy.html banner+disable; monitor
|
||
> (warn80/crit90) watches the same vol; log rotation baked into the golden daemon.json.
|
||
> - Live-validated by DESTROYING + RE-PROVISIONING 9201 from the split golden: split layout, overlay2,
|
||
> images on data vol, lean rootfs, OS isolation (data vol 100% → rootfs healthy), deploy gate 507,
|
||
> ActualBudget volume on data vol, hub config pull, external access via CF. RomM/USB re-enroll = the
|
||
> documented final restore step (RomM data safe on host USB). See memory [[os-data-split]].
|
||
>
|
||
> **2026-06-13 — v0.57.0 UI fixes (Part A of the UI-fixes/storage-spike spec):**
|
||
> - A1: fixed the RIGHT storage list — `#host-storage-bars` (the JS-filled, agent-PVE-storage list:
|
||
> `local`/`local-lvm`/`felhom-pbs`/`felhom-usb`), which reordered on every poll. Now
|
||
> `enrichHostStorageTargets` sorts `/api/host-metrics` server-side + adds friendly Hungarian
|
||
> labels/purpose. Display-only — PVE storage ids never renamed. (v0.56.0's 4C had sorted the OTHER,
|
||
> server-rendered user-data list.)
|
||
> - A2: per-app Tier-2 config panel at `GET/POST /stacks/{name}/backup`; the dead-end "Beállítás" button
|
||
> (was → deploy page) is repointed there. Pin a target drive / toggle Tier 2 off; prefs
|
||
> (`UserDisabled`/`PreferredTarget`) persist on `CrossDriveBackup` and survive the runner's status
|
||
> writes (`withTier2Prefs`). Always visible incl. single-SSD + non-HDD (PBS-context) apps.
|
||
> - Part B (storage OS/data split spike) = build-nothing; findings → `felhom-agent/REPORT-storage-split-spike.md`.
|
||
> - Live-validated on guest 9201; build/deploy = golden bootstrap (`/etc/felhom-controller-image` + restart `felhom-controller-bootstrap.service`).
|
||
>
|
||
> **2026-06-13 — v0.56.0 Phase 4: FileBrowser scoping + UI polish (SLICE COMPLETE):**
|
||
> - 4A: FileBrowser bind scoped to `<drive>/appdata` (recovery units + Tier 2 copies under `backups/`
|
||
> NOT mounted → customer can't browse/delete the restore source). 4B: deploy storage step states
|
||
> files-on-drive / DB-on-fast-SSD. 4C: `buildStorageBars` stable sort + purpose description on the
|
||
> monitoring list (user-data drives only; agent local/local-lvm/pbs live on the storage page, not here).
|
||
> - Live-validated (9201): FileBrowser mount `/mnt/felhom-usb/appdata -> /srv/felhom-usb` (backups hidden);
|
||
> deploy + monitoring text rendered. **All 5 phases (1, 2, 2b, 3, 4) shipped + live-validated, v0.52→v0.56.**
|
||
>
|
||
> **2026-06-13 — v0.55.0 Phase 3: auto off-drive Tier 2 (rootfs-headroom guard):**
|
||
> - `internal/backup/tier2.go`: rsync `-a --delete` of each HDD app's recovery unit + appdata → a
|
||
> DIFFERENT physical disk (`<target>/backups/secondary/<app>/`). Auto target: prefer another registered
|
||
> drive (off-disk via `system.SamePhysicalDevice`), else internal SSD for SMALL units only.
|
||
> - **Rootfs-headroom guard** (`tier2FitsHeadroom`, unit-tested): SSD = ~8G guest rootfs, so REFUSE
|
||
> unless the unit fits leaving reserve = max(2G, 20%) free; honest "needs 2nd HDD" status when nothing
|
||
> fits — never fills the rootfs. Status via surviving `settings.CrossDriveBackup`; "2. mentés" UI card
|
||
> now populated (`buildAppBackupRows`). Daily `tier2-backup` 03:30 + `POST /api/backup/tier2`.
|
||
> - **Live-validated (9201):** happy path (RomM → SSD, off felhom-usb, 77KB, "[SSD: DB/config only]");
|
||
> refuse path (1G userdata dummy → REFUSED with honest msg, rootfs not filled); UI card shows
|
||
> "Sikeres → belső SSD (csak DB/konfiguráció)". Demo cleaned.
|
||
> - Next: Phase 4 (FileBrowser scoping + deploy-UI DB-on-SSD note + monitoring sort).
|
||
>
|
||
> **2026-06-13 — v0.53.0/v0.53.1 Phase 2: per-app recovery unit (capture side, SECRET-FREE):**
|
||
> - Each app's `backups/primary/<app>/` becomes a self-contained recovery unit: `compose/`
|
||
> (docker-compose.yml + .felhom.yml + **secret-stripped** app.yaml) + db-dumps/ + volume-dumps/ +
|
||
> `manifest.json` (image pins, secret env-var NAMES, data_key names, checksums, secret_source note).
|
||
> - **Secret-free by design.** Decided after reading the ACTUAL hub code: hub is zero-knowledge (no app
|
||
> secrets); app.yaml + key live on the guest rootfs → in the PBS whole-guest snapshot. So the unit
|
||
> stores no secret/data-key/image; restore recovers secrets from the guest's app.yaml (live/PBS),
|
||
> regenerates nothing. `data_key` (DeployField.DataKey; AdventureLog SECRET_KEY marked) = fail-closed
|
||
> restore annotation only.
|
||
> - Capture needs no decryption (non-secret env is plaintext; excludes secret-named + encrypted keys).
|
||
> Wired into RunDBDumps AND the periodic RefreshCache (idempotent checksum-skip → no USB thrash).
|
||
> - **Deploy mechanism resolved:** controller in guest 9201 is golden/bootstrap-managed —
|
||
> `felhom-controller-bootstrap.service` docker-runs the tag from `/etc/felhom-controller-image`
|
||
> (gitea anon-pull). Deploy = build+push → anon-pull → update tag file → restart the service.
|
||
> - **Live-validated (9201):** RomM unit captured (images=3, secrets=3, data_keys=0), secret-leak grep
|
||
> = NO_LEAK.
|
||
> - **v0.54.0 Phase 2b (restore-from-unit + fail-closed gate):** `RestoreFromRecoveryUnit` recreates an
|
||
> app from its unit + secrets recovered from the GUEST's live app.yaml (`RecoverStackSecrets`,
|
||
> `stacks.RedeployFromEnv`), regenerating nothing. `reconcileRestoreSecrets` (pure, unit-tested) is the
|
||
> fail-closed gate: missing/empty data-key → REFUSE (needs PBS whole-guest restore); missing resettable
|
||
> secret → warn+proceed. Wired into `/backup/restore`. Gate + orchestration + data_key parsing
|
||
> unit/integration-tested; deployed v0.54.0 healthy.
|
||
> - **LIVE-validated (9201, AdventureLog):** unit manifest `data_key_env_vars:[SECRET_KEY]`
|
||
> (catalog→manifest live); with SECRET_KEY made unrecoverable, `POST /backup/restore` REFUSED with the
|
||
> exact fail-closed message BEFORE any compose-up. Demo has NO dashboard password → API open (auth+CSRF
|
||
> skipped), driven via public URL. NOTE: full deploy-with-data→restore e2e blocked because AdventureLog
|
||
> images don't fit the 8G guest rootfs ("no space left") — that's the Phase 3 rootfs-headroom concern
|
||
> seen live. Demo left clean (AdventureLog reverted to not-deployed).
|
||
> - Next: Phase 3 (Tier 2 auto off-drive, rootfs-headroom guard), Phase 4 (FileBrowser + UI).
|
||
>
|
||
> **2026-06-13 — v0.52.0 Phase 1 GATE: deploy-side double-nest fix (catalog) + path-agreement test:**
|
||
> - The `felhom-data` double-nest lived in the **app-catalog compose templates**
|
||
> (`${HDD_PATH}/felhom-data/appdata/<app>`), not in `deploy.go`. On a Model-A in-guest drive the mount
|
||
> already IS the `felhom-data` namespace, so it double-nested on disk while the v0.51.0 backup helpers
|
||
> resolved single-nested → divergence. Fixed all four HDD templates (romm, nextcloud, immich,
|
||
> paperless-ngx) → `${HDD_PATH}/appdata/<app>`.
|
||
> - New `internal/stacks/hddpath_agreement_test.go` locks deploy-resolver (`ParseComposeHDDMounts`) ==
|
||
> backup helper (`AppDataDir(NamespaceRoot(.,true))`). No controller runtime change → no image rebuild
|
||
> (deployed stays 0.51.0, functionally current; golden not rebaked for a no-op).
|
||
> - **Live (guest 9201):** git-sync auto-delivered the fix to all four stack files; RomM migrated
|
||
> (stop→move→verify→redeploy) from `/mnt/felhom-usb/felhom-data/appdata/romm` →
|
||
> `/mnt/felhom-usb/appdata/romm`, healthy + HTTP 200, no data loss, old namespace empty. **GATE PASSED.**
|
||
> - Next: Phase 2 (per-app recovery unit), Phase 3 (auto-enabled off-drive Tier 2 w/ rootfs-headroom
|
||
> guard), Phase 4 (FileBrowser scoping + deploy-UI DB-on-SSD note + monitoring sort).
|
||
>
|
||
> **2026-06-12 — storage UX polish (v0.45.0), pairs with felhom-agent v0.24.0:**
|
||
> - **Agent eject role-gate (Part A, felhom-agent v0.24.0):** `POST /disks/eject` now refuses to
|
||
> unmount system/backup storage *at the agent* (fail-safe to protected on ambiguity) — the UI hiding
|
||
> the button was never the control. Validated live on guest 9201 (eject `/var/lib/vz` → 403, no unmount).
|
||
> - **Controller (Part B, v0.45.0):** B1 deterministic `/api/disks` order (user-data→system→backup,
|
||
> alpha within); B2 init wizard excludes mounted drives; B3 **Regisztrálás** primary action for a
|
||
> mounted-but-unregistered user-data drive (`POST /api/storage/register`); B4 per-card purpose
|
||
> descriptions + app-backing tags + tiering note (`local` & `local-lvm` both kept); B5 eject already
|
||
> names affected apps. All validated live on guest 9201.
|
||
> - **Golden REBAKED** to controller 0.45.0 (`/root/build-golden.sh`; gitea allows anon pull, no creds);
|
||
> archive `local:backup/vzdump-lxc-9100-2026_06_12-09_53_03.tar.zst`, build guest 9100 purged.
|
||
|
||
---
|
||
|
||
## About Viktor (project owner)
|
||
|
||
- Works at Deutsche Telekom (Budapest), building Felhom.eu as a side business
|
||
- Felhom.eu: managed home-server service for Hungarian households
|
||
- Technical but prefers pragmatic solutions over over-engineering
|
||
- Runs all infrastructure on Gitea (gitea.dooplex.hu), k3s cluster for management
|
||
- Customer deployments use Docker Compose (not Kubernetes) for simplicity
|
||
|
||
### felhom-controller (this repo)
|
||
- **Version:** v0.16.1
|
||
- **Phase 1:** ✅ COMPLETE — Stack Manager + Deploy Flow
|
||
- **Phase 2:** ✅ COMPLETE — Monitoring & Health (scheduler, CPU/temp, healthchecks.io pings)
|
||
- **Phase 3:** ✅ COMPLETE — Backups (DB dumps, restic integration, manual trigger, **dedicated backup page**)
|
||
- **Phase 4:** ✅ COMPLETE — Monitoring Page with Metrics Store (SQLite, Chart.js, system + container metrics)
|
||
- **Phase 5:** ✅ COMPLETE — Authentication, Persistence & Settings Page (settings.json, password change, session management)
|
||
- **Phase 6:** ✅ COMPLETE — Monitoring Warnings, Dashboard Alerts & Notification System
|
||
- **Phase 7:** ✅ COMPLETE — Storage Overview, Per-App Backup Toggles & Limited Restore
|
||
- **Phase A:** ✅ COMPLETE — Storage Paths Foundation (registry, auto-discovery, per-app HDD_PATH, deploy dropdown, health monitoring)
|
||
- **Phase B:** ✅ COMPLETE — Storage Management UI Polish & Health Severity Fix (flash messages, label editing, app details, FS info, deploy free space, backup context)
|
||
- **Phase C:** ✅ COMPLETE — Storage Init Wizard, Data Migration & Startup Fix (disk scan/format/mount wizard, rsync-based migration, startup pings)
|
||
- **v0.11.1 bugfix:** ✅ COMPLETE — Storage Scan: system disk detection via host fstab + blkid UUID resolution; FSType enrichment via `blkid -o export`
|
||
- **v0.11.2 bugfix:** ✅ COMPLETE — /host-dev mount for block device access; `HostDevicePath()` helper; all format/scan/safety ops use /host-dev
|
||
- **v0.11.3 bugfix:** ✅ COMPLETE — Added `fdisk` package to Dockerfile (provides `sfdisk`; not in `util-linux` on Debian bookworm)
|
||
- **v0.11.4 bugfix:** ✅ COMPLETE — FormatAndMount: fixed sfdisk (wipefs+force+`,,`), mount (explicit device path), mount propagation (rshared), ASCII label, smart partition skip, findmnt verification
|
||
- **v0.11.6:** ✅ COMPLETE — FileBrowser auto-mount sync (`syncFileBrowserMounts()`) + 3 UI fixes (badge color, progress bar, button text)
|
||
- **v0.11.7:** ✅ COMPLETE — Stale data cleanup + FileBrowser sync after migration + deploy page title fix
|
||
- **v0.11.8:** ✅ COMPLETE — Per-App Cross-Drive Backup (3-2-1 rule): rsync/restic to secondary drive, deploy page UI, backup page summary, scheduler jobs, API endpoints
|
||
- **v0.11.9:** ✅ COMPLETE — UI Polish Fixes: spacing, tooltip on "Módszer", status dot instead of disabled checkbox, progressive disclosure, emoji cleanup
|
||
- **First app deployed:** Paperless-ngx on demo-felhom.eu (2026-02-13)
|
||
- **Running on:** demo-felhom (N100 mini PC) at 192.168.0.162:8080, felhotest (Proxmox VM) at router.abonet.hu:33022
|
||
- **All Phase 1-5 features working:** deploy, start/stop/restart/update, logs, health-aware states, auth, monitoring, backups, backup detail page, system monitoring page, settings page
|
||
|
||
## Architecture decisions
|
||
|
||
| Decision | Rationale |
|
||
|----------|-----------|
|
||
| Go stdlib for web (no Gin/Echo) | Minimal dependencies, single binary, easy to embed templates |
|
||
| Templates as go:embed HTML/CSS files | Zero runtime file dependencies (compiled into binary), but each template is a separate editable file |
|
||
| Docker Compose for customers (not k8s) | Simpler troubleshooting, customers don't need k8s knowledge |
|
||
| k3s for management infra only | Viktor's own services (gitea, monitoring, website) run on k3s |
|
||
| Cloudflare Tunnel for remote access | No port forwarding needed, works behind any NAT |
|
||
| app.yaml per stack | Separates deploy config from compose files, survives git pulls |
|
||
| Password fields require explicit input | Prevents accidental empty-password deployments |
|
||
| Health-aware state from Docker Status field | Docker's State says "running" even for unhealthy containers |
|
||
| Memory limits via deploy.resources.limits | Prevents runaway containers; ~50% headroom over expected usage |
|
||
| System info from /proc/meminfo + statfs | No external dependencies, cheap to read on each page load |
|
||
| mem_request vs mem_limit (K8s-inspired) | Requests = expected usage (hard block), limits = peak (overcommit OK) |
|
||
| 384MB reserved for system | Prevents deploying apps that would starve the OS/controller |
|
||
| Logo SVG embedded as Go constant | Same approach as CSS/HTML — zero external file deps |
|
||
| Git sync via os/exec git CLI | No Go git library needed, git is in the container image |
|
||
| SHA-256 for content comparison | Only copy changed files, avoid unnecessary disk writes |
|
||
| 30s debounce on manual sync | Prevents spamming the git server |
|
||
| Orphan = deployed but not in catalog | Safe lifecycle: remove from catalog → mark orphaned → user deletes via UI |
|
||
| FileBrowser as infra (not catalog) | Needed even after apps deleted (user browses HDD data); deployed by setup script |
|
||
| Protected HDD paths | Safety net: never delete top-level HDD dirs (media, storage, Dokumentumok, appdata) |
|
||
| Central scheduler (not ad-hoc goroutines) | Single place to register/monitor all periodic tasks, graceful shutdown, skip-if-running |
|
||
| CPU sampling via background goroutine | /proc/stat delta needs two readings — collector runs every 5s, GetInfo() reads cached value |
|
||
| Temperature from /host/sys (Docker mount) | Container can't read host /sys directly — mount /sys:/host/sys:ro, try /host/sys first |
|
||
| Restic password auto-generated | No manual setup needed — generated on first backup run, stored in named volume |
|
||
| DB discovery via docker inspect | No config needed — discovers postgres/mariadb containers by image name + env vars |
|
||
| Backup orchestrator with running flag | Prevents concurrent backups, supports both scheduled and manual trigger |
|
||
| modernc.org/sqlite (pure Go) | No CGO/gcc needed in Docker build stage — keeps `CGO_ENABLED=0` static binary |
|
||
| AlertManager state-based refresh | Alerts regenerated every 5min from health report — no persistent storage needed, always reflects current state |
|
||
| Notification relay via hub | Controller → hub → Resend → email. Hub acts as central relay: knows customer email, handles Resend API. Controller only needs hub URL + API key |
|
||
| In-memory notification cooldowns | Per-event-type cooldown map (default 6h). Lost on restart = acceptable (better to re-notify than miss). No persistence needed |
|
||
| Health status change detection | Only notify on degradation (ok→warn, ok→fail, warn→fail). Avoids spam on flapping. First run records baseline, doesn't notify |
|
||
| Resend HTTP API (no SMTP) | Direct POST to api.resend.com — same pattern as website contact-mailer. Simpler than SMTP setup, good deliverability |
|
||
| Preferences sync on save + startup | Controller pushes prefs to hub (not pull). Startup sync handles hub DB rebuild. Local save always succeeds even if sync fails |
|
||
| Chart.js embedded locally | Customer hardware may not have internet — CDN not reliable for offline environments |
|
||
| StackDataProvider interface | backup package needs stack data but can't import stacks (circular). Interface in backup, thin adapter in main.go |
|
||
| Password sync to hub via report | Restic password in Docker named volume on SSD. Hub sync provides redundancy for disaster recovery |
|
||
| App backup via HDD mounts only | Docker volumes at /var/lib/docker/volumes/ not mounted in controller. HDD data is the important user data; DB in volumes covered by nightly dump |
|
||
| Restore uses running mutex | Prevents concurrent backup+restore on same restic repo. Reuses existing `m.running` flag |
|
||
| Storage paths registry in settings.json | Multi-storage support: each app's HDD_PATH from app.yaml is authoritative. Auto-discovery on startup avoids manual config. Registry enables UI management + health monitoring per path |
|
||
| /mnt:/mnt:rw mount in controller | Replaces per-path HDD_PATH mount. Enables multi-storage + restore writes. All customer HDD mounts are under /mnt/ by convention |
|
||
| Per-app HDD_PATH resolution (app.yaml > global) | App's own env HDD_PATH is Priority 1, registered storage paths as fallback. Eliminates dependency on global controller.yaml hdd_path |
|
||
| Mount-point detection via syscall.Stat_t.Dev | Compares device ID of path vs parent dir — reliable check that path is on separate filesystem. Prevents data writes to SSD |
|
||
| Health severity: mount-point = warning | Non-mount-point is informational, not a service failure. FAIL reserved for genuinely broken things. Avoids false alarms on demo/test environments |
|
||
| FS info via findmnt + sysfs | `findmnt -n -o SOURCE,FSTYPE --target <path>` for filesystem type/device. `/sys/block/<dev>/device/model` for disk model. Best-effort, returns nil on failure |
|
||
| Query param flash messages | Stateless, no session store needed. Consistent with backup page pattern. `?storage_msg=success&storage_detail=...` |
|
||
| StorageLabels map on stacks page | Separate map passed to template (not modifying Stack struct). Built from deployed apps' HDD_PATH → registered path label lookup |
|
||
| Metrics downsampling via SQL | Bucket-based AVG in GROUP BY keeps Chart.js responsive with up to 30 days of data |
|
||
| 60s metrics collection interval | Good balance of resolution vs. storage — ~44K rows/month for system metrics |
|
||
| /etc/os-release mounted read-only | Container can't read host OS info directly — mount to /host/etc/os-release:ro |
|
||
|
||
## Key file locations on demo-felhom
|
||
|
||
```
|
||
/opt/docker/felhom-controller/ # Controller compose + config
|
||
├── controller.yaml # Customer config (domain, auth, paths)
|
||
├── docker-compose.yml # Controller's own compose
|
||
└── data/ # Controller persistent data (named volume)
|
||
|
||
/opt/docker/stacks/ # All app stacks
|
||
├── traefik/ # Reverse proxy (protected)
|
||
├── cloudflared/ # Tunnel (protected)
|
||
├── paperless-ngx/ # First deployed app ✅
|
||
│ ├── docker-compose.yml
|
||
│ ├── .felhom.yml # App metadata
|
||
│ └── app.yaml # Deploy config (env vars, locked fields)
|
||
└── whoami/ # Test stack (not deployed)
|
||
|
||
/mnt/hdd_placeholder/storage/ # HDD storage for apps
|
||
└── paperless/
|
||
├── consume/ # Drop files here for OCR
|
||
├── media/ # Processed documents
|
||
└── export/ # Backup exports
|
||
```
|
||
|
||
## Related repositories and their state
|
||
|
||
| Repository | Status | Notes |
|
||
|------------|--------|-------|
|
||
| felhom-controller | Active | This repo. Controller code + deploy scripts |
|
||
| app-catalog-felhom.eu | Active | 10 app templates, all with .felhom.yml metadata + memory limits |
|
||
| felhom.eu | Active | Website + hub/ subfolder (felhom-hub service) + k8s manifests |
|
||
| homelab-manifests | Stable | k3s cluster running (dooplex.hu services) |
|
||
| misc-scripts | Utility | collect-repo.sh, backup helpers |
|
||
|
||
## Gotchas & lessons learned
|
||
|
||
- `docker compose restart` ≠ `docker compose up -d` — restart doesn't pick up new images
|
||
- Go maps have random iteration order — always sort slices before displaying
|
||
- Docker `.State`="running" doesn't mean healthy — check `.Status` for "(health: starting)" / "(unhealthy)"
|
||
- Paperless-ngx needs `PAPERLESS_OCR_LANGUAGES` (plural) to install language packs, `PAPERLESS_OCR_LANGUAGE` (singular) to select
|
||
- In-memory Deployed flag must be set BEFORE `docker compose up -d` (not after) — compose can take 30-60s for image pulls, during which the UI would show a stale "Telepítés" button
|
||
- Cloudflare Tunnel handles *.demo-felhom.eu → Traefik handles Host()-based routing to containers
|
||
- BIOS "AC Power Recovery" must be enabled on N100 for auto-restart after power outage
|
||
- `docker compose up -d` returns exit 0 even when containers immediately crash-loop — need post-start status check to detect this
|
||
- When logging env vars for debugging, only log keys (not values) to avoid leaking secrets in log files
|
||
- Mealie image (`ghcr.io/mealie-recipes/mealie`) doesn't include wget/curl — use Python TCP socket check for healthcheck
|
||
- Mealie DB migrations on first start take ~40s (alembic) — use `start_period: 60s` to avoid premature unhealthy status
|
||
- Alpine-based images (filebrowser, vaultwarden) have wget via BusyBox — healthchecks with `wget --spider` work fine
|
||
- Deploy `sed` command to update image version must target only the `image:` line — naive `sed 's|name:OLD|name:NEW|'` also matches the service name line (e.g., `felhom-controller:` → `felhom-controller:0.2.12`), breaking YAML. Use `sudo sed -i 's|image:.*felhom-controller:[^ ]*|image: ...felhom-controller:NEW|'` or similar scoped pattern
|
||
- Hungarian quotation marks `„"` in YAML: `„` (U+201E) is safe inside YAML double-quoted strings, but the closing `"` must NOT be ASCII `"` (0x22) — it terminates the YAML string. Use `\"` escape or Unicode `"` (U+201D). This caused a silent parse failure for the entire `.felhom.yml` file
|
||
- Never silently swallow parse errors — always log them. Silent failures make debugging impossible (took a dedicated debug session to find a simple quoting issue) |