admin/felhom-controller
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(report): v0.154.0 R-48 wizard shipped; Part 3 STOPPED (v0.90.1 is wrapper-only, not a binary fix)

Browse Source
This commit is contained in:
authentik Default Admin
2026-07-21 08:36:50 +02:00
parent 3a9d744360
commit 70cb21b058
1 changed files with 232 additions and 269 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+232 -269
View File
@@ -1,314 +1,277 @@
# REPORT — R-47: the DB replay must not race the app (both restore paths) · felhom-controller v0.153.0 # REPORT — v0.154.0 (R-48 restore wizard) + Part 2 docs; **Part 3 STOPPED**, STOP-1 pending
**Date:** 2026-07-20 · **Repo:** `felhom-controller` (v0.152.0 → **v0.153.0**) · Trunk, pushed to Session date: **2026-07-21**. Executed on DooPlex as `kisfenyo`.
`main`. · **Baseline:** `main` @ `fd40b29` (clean, equal to `origin/main` at session start)
--- ## 0. Outcome at a glance
## 1. What was wrong | Leg | Status |
`felhom.eu/documentation/audits/DIAG-immich-restore-round2-2026-07-19.md`, finding **H4**. The
offsite reconstitution ran its designed sequence — safety dump → stop → start → replay — and the
replay aborted:
```
10:58:25 controller: replaying DB dump into immich-postgres
10:58:33 immich-server: "Reindexing clip_index" -> "Reindexed clip_index"
10:58:35 controller: ERROR relation "clip_index" already exists - exit status 3
```
`ImportDump` needs a running database container, so the code started the WHOLE stack first. That gave
the application an eight-second window to rebuild the very schema objects the dump was about to
create; under `ON_ERROR_STOP=1` the collision aborted the script. The data survived only because
`pg_dump` emits COPY before CREATE INDEX — a collision earlier in the script would have left a
genuinely half-restored database and reported it identically.
**Class defect.** The local `RestoreFromRecoveryUnit` had the same start-then-replay shape, hidden
inside `RecreateStackFromUnit` (which ended in a full `compose up -d`). Both are fixed here.
## 2. What was built
**Part 1 — the seams**
| Change | File |
|---|---| |---|---|
| `dbTypeForImage` extracted from `DiscoverDatabases` (behaviour byte-equivalent) and shared | `internal/appbackup/dbdump.go`, `internal/appbackup/dbservices.go` (new) | | **Part 1 — R-48 restore wizard (controller v0.154.0)** | **DONE.** Committed `3a9d744`, pushed, image `0.154.0` built + pushed. **Deliberately NOT deployed** (by design — the floor save is the deploy). |
| `DBServiceNames(composePath)` — sorted compose SERVICE names holding a DB; yaml.v3 `services:` map parse | `internal/appbackup/dbservices.go` (new) | | **Part 2 — capability-map cell fix (felhom.eu)** | **DONE.** Committed `ce8c539`, pushed. |
| `Manager.StartStackServices(name, services)` — scoped `up -d`, **refuses an empty list** | `internal/stacks/manager.go` | | **Part 3 — agent 0.90.1 publish + deploy** | **STOPPED before publishing — premise does not hold.** See §5. Nothing published, nothing deployed, felhom-pve untouched. |
| `RedeployFromEnv` split; persist half is `PersistUnitRedeployConfig` (starts nothing) | `internal/stacks/deploy.go` | | **STOP-1 — floor save + single-fire assertion** | **PENDING — needs the operator.** Preconditions verified green (§2). |
| `StackDataProvider`: `RecreateStackFromUnit` → `RecreateStackDefinitionFromUnit` (+ `StartStackServices`) | `internal/appbackup/appdata.go` | | **Phase D — the two post-evidence doc flips** | **NOT DONE**, correctly: both are gated on evidence that does not exist yet (§7). |
| Adapter: definition-only recreate + delegation | `cmd/controller/main.go` |
| `DBServiceNames` forwarder | `internal/backup/appbackup_bridge.go` |
**Part 2 — offsite** (`internal/backup/offbox_reconstitute.go`): DB services resolved from the LIVE ## 1. Baselines (re-confirmed live at session start)
compose before any mutation; fail-closed refusal when a DB exists but no service is identifiable;
sequence is now **stop → files → `StartStackServices(dbServices)` → replay → `StartStack` (full) →
health wait**; both failure exits from the window do a best-effort full start.
**Part 3 — local** (`internal/backup/restore_unit.go`): DB services resolved from the UNIT's compose | Repo | `main` @ start | Clean + synced | End state |
(it is about to become the live one) plus `hasReplayableDump` (excludes `pre-restore-` safety dumps); |---|---|---|---|
same fail-closed gate before the first mutation; sequence is now **stop → volumes → | felhom-controller | `b30e2e5` | yes | `3a9d744` (v0.154.0) |
`RecreateStackDefinitionFromUnit` → `StartStackServices` → replay → `StartStack` (full) → health | felhom.eu | `fb0b8c1` | yes | `ce8c539` |
wait**, with the pre-existing `dataErr` / "completed with data errors" semantics preserved. | felhom-agent | `8c55ac7` | yes | **unchanged — no commit, no publish, no deploy** |
Untouched, as specified: `restore_db.go`, `ImportDump`, `waitDBReady`, the dump flags ## 2. Phase-0 probes
(`--clean --if-exists`, `ON_ERROR_STOP=1`), `mapOffsiteRestorePaths`, the copiers, the honesty
surfaces, `IsDownState`/alerting (R-51), and the agent/hub.
## 3. Tests — 19 new, Groups A–G ### P1 — self-update enabled and hub-wired on guest 9201: **PASS**
From `/var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml`:
```yaml
self_update:
auto_update: false
check_interval: 6h
enabled: true
health_timeout_seconds: 60
image: gitea.dooplex.hu/admin/felhom-controller
```
`auto_update: false` is **not** a blocker, and this was verified by reading the code rather than
assumed: `MaybeAutoUpdate` (`internal/selfupdate/updater.go`) never consults `cfg.AutoUpdate`. That
flag is the customer's opt-in to chase *latest*; the FLOOR path is the managed one and is independent
of it.
Stronger evidence than the config — the path has already fired on this box.
`data/update-state.json`:
```json
{ "status": "success", "previous_version": "0.143.0", "target_version": "0.145.0",
"initiated_by": "auto-floor", "initiated_at": "2026-07-18T16:32:29Z",
"completed_at": "2026-07-18T16:32:34Z" }
```
`initiated_by: "auto-floor"` proves the agent swapper is wired (a nil agent short-circuits with
"no agent — auto-update unavailable") and that a floor-driven swap completes in ~5 s on this box.
Anti-flap will **not** block the 0.154.0 swap: the persisted `target_version` is `0.145.0`, not
`0.154.0`, and the in-process `lastAutoFloorAttempt` is unset for it (the current-≥-floor branch
returns before setting it). Live guest state at probe time: image `0.153.0`, `Up 15 hours (healthy)`,
container `StartedAt 2026-07-20 15:03:11 UTC`.
Registry precondition for the floor validation also verified: `0.154.0` is the **highest** semver tag
in `admin/felhom-controller` (23 tags; top four `0.151.0, 0.152.0, 0.153.0, 0.154.0`), so the
`floor <= latest` gate passes rather than deferring.
### P2 — agent version: **PASS, with a finding that changed the plan**
`cmd/felhom-agent/main.go:59` holds `var version = "0.89.0"` as a *fallback only*; the real version is
ldflags-injected (`-X main.version`) by `scripts/publish-agent.sh`. Built at `main` and verified by
self-report, not by CHANGELOG:
```
$ felhom-agent --version
felhom-agent 0.90.1
sha256 ba1d029602c96b1b743a4dca3ddbd9b63415fcb47021b91a21f8c32dcc534870 13 734 067 bytes
```
The R-39 fix commit `9596d5a` is an ancestor of `main` and is not reverted. Agent green gate passes.
**The finding is in §5** — this binary does not contain the R-39 fix, and cannot.
### P3 — before-state of `/backups/restore`: **CAPTURED**
Method: **endpoint-level** (no browser on DooPlex). Authenticated session against the container IP
with the `Host:` header, from inside guest 9201. `LOGIN_HTTP=302`, `PAGE_HTTP=200`, 58 115 bytes.
Offsite section, buttons in render order — this **is** the R-48 defect:
```
immich / bookstack / calibre-web each:
- Visszaállítás ellenőrzéshez (konfiguráció + adatbázis)
- Teljes visszaállítás előkészítése
immich additionally (scratch prepared):
- Helyreállítás az élő adatok közé (csak a hiányzó fájlok) <- data CANNOT come back
- Teljes visszaállítás (fájlok + adatbázis) <- data CAN come back
```
Form actions on the page: 6× `/backup/offbox/restore`, 1× `place`, 1× `reconstitute`. The two decisive
controls are adjacent siblings differing only by label. Snapshot retained at
`scratchpad/restore-before.html`.
## 3. Part 1 — what shipped (commit `3a9d744`)
Files touched: `internal/web/restore_wizard.go` (new), `templates/backups_restore_wizard.html` (new),
`internal/web/restore_wizard_test.go` (new), `templates/backups_restore.html`, `offbox_handlers.go`,
`server.go`, `templates/style.css`, plus `CHANGELOG.md` / `CONTEXT.md` / `REUSE.md` /
`controller/README.md`.
- **One entry per app.** The five inline forms per row collapse to a single „Visszaállítás…" link to
`GET /backups/restore/app?name=<app>`.
- **Three intent CARDS** with consequence sentences, danger styling on card 3, the R-43
double-confirm and its pair-honesty facts carried over **verbatim**.
- **`deriveWizardStep` is pure** over (op running, size-gate flash, scratch ready). Precedence is
strict and load-bearing: a running op outranks a stale `?full_prep=`.
- **No new mutation endpoint.** One GET route added; every card posts to the pre-existing
`/backup/offbox/{restore,place,reconstitute}` with unchanged field names and gates.
- Untouched as instructed: the shares block, the local restore panel, the .fab block,
`internal/backup`, `internal/appbackup`, `internal/selfupdate`.
**Bug found and fixed on the way (not in the spec).** `offboxRedirectTo` hardcoded `"?"` when
appending its flash. Retargeting redirects at a URL that already carries `?name=<app>` would have
produced `...?name=immich?flash=...`, burying the flash inside the `name` value — the wizard would
then have refused its own app with "nincs kijelölve" after every action. The separator is now chosen.
### Test results
Full suite green: `go build ./... && go vet ./... && go test ./...` — **0 failures**.
All six controller design gates pass (`template_id`, `emoji`, `mojibake`, `native_confirm`,
`offbox_rename`, `app_row_dedup`).
| Group | Test | Result | | Group | Test | Result |
|---|---|---| |---|---|---|
| A | `TestReconstituteReplaysWithOnlyTheDBServiceUp` — order **plus state-at-replay-time** | PASS | | B | `TestDeriveWizardStep_Table` (7 rows) | PASS |
| A | `TestReconstituteReplaysDBAndOrdersOperations` (existing, sequence assertion updated) | PASS | | C | `TestResolveWizardApp_Refusals`, `TestRestoreWizardHandler_UnconfiguredRedirects` | PASS (302, no 500) |
| B | `TestReconstituteNoDBAppNeverStartsServicesOnly` — negative, zero scoped starts | PASS | | A | `TestRestoreList_SingleEntryPerApp` | PASS |
| C | `TestReconstituteRefusesWhenNoDBServiceIdentifiable` — zero-mutation effect | PASS | | C | `TestRestoreWizard_ThreeIntentCards`, `TestRestoreWizard_NoScratchLocksDataIntents` | PASS |
| C | `TestRestoreFromUnitRefusesWhenNoDBServiceIdentifiable` — zero-mutation effect | PASS | | E | `TestRestoreWizard_OpRunningSuppressesAllMutations` | PASS |
| D | `TestRestoreFromUnitReplaysWithOnlyTheDBServiceUp` | PASS | | D | `TestRestoreWizard_NoNewMutationEndpoints`, `TestRestoreWizard_FieldContract` | PASS |
| D | `TestRestoreFromUnitNoDumpsTakesOneFullStart` | PASS |
| D | `TestRestoreFromUnitIgnoresSafetyDumpsWhenDecidingToReplay` | PASS |
| E | `TestReconstituteReplayFailureStillBringsTheStackUp` | PASS |
| E | `TestReconstituteDBOnlyStartFailureStillBringsTheStackUp` | PASS |
| E | `TestRestoreFromUnitReplayFailureStillBringsTheStackUp` | PASS |
| F | `TestDBTypeForImage`, `TestDBServiceNames` (8 sub-cases), `TestDBServiceNames_TopLevelKeysAreNotServices`, `TestDBServiceNames_UnreadableAndUnparseableError`, `TestDiscoverAndComposeAgreeOnTheSameImages` | PASS |
| G | `TestStartStackServicesRefusesEmptyList`, `TestPersistUnitRedeployConfigPersistsWithoutStarting`, `TestPersistUnitRedeployConfigRejectsUnknownStack` | PASS |
The core assertion is deliberately not "no error": a recording provider captures whether the FULL Two pre-existing tests were coupled to the old IA and were updated, not deleted:
stack had been started at the moment the import fired. Asserting only `err == nil` passes on the `TestAppRow_RestoreLists` and `TestBackupsSplit_SectionsOnExactlyOnePage` asserted
pre-fix shape — which is exactly how this shipped. `action="/backup/offbox/restore"` on the list page — now inverted to assert those forms are **absent**
there and the single wizard entry is present.
The compose-parser decoys use the catalog's REAL immich template shape (`immich_ml_cache:`, ### Group-B red-proof (run, then reverted)
`immich_postgres_data:` as top-level `volumes:` keys, `ghcr.io/immich-app/postgres:16-vectorchord…`
as the pin) — the exact input a line scan would misread.
### Companion red-proofs — three run, all reverted, tree clean Replaced the body of `deriveWizardStep` with the trivial `return restoreWizardView{Step:
wizStepIntent, VerifyEnabled: true}`. Result — **all 7 table rows FAILED**, plus the execution render
| # | Pre-fix shape restored | Failure observed | test:
|---|---|---|
| 1 | offsite: `StartStackServices` → full `StartStack` before the replay | `TestReconstituteReplaysWithOnlyTheDBServiceUp`: *"the database service was NOT started before the replay"*; `TestReconstituteReplaysDBAndOrdersOperations`: sequence `"stop,start,start"` |
| 2 | local: full `StartStack` inserted before the replay | `TestRestoreFromUnitReplaysWithOnlyTheDBServiceUp`: *"the FULL stack was already up when the replay fired — the H4 race, on the local path"* |
| 3 | both fail-closed gates deleted | both `RefusesWhenNoDBServiceIdentifiable` tests: *"expected a refusal…"* |
### Green gate
`go build ./... && go vet ./... && go test ./...` — **23/23 packages green**, exit 0
(`internal/backup` 174 s). New tests by package: backup +11, appbackup +5, stacks +3.
## 4. Deployment
Built + pushed from the clean tree at `78ff991`: `gitea.dooplex.hu/admin/felhom-controller:0.153.0`,
digest `sha256:cc02c950c47456dac01aa754f23befbdfb3d4897b0a9863aafa3589cff95b8ab`. Deployed to guest
9201 via the bootstrap path (pull → `/etc/felhom-controller-image` → restart bootstrap service).
Verified: `gitea.dooplex.hu/admin/felhom-controller:0.153.0 | Up (healthy)`, clean startup, and
`Event pushed: controller_started (info) — Controller elindult (0.153.0)`.
## 4b. STOP-1 — supervised live leg: **PASSED** (2026-07-20, operator present)
Method: **endpoint-level** — the exact endpoints the UI posts to, driven with an authenticated
session inside the guest (no browser on DooPlex; no state hand-setting, no `docker exec` shortcut
around the pipeline). App: **immich** (a real DB-indexed app), snapshot `49e7cb46` — the SAME
snapshot that aborted in round 2.
Baseline before the run: **11 assets, all `active`**.
1. `POST /backup/offbox/restore` `app=immich mode=full confirm=1` → 302; scratch staged
**15:30:08 UTC**: `restored immich (49e7cb46, full=true) → …/backups/offsite-restore/immich`.
2. `POST /backup/offbox/reconstitute` `app=immich confirm=1` → 302, fired **15:39:36 UTC**.
**Controller log — the ordering, live:**
``` ```
15:39:42 [offbox] immich: pre-restore safety dump written -> pre-restore-20260720T153941Z-immich-postgres.sql (49.9 MB) --- FAIL: TestDeriveWizardStep_Table/op_running_(this_app)_→_execution;_nothing_offered
15:39:42 [stacks] Stopping stack: immich --- FAIL: TestDeriveWizardStep_Table/op_running_OUTRANKS_a_stale_full_prep_flash…
15:39:43 [stacks] Starting stack immich services only: [immich-postgres] <- THE FIX --- FAIL: TestDeriveWizardStep_Table/scratch_ready_→_intent,_and_BOTH_data-touching_intents_unlock…
15:39:43 [backup] Restore immich: replaying DB dump into immich-postgres (postgres) --- FAIL: TestDeriveWizardStep_Table/full_prep_flash_for_THIS_app_→_prepare-confirm…
15:40:03 [backup] Restore immich: replayed 1 DB dump(s) <- rc-0, 20 s (+3 more rows)
15:40:03 [stacks] Starting stack: immich <- full start, only now --- FAIL: TestRestoreWizard_OpRunningSuppressesAllMutations
15:40:27 [offbox] reconstituted immich from snapshot 49e7cb46: 6 file(s) placed, restore_wizard_test.go:253: the execution step must render NO mutation form
1 DB dump(s) replayed, safety dump=pre-restore-…, skewed=false restore_wizard_test.go:256: the execution card must name what is actually running
``` ```
**Verification through the system's own surfaces:** Implementation restored; suite green; `git diff` clean before commit.
| Check | Round 2 (v0.148.0) | This run (v0.153.0) | One assertion in `TestRestoreWizard_ThreeIntentCards` was itself caught being hollow during
|---|---|---| authoring: a bare substring check for the skew/empty warnings passed on a *clean* pair, because the
| `already exists` / replay abort | `ERROR relation "clip_index" already exists` (exit 3) | **none** — replay exited 0 | `confirmFullRestore` JS repeats both sentences as string literals. Tightened to assert the rendered
| App up during replay | yes (immich-server rebuilt `clip_index` mid-replay) | **no** — only `immich-postgres` was up | banner markup and the `data-restore-*` attributes instead.
| Operation outcome | reported FAILURE | **success**, `dbs=1` |
| immich's own schema verdict | **schema drift** reported | **`No schema drift detected`** — twice (Microservices + Api) |
| Assets | 11 (recovered by luck of `pg_dump` ordering) | **11, all `active`** |
| Containers | — | all four immich containers `Up (healthy)` |
| `public` indexes | incomplete (aborted script) | **231** |
The decisive line is `Starting stack immich services only: [immich-postgres]` followed by a replay ### Image
that exits 0 — the window in which H4 occurred no longer exists. The DB service name was resolved
from the LIVE compose by `DBServiceNames`, unassisted.
Credentials handling per GL-1: the operator's password was supplied file→file, never echoed, and the `gitea.dooplex.hu/admin/felhom-controller:0.154.0` built + pushed, 145 MB. Self-report verified:
file plus both cookie jars (host and guest) were shredded at the end of the run.
## 4c. Phase C — golden **0.153.0** baked and published (2026-07-20)
Probes first, all before any mutation:
| Probe | Result |
|---|---|
| **P1** — read `180:/mnt/5_hdd/felhom.eu/drill/bake-0.146.0.log` end to end | recipe extracted: `pct`-based LXC 9100 on felhom-pve, args `9100 <template> local-lvm local vmbr0 <controller-image>`, creds via `REGISTRY_USER`/`REGISTRY_TOKEN`, publish to the gitea generic package registry |
| **P2** — build environment matches the log's starting point | 9100 absent (clean slate ✓); local-lvm 277 G / local 80 G free ✓; **DEVIATION: the Debian 13 template cache was EMPTY** (the 0.146.0 bake had it cached) |
| **P3** — registry reachable from the vacation site (**load-bearing**) | **PASS.** `gitea.dooplex.hu` → `37.191.56.193` (public path); registry `/v2/` → 401 (alive, auth-gated); gitea API → 200; Docker Hub `/v2/` → 401. Independently corroborated: guest 9201 pulled `felhom-controller:0.153.0` from that registry the same afternoon |
**P2 deviation, handled explicitly:** `pveam download local debian-13-standard_13.6-1_amd64.tar.zst`
(129 954 319 B, checksum verified, 22 MB/s). This is the ONE departure from the 0.146.0 recipe and
it is environmental, not a build-golden defect. Operator approved before it ran.
Bake: `build-golden.sh` **v2.1.0** (confirmed on the host copy), run **detached under `setsid`+`nohup`**
so an SSH blip over the tailnet could not orphan a half-built guest. Controller image
`gitea.dooplex.hu/admin/felhom-controller:0.153.0`.
### Gate table — vs the 0.146.0 row
| Gate | 0.146.0 | **0.153.0** |
|---|---|---|
| Script version | v2.0.0 | **v2.1.0** |
| Docker in build guest | overlay2, separate `/var/lib/docker` + `/mnt/sys_drive` mounts | same ✓ |
| Controller image baked | 0.146.0 | **0.153.0** ✓ |
| **Infra images baked** | **3** (historical fallback list) | **4** — `traefik:v3.6.7`, `cloudflare/cloudflared:2026.6.0`, `gtstef/filebrowser:1.3.3-stable`, **`gitea.dooplex.hu/admin/felhom-samba:1.1.0`** ✓ |
| Source of the infra list | hard-coded fallback | **`--print-infra-images` from the 0.153.0 binary itself** — no WARN line in the log, so the fallback never fired |
| `docker image ls` inside the guest ⊇ that list | — | **verified** — all four present, plus the baked controller |
| Archive | 584 MB | **618 MB**, volid `local:backup/vzdump-lxc-9100-2026_07_20-17_50_57.tar.zst` |
| Publish: pre-delete | HTTP 404 | **HTTP 404** ✓ |
| Publish: upload | HTTP 201 | **HTTP 201** ✓ |
| Anonymous GET + byte-match | — | **HTTP 200, 648 549 366 B, sha256 matches exactly** ✓ |
| Ranged request (resumable restore) | — | **HTTP 206**, 1024 B ✓ |
| Hub can fetch it from its own network position | — | **verified** — `wget` from the `hub-…` pod exits 0 |
``` ```
GOLDEN_VERSION=0.153.0 felhom-controller 0.154.0 (built 2026-07-21T06:31:08Z, commit 3a9d744)
GOLDEN_SHA256=15fdd191f3c660a60dc8651111053dd84281aeebc6c4c0f9ecdd3a87cb45a9d0
``` ```
**GL-1 teardown:** the token was written to a `0600` env file via STDIN (never in argv on either commit matches `HEAD`. **Not deployed to 9201** — by design; the floor save is the deploy.
host), sourced by the detached run, and **shredded**. Token-leak grep over the bake transcript
(host copy and the retained copy) = **0**; the build guest holds no registry credential; build guest
9100 left **stopped** (destroy with `pct destroy 9100 --purge`); the 618 MB local verification copy
was deleted. Log retained beside its predecessor: `180:/mnt/5_hdd/felhom.eu/drill/bake-0.153.0.log`.
**Honest scope note:** I verified the golden artifact is published, byte-correct and fetchable *by ## 4. Part 2 — capability-map cell (`ce8c539`)
the hub*. I did **not** render the hub's golden **dropdown** — that is the password-gated operator
UI, and it remains part of STOP-2.
## 4d. STOP-2 — hub saves: **DONE** (operator, 2026-07-20) Row 61's status cell read `**PARTIAL** — *scope corrected 2026-07-19*` while its own note body already
ended with the 2026-07-20 destructive drill and the words "the row now earns PROVEN-LIVE". Cell
corrected to `**PROVEN-LIVE** (2026-07-20)`; the full note history retained verbatim.
Both password-gated saves applied in the correct order and confirmed by screenshot: ## 5. Part 3 — **STOPPED. The premise is wrong; publishing would encode a false claim.**
| Setting | Value | Part 3 asked me to publish + deploy v0.90.1 as "R-39's cheap half — the PBS wrapper argv fix", while
|---|---| Rule 12 forbids touching the wrapper. Those two cannot both be satisfied, because **the fix IS the
| Day-0 artifact — Golden | **0.153.0**, sha256 `15fdd191f3c660a60dc8651111053dd84281aeebc6c4c0f9ecdd3a87cb45a9d0` (matches the bake exactly) | wrapper**. Evidence, in order:
| Day-0 artifact — Agent | 0.90.0 (unchanged) |
| Min agent | 0.90.0 (unchanged — v0.153.0 declares no coupling) |
| Managed-updates global floor | **v0.153.0**, source `DB (hub_settings)`, saved **last** ✓ |
**Agent 0.90.1 is absent from the dropdown — correct, not a defect.** It was built on the old **(a) v0.90.1 contains zero non-test Go changes.** `git show 9596d5a --stat`:
Windows workstation but **never published**: `felhom-agent/0.90.1` in Gitea returns **404** (0.90.0
returns 206), and felhom-pve reports `felhom-agent 0.90.0`. The hub lists only published versions.
The v0.90.1 source IS committed (`9596d5a`, the R-39 `pvesm set --server` hotfix), so shipping it is
a build+publish away — tracked under R-39, out of scope here.
Build guest **9100 destroyed** (`pct destroy 9100 --purge`, all three LVs removed); only 9201
remains on the host.
## 4e. C6 — the DESTRUCTIVE customer-restore drill: **PASSED** (operator, 2026-07-20)
This is the run that today's earlier reconstitute could not be: **data was genuinely destroyed
first.** Operator deleted the photos in immich's own UI and **emptied the trash** (the step whose
absence makes a drill prove nothing), then ran the restore from the customer-facing UI —
„Teljes visszaállítás (fájlok + adatbázis)" on `/backups/restore`.
``` ```
17:13:42 [offbox] immich: pre-restore safety dump written (49.6 MB) CHANGELOG.md | 51 +
17:13:42 [stacks] Stopping stack: immich REPORT.md | 184 +-
17:13:42 [stacks] Starting stack immich services only: [immich-postgres] configs/felhom-pbs-apply | 11 +- <- the wrapper: the actual fix
17:13:43 [backup] Restore immich: replaying DB dump into immich-postgres (postgres) internal/pbsdr/manager_test.go | 48 + <- the red-proof test
17:14:03 [backup] Restore immich: replayed 1 DB dump(s) <- rc-0
17:14:03 [stacks] Starting stack: immich
17:14:27 [offbox] reconstituted immich from snapshot 49e7cb46:
40 file(s) placed, 1 DB dump(s) replayed, skewed=false
``` ```
**`40 file(s) placed` is the proof of destructiveness** — the 15:40 non-destructive run over an Its own commit message says so outright: *"Config-only (wrapper + red-proof); **the Go binary is
already-good tree placed only 6. The files were really gone and really came back. unchanged**, so this ships with the next agent deploy as a config artifact."* The 0.90.1 binary is
functionally identical to the 0.90.0 already running — only the ldflags string differs.
Verified after: **11 assets `active`**, all four containers healthy, immich's own DatabaseService **(b) felhom-pve already has the fix.** The wrapper was hotfixed in place on 2026-07-18:
logged **`No schema drift detected`** (17:14:20), and the operator confirmed the photos are visible
in the timeline. Full UI path, no endpoint shortcuts — the customer's own buttons.
## 4f. R-23(a) — operator-UI save→apply round trip: **evidence banked** (2026-07-20) ```
/usr/local/sbin/felhom-pbs-apply:107: args=(--fingerprint "$fp") <- fixed (no --server)
/usr/local/sbin/felhom-pbs-apply.bak-20260718-preR39 <- rollback retained
```
Not a task deliverable; captured opportunistically because the STOP-2 floor save happened to be the **(c) New hosts already get the fix too, independent of the binary version.**
exact trigger R-23(a) needs, and the evidence was still in the debug ring. `felhom-host-install.sh:1914` fetches `configs/felhom-pbs-apply` via `fetch_raw`, which pulls from
`raw/branch/main` — **unversioned**. `9596d5a` is an ancestor of `main`, so every install since
2026-07-18 has installed the fixed wrapper.
| Source | Time | Event | **Conclusion.** R-39's cheap half is *already closed in the field*, through a channel that carries no
|---|---|---| version number. Publishing + deploying the 0.90.1 binary would:
| Hub (CEST) | 18:56:27 | `Global controller-version floor set to "0.153.0"` |
| Controller (UTC) | **16:56:27Z** | `wait woke: generation=1 — firing out-of-cycle report` |
| Controller (UTC) | 16:56:29Z | `Building system report` |
`18:56:27 CEST` == `16:56:27 UTC` — **the same second**; the out-of-cycle report follows 2 s later. - deliver **no functional change** to felhom-pve;
The ring also shows `wait baseline generation=0` at startup (baseline recorded WITHOUT firing, as - restart the agent on a **production host at a remote site** for that nothing;
designed) then `generation=1`, so the generation advanced past 0. - and, once the operator saved Day-0 manifest Agent → 0.90.1 in the same sitting (as STOP-1
contemplates), **advertise that the R-39 fix shipped as a versioned artifact — which it never did.**
A fresh box gets the fixed wrapper from `main` whether it installs binary 0.90.0 or 0.90.1, so the
version number would be a false signal about fleet state.
**The wake is `logx.Debugf`, so `docker logs` at INFO shows nothing** — this evidence exists only in Per §0 ("if anything requires touching agent source → STOP and report") and the Part-3 STOP rule, I
the debug ring (`GET /api/debug/logs?level=DEBUG`, never gated on `logging.level`). Anyone hunting stopped **before** the publish. **Nothing was published. Nothing was deployed. `felhom-agent` has no
for it in stdout will wrongly conclude the box did not wake. That is arguably R-23(b)'s complaint new commit and felhom-pve is byte-for-byte as found.** The built binary sits unpublished in the
generalised: the Direction-2 path's most interesting moment is invisible at the default level. session scratchpad; the registry pre-check (the diagnosed state) was confirmed and left as-is:
**Still unproven, and R-23(a) is NOT complete:** the **self-restart** leg. The floor was set to ```
0.153.0 on a box already running 0.153.0, so there was no work to do and no restart — the GET .../generic/felhom-agent/0.90.1/felhom-agent -> 404 (authed and anonymous)
"restart **single-fire**, no storm, once the generation has advanced past 0" assertion remains GET .../generic/felhom-agent/0.90.0/felhom-agent -> 200 (control)
unexercised. Finish it by bumping the floor to a version the box does NOT yet run, with the debug ```
ring open, and asserting **exactly one** restart.
## 5. NOT yet live-validated — remaining human/supervised work **This needs an operator ruling** — see §8.
- ~~**STOP-1**~~ — **DONE 2026-07-20, PASSED.** See §4b for the log ordering, timestamps and the ## 6. STOP-1 — not yet run (needs the operator)
before/after comparison against round 2. Residual: the immich **timeline screenshot** still cannot
be captured from here (no browser on DooPlex) — that one visual remains the operator's.
- ~~**Phase C**~~ — **DONE 2026-07-20.** Golden 0.153.0 baked, published, all gates green
(§4c). `GOLDEN_SHA256=15fdd191f3c660a60dc8651111053dd84281aeebc6c4c0f9ecdd3a87cb45a9d0`.
Housekeeping left for whoever is next at the host: `pct destroy 9100 --purge`.
- ~~**STOP-2**~~ — **DONE** (§4d). Golden 0.153.0 + floor v0.153.0 saved in the right order.
- ~~**C6 destructive customer-restore drill**~~ — **DONE 2026-07-20, PASSED** (§4e): trash emptied,
restored through the customer UI, 40 files placed, 11 assets back, no drift.
- **R-23(a) — PARTIALLY banked** (§4f). Save→wake→report proven to the same second. **The
self-restart single-fire leg is still open**: bump the floor to a version the box does NOT run,
with the debug ring open, and assert exactly one restart.
- **R-48 — restore-page layout still janky** (operator-confirmed again 2026-07-20 during the C6 run:
the overlong button row persists). Unchanged by this task and still the highest-value UX item on
that page, since it is the finding that CAUSED the round-1 incident.
## 6. Observations (out of scope, recorded not acted on) Preconditions are green (§2 P1) and the target image is published and verified. The assertion is
ready to run the moment Viktor saves the floor; I have not observed a swap, so **no R-23(a) evidence
exists yet and the map row is untouched.**
- **The `pre-restore-` prefix is load-bearing in three separate places** (the replay's exact-name ## 7. Doc flips deliberately NOT made
match, `OffsiteScratchPair`'s dump sniff, and now `hasReplayableDump`) with no shared predicate
deciding "is this file a replay source". A fourth consumer that forgets the exclusion would arm the - **R-23(a) → banked**: requires STOP-1 single-fire evidence. Does not exist. Not flipped.
DB-only window for an app that has nothing to replay. Worth a single helper at some point. - **"config/state change round-trips in seconds" PARTIAL → PROVEN-LIVE**: same gate. Not flipped.
- **`reimportDBDumpsFrom`'s own `hasDump` scan does NOT exclude the safety prefix** (unchanged here — - **R-48 → SHIPPED** and the **R-39 artifact note** in `ROADMAP.md`: not written this session. R-48's
`restore_db.go` was explicitly out of scope). Harmless today because the per-DB lookup is an exact is merely deferred until the live click-through; the R-39 line is deferred **because the sentence
`<stack>-<dbtype>.sql` match, so a directory holding only safety dumps merely produces the the task specified would have been false** (§5).
"no matching running DB container" WARN instead of a clean zero.
- **`dbTypeForImage` maps both `mysql` and `mariadb` to `DBTypeMariaDB`.** Pre-existing and correct ## 8. Remaining — human
for the current catalog (the `mariadb` client speaks to both), but it is an assumption, not an
invariant, and it is now written down in one place instead of two. 1. **Ruling on Part 3 (blocking).** Options as I see them: (i) publish 0.90.1 anyway as a
- **`RedeployFromEnv` has no end-to-end test** (it shells out to compose), so the split's equivalence version-hygiene artifact, with the CHANGELOG stating plainly that the binary is unchanged and the
is asserted on the persist half only. That is the half the split could break; the tail is R-39 fix rode the wrapper — cheap, but the Day-0 manifest still implies more than it delivers;
byte-identical code that was moved, not rewritten. (ii) leave 0.90.0 as the published binary and close R-39's cheap half in the docs, recording that
- **R-29a (`estimate.go` gate finding)** remains open and was not touched. the wrapper is delivered unversioned from `main`; (iii) treat "a privileged host artifact tracks
- **CORRECTION to an earlier claim in this session.** Mid-run I reported that the controller password `main` with no version" as the real defect and spec that as TASK-B alongside R-39's primary half.
in `180:~/.config/credentials` was **stale** and had been superseded by the customer-claim flow. **Recommendation: (ii) now, (iii) as the actual fix** — (iii) is what stops a root-owned wrapper
**That was wrong.** The values in that file are single-quoted (`PASSWORD='…'`) and my from silently tracking `main`, which is the same class of problem R-39 already demonstrated once.
`cut -d= -f2-` extraction kept the quotes, so a 15-character string with literal `'` on both ends 2. **STOP-1**: floor save 0.153.0 → **v0.154.0**, password-gated. I observe and assert single-fire.
was sent instead of the 13-character password. Re-tested with the quotes stripped: `302` + session 3. **Wizard click-through + one non-destructive Ellenőrzés** through the new surface.
cookie. The credential was correct the entire time; no rotation, no drift, nothing to reconcile. 4. **TASK-B** ruling still pending (R-39 primary half: the descriptor-hash re-apply gap).
The lesson recorded in memory is the general one: **an auth failure is evidence about the bytes you 5. **C6 customer-restore pass** for the alpha.
sent, not proof about the stored secret** — a 2-character length discrepancy is this bug's
signature, and it was stated confidently three times before being caught. ## 9. Observations
- **`docker_run_volume_path_gate.py` fails at `HEAD`, unrelated to this work.**
`internal/appexport/estimate.go:179` has an unreviewed `docker run -v`. Verified pre-existing by
stashing my changes and re-running (still rc=1). Left alone — out of scope — but it means that gate
is currently not a usable signal.
- **The `?`-vs-`&` redirect bug (§3) is the kind the wizard makes more likely**, since app-scoped
pages now carry query strings as a matter of course. `offboxRedirectTo`'s REUSE.md row now carries
the trap.
- **`felhom-pve` is up and healthy** despite the vacation note in `CLAUDE.md` saying the agent is
down: `systemctl is-active` → `active`, `felhom-agent 0.90.0`. The F1 static-`.162` mitigation is
holding; the `CLAUDE.md` TEMPORARY block is stale on that point.
- **P3's accented-grep trap fired as documented.** ASCII-safe substrings were required; an accented
pattern through `ssh → pct exec → bash -c` returned a false 0. Used a local Python pass over the
fetched bytes instead of grepping remotely.