admin/felhom-controller
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(report): v0.154.0 R-48 wizard shipped; Part 3 STOPPED (v0.90.1 is wrapper-only, not a binary fix)

Browse Source
This commit is contained in:
authentik Default Admin
2026-07-21 08:36:50 +02:00
parent 3a9d744360
commit 70cb21b058
1 changed files with 232 additions and 269 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+232 -269
View File
@@ -1,314 +1,277 @@
# REPORT — R-47: the DB replay must not race the app (both restore paths) · felhom-controller v0.153.0
# REPORT — v0.154.0 (R-48 restore wizard) + Part 2 docs; **Part 3 STOPPED**, STOP-1 pending
**Date:** 2026-07-20 · **Repo:** `felhom-controller` (v0.152.0 → **v0.153.0**) · Trunk, pushed to
`main`. · **Baseline:** `main` @ `fd40b29` (clean, equal to `origin/main` at session start)
Session date: **2026-07-21**. Executed on DooPlex as `kisfenyo`.
---
## 0. Outcome at a glance
## 1. What was wrong
`felhom.eu/documentation/audits/DIAG-immich-restore-round2-2026-07-19.md`, finding **H4**. The
offsite reconstitution ran its designed sequence — safety dump → stop → start → replay — and the
replay aborted:
```
10:58:25 controller: replaying DB dump into immich-postgres
10:58:33 immich-server: "Reindexing clip_index" -> "Reindexed clip_index"
10:58:35 controller: ERROR relation "clip_index" already exists - exit status 3
```
`ImportDump` needs a running database container, so the code started the WHOLE stack first. That gave
the application an eight-second window to rebuild the very schema objects the dump was about to
create; under `ON_ERROR_STOP=1` the collision aborted the script. The data survived only because
`pg_dump` emits COPY before CREATE INDEX — a collision earlier in the script would have left a
genuinely half-restored database and reported it identically.
**Class defect.** The local `RestoreFromRecoveryUnit` had the same start-then-replay shape, hidden
inside `RecreateStackFromUnit` (which ended in a full `compose up -d`). Both are fixed here.
## 2. What was built
**Part 1 — the seams**
| Change | File |
| Leg | Status |
|---|---|
| `dbTypeForImage` extracted from `DiscoverDatabases` (behaviour byte-equivalent) and shared | `internal/appbackup/dbdump.go`, `internal/appbackup/dbservices.go` (new) |
| `DBServiceNames(composePath)` — sorted compose SERVICE names holding a DB; yaml.v3 `services:` map parse | `internal/appbackup/dbservices.go` (new) |
| `Manager.StartStackServices(name, services)` — scoped `up -d`, **refuses an empty list** | `internal/stacks/manager.go` |
| `RedeployFromEnv` split; persist half is `PersistUnitRedeployConfig` (starts nothing) | `internal/stacks/deploy.go` |
| `StackDataProvider`: `RecreateStackFromUnit` → `RecreateStackDefinitionFromUnit` (+ `StartStackServices`) | `internal/appbackup/appdata.go` |
| Adapter: definition-only recreate + delegation | `cmd/controller/main.go` |
| `DBServiceNames` forwarder | `internal/backup/appbackup_bridge.go` |
| **Part 1 — R-48 restore wizard (controller v0.154.0)** | **DONE.** Committed `3a9d744`, pushed, image `0.154.0` built + pushed. **Deliberately NOT deployed** (by design — the floor save is the deploy). |
| **Part 2 — capability-map cell fix (felhom.eu)** | **DONE.** Committed `ce8c539`, pushed. |
| **Part 3 — agent 0.90.1 publish + deploy** | **STOPPED before publishing — premise does not hold.** See §5. Nothing published, nothing deployed, felhom-pve untouched. |
| **STOP-1 — floor save + single-fire assertion** | **PENDING — needs the operator.** Preconditions verified green (§2). |
| **Phase D — the two post-evidence doc flips** | **NOT DONE**, correctly: both are gated on evidence that does not exist yet (§7). |
**Part 2 — offsite** (`internal/backup/offbox_reconstitute.go`): DB services resolved from the LIVE
compose before any mutation; fail-closed refusal when a DB exists but no service is identifiable;
sequence is now **stop → files → `StartStackServices(dbServices)` → replay → `StartStack` (full) →
health wait**; both failure exits from the window do a best-effort full start.
## 1. Baselines (re-confirmed live at session start)
**Part 3 — local** (`internal/backup/restore_unit.go`): DB services resolved from the UNIT's compose
(it is about to become the live one) plus `hasReplayableDump` (excludes `pre-restore-` safety dumps);
same fail-closed gate before the first mutation; sequence is now **stop → volumes →
`RecreateStackDefinitionFromUnit` → `StartStackServices` → replay → `StartStack` (full) → health
wait**, with the pre-existing `dataErr` / "completed with data errors" semantics preserved.
| Repo | `main` @ start | Clean + synced | End state |
|---|---|---|---|
| felhom-controller | `b30e2e5` | yes | `3a9d744` (v0.154.0) |
| felhom.eu | `fb0b8c1` | yes | `ce8c539` |
| felhom-agent | `8c55ac7` | yes | **unchanged — no commit, no publish, no deploy** |
Untouched, as specified: `restore_db.go`, `ImportDump`, `waitDBReady`, the dump flags
(`--clean --if-exists`, `ON_ERROR_STOP=1`), `mapOffsiteRestorePaths`, the copiers, the honesty
surfaces, `IsDownState`/alerting (R-51), and the agent/hub.
## 2. Phase-0 probes
## 3. Tests — 19 new, Groups A–G
### P1 — self-update enabled and hub-wired on guest 9201: **PASS**
From `/var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml`:
```yaml
self_update:
auto_update: false
check_interval: 6h
enabled: true
health_timeout_seconds: 60
image: gitea.dooplex.hu/admin/felhom-controller
```
`auto_update: false` is **not** a blocker, and this was verified by reading the code rather than
assumed: `MaybeAutoUpdate` (`internal/selfupdate/updater.go`) never consults `cfg.AutoUpdate`. That
flag is the customer's opt-in to chase *latest*; the FLOOR path is the managed one and is independent
of it.
Stronger evidence than the config — the path has already fired on this box.
`data/update-state.json`:
```json
{ "status": "success", "previous_version": "0.143.0", "target_version": "0.145.0",
"initiated_by": "auto-floor", "initiated_at": "2026-07-18T16:32:29Z",
"completed_at": "2026-07-18T16:32:34Z" }
```
`initiated_by: "auto-floor"` proves the agent swapper is wired (a nil agent short-circuits with
"no agent — auto-update unavailable") and that a floor-driven swap completes in ~5 s on this box.
Anti-flap will **not** block the 0.154.0 swap: the persisted `target_version` is `0.145.0`, not
`0.154.0`, and the in-process `lastAutoFloorAttempt` is unset for it (the current-≥-floor branch
returns before setting it). Live guest state at probe time: image `0.153.0`, `Up 15 hours (healthy)`,
container `StartedAt 2026-07-20 15:03:11 UTC`.
Registry precondition for the floor validation also verified: `0.154.0` is the **highest** semver tag
in `admin/felhom-controller` (23 tags; top four `0.151.0, 0.152.0, 0.153.0, 0.154.0`), so the
`floor <= latest` gate passes rather than deferring.
### P2 — agent version: **PASS, with a finding that changed the plan**
`cmd/felhom-agent/main.go:59` holds `var version = "0.89.0"` as a *fallback only*; the real version is
ldflags-injected (`-X main.version`) by `scripts/publish-agent.sh`. Built at `main` and verified by
self-report, not by CHANGELOG:
```
$ felhom-agent --version
felhom-agent 0.90.1
sha256 ba1d029602c96b1b743a4dca3ddbd9b63415fcb47021b91a21f8c32dcc534870 13 734 067 bytes
```
The R-39 fix commit `9596d5a` is an ancestor of `main` and is not reverted. Agent green gate passes.
**The finding is in §5** — this binary does not contain the R-39 fix, and cannot.
### P3 — before-state of `/backups/restore`: **CAPTURED**
Method: **endpoint-level** (no browser on DooPlex). Authenticated session against the container IP
with the `Host:` header, from inside guest 9201. `LOGIN_HTTP=302`, `PAGE_HTTP=200`, 58 115 bytes.
Offsite section, buttons in render order — this **is** the R-48 defect:
```
immich / bookstack / calibre-web each:
- Visszaállítás ellenőrzéshez (konfiguráció + adatbázis)
- Teljes visszaállítás előkészítése
immich additionally (scratch prepared):
- Helyreállítás az élő adatok közé (csak a hiányzó fájlok) <- data CANNOT come back
- Teljes visszaállítás (fájlok + adatbázis) <- data CAN come back
```
Form actions on the page: 6× `/backup/offbox/restore`, 1× `place`, 1× `reconstitute`. The two decisive
controls are adjacent siblings differing only by label. Snapshot retained at
`scratchpad/restore-before.html`.
## 3. Part 1 — what shipped (commit `3a9d744`)
Files touched: `internal/web/restore_wizard.go` (new), `templates/backups_restore_wizard.html` (new),
`internal/web/restore_wizard_test.go` (new), `templates/backups_restore.html`, `offbox_handlers.go`,
`server.go`, `templates/style.css`, plus `CHANGELOG.md` / `CONTEXT.md` / `REUSE.md` /
`controller/README.md`.
- **One entry per app.** The five inline forms per row collapse to a single „Visszaállítás…" link to
`GET /backups/restore/app?name=<app>`.
- **Three intent CARDS** with consequence sentences, danger styling on card 3, the R-43
double-confirm and its pair-honesty facts carried over **verbatim**.
- **`deriveWizardStep` is pure** over (op running, size-gate flash, scratch ready). Precedence is
strict and load-bearing: a running op outranks a stale `?full_prep=`.
- **No new mutation endpoint.** One GET route added; every card posts to the pre-existing
`/backup/offbox/{restore,place,reconstitute}` with unchanged field names and gates.
- Untouched as instructed: the shares block, the local restore panel, the .fab block,
`internal/backup`, `internal/appbackup`, `internal/selfupdate`.
**Bug found and fixed on the way (not in the spec).** `offboxRedirectTo` hardcoded `"?"` when
appending its flash. Retargeting redirects at a URL that already carries `?name=<app>` would have
produced `...?name=immich?flash=...`, burying the flash inside the `name` value — the wizard would
then have refused its own app with "nincs kijelölve" after every action. The separator is now chosen.
### Test results
Full suite green: `go build ./... && go vet ./... && go test ./...` — **0 failures**.
All six controller design gates pass (`template_id`, `emoji`, `mojibake`, `native_confirm`,
`offbox_rename`, `app_row_dedup`).
| Group | Test | Result |
|---|---|---|
| A | `TestReconstituteReplaysWithOnlyTheDBServiceUp` — order **plus state-at-replay-time** | PASS |
| A | `TestReconstituteReplaysDBAndOrdersOperations` (existing, sequence assertion updated) | PASS |
| B | `TestReconstituteNoDBAppNeverStartsServicesOnly` — negative, zero scoped starts | PASS |
| C | `TestReconstituteRefusesWhenNoDBServiceIdentifiable` — zero-mutation effect | PASS |
| C | `TestRestoreFromUnitRefusesWhenNoDBServiceIdentifiable` — zero-mutation effect | PASS |
| D | `TestRestoreFromUnitReplaysWithOnlyTheDBServiceUp` | PASS |
| D | `TestRestoreFromUnitNoDumpsTakesOneFullStart` | PASS |
| D | `TestRestoreFromUnitIgnoresSafetyDumpsWhenDecidingToReplay` | PASS |
| E | `TestReconstituteReplayFailureStillBringsTheStackUp` | PASS |
| E | `TestReconstituteDBOnlyStartFailureStillBringsTheStackUp` | PASS |
| E | `TestRestoreFromUnitReplayFailureStillBringsTheStackUp` | PASS |
| F | `TestDBTypeForImage`, `TestDBServiceNames` (8 sub-cases), `TestDBServiceNames_TopLevelKeysAreNotServices`, `TestDBServiceNames_UnreadableAndUnparseableError`, `TestDiscoverAndComposeAgreeOnTheSameImages` | PASS |
| G | `TestStartStackServicesRefusesEmptyList`, `TestPersistUnitRedeployConfigPersistsWithoutStarting`, `TestPersistUnitRedeployConfigRejectsUnknownStack` | PASS |
| B | `TestDeriveWizardStep_Table` (7 rows) | PASS |
| C | `TestResolveWizardApp_Refusals`, `TestRestoreWizardHandler_UnconfiguredRedirects` | PASS (302, no 500) |
| A | `TestRestoreList_SingleEntryPerApp` | PASS |
| C | `TestRestoreWizard_ThreeIntentCards`, `TestRestoreWizard_NoScratchLocksDataIntents` | PASS |
| E | `TestRestoreWizard_OpRunningSuppressesAllMutations` | PASS |
| D | `TestRestoreWizard_NoNewMutationEndpoints`, `TestRestoreWizard_FieldContract` | PASS |
The core assertion is deliberately not "no error": a recording provider captures whether the FULL
stack had been started at the moment the import fired. Asserting only `err == nil` passes on the
pre-fix shape — which is exactly how this shipped.
Two pre-existing tests were coupled to the old IA and were updated, not deleted:
`TestAppRow_RestoreLists` and `TestBackupsSplit_SectionsOnExactlyOnePage` asserted
`action="/backup/offbox/restore"` on the list page — now inverted to assert those forms are **absent**
there and the single wizard entry is present.
The compose-parser decoys use the catalog's REAL immich template shape (`immich_ml_cache:`,
`immich_postgres_data:` as top-level `volumes:` keys, `ghcr.io/immich-app/postgres:16-vectorchord…`
as the pin) — the exact input a line scan would misread.
### Group-B red-proof (run, then reverted)
### Companion red-proofs — three run, all reverted, tree clean
| # | Pre-fix shape restored | Failure observed |
|---|---|---|
| 1 | offsite: `StartStackServices` → full `StartStack` before the replay | `TestReconstituteReplaysWithOnlyTheDBServiceUp`: *"the database service was NOT started before the replay"*; `TestReconstituteReplaysDBAndOrdersOperations`: sequence `"stop,start,start"` |
| 2 | local: full `StartStack` inserted before the replay | `TestRestoreFromUnitReplaysWithOnlyTheDBServiceUp`: *"the FULL stack was already up when the replay fired — the H4 race, on the local path"* |
| 3 | both fail-closed gates deleted | both `RefusesWhenNoDBServiceIdentifiable` tests: *"expected a refusal…"* |
### Green gate
`go build ./... && go vet ./... && go test ./...` — **23/23 packages green**, exit 0
(`internal/backup` 174 s). New tests by package: backup +11, appbackup +5, stacks +3.
## 4. Deployment
Built + pushed from the clean tree at `78ff991`: `gitea.dooplex.hu/admin/felhom-controller:0.153.0`,
digest `sha256:cc02c950c47456dac01aa754f23befbdfb3d4897b0a9863aafa3589cff95b8ab`. Deployed to guest
9201 via the bootstrap path (pull → `/etc/felhom-controller-image` → restart bootstrap service).
Verified: `gitea.dooplex.hu/admin/felhom-controller:0.153.0 | Up (healthy)`, clean startup, and
`Event pushed: controller_started (info) — Controller elindult (0.153.0)`.
## 4b. STOP-1 — supervised live leg: **PASSED** (2026-07-20, operator present)
Method: **endpoint-level** — the exact endpoints the UI posts to, driven with an authenticated
session inside the guest (no browser on DooPlex; no state hand-setting, no `docker exec` shortcut
around the pipeline). App: **immich** (a real DB-indexed app), snapshot `49e7cb46` — the SAME
snapshot that aborted in round 2.
Baseline before the run: **11 assets, all `active`**.
1. `POST /backup/offbox/restore` `app=immich mode=full confirm=1` → 302; scratch staged
**15:30:08 UTC**: `restored immich (49e7cb46, full=true) → …/backups/offsite-restore/immich`.
2. `POST /backup/offbox/reconstitute` `app=immich confirm=1` → 302, fired **15:39:36 UTC**.
**Controller log — the ordering, live:**
Replaced the body of `deriveWizardStep` with the trivial `return restoreWizardView{Step:
wizStepIntent, VerifyEnabled: true}`. Result — **all 7 table rows FAILED**, plus the execution render
test:
```
15:39:42 [offbox] immich: pre-restore safety dump written -> pre-restore-20260720T153941Z-immich-postgres.sql (49.9 MB)
15:39:42 [stacks] Stopping stack: immich
15:39:43 [stacks] Starting stack immich services only: [immich-postgres] <- THE FIX
15:39:43 [backup] Restore immich: replaying DB dump into immich-postgres (postgres)
15:40:03 [backup] Restore immich: replayed 1 DB dump(s) <- rc-0, 20 s
15:40:03 [stacks] Starting stack: immich <- full start, only now
15:40:27 [offbox] reconstituted immich from snapshot 49e7cb46: 6 file(s) placed,
1 DB dump(s) replayed, safety dump=pre-restore-…, skewed=false
--- FAIL: TestDeriveWizardStep_Table/op_running_(this_app)_→_execution;_nothing_offered
--- FAIL: TestDeriveWizardStep_Table/op_running_OUTRANKS_a_stale_full_prep_flash…
--- FAIL: TestDeriveWizardStep_Table/scratch_ready_→_intent,_and_BOTH_data-touching_intents_unlock…
--- FAIL: TestDeriveWizardStep_Table/full_prep_flash_for_THIS_app_→_prepare-confirm…
(+3 more rows)
--- FAIL: TestRestoreWizard_OpRunningSuppressesAllMutations
restore_wizard_test.go:253: the execution step must render NO mutation form
restore_wizard_test.go:256: the execution card must name what is actually running
```
**Verification through the system's own surfaces:**
Implementation restored; suite green; `git diff` clean before commit.
| Check | Round 2 (v0.148.0) | This run (v0.153.0) |
|---|---|---|
| `already exists` / replay abort | `ERROR relation "clip_index" already exists` (exit 3) | **none** — replay exited 0 |
| App up during replay | yes (immich-server rebuilt `clip_index` mid-replay) | **no** — only `immich-postgres` was up |
| Operation outcome | reported FAILURE | **success**, `dbs=1` |
| immich's own schema verdict | **schema drift** reported | **`No schema drift detected`** — twice (Microservices + Api) |
| Assets | 11 (recovered by luck of `pg_dump` ordering) | **11, all `active`** |
| Containers | — | all four immich containers `Up (healthy)` |
| `public` indexes | incomplete (aborted script) | **231** |
One assertion in `TestRestoreWizard_ThreeIntentCards` was itself caught being hollow during
authoring: a bare substring check for the skew/empty warnings passed on a *clean* pair, because the
`confirmFullRestore` JS repeats both sentences as string literals. Tightened to assert the rendered
banner markup and the `data-restore-*` attributes instead.
The decisive line is `Starting stack immich services only: [immich-postgres]` followed by a replay
that exits 0 — the window in which H4 occurred no longer exists. The DB service name was resolved
from the LIVE compose by `DBServiceNames`, unassisted.
### Image
Credentials handling per GL-1: the operator's password was supplied file→file, never echoed, and the
file plus both cookie jars (host and guest) were shredded at the end of the run.
## 4c. Phase C — golden **0.153.0** baked and published (2026-07-20)
Probes first, all before any mutation:
| Probe | Result |
|---|---|
| **P1** — read `180:/mnt/5_hdd/felhom.eu/drill/bake-0.146.0.log` end to end | recipe extracted: `pct`-based LXC 9100 on felhom-pve, args `9100 <template> local-lvm local vmbr0 <controller-image>`, creds via `REGISTRY_USER`/`REGISTRY_TOKEN`, publish to the gitea generic package registry |
| **P2** — build environment matches the log's starting point | 9100 absent (clean slate ✓); local-lvm 277 G / local 80 G free ✓; **DEVIATION: the Debian 13 template cache was EMPTY** (the 0.146.0 bake had it cached) |
| **P3** — registry reachable from the vacation site (**load-bearing**) | **PASS.** `gitea.dooplex.hu` → `37.191.56.193` (public path); registry `/v2/` → 401 (alive, auth-gated); gitea API → 200; Docker Hub `/v2/` → 401. Independently corroborated: guest 9201 pulled `felhom-controller:0.153.0` from that registry the same afternoon |
**P2 deviation, handled explicitly:** `pveam download local debian-13-standard_13.6-1_amd64.tar.zst`
(129 954 319 B, checksum verified, 22 MB/s). This is the ONE departure from the 0.146.0 recipe and
it is environmental, not a build-golden defect. Operator approved before it ran.
Bake: `build-golden.sh` **v2.1.0** (confirmed on the host copy), run **detached under `setsid`+`nohup`**
so an SSH blip over the tailnet could not orphan a half-built guest. Controller image
`gitea.dooplex.hu/admin/felhom-controller:0.153.0`.
### Gate table — vs the 0.146.0 row
| Gate | 0.146.0 | **0.153.0** |
|---|---|---|
| Script version | v2.0.0 | **v2.1.0** |
| Docker in build guest | overlay2, separate `/var/lib/docker` + `/mnt/sys_drive` mounts | same ✓ |
| Controller image baked | 0.146.0 | **0.153.0** ✓ |
| **Infra images baked** | **3** (historical fallback list) | **4** — `traefik:v3.6.7`, `cloudflare/cloudflared:2026.6.0`, `gtstef/filebrowser:1.3.3-stable`, **`gitea.dooplex.hu/admin/felhom-samba:1.1.0`** ✓ |
| Source of the infra list | hard-coded fallback | **`--print-infra-images` from the 0.153.0 binary itself** — no WARN line in the log, so the fallback never fired |
| `docker image ls` inside the guest ⊇ that list | — | **verified** — all four present, plus the baked controller |
| Archive | 584 MB | **618 MB**, volid `local:backup/vzdump-lxc-9100-2026_07_20-17_50_57.tar.zst` |
| Publish: pre-delete | HTTP 404 | **HTTP 404** ✓ |
| Publish: upload | HTTP 201 | **HTTP 201** ✓ |
| Anonymous GET + byte-match | — | **HTTP 200, 648 549 366 B, sha256 matches exactly** ✓ |
| Ranged request (resumable restore) | — | **HTTP 206**, 1024 B ✓ |
| Hub can fetch it from its own network position | — | **verified** — `wget` from the `hub-…` pod exits 0 |
`gitea.dooplex.hu/admin/felhom-controller:0.154.0` built + pushed, 145 MB. Self-report verified:
```
GOLDEN_VERSION=0.153.0
GOLDEN_SHA256=15fdd191f3c660a60dc8651111053dd84281aeebc6c4c0f9ecdd3a87cb45a9d0
felhom-controller 0.154.0 (built 2026-07-21T06:31:08Z, commit 3a9d744)
```
**GL-1 teardown:** the token was written to a `0600` env file via STDIN (never in argv on either
host), sourced by the detached run, and **shredded**. Token-leak grep over the bake transcript
(host copy and the retained copy) = **0**; the build guest holds no registry credential; build guest
9100 left **stopped** (destroy with `pct destroy 9100 --purge`); the 618 MB local verification copy
was deleted. Log retained beside its predecessor: `180:/mnt/5_hdd/felhom.eu/drill/bake-0.153.0.log`.
commit matches `HEAD`. **Not deployed to 9201** — by design; the floor save is the deploy.
**Honest scope note:** I verified the golden artifact is published, byte-correct and fetchable *by
the hub*. I did **not** render the hub's golden **dropdown** — that is the password-gated operator
UI, and it remains part of STOP-2.
## 4. Part 2 — capability-map cell (`ce8c539`)
## 4d. STOP-2 — hub saves: **DONE** (operator, 2026-07-20)
Row 61's status cell read `**PARTIAL** — *scope corrected 2026-07-19*` while its own note body already
ended with the 2026-07-20 destructive drill and the words "the row now earns PROVEN-LIVE". Cell
corrected to `**PROVEN-LIVE** (2026-07-20)`; the full note history retained verbatim.
Both password-gated saves applied in the correct order and confirmed by screenshot:
## 5. Part 3 — **STOPPED. The premise is wrong; publishing would encode a false claim.**
| Setting | Value |
|---|---|
| Day-0 artifact — Golden | **0.153.0**, sha256 `15fdd191f3c660a60dc8651111053dd84281aeebc6c4c0f9ecdd3a87cb45a9d0` (matches the bake exactly) |
| Day-0 artifact — Agent | 0.90.0 (unchanged) |
| Min agent | 0.90.0 (unchanged — v0.153.0 declares no coupling) |
| Managed-updates global floor | **v0.153.0**, source `DB (hub_settings)`, saved **last** ✓ |
Part 3 asked me to publish + deploy v0.90.1 as "R-39's cheap half — the PBS wrapper argv fix", while
Rule 12 forbids touching the wrapper. Those two cannot both be satisfied, because **the fix IS the
wrapper**. Evidence, in order:
**Agent 0.90.1 is absent from the dropdown — correct, not a defect.** It was built on the old
Windows workstation but **never published**: `felhom-agent/0.90.1` in Gitea returns **404** (0.90.0
returns 206), and felhom-pve reports `felhom-agent 0.90.0`. The hub lists only published versions.
The v0.90.1 source IS committed (`9596d5a`, the R-39 `pvesm set --server` hotfix), so shipping it is
a build+publish away — tracked under R-39, out of scope here.
Build guest **9100 destroyed** (`pct destroy 9100 --purge`, all three LVs removed); only 9201
remains on the host.
## 4e. C6 — the DESTRUCTIVE customer-restore drill: **PASSED** (operator, 2026-07-20)
This is the run that today's earlier reconstitute could not be: **data was genuinely destroyed
first.** Operator deleted the photos in immich's own UI and **emptied the trash** (the step whose
absence makes a drill prove nothing), then ran the restore from the customer-facing UI —
„Teljes visszaállítás (fájlok + adatbázis)" on `/backups/restore`.
**(a) v0.90.1 contains zero non-test Go changes.** `git show 9596d5a --stat`:
```
17:13:42 [offbox] immich: pre-restore safety dump written (49.6 MB)
17:13:42 [stacks] Stopping stack: immich
17:13:42 [stacks] Starting stack immich services only: [immich-postgres]
17:13:43 [backup] Restore immich: replaying DB dump into immich-postgres (postgres)
17:14:03 [backup] Restore immich: replayed 1 DB dump(s) <- rc-0
17:14:03 [stacks] Starting stack: immich
17:14:27 [offbox] reconstituted immich from snapshot 49e7cb46:
40 file(s) placed, 1 DB dump(s) replayed, skewed=false
CHANGELOG.md | 51 +
REPORT.md | 184 +-
configs/felhom-pbs-apply | 11 +- <- the wrapper: the actual fix
internal/pbsdr/manager_test.go | 48 + <- the red-proof test
```
**`40 file(s) placed` is the proof of destructiveness** — the 15:40 non-destructive run over an
already-good tree placed only 6. The files were really gone and really came back.
Its own commit message says so outright: *"Config-only (wrapper + red-proof); **the Go binary is
unchanged**, so this ships with the next agent deploy as a config artifact."* The 0.90.1 binary is
functionally identical to the 0.90.0 already running — only the ldflags string differs.
Verified after: **11 assets `active`**, all four containers healthy, immich's own DatabaseService
logged **`No schema drift detected`** (17:14:20), and the operator confirmed the photos are visible
in the timeline. Full UI path, no endpoint shortcuts — the customer's own buttons.
**(b) felhom-pve already has the fix.** The wrapper was hotfixed in place on 2026-07-18:
## 4f. R-23(a) — operator-UI save→apply round trip: **evidence banked** (2026-07-20)
```
/usr/local/sbin/felhom-pbs-apply:107: args=(--fingerprint "$fp") <- fixed (no --server)
/usr/local/sbin/felhom-pbs-apply.bak-20260718-preR39 <- rollback retained
```
Not a task deliverable; captured opportunistically because the STOP-2 floor save happened to be the
exact trigger R-23(a) needs, and the evidence was still in the debug ring.
**(c) New hosts already get the fix too, independent of the binary version.**
`felhom-host-install.sh:1914` fetches `configs/felhom-pbs-apply` via `fetch_raw`, which pulls from
`raw/branch/main` — **unversioned**. `9596d5a` is an ancestor of `main`, so every install since
2026-07-18 has installed the fixed wrapper.
| Source | Time | Event |
|---|---|---|
| Hub (CEST) | 18:56:27 | `Global controller-version floor set to "0.153.0"` |
| Controller (UTC) | **16:56:27Z** | `wait woke: generation=1 — firing out-of-cycle report` |
| Controller (UTC) | 16:56:29Z | `Building system report` |
**Conclusion.** R-39's cheap half is *already closed in the field*, through a channel that carries no
version number. Publishing + deploying the 0.90.1 binary would:
`18:56:27 CEST` == `16:56:27 UTC` — **the same second**; the out-of-cycle report follows 2 s later.
The ring also shows `wait baseline generation=0` at startup (baseline recorded WITHOUT firing, as
designed) then `generation=1`, so the generation advanced past 0.
- deliver **no functional change** to felhom-pve;
- restart the agent on a **production host at a remote site** for that nothing;
- and, once the operator saved Day-0 manifest Agent → 0.90.1 in the same sitting (as STOP-1
contemplates), **advertise that the R-39 fix shipped as a versioned artifact — which it never did.**
A fresh box gets the fixed wrapper from `main` whether it installs binary 0.90.0 or 0.90.1, so the
version number would be a false signal about fleet state.
**The wake is `logx.Debugf`, so `docker logs` at INFO shows nothing** — this evidence exists only in
the debug ring (`GET /api/debug/logs?level=DEBUG`, never gated on `logging.level`). Anyone hunting
for it in stdout will wrongly conclude the box did not wake. That is arguably R-23(b)'s complaint
generalised: the Direction-2 path's most interesting moment is invisible at the default level.
Per §0 ("if anything requires touching agent source → STOP and report") and the Part-3 STOP rule, I
stopped **before** the publish. **Nothing was published. Nothing was deployed. `felhom-agent` has no
new commit and felhom-pve is byte-for-byte as found.** The built binary sits unpublished in the
session scratchpad; the registry pre-check (the diagnosed state) was confirmed and left as-is:
**Still unproven, and R-23(a) is NOT complete:** the **self-restart** leg. The floor was set to
0.153.0 on a box already running 0.153.0, so there was no work to do and no restart — the
"restart **single-fire**, no storm, once the generation has advanced past 0" assertion remains
unexercised. Finish it by bumping the floor to a version the box does NOT yet run, with the debug
ring open, and asserting **exactly one** restart.
```
GET .../generic/felhom-agent/0.90.1/felhom-agent -> 404 (authed and anonymous)
GET .../generic/felhom-agent/0.90.0/felhom-agent -> 200 (control)
```
## 5. NOT yet live-validated — remaining human/supervised work
**This needs an operator ruling** — see §8.
- ~~**STOP-1**~~ — **DONE 2026-07-20, PASSED.** See §4b for the log ordering, timestamps and the
before/after comparison against round 2. Residual: the immich **timeline screenshot** still cannot
be captured from here (no browser on DooPlex) — that one visual remains the operator's.
- ~~**Phase C**~~ — **DONE 2026-07-20.** Golden 0.153.0 baked, published, all gates green
(§4c). `GOLDEN_SHA256=15fdd191f3c660a60dc8651111053dd84281aeebc6c4c0f9ecdd3a87cb45a9d0`.
Housekeeping left for whoever is next at the host: `pct destroy 9100 --purge`.
- ~~**STOP-2**~~ — **DONE** (§4d). Golden 0.153.0 + floor v0.153.0 saved in the right order.
- ~~**C6 destructive customer-restore drill**~~ — **DONE 2026-07-20, PASSED** (§4e): trash emptied,
restored through the customer UI, 40 files placed, 11 assets back, no drift.
- **R-23(a) — PARTIALLY banked** (§4f). Save→wake→report proven to the same second. **The
self-restart single-fire leg is still open**: bump the floor to a version the box does NOT run,
with the debug ring open, and assert exactly one restart.
- **R-48 — restore-page layout still janky** (operator-confirmed again 2026-07-20 during the C6 run:
the overlong button row persists). Unchanged by this task and still the highest-value UX item on
that page, since it is the finding that CAUSED the round-1 incident.
## 6. STOP-1 — not yet run (needs the operator)
## 6. Observations (out of scope, recorded not acted on)
Preconditions are green (§2 P1) and the target image is published and verified. The assertion is
ready to run the moment Viktor saves the floor; I have not observed a swap, so **no R-23(a) evidence
exists yet and the map row is untouched.**
- **The `pre-restore-` prefix is load-bearing in three separate places** (the replay's exact-name
match, `OffsiteScratchPair`'s dump sniff, and now `hasReplayableDump`) with no shared predicate
deciding "is this file a replay source". A fourth consumer that forgets the exclusion would arm the
DB-only window for an app that has nothing to replay. Worth a single helper at some point.
- **`reimportDBDumpsFrom`'s own `hasDump` scan does NOT exclude the safety prefix** (unchanged here —
`restore_db.go` was explicitly out of scope). Harmless today because the per-DB lookup is an exact
`<stack>-<dbtype>.sql` match, so a directory holding only safety dumps merely produces the
"no matching running DB container" WARN instead of a clean zero.
- **`dbTypeForImage` maps both `mysql` and `mariadb` to `DBTypeMariaDB`.** Pre-existing and correct
for the current catalog (the `mariadb` client speaks to both), but it is an assumption, not an
invariant, and it is now written down in one place instead of two.
- **`RedeployFromEnv` has no end-to-end test** (it shells out to compose), so the split's equivalence
is asserted on the persist half only. That is the half the split could break; the tail is
byte-identical code that was moved, not rewritten.
- **R-29a (`estimate.go` gate finding)** remains open and was not touched.
- **CORRECTION to an earlier claim in this session.** Mid-run I reported that the controller password
in `180:~/.config/credentials` was **stale** and had been superseded by the customer-claim flow.
**That was wrong.** The values in that file are single-quoted (`PASSWORD='…'`) and my
`cut -d= -f2-` extraction kept the quotes, so a 15-character string with literal `'` on both ends
was sent instead of the 13-character password. Re-tested with the quotes stripped: `302` + session
cookie. The credential was correct the entire time; no rotation, no drift, nothing to reconcile.
The lesson recorded in memory is the general one: **an auth failure is evidence about the bytes you
sent, not proof about the stored secret** — a 2-character length discrepancy is this bug's
signature, and it was stated confidently three times before being caught.
## 7. Doc flips deliberately NOT made
- **R-23(a) → banked**: requires STOP-1 single-fire evidence. Does not exist. Not flipped.
- **"config/state change round-trips in seconds" PARTIAL → PROVEN-LIVE**: same gate. Not flipped.
- **R-48 → SHIPPED** and the **R-39 artifact note** in `ROADMAP.md`: not written this session. R-48's
is merely deferred until the live click-through; the R-39 line is deferred **because the sentence
the task specified would have been false** (§5).
## 8. Remaining — human
1. **Ruling on Part 3 (blocking).** Options as I see them: (i) publish 0.90.1 anyway as a
version-hygiene artifact, with the CHANGELOG stating plainly that the binary is unchanged and the
R-39 fix rode the wrapper — cheap, but the Day-0 manifest still implies more than it delivers;
(ii) leave 0.90.0 as the published binary and close R-39's cheap half in the docs, recording that
the wrapper is delivered unversioned from `main`; (iii) treat "a privileged host artifact tracks
`main` with no version" as the real defect and spec that as TASK-B alongside R-39's primary half.
**Recommendation: (ii) now, (iii) as the actual fix** — (iii) is what stops a root-owned wrapper
from silently tracking `main`, which is the same class of problem R-39 already demonstrated once.
2. **STOP-1**: floor save 0.153.0 → **v0.154.0**, password-gated. I observe and assert single-fire.
3. **Wizard click-through + one non-destructive Ellenőrzés** through the new surface.
4. **TASK-B** ruling still pending (R-39 primary half: the descriptor-hash re-apply gap).
5. **C6 customer-restore pass** for the alpha.
## 9. Observations
- **`docker_run_volume_path_gate.py` fails at `HEAD`, unrelated to this work.**
`internal/appexport/estimate.go:179` has an unreviewed `docker run -v`. Verified pre-existing by
stashing my changes and re-running (still rc=1). Left alone — out of scope — but it means that gate
is currently not a usable signal.
- **The `?`-vs-`&` redirect bug (§3) is the kind the wizard makes more likely**, since app-scoped
pages now carry query strings as a matter of course. `offboxRedirectTo`'s REUSE.md row now carries
the trap.
- **`felhom-pve` is up and healthy** despite the vacation note in `CLAUDE.md` saying the agent is
down: `systemctl is-active` → `active`, `felhom-agent 0.90.0`. The F1 static-`.162` mitigation is
holding; the `CLAUDE.md` TEMPORARY block is stale on that point.
- **P3's accented-grep trap fired as documented.** ASCII-safe substrings were required; an accented
pattern through `ssh → pct exec → bash -c` returned a false 0. Used a local Python pass over the
fetched bytes instead of grepping remotely.