Dead primary: degraded in 13 s, exactly one app_start_failed, banner rendered and
self-cleared. Boot orphan: recovered in one attempt with zero alerts. Dead dhclient:
detected in 57 s on process liveness while the lease was still live, healed 120 s after the
kill — the tunnel never dropped, so the outage was prevented rather than observed.
P1 answered as a by-product: bookstack StartedAt == the moment bootrecon StartStack
returned, so unless-stopped did NOT resurrect it. F5 hypothesis confirmed.
New R-55, surfaced by the leg designed to prove the opposite: the boot bind gate recreates
and STARTS every deployed drive-backed app unconditionally, so a customer Stop does not
survive a reboot for those apps. Predates R-52 and does not implicate it, but it narrows
R-52's practical scope and needs a ruling.
R-51's roadmap diagnosis is corrected at the source: aggregation returned StateRunning
("partial") for a running/stopped mix, so the stack read RUNNING and IsDownState was never
consulted about at all — the constraint that row protects was never in tension
with the fix.
New R-54 row closes the INCIDENT-guest-dhclient-killed-2026-07-20 §5 OPEN RISK, and records
the design fact that makes it work: liveness of the DHCP client is itself a probe, because
the damage is timed and the address outlives its cause by 1-2 hours. The static-guest leg is
deliberately deferred to R-50.
New capability-map row is IMPLEMENTED, not PROVEN-LIVE: one leg is live (the watchdog's
healthy cycle on felhom-pve), the three that matter are destructive and operator-present and
have not run.
PROMPT-TEMPLATE §10 gains the seam-discipline row, including that a strings.Contains source
assertion is NOT sufficient — a commented-out call still contains the string.
The operator pressed Re-issue PBS credentials and the chain closed in 13 seconds. The
identical click on 2026-07-18 did nothing at all.
hub 08:39:31Z fresh mint, generation 0 -> 1; descriptor gains secret_generation: 1
(token_id + fingerprint BYTE-IDENTICAL — the invisible re-key shape)
agent 10:39:34 felhom-pbs-apply read felhom-pbs (leg b: the impossible read)
agent 10:39:38 ERROR REJECTED ... applied and DEAD, previous_state=applied
(leg c: the R-39 state, loud)
hub 08:39:45Z consumed_at stamped
agent 10:39:45 one-time token secret consumed (leg a: NO short-circuit)
agent 10:39:45 reconcile (set-only, no --server)
agent 10:39:47 pbsdr: converged state=applied
Corroboration: marker hash moved to afbb3b41… (it was byte-identical to the pre-reissue
marker in the failure); secret mtime 2026-07-18 -> 2026-07-21 10:39:45; new credential
probes 200; three consecutive reports trace applied -> auth_failed -> applied; ZERO
self-heal escalations, one mint, one consume, no consumed-failed.json — the box healed
through the descriptor path before the damper was ever needed.
Recorded for future runbooks: the operator first pressed the OFFSITE re-issue (two
distinct Re-issue actions exist). Harmless to PBS-DR, but it rotated the restic password
and correctly marked the escrow STALE, so the ceremony had to be re-run. Name the surface
explicitly next time.
The v0.68.0 row wrapped itself in a <div>, but the artifacts <form> IS the CSS grid
(display:grid, no inner container). The stray </div> closed the surrounding card from
inside the form and the new <div> was never closed — it swallowed the submit button and
ran to </form>, so the row rendered outside the card and Save landed inline. Reported by
the operator on first use.
The field still submitted (it stayed inside the form), so this was layout damage rather
than data loss, but the unbalanced markup put every section below it in the wrong
container.
Fixed as plain grid cells (grid-column: 2/4), no nested elements.
There was no render assertion on this form at all, which is why a hand-edit broke it
silently. The new test asserts the field is inside the form, the button has not escaped,
the form contains ZERO divs, whole-page div balance holds, and the sections after it
survive. Red-proofed against the broken shape.
R-39's three legs are closed and deployed: the hub stamps a monotonic secret_generation
so a re-key finally moves the descriptor hash; the wrapper gains a narrow read verb so
the non-root agent can read the credential it writes; and ProbeAuth turns a 401 into a
loud auth_failed the existing damper escalates to a fresh mint. Plus a consumed_at
honesty gauge for the applied-but-never-consumed disagreement.
Recorded in the R-39 row, because both are the kind of thing a future reader needs:
- A load-bearing fact the spec did not flag, checked rather than trusted: Apply bails out
if the storage status probe ERRORS and adopt converges without consuming when the
storage reads active, so the fix depended on PVE's 401 behaviour. PVE's storage_info
wraps activation in eval{} and leaves active=0, so a 401 returns HTTP 200 with
active:0 — never an API error. The chain is sound by proof, not inference.
- A defect I shipped and caught: v0.91.0 built the probe seam and main.go never wired it,
so the leg was inert while every test passed. Same class as controller v0.154.0 the day
before. Fixed in v0.91.1 (artifact superseded, not overwritten); v0.91.2 made a healthy
probe observable so "no auth_failed" can never again be confused with "never probed".
The DR-tier capability row is deliberately NOT upgraded to PROVEN-LIVE: the decisive
evidence is STOP-2, the operator pressing Re-issue and the box converging where the
identical click did nothing on 2026-07-18.
R-50b(a) shipped — wrapper sha256 in the manifest + agent reporting + host drift surface,
with unknown-on-either-side reading as quiet rather than drift. (b)/(c) remain open: the
wrapper is still fetched unversioned from raw/branch/main.
Completes the hub half of R-39's fleet fix on top of the generation core (c484aa2).
pbsdrheal gains an auth_failed TRIGGER — a new trigger in the existing machine, not a
new machine. A box whose credential PBS rejects escalates to a fresh mint, never a
re-stage (which would re-feed the secret PBS just rejected), through the EXISTING damper:
a 401 flap must not become a secret-minting chain. With the generation stamp this closes
the loop end to end — agent proves the 401, hub re-keys, generation advances, descriptor
hash moves, agent re-consumes.
consumed_at honesty gauge: a staged secret still unconsumed past a 15-minute grace while
the box reports `applied` is surfaced with its own event. That is the exact 2026-07-18
fingerprint and a disagreement no single tier can see alone. Deliberately a SURFACE, not
a heal — auto-re-issuing on it would mint a second secret on top of an unconsumed one,
which is the mint/consume race R-39(a) already recorded. One event per distinct report,
and an honestly-stuck box does not double-report (its unconsumed secret is the symptom
being healed, not a contradiction).
R-50b(a): ArtifactManifest.WrapperSHA256 + operator field + host-page drift surface. The
PBS wrapper is root-owned 0755 and the pinned sudoers vector, yet installed unversioned
from raw/branch/main and absent from every manifest. Agents >=0.91.0 report the installed
hash; a mismatch is surfaced. An unknown on EITHER side reads as quiet, never as drift —
lighting every host amber on rollout day is how a warning becomes background noise. The
delivery channel itself stays R-50b(b)/(c).
Compatibility unchanged: safe for 0.90.0 agents (unknown JSON key dropped); the re-arm
and auth-honesty guarantees need agent >=0.91.0, so MinAgent moves only after the fleet
has self-updated.
Tests: auth_failed escalate/debounce/recovery-forgets-streak; honesty gauge incl. grace
window, the restage edge (consumed_at deliberately NULLed), consumed-never-alarms, and
honest-stuck-no-double-report; wrapper drift incl. both unknown directions. Red-proof run
at the assertion level: removing the auth_failed arm fails the escalation tests with
reissues=0.
The fleet half of R-39. An ep0 credential re-issue re-keys the SECRET of an existing
token, so token_id, fingerprint, datastore and namespace all come back byte-identical.
The agent re-applies on the descriptor's CONTENT HASH, so a re-issue was invisible to a
converged box: it short-circuited, never consumed the fresh secret, and served a revoked
credential while reporting `applied` — the N100 failure of 2026-07-18.
host_pbs_secrets gains a monotonic per-host `generation`, advanced by every fresh MINT and
by nothing else, stamped into the descriptor as `secret_generation`. That is now the only
field a re-key moves, and it is what re-arms the agent.
DEVIATION FROM SPEC, deliberate: the brief said to return "the new row's id (int64) …
no schema change". There is no row id — host_pbs_secrets is keyed by host_id and UPSERTed
last-write-wins, so a new row never exists, and created_at collides for two mints in the
same second. An additive counter column is the only monotonic source; it uses the repo's
existing idempotent ALTER-TABLE idiom.
RestageHostPBSSecret deliberately does NOT advance it: a re-stage re-arms the SAME secret,
the descriptor content genuinely has not changed, and a bump would cause a pointless agent
refetch loop (that method's own contract says so).
Also corrects a comment that asserted the re-issue refreshes the descriptor "with the NEW
token_id/fingerprint". That is false for a re-key, and believing it is why the descriptor
was never expected to be identical in the first place.
omitempty is load-bearing: a zero generation must not start emitting a new key into every
pre-existing descriptor, which would itself be a fleet-wide spurious re-apply.
Compatibility: agents below 0.91.0 drop the unknown JSON key and behave exactly as today —
inert, not breaking (Scenario C).
Tests: store-level monotonicity + per-host isolation + restage-leaves-it-alone; descriptor
byte-change, omitempty, and sibling-key round-trip; and a FLOW-level test driving
ReissuePBSDR against a fake that models a real re-key. Red-proof run at the assertion
level (not the compiler): commenting out the stamp makes the flow test fail with both
byte-identical blocks printed.
The operator moved the global floor to a version the box did NOT run (0.153.0 ->
v0.154.0) and the managed self-update fired exactly once:
06:57:13Z UpdateState pending, initiated_by=auto-floor
06:57:17Z agent: controller swap requested 0.153.0 -> 0.154.0
06:57:21Z container restarted
06:57:29Z agent: new controller healthy (16 s save -> healthy)
Over a 39-minute window: swap requests 1, agent-driven bootstrap restarts 1,
rollbacks 0, container RestartCount 0. VerifyStartup confirmed on the next boot;
the following periodic check logged "Current version 0.154.0 is up to date" —
the at/above-floor branch correctly doing nothing.
The 2026-07-20 attempt proved nothing because it targeted an already-running
version; that was the whole reason this leg stayed open.
Disclosed in both rows: a hand-deploy of v0.155.0 at 07:17:10 falls inside the
observation window and is what StartedAt shows afterwards. It never goes through
SwapController, so the swap-count assertions hold across the full window — and it
incidentally re-confirmed the at/above-floor branch (0.155.0 running against a
0.154.0 floor -> updater did nothing).
R-23(b) (cosmetic Waiter "recovered" log timing) remains open.
R-48 — the offsite restore controls collapse to one „Visszaállítás…" entry per app plus
a per-app wizard with three described intent cards. Shipped in controller v0.154.0
(3a9d744). Live click-through still pending the operator's floor save.
R-39 — the planned v0.90.1 artifact publish was CANCELLED as a false signal (operator
ruling 2026-07-21). 9596d5a changes zero non-test Go files; its own message says "the Go
binary is unchanged". The fix is the felhom-pbs-apply wrapper, which felhom-pve has
carried since 2026-07-18 and which every new install fetches from raw/branch/main
regardless of binary version. Publishing would have delivered no behaviour change and
advertised a versioned fix the artifact channel never carried.
R-50b (new) — that stop surfaced the real defect: a root-owned privileged host artifact
is delivered unversioned from main, absent from the Day-0 manifest, so the fleet has no
way to answer which wrapper a given host is running.
Row 61's status cell still read "PARTIAL — scope corrected 2026-07-19" while the note
body already ended with the destructive drill of 2026-07-20 and the words "the row now
earns PROVEN-LIVE": 40 files placed after the operator deleted the photos in immich's own
UI and emptied the trash, 1 DB dump replayed rc-0, 11 assets active, no schema drift,
timeline confirmed — taken through the customer-facing buttons, not endpoint shortcuts.
Cell-only correction; the full note history is retained verbatim.
Capability map — the destroy-then-recover drill ran through the customer UI:
photos deleted, TRASH EMPTIED, full files+database restore. 40 files placed
against 6 in the earlier non-destructive run, 1 DB dump replayed rc-0, 11
assets active, no drift, timeline confirmed. That is the proof the 6D
downgrade asked for, so the offsite-restore row earns PROVEN-LIVE. The
customer-restore row records the honest residual: an operator ran it, so the
row's literal 'a customer, not the operator' wording still owes one pass.
ROADMAP R-23(a) — the STOP-2 floor save released the held wait in the SAME
SECOND (hub 18:56:27 CEST = controller 16:56:27Z), out-of-cycle report 2s
later, generation advanced 0 -> 1. Still open: the self-restart single-fire
leg, since the floor was set to a version the box already ran.
Trap recorded: the wake is logx.Debugf, so it is invisible in docker logs at
INFO and lives only in the debug ring.
The operator confirmed the immich timeline renders correctly after the
2026-07-20 reconstitute, which was the last named residual. But that run
restored over an already-good state (11 assets before, 11 after), so it
proves the H4 window is closed and the operation is clean — it is NOT a
destroy-then-recover drill.
Flipping the row on it would repeat the 6D overclaim verbatim: 'immich
end-to-end from offsite alone' must not be claimed by a run in which
nothing was ever lost. Row stays PARTIAL pending a real destructive drill.
ROADMAP R-47 and the capability map's offsite-restore row now carry the live
evidence: a supervised reconstitute of immich from snapshot 49e7cb46 — the
same snapshot that aborted in round 2 — ran stop -> DB-service-only start ->
replay rc-0 -> full start, with no 'already exists', a SUCCESS outcome, and
immich's own 'No schema drift detected' twice where round 2 left it
reporting drift.
The offsite-restore row's residual for a PROVEN-LIVE flip is now only the
immich timeline screenshot (no browser on DooPlex).
Golden 0.153.0 baked and published the same day, sha256 15fdd191f3c660a6...,
first golden carrying all four infra images.
ROADMAP: R-47 collapsed to a shipped one-liner naming both paths (offsite
ReconstituteFromOffsite and local RestoreFromRecoveryUnit), the DB-only
window mechanism, the fail-closed rule, and the no-agent-coupling
declaration (MinAgent stays 0.90.0).
Capability map: the offsite-restore PARTIAL row and the customer-restore
MISSING row both note that H4 is closed in controller v0.153.0 and that they
now await only one clean acceptance run.
Also corrects the stale "closes in v0.149" wording in BOTH rows — v0.149.0
was the F3 dashboard BackupStatus fix; R-47 shipped in v0.153.0.
Capture disproved addendum 1's guess: macOS DOES send a correct NBNS query and
IS answered correctly in 140us; it simply never acts on it. Records the proven
per-client matrix and keeps sidebar discovery explicitly unproven.
The v0.150.0 'orphaned dhclient' cleanup killed guest 9201's DHCP client, not a
host one: LXC guests share the host PID namespace, so the missing eth0, pidfile
and lease file were all true ON THE HOST and all irrelevant. Lease expired ~80
min later, guest lost its address, tunnel/hub/catalog/agent all went dark.
Restored; records the one-command ownership check (/proc/<pid>/cgroup) and
leaves the unsupervised-dhclient risk open for a ruling with R-50.
Also records against R-45 the lesson the loop taught: a phase a client answers
with a one-shot action must be an EDGE the registry serves once, never a level
synthesised into the phase channel.
felhom-pve joins the tailnet as a host package (100.70.170.35, expiry
disabled); DooPlex already advertised 192.168.0.0/24 via its GitOps k3s
tailscale pod (100.107.87.53). ssh felhom-pve now targets the tailnet IP
(direct-over-LAN at home, tunnel when away); felhom-pve-lan = LAN fallback.
--accept-dns=false on the host. Host converted static->DHCP (reservation
keeps .162 at home). Measured: --accept-routes hairpins the local subnet
at home -> travel-only opt-in. PBS is offsite (own WireGuard tunnel), needs
none of this. Recorded not fixed: controller<->agent plane hard-pinned to
192.168.0.162 (agent listen_addr + guest bootstrap.json) -> not yet portable
off-LAN. No code changed. Full doc: documentation/operations/tailscale.md
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XMForrE4c1wZxd9LukxYVt
VERDICT. H1 confirmed: the reconstitution never ran. /backup/offbox/reconstitute
was never hit (reconstituted/safety-dump/replay line counts all 0); the operator
clicked the old missing-only button, which merged 34 files and left the DB
untouched. H2 rejected (the full path never executed; staging used the correct
snapshot 49e7cb46). H3 rejected (no replay happened; DB was genuinely asset:0).
R-44 is exonerated and working: 49e7cb46 carries offsite_run_id 20260719T102851Z
and its dump probes to asset:11 / user:1 — a properly coherent pair, against
round 1's asset:0 / user:0.
H4 (NEW, the more important result). Phase-3 recovery ran the real sequence and
the v0.148.0 path executed correctly — safety dump, stop, start, replay — then
the replay ABORTED: relation "clip_index" already exists. Proven to the second:
10:58:25 controller begins replay
10:58:33 immich-server "Reindexing clip_index" -> "Reindexed clip_index"
10:58:35 dump's CREATE INDEX clip_index fails, exit 3
The replay races the app's own schema repair, because reconstitution starts the
stack before replaying (ImportDump needs a live container). The photos survived
only because pg_dump emits COPY data before CREATE INDEX — luck, not design. The
same start-then-replay shape is on the LOCAL path, so it is a class defect.
Recovery: PARTIAL and STOPPED as instructed. All 11 assets are back and visible
by every DB criterion (status=active, deletedAt null, all files resolve), but the
operation reported failure and immich reports schema drift. Not clean acceptance
evidence; no second attempt made; safety dump read only.
Findings -> R-47 (the replay race), R-48 (restore controls separable only by
layout — the proven cause of this incident), R-49 (1.1 GB of a 1.2 GB immich
backup is ML model cache + a postgres data-dir tar duplicating the .sql dump +
immich's own nightly dump; recorded, not changed).
RULINGS LANDED.
- 00-capability-map.md:61 -> PARTIAL, scope-corrected: 6D's destruction hit the
FILE TREE only; the DB survived in its named volume, so "end-to-end from
offsite alone" overclaimed scope.
- The 704.6 MiB "discrepancy" -> CLOSED, not a defect: it was immich's own
Tarhely widget, never a controller page. Same for round 2's 650MiB->1.4GiB.
- customer-restore row: partial evidence recorded, NOT flipped.
Diagnosis only — no code, label or layout changes; no forget/prune.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
ROADMAP R-43/R-44 → SHIPPED with what actually landed:
- R-43: ReconstituteFromOffsite — safety dump → stop → files overwritten to
the snapshot version → start → the snapshot's own dump replayed from the
SCRATCH unit (the live unit is still never overwritten). Invariants:
nothing is ever deleted, and the undo is verified on disk before the act.
- R-44: every run, manual AND nightly, refreshes dumps + units BEFORE the
capture, so each snapshot is a coherent {DB@T, files@T} pair stamped with
offsite_run_id + dumps_at. Nightly ordering is now structural rather than a
coincidence of two scheduler entries at 02:30 and 04:15.
DIAG doc gains a "fixed in v0.148.0" header pointing at both.
Capability map deliberately NOT flipped: the DB half now exists but is PARTIAL,
not PROVEN-LIVE — shipped and deployed with no live acceptance behind it. The
customer-restore row stays MISSING, now blocked only on the evidence run rather
than on missing capability, and R-3 stays DRAFT.
Still open: the 00-capability-map.md:61 ruling — did CAMPAIGN-6D's "immich
end-to-end from offsite alone" exercise the DB half, or only the file half?
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
R-38 flips to SHIPPED with the live-canary evidence. Two new items registered
from tonight's work: R-45 (unified async-job feedback — five one-off progress
surfaces now exist and agree on nothing; the two lessons any framework must
encode are "probe the terminal state, never infer it" and "a source reporting
nothing is normal, not broken") and R-46 (verification copies are now visible
but still cannot be browsed and never expire — the invisible-result gap and the
disk-lifecycle gap, designed together because they are the same object).
Pre-invite checklist gains the golden note: build-golden v2.1.0 makes goldens
>= 0.147.x carry all four infra images; no rebuild for this alone.
REPORT-<topic>.md per the parallel-session rule — another session is writing in
this clone tonight, so the shared REPORT.md is left alone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Git archaeology: the grid was never a deliberate removal. It lived as a fixed
`body::before` in index.html's inline <style> block and was dropped when bed8675
("D3 Part 2: index + kapcsolat on design system v2") migrated the page onto
site.css. dd54e4c, which created site.css, has no body::before — it was a
porting omission, and nothing took its place. No asset was lost; it was pure CSS.
Restored at its original geometry and subtlety (50px cells, 1px lines, 3%), not
redesigned. Only change: the accent is the v2 --blue #0083D8 instead of the
retired legacy #0088cc, which site_gates.py bans. Scoped to body.page-index
because index is the only page that ever had it.
site.css cache-bust bumped to v=2 across all seven pages (nginx caches 7d).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
The canary boot showed the entry sitting noticeably left of the centered logo.
gfxmenu's boot_menu draws items left-aligned inside the box with a ~29px inset
and offers no item alignment, so centering means placing the box: left 25%->40%,
width 50%->35%. Measured off the canary screenshot, not guessed; the comment
records the measurement so a future label change gets re-measured instead of
re-guessed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
The v1.22.0 gate asserts the rendered grub.cfg has no live reference to
proxtui/proxdebug/nomodeset/Rescue Boot/memtest/fwsetup. It grepped the whole
file, so the template's own header — which documents exactly which stock entries
were dropped, and names them — tripped it. Caught on the first canary build:
fail-closed, no ISO produced, which is the behavior we want from a safety gate
that is wrong.
Strip comments before matching. A comment naming a removed entry is the point;
a directive using one is the bug.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Viktor deleted 11 immich photos to test offsite restore; both place runs
merged 0 files and flashed success. Two-layered root cause:
- A UI delete means TRASH, not deletion — no file left the disk, so there
was genuinely nothing missing to restore. The test proved nothing.
- A valid test would also have failed: no offsite path loads a database.
All three buttons are file-only (offbox_restore.go:171-232, :346-421).
Files return; the library stays empty. The only DB replay is
RestoreFromRecoveryUnit, reachable solely from the LOCAL restore panel.
Compounding: the manual push never refreshes dumps (offbox_handlers.go:
203-227) — today's unit dump is from 02:30 and probes to asset:0 / user:0 /
album:0, predating both the admin user and the photos. Had a full restore
loaded it, it would have written an empty DB over the live one, destroying
the trashed rows that were the only surviving recovery path.
Answer to "can a customer trust same-day offsite?" for a DB-indexed app:
no — files come back, content does not.
No product code changed. Photos left in trash (recovery not wanted).
Capability map: customer-restore row stays MISSING by finding, not by
absence of attempt; offsite-restore row flagged scope-contested pending
a ruling on whether CAMPAIGN-6D exercised the DB half.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
Two jobs, one repack pass.
BRANDING. Every ISO now carries a Felhom boot screen built from the website's
og-image_2.png at repack time (ImageMagick in the assistant container), so the
boot card has ONE source and not a second pre-rendered copy in the repo to
drift. The card is scaled onto a 1024x768 gfxterm canvas, top-centered, and the
card's own subtle background grid is continued across the letterbox fill
PHASE-LOCKED to where the card's grid lands — the fill is seamless rather than a
square of grid floating in flat navy. Menu positioning needs a gfxmenu theme
(plain background_image cannot move the menu off the wordmark), so the stock
pvetheme is replaced by felhomtheme, which puts the menu in the lower third the
layout deliberately leaves empty.
SAFETY — the half that matters. The stock PVE menu offers Graphical, Terminal
UI and serial installers plus an Advanced Options submenu (nomodeset x2, three
debug variants, Rescue Boot, memtest, UEFI settings). Every one of them reaches
the MANUAL installer, whose first question is which disk to wipe. A customer, or
their helpful nephew, must not be able to get there from a boot menu. They are
not hidden and not password-gated: they are NOT EMITTED. What ships is one
entry, 'Felhom telepítés', default, 5s.
Boot behavior is unchanged. The kernel/append and initrd lines are lifted
VERBATIM from the ISO's own 'Install Proxmox VE (Automated)' entry rather than
frozen into a copy here, so a PVE bump tracks automatically; the build fails if
they cannot be found, if the append line has lost proxmox-start-auto-installer,
or if auto-installer-mode.toml is absent (which would mean the one Felhom-
labelled entry boots a manual installer). The rendered menu is then gated for
exactly 1 entry, 0 submenus, and zero references to proxtui/proxdebug/nomodeset/
Rescue Boot/memtest/fwsetup — and re-verified by reading the menu back OUT of
the finished ISO, not merely out of the extract tree.
mkimage-surgery.sh -> iso-repack.sh: branding and the slice-B loader swap need
the same extract -> modify -> re-master cycle, so they share one pass instead of
re-mastering twice. The mkimage recipe is untouched. The embedded module list is
still derived from the STOCK grub.cfg (snapshotted before branding rewrites it),
plus gfxmenu's bitmap/bitmap_scale/trig renderer deps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Fills the gap flagged in §8.1. Upstream notes for bookstack, immich, nextcloud,
n8n, grafana, vikunja, privatebin, radarr, romm; plus the ones this sweep found
EMPIRICALLY on a real deploy (outline PGSSLMODE, tandoor port+ALLOWED_HOSTS,
zipline DATABASE_URL/healthcheck rename, homebox tag prefix + pepper, wger
DJANGO_DB_* + :8000) -- which are the more useful half, being what actually
happens rather than what a changelog remembered to mention.
Four majors (calcom, claper, gramps-web, onlyoffice) deployed clean but have no
retrieved upstream one-liner; recorded as not-retrieved, not fabricated.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
An offsite run after the trio's fresh redeploy shows the snapshot TAGS are stable
across uninstall -> volume-destroy -> reinstall: calibre-web 243ad08c -> 9698ba74
and immich 6df12205 -> ea511336 both keep their exact tag sets, so a
'restic --tag <app>' query still spans the reinstall boundary and pre-uninstall
restore points stay addressable.
Side effect: the standing 'kimaradtak: bookstack' offsite warning is now CLEARED.
bookstack had no offsite snapshot because it had no tier-1 backup to push; the
fresh redeploy produced one, so its first-ever offsite snapshot landed (run ok,
1m13s, 6 snapshots, 211.6 MB, last_warning None).
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Adds the CAMPAIGN 7 audit doc (53/53 apps deployed and validated on the demo box
through the real controller endpoints; 45 pass, 13 template fixes, ranked
findings) and the uninstall-semantics map row it proves out: remove requires stop
first, named docker volumes are ALWAYS destroyed including the DB, HDD data and
backups survive unless explicitly ticked, and the per-app offsite toggle outlives
the app while tier-2 config is cleared.
ROADMAP +3:
R-40 the update path cannot express a multi-hop major upgrade (nextcloud
forbids skipping majors; the template moved 31 -> 34)
R-41 no standing 'does every template still deploy?' check -- the sweep was
the first thing to try, and found 5 apps that had NEVER been deployable
R-42 ruling needed on DB/cache sidecar majors (deliberately not bumped)
CLAUDE.md: REPORT.md is overwritten, so a second concurrent session in this repo
writes REPORT-<topic>.md instead -- applied to this run's own report.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
R-36: both halves delivered — the enabled-but-unprovisioned warning on the
customer page (reusing the same predicate the offsite re-issue handler refuses
on), and the related sub-item, auto-minting the self-bind link at customer
creation AND RESET completion so the console banner's promised email is already
true. Records the gap found while wiring it: PurgeCustomerResetDBState does not
clear selfbind_tokens, so a pre-RESET link would have survived the reset; the
skip paths now clear stale tokens.
R-37: the post-RESET staleness banner, narrow by design — an in-flight reset
does not trigger it, it clears itself on the first post-RESET report, and ties
resolve to STALE because SQLite timestamps are second-resolution and a
same-second report almost certainly predates the reset.
Both red-proofed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Deploys hub v0.67.0 (auto-minted self-bind link, post-RESET staleness banner,
unprovisioned-offsite warning, pbsdr_reissued flash text). The manifest is the
truth — the code push and image build deploy nothing until this tag moves.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Four small items, each a case where the hub already knew something and said
nothing. Green: build, vet, tests all pass.
(a) Self-bind link is minted automatically at customer creation AND at RESET
completion (R-36 sub-item). The console banner tells the customer to open
"az e-mailben kapott link"; until now that email existed only once the
operator remembered the button, so the banner could point at something that
did not exist — during the 2026-07-18 rehearsal the box waited ~11.7 min on
exactly that. handleSelfBindLinkSend's body was extracted into a shared
mintAndSendSelfBindLink core so the button and the auto-mint callers cannot
drift apart on the honesty rules: F1 (no address -> mint nothing) and F2
(send failed -> delete the token, never leave it live). The wrapper NEVER
fails the operation it rides on — a create that provisioned Cloudflare,
offsite and PBS must not 500 over a courtesy email.
Gap found and closed while wiring it: PurgeCustomerResetDBState does NOT
clear selfbind_tokens, so a link minted BEFORE a reset would have stayed
live across it. A successful mint already replaces it (delete-then-insert,
single-active); the skip paths would not have, so they now clear stale
tokens too. Invariant: after auto-mint runs the only live link is one it
just issued, or none.
(b) Post-RESET staleness banner (R-37). When a RESET COMPLETED after the newest
report, every health figure on the page describes a lifecycle that no longer
exists, and the page kept showing pre-RESET warnings as current. Narrow on
purpose: an in-flight reset does not trigger it, and it clears itself when a
report arrives. Ties resolve to STALE — SQLite timestamps are second-
resolution and a same-second report almost certainly predates the reset;
erring the other way would hide the banner exactly when it matters.
(c) Unprovisioned-offsite warning (R-36 interim). enabled==true with type=="" is
a real, stable, silent state: provisioning is Save-triggered and the
re-enroll auto-re-issue deliberately skips an unprovisioned target, so
nothing self-heals it. Reuses the exact predicate the offsite re-issue
handler already refuses on.
(d) pbsdr_reissued rendered an EMPTY flash box — the key had no template branch,
so re-issuing PBS credentials showed a success box with no words (observed
live 2026-07-18). Now describes what was staged plus the R-39 caveat:
confirm `pvesm status` shows the entry active, because a converged agent can
report `applied` while the storage still 401s.
New .flash-warn (amber, --warn tokens) for the deviation tier between success
and error — exception-color principle: only on deviation, never on a healthy
page.
Tests assert each banner is ABSENT in the nominal cases as well as present in
the deviating one — a banner that always renders is worse than none. Both
red-proofed: deleting the pbsdr_reissued branch reproduces the original empty
box; neutering the staleness predicate fails the banner assertion. New
read-only store accessor CountSelfBindTokens makes the single-active invariant
assertable.
NOT in this train: the R-39 hub-side generation-bump fix the pre-travel task
made conditional. Its condition was REFUTED (SetHostDesired bumps
unconditionally; applyPBSDR is idempotent as documented) — the real mechanism is
the agent's descriptor-hash convergence and needs its own spec.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
admin