Read-only probes on demo-felhom: both disks (system SSD /dev/sda, USB /dev/sdb) report
PASSED via the exact allowlisted 'smartctl -a -j <dev>'. Card shows Nincs adat because the
agent never reads: 'local' is a dir on LVM pve-root (backing='' + smartDeviceFor has no
dm/LVM branch), and the USB is surfaced via the non-enriched driveTargets.Known union path.
-d sat NOT needed (bridge passes SMART through; sudoers grants only -a -j). Graded fix
directions B(low-risk, USB) > A(system SSD, medium) > C(reject). STOP — no fix implemented.
Virgin-ISO nested drill closed the train: dead-NIC install baked the
fallback (incl. the dead default gateway), the R-59 screen painted
(capture committed beside the spike doc), the cable move healed +
registered at the hub in 23s unaided, and the build's rootpw file
matched the installed box's shadow hash. R-59 SHIPPED with the recorded
deviation (first-boot gate; installer-initrd abort out of scope by
operator ack). R-60 SHIPPED (spike + drill cited; F-P9 route-flush fix
included). R-61 slice 1 SHIPPED. New R-62 row (hub delete-dialog
cosmetics, XS). Capability map: new PROVEN-LIVE row (nested != metal,
said so). Cleanup verified: felhom-pve interfaces byte-identical,
bridge/VMs/ISO removed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
Nested VM 9310 on felhom-pve (throwaway vmbr9 rig): re-point
bridge-ports -> ifreload (0.36s, raises the port itself) -> dhclient
lease 3.3s -> hub HTTPS 302 in 160ms; a dead candidate costs exactly
its timeout bound (rc=124 at 20s). Bonus finding: the auto-installer
picks the NIC that leases and bakes the lease as STATIC — so wrong-NIC
boxes come in two variants (192.168.100.2 fallback bake vs
truth-changed-after-install), and the gate must trigger on
hub-unreachable, never on the fallback signature. Eight design inputs
(F-P1..F-P8) recorded for the Part-3 gate. STOP-0 discharged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
Capture disproved addendum 1's guess: macOS DOES send a correct NBNS query and
IS answered correctly in 140us; it simply never acts on it. Records the proven
per-client matrix and keeps sidebar discovery explicitly unproven.
The v0.150.0 'orphaned dhclient' cleanup killed guest 9201's DHCP client, not a
host one: LXC guests share the host PID namespace, so the missing eth0, pidfile
and lease file were all true ON THE HOST and all irrelevant. Lease expired ~80
min later, guest lost its address, tunnel/hub/catalog/agent all went dark.
Restored; records the one-command ownership check (/proc/<pid>/cgroup) and
leaves the unsupervised-dhclient risk open for a ruling with R-50.
Also records against R-45 the lesson the loop taught: a phase a client answers
with a one-shot action must be an EDGE the registry serves once, never a level
synthesised into the phase channel.
VERDICT. H1 confirmed: the reconstitution never ran. /backup/offbox/reconstitute
was never hit (reconstituted/safety-dump/replay line counts all 0); the operator
clicked the old missing-only button, which merged 34 files and left the DB
untouched. H2 rejected (the full path never executed; staging used the correct
snapshot 49e7cb46). H3 rejected (no replay happened; DB was genuinely asset:0).
R-44 is exonerated and working: 49e7cb46 carries offsite_run_id 20260719T102851Z
and its dump probes to asset:11 / user:1 — a properly coherent pair, against
round 1's asset:0 / user:0.
H4 (NEW, the more important result). Phase-3 recovery ran the real sequence and
the v0.148.0 path executed correctly — safety dump, stop, start, replay — then
the replay ABORTED: relation "clip_index" already exists. Proven to the second:
10:58:25 controller begins replay
10:58:33 immich-server "Reindexing clip_index" -> "Reindexed clip_index"
10:58:35 dump's CREATE INDEX clip_index fails, exit 3
The replay races the app's own schema repair, because reconstitution starts the
stack before replaying (ImportDump needs a live container). The photos survived
only because pg_dump emits COPY data before CREATE INDEX — luck, not design. The
same start-then-replay shape is on the LOCAL path, so it is a class defect.
Recovery: PARTIAL and STOPPED as instructed. All 11 assets are back and visible
by every DB criterion (status=active, deletedAt null, all files resolve), but the
operation reported failure and immich reports schema drift. Not clean acceptance
evidence; no second attempt made; safety dump read only.
Findings -> R-47 (the replay race), R-48 (restore controls separable only by
layout — the proven cause of this incident), R-49 (1.1 GB of a 1.2 GB immich
backup is ML model cache + a postgres data-dir tar duplicating the .sql dump +
immich's own nightly dump; recorded, not changed).
RULINGS LANDED.
- 00-capability-map.md:61 -> PARTIAL, scope-corrected: 6D's destruction hit the
FILE TREE only; the DB survived in its named volume, so "end-to-end from
offsite alone" overclaimed scope.
- The 704.6 MiB "discrepancy" -> CLOSED, not a defect: it was immich's own
Tarhely widget, never a controller page. Same for round 2's 650MiB->1.4GiB.
- customer-restore row: partial evidence recorded, NOT flipped.
Diagnosis only — no code, label or layout changes; no forget/prune.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
ROADMAP R-43/R-44 → SHIPPED with what actually landed:
- R-43: ReconstituteFromOffsite — safety dump → stop → files overwritten to
the snapshot version → start → the snapshot's own dump replayed from the
SCRATCH unit (the live unit is still never overwritten). Invariants:
nothing is ever deleted, and the undo is verified on disk before the act.
- R-44: every run, manual AND nightly, refreshes dumps + units BEFORE the
capture, so each snapshot is a coherent {DB@T, files@T} pair stamped with
offsite_run_id + dumps_at. Nightly ordering is now structural rather than a
coincidence of two scheduler entries at 02:30 and 04:15.
DIAG doc gains a "fixed in v0.148.0" header pointing at both.
Capability map deliberately NOT flipped: the DB half now exists but is PARTIAL,
not PROVEN-LIVE — shipped and deployed with no live acceptance behind it. The
customer-restore row stays MISSING, now blocked only on the evidence run rather
than on missing capability, and R-3 stays DRAFT.
Still open: the 00-capability-map.md:61 ruling — did CAMPAIGN-6D's "immich
end-to-end from offsite alone" exercise the DB half, or only the file half?
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
Viktor deleted 11 immich photos to test offsite restore; both place runs
merged 0 files and flashed success. Two-layered root cause:
- A UI delete means TRASH, not deletion — no file left the disk, so there
was genuinely nothing missing to restore. The test proved nothing.
- A valid test would also have failed: no offsite path loads a database.
All three buttons are file-only (offbox_restore.go:171-232, :346-421).
Files return; the library stays empty. The only DB replay is
RestoreFromRecoveryUnit, reachable solely from the LOCAL restore panel.
Compounding: the manual push never refreshes dumps (offbox_handlers.go:
203-227) — today's unit dump is from 02:30 and probes to asset:0 / user:0 /
album:0, predating both the admin user and the photos. Had a full restore
loaded it, it would have written an empty DB over the live one, destroying
the trashed rows that were the only surviving recovery path.
Answer to "can a customer trust same-day offsite?" for a DB-indexed app:
no — files come back, content does not.
No product code changed. Photos left in trash (recovery not wanted).
Capability map: customer-restore row stays MISSING by finding, not by
absence of attempt; offsite-restore row flagged scope-contested pending
a ruling on whether CAMPAIGN-6D exercised the DB half.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
Fills the gap flagged in §8.1. Upstream notes for bookstack, immich, nextcloud,
n8n, grafana, vikunja, privatebin, radarr, romm; plus the ones this sweep found
EMPIRICALLY on a real deploy (outline PGSSLMODE, tandoor port+ALLOWED_HOSTS,
zipline DATABASE_URL/healthcheck rename, homebox tag prefix + pepper, wger
DJANGO_DB_* + :8000) -- which are the more useful half, being what actually
happens rather than what a changelog remembered to mention.
Four majors (calcom, claper, gramps-web, onlyoffice) deployed clean but have no
retrieved upstream one-liner; recorded as not-retrieved, not fabricated.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
An offsite run after the trio's fresh redeploy shows the snapshot TAGS are stable
across uninstall -> volume-destroy -> reinstall: calibre-web 243ad08c -> 9698ba74
and immich 6df12205 -> ea511336 both keep their exact tag sets, so a
'restic --tag <app>' query still spans the reinstall boundary and pre-uninstall
restore points stay addressable.
Side effect: the standing 'kimaradtak: bookstack' offsite warning is now CLEARED.
bookstack had no offsite snapshot because it had no tier-1 backup to push; the
fresh redeploy produced one, so its first-ever offsite snapshot landed (run ok,
1m13s, 6 snapshots, 211.6 MB, last_warning None).
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Adds the CAMPAIGN 7 audit doc (53/53 apps deployed and validated on the demo box
through the real controller endpoints; 45 pass, 13 template fixes, ranked
findings) and the uninstall-semantics map row it proves out: remove requires stop
first, named docker volumes are ALWAYS destroyed including the DB, HDD data and
backups survive unless explicitly ticked, and the per-app offsite toggle outlives
the app while tier-2 config is cleared.
ROADMAP +3:
R-40 the update path cannot express a multi-hop major upgrade (nextcloud
forbids skipping majors; the template moved 31 -> 34)
R-41 no standing 'does every template still deploy?' check -- the sweep was
the first thing to try, and found 5 apps that had NEVER been deployable
R-42 ruling needed on DB/cache sidecar majors (deliberately not bumped)
CLAUDE.md: REPORT.md is overwritten, so a second concurrent session in this repo
writes REPORT-<topic>.md instead -- applied to this run's own report.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Human Explorer leg exposed the split: FELHOM-SPIKE renders (WSD PASS) but the
double-click fails 0x80070035 — flat name resolves by no path (DNS/LLMNR/NetBIOS
all silent; disable netbios=yes). By-IP mount works => SMB is healthy, the gap is
name resolution. Fix verified live: adding nmbd (NetBIOS) => nbtstat lists
FELHOM-SPIKE, ping resolves, \FELHOM-SPIKE\spike-share mounts by name. R-7 must
ship smbd+wsdd+nmbd (+avahi/.local), not wsdd alone.
Verdict: appliance guest is LAN-bridged; multicast discovery works only in the
guest netns (guest-direct or docker --network host) — default bridge is deaf to
LAN multicast. Real samba+wsdd on host-net: Windows 11 ProbeMatch + 445 + SMB
round-trip PASS; SSDP MediaServer:1 reaches LAN clients. R-7 => host-network
LAN-bound; R-8 Jellyfin-DLNA plausible if host-network. ROADMAP R-6 -> spiked,
R-7/R-8 unblocked. S4.4 Explorer render pending human.
Empirical spike (demo arc, docs-only, no product code). Measures the two hub->box
'sync now' transports. Verdict: (b) long-poll = primary for user config (hub-only,
reaches controller, no ep0/WG/agent coupling; binding ceiling = twin 60s nginx
proxy_read_timeout + hub WriteTimeout, both config lifts); (a) ep0-relayed poke
= agent-plane nudge (~0.42s, zero ep0/box infra change, EKEYREJECTED-confined).
Viktor Q2 answered: hub.felhom.eu is DNS-only, no Cloudflare. All live mutations
reverted; arc re-verified green.
Closes the two deferred deep campaigns from 6D on live infra: tier-2 v2 layout
engine (A1-A4) and F7 atomic-write + restic C2 self-heal (B1-B4). No production
code shipped. Two LOW findings (F-6E-1 demo drive drift; T-6E-1 DB-dump dir-fsync
parity). Drill repo torn down; fixtures reverted; box healthy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Refutes the surviving theory: on the drill host the pbs_dr descriptor is present
and enabled, the WG peer exists, verify passes — the block is a one-time secret
consumed 07-12 that no path re-mints after the agent loses its converged marker
(re-install/rollback; same WG pubkey -> changed==false -> cascade can't re-fire).
Live-proven: staging any consumable secret converges in one tick (existing token,
zero ep0 churn); re-asserting a converged descriptor is a clean idempotent no-op.
Safe re-trigger = re-serve a secret gated on agent waiting_secret/consumed_failed,
never blind-timer Reissue. Design-inputs table handed to the self-heal TASK spec.
Drill left CONVERGED (P-DAY0-DEEP PBS-DR leg now GREEN).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Fresh-appliance PBS-DR consume races the WG handshake at install (fingerprint probe timeout),
consumes nothing, and does not re-request the descriptor once the tunnel is up; reused-peer
re-provision never re-issues it. No operational re-trigger. Escrow ceremony + offsite round-trip
blocked on this fresh guest → friend-alpha alpha #1 escrow must be supervised live (customer-holds-R).
Drill deviation (CC-held R) never exercised. Recorded, not fixed inline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
immich enlargement blocked at real 50 GiB quota via sparse mandatory fixture; unit-only
continues; EnlargedBlocked edge-fires once; both operator + customer emails delivered;
fully reverted. G4 notification email restored via the real form (wipe-bug regression).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Core-promise legs proven on real data: .fab >=1GiB full circle (byte-identical mandatory,
zero sibling contamination, DB round-trip) + SQ3 immich offsite restore-to-live (functional
from offsite alone). Deferred: P3-BROWSER/P3-DELIVERY/P-TIER2-deep/P4-timing/P-DAY0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
Unattended close-out of the C6 remainder. Flagship finding C6B-F1 (CRITICAL):
.fab export silently produces hollow, data-free bundles for the standard
${USERDATA_PATH} needs_hdd convention (12/13 media apps) — sonarr 4.17 GB -> a
2308-byte config-only bundle, past the v0.125.0 anti-hollow guard. Transport
(upload/download, real CF edge, 413, byte-integrity) all PASS. C6B-F2 (MED,
confounded): net-storage removal leaves an orphaned autofs mount. Browser-
interactive + deep backup-tier items split to CAMPAIGN-6C. No code changed; no
secret/R/blob committed; campaign credential left for Viktor to rotate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A45Qop8YY8tS94bz63LFne
CAMPAIGN-5 findings doc (2026-07-14): NAS re-arm ring core + v0.129.0 fix live-proof
Findings-only. v0.129.0 F-A/F-B/F-C all CONFIRMED FIXED live on the fleet. NAS
re-arm ring core PASS on a fresh campaign NFS share: F10 start-limit clear +
reset-failed+rearmed, F9 per-share verdict (no empty sweep), F1/F2 clean removal,
F12 unit cycle-free. F8 improved (stub/mounted:false during outage) but reachable:true
still server-level. Reboot half of the matrix + upload/browser planes deferred with
procedures. No credential/R/blob committed.
Claude-Session: https://claude.ai/code/session_01LbMm4T7Ayzs1unB9pN6Uqd
@
admin