Unconditional and three sessions overdue, so it commits before any code is
touched — E-2d itself stopped at Phase 0 and banked nothing.
E-2d row: 822 words -> 121, and the contradiction resolved. Its State read
CLOSED — PARTIALLY PROVEN while the cell's final sentence read "This row stays
OPEN only for the residue"; a reader could not tell which. It is CLOSED, with
R-116 the single named open leg.
Nothing unique was binned. Three facts existed ONLY in that cell and are moved
into audits/E2D-fresh-vm-2026-07-29.md as a new §1a: the local-lvm fence figures
with the 888 GB nvme alternative, the exactMount subdirectory caveat and why the
subdirectory is nonetheless the safe placement (no durable_id collision), and
the ISO/PAIRING -> DIRECT fall-through derived at source with its line
citations. drill-r50's blocked status was already in both audits.
Capability map: it had ZERO rows for the backup-target work — grep gives 0 hits
for backup_target and one for "E-2" that is a campaign date string. Three
scenario rows added, at today's honest status, not the value hoped for later:
C. Protection & recovery — installer Case A/B, DEGRADED recorded not hidden
PROVEN-LIVE, cites E2D-fresh-vm C1+C2
D. Storage & devices — the offer, and that registration confers no role
PROVEN-LIVE, cites SESSION-C C4 + the decline path
F. Notifications & monitoring — the absent-target alarm and its pairing
PARTIAL, cites SESSION-C C5, leg named, -> R-116
Row F is PARTIAL today per the doc's own strict enum (a leg not exercised live
is PARTIAL with the leg named, never PROVEN-LIVE). A later session may flip it;
this commit must not.
Full ISO/PAIRING run on a fresh nested box. Agent 0.114.0 came from the Day-0
manifest -- the SHIPPED binary -- so C5 tested the real artifact. Controller
0.186.0 hand-deployed after install per the §3.1 ruling; the vouched golden
bakes 0.185.1, so C3/C4 prove the code not the shipped golden, and that lag is
filed against R-115 rather than a new ID.
R-113 PROVEN: detach 18:43:50, gate fired 18:43:54 -- four seconds, where E-2d
measured zero over 4.5 minutes -- and SetDisconnected was reached. It fired on
exactly the shape that defeated it: raw /mnt/mentes NOT mounted while the bind
/mnt/felhom-drives/mentes still read /dev/sdb[/felhom-data].
R-114 PROVEN: with the target absent the page rendered the absent copy, the
system-disk copy 0 and the offer block 0. Both of E-2d's falsehoods are gone.
R-112 PROVEN: the banner reached a customer's page for the first time. Healthy
renders nothing, proven POSITIVELY -- idle delta 0 /backup/tiers calls, page
load delta +1, single caller, so the seam ran and chose silence.
C5 FAILED on a fourth, separate defect. The alarm fires but as the GENERIC
storage_disconnected, while the recovery is the SPECIFIC backup_target_restored
-- a pair an operator cannot match, which is what notifyDriveReturned's own
comment forbids. backup_target_absent count 0 across the run. Root cause: the
drive is TWO /disks rows and BackupTarget and GuestPath sit on different ones;
absent they separate, on return they rejoin. v0.184.1 fixed the keying, not
this. Only reachable because R-113 made the gate fire at all. Filed as R-116.
Mirror + over-correction guard PASS: non-target drive -> storage_disconnected,
backup_target_absent 0; both drives present -> 0 ABSENT lines and the target
stayed healthy. Caveat recorded: the mirror passes trivially because the target
also produced the generic event.
E-2 and E-2d CLOSED as partially proven with R-116 the one named open leg, per
the runbook's §9 rule decided in advance rather than mid-run.
Capability map NOT touched: it has no E-2 rows at all, so nothing could move to
PROVEN-LIVE. Creating them is a design act, not a validation act.
Teardown clean: pvesm status after == before (local-lvm 38.78%), guest 9201 and
drill-r50 untouched. Customer delete attempted and correctly refused while the
host still reads ONLINE; command recorded for once it ages to DOWN.
The delete was correctly refused at four successive gates while the host still
read ONLINE (acknowledgements -> typed confirm_id -> expect_hosts stale-preview
-> "host is ONLINE"). Rather than force it, the run waited for the destroyed
host to age to DOWN; delete-impact then reported deletable:true and the
documented cascade ran:
host deleted (escrow demoted to retained custody), tenantsync deprovisioned,
PBS tenancy deprovisioned, claim reset to unclaimed, residue purged
(reports=5 app_telemetry=5 notif_prefs=1 appliance_registrations=1)
Verified after: 0 occurrences of "e2d" anywhere on the hosts page; demo-felhom
and demo-hp ONLINE on agent 0.113.0; drill-r50 and peti-felhom unchanged;
demo-hp carries only guest 9201 and VM 300.
Scoping checked rather than assumed: the single purged appliance_registration
was this run's own appliance (810d10c5, bound to e2d-fresh). The unrelated stale
2026-07-25 appliance (206c8838 / QWA-WJE) was NOT touched by the cascade — the
operator removed it separately.
- OPEN-ITEMS.md: the drill-cleanup WATCHING row is removed (done, not open).
- audits/E2D-fresh-vm-2026-07-29.md §8 + REPORT-e2d.md: teardown recorded as
complete, with the cascade output and the appliance-scoping note.
Full ISO/PAIRING route on a nested PVE VM on demo-hp, after R-111 was fixed
earlier in the session. Bind -> running controller in 3m35s. The install fetched
the artifacts published an hour before and restored the golden baked 20 minutes
before, so the publish train is proven end to end on a real install.
C1 PROVEN: "felhom-host-install v1.22.0", "Day-0 provision SUCCESS", guest 9201
running, bootstrap unit wrote its done-flag and self-disabled. This retires E-2's
"installer-logic-tested, not install-tested".
C2 PROVEN: both DEGRADED warning lines verbatim, backup.local_backup_target=local,
no felhom-backup storage created, and the install did not abort.
C3/C4 PARTIAL and C5 FAILED — three findings, none fixed:
R-112 (P1): E-2's degraded banner and offer have NO UI CONSUMER. The endpoint
returns byte-exact copy; grep 'backup-target' across every html/js/css is 0 hits
and no page handler injects the state. Templates fetch 18 distinct /api/storage/*
endpoints; these two are the only ones with zero references. v0.185.1 fixed the
router mount and stopped one layer short of the render. Fifth instance of
seam-built-but-never-wired.
R-113 (P1): the drive-absent gate CANNOT FIRE on device loss. planDriveGates
reads presence from BoundUnderParent = "is this path in the guest's mountinfo".
The raw mount is a device-bound systemd unit and dies with the device; the
agent's own bind is not device-bound and outlives it, so the gate sees "present"
forever. Live: agent reported the drive absent every 20s for 4.5 minutes, the
controller logged 0 [gate] lines, the hub received zero events -- neither
backup_target_absent nor the generic storage_disconnected. Sixth instance of the
class: E-2b wired the seam to a condition that cannot occur.
R-114: on target-drive loss the message claims the backup is on the system disk
(false) and offers the drive that just vanished. Invisible only because of R-112,
so it must be fixed BEFORE R-112 is wired.
Also filed as a second instance under R-110 rather than a new ID: host-install
fetches nine files from raw/branch/main and the hub vouches a sha for one;
E-2a's wrapper is installed 0755 to /usr/local/sbin, root-fenced in sudoers,
validated only by bash -n.
C4 is fully proven at API level: decline path (registration confers no role),
restart_required:true, agent did NOT self-restart (in-flight check performed and
recorded first), E-2a wrapper created the storage at the drive's own mountpoint,
and healthy renders nothing.
Teardown: VM destroyed, scratch storage removed, pvesm status after == before
(local-lvm 38.77%), guest 9201 and drill-r50 untouched. Hub records for e2d-fresh
remain -- delete correctly refused at four gates, finally "host is ONLINE";
deletable once it ages to DOWN. Command recorded in OPEN-ITEMS.md.
capability-map NOT touched: the customer-facing legs are broken rather than
proven, and the map has no E-2 rows at all.
No VM created, no install run, no box touched. The run stopped at the Phase 0
gate per runbook §3, before provisioning.
felhom-host-install.sh does not install what is on main. resolve_artifacts()
(:423-436) reads the hub-vouched manifest and fetches Gitea GENERIC PACKAGES
(agent :1945, golden :2573). Gitea's newest are agent 0.96.0 and golden 0.161.0;
the hub manifest selects exactly those; the global floor v0.156.0 is below the
golden's 0.161.0 so nothing self-updates. A fresh box therefore lands on agent
0.96.0 + controller 0.161.0 against main's 0.113.0 / 0.185.1. Agent 0.113.0
reached both demo boxes by direct deploy and is not in the channel at all.
Claim impact, each pinned to its introducing commit:
- C1 (real rc=0 1.22.0 install) and C2 (Case B natural) — ACHIEVABLE, not run;
both are installer-side and host-install is served at 1.22.0.
- C3 — BLOCKED: banner + GET /api/storage/backup-target are controller v0.185.1
(cdaeb36), copy v0.185.0 (3f7cf2a). Unblocks cheaply by raising the hub floor
to >=0.185.0; measured fleet impact nil (both demo boxes already 0.185.1).
- C4 — BLOCKED: needs controller v0.185.1 + agent v0.113.0 (58b598b).
- C5 — BLOCKED: needs controller v0.184.0 (c1a63de) + agent v0.112.0.
Filed R-111 (P1): 17 unpublished agent releases (v0.97.0-v0.113.0) strand the
entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT, so a new customer's
box installs without them. Mirror of R-110, not a duplicate.
- audits/E2D-fresh-vm-2026-07-29.md — all four Phase 0 answers recorded so a
resumed run does not re-derive them (cadence 30s; hot-detach available; ISO
present; local-lvm fence re-measured at 38.77%, unchanged).
- OPEN-ITEMS.md — R-111 opened; E-2d re-stated, NOT closed.
- ROADMAP.md — R-111 under P1.
- capability map NOT touched: nothing was proven live.
The §5.1a operator STOP is retired — HUB_PW is in ~/.config/credentials and hub
auth was verified, so CC can bind on a resumed run.
Phase 0 sized C9-F1 properly before anything was designed: 43 of the 53 catalog apps have NO
subtree the Tier-2 restore can read (not 2), 9 are covered only for their file legs and never
their database or volumes, 1 is stateless. The asymmetry is Tier-2's alone — Tier-1 and offsite
both restore the unit and replay volume dumps, so BookStack always had a working restore and
only this button lied.
Shipped: the restore refuses BEFORE stopping the app and names the action that does work; a run
that proceeds claims only what it EXAMINED and discloses that the database and volumes are not
covered. C9-F2 alarms after a 5-minute sustained-restarting threshold, set above the 120s deploy
timeout, Mealie's 60s start_period and R-97b's 180s grace; StateRestarting is deliberately NOT
added to IsDownState.
Live: silent through ten 30s samples then app_start_failed at 5m25s, heartbeat now reads
"1 currently down" where Campaign 9 recorded 0; a real deploy stayed silent; bookstack refused
with its uptime unbroken; paperless re-restored 43/43 byte-identical, 16/16 docs clean.
Filed, not fixed: C9-F1b (route to the Tier-1 restore — its own task because it puts a
destructive operation behind a non-destructive button) and C9-F4 (nothing reads the Tier-2
copy's recovery-unit/ mirror, so the second local copy that exists for drive loss is unreachable
by any customer action — potentially larger than C9-F1).
Phase A is the headline and it passed on live hardware, through the real endpoints the UI
posts to: a customer who deletes files — or their entire app data directory — gets everything
back byte-identical, and the app works afterwards (paperless served the restored bytes over
its own API at the exact pre-deletion sha256). A1's two non-destruction promises both hold.
Three defects, recorded not fixed:
C9-F1 (HIGH) the Tier-2 restore button is offered for apps it can never restore, takes a
real outage, and reports "nothing was missing" — indistinguishable from a
genuine result, while 156 MB of that app's data sits unread in the same copy.
C9-F2 (HIGH) an app in a crash loop never alarms on any channel; StateRestarting is in no
down-set, so F-OBS's own heartbeat printed "0 currently down" for 9 minutes.
C9-F3 (MEDIUM) an interrupted offsite run leaves a lock the self-heal cannot reach; the tier
is dead until a human unlocks and the operator is told "unknown reason".
This answers Phase C item 8.
Two candidates were deliberately NOT filed: a recovery-unit poisoning the catalog sync healed
in ~3 min, and a snapshot_id that is documented as logging-only. Reporting either would have
been reporting an artifact.
Stopped at the end of Phase B (plus D10), then full recovery — both boxes healthy, real
cadences, offsite tier proven working again, no leaked scratch guests, peti untouched.
D11's approved staleAfter compression turned out not to be a knob; reported, not worked around.
F-LEAK's first fix (pool adoption) was refuted live and is recorded as such. R-100 is
F-CRIT-2's defect class on the hub for the restic tier: isStale reads only LastRun, which
the controller writes unconditionally on failure. Investigated, not fixed.
Both marked SHIPPED + PROVEN-LIVE in OPEN-ITEMS and the campaign doc. All three
of Campaign 8's alarm findings are now closed (F-CRIT-1, F-CRIT-2, F-A1).
Adds the standing rule earned by this arc to the versioned workspace CLAUDE.md:
a comment asserting an invariant needs a test pinning it, or it is a wish — with
all six shipped-false-guarantee instances catalogued, and the corollary that a
test should assert the CONSEQUENCE (does the alarm fire?) not the MECHANISM
(does suppression expire?).
F-CRIT-2 marked SHIPPED + PROVEN-LIVE: NewestArchiveTime now counts only
plausibly-complete entries (measured 1 MiB floor), verified by replaying
campaign fault 2 on demo-hp in both directions.
Files two follow-ups:
- R-99: prune never REMOVES phantoms. It does NOT count them toward keep-last
(dry-run kept 2 real + the phantom), so there is no retention/data-loss bug —
but they accumulate one per aborted upload.
- F-CRIT-1: filed READY-HIGHEST. Campaign 8's other HIGH finding, untouched by
this fix.
GC run 2 (04:44Z, past the atime boundary) reclaimed 0 B, and the cause is the
campaign itself: the datastore is relatime, so the night's three full verifies,
~14 backups, two GC runs and several restores kept refreshing chunk atimes.
Sampled atimes confirm no chunk was ever 24h5m cold. Not a defect - GC made a
correct, deliberate decision both times - but a real operational property:
on a relatime datastore, activity defers reclamation. The spec's intended
experiment therefore could not complete inside the window; recorded as such.
Fault 13 now complete: backup-vs-restore-test single-flight holds, and a full
two-tier backup ran successfully DURING GC run 2 (both artefacts landed, GC
TASK OK, zero .bad chunks). True three-way overlap remains untested.
Final fleet verification clean on all three hosts. Campaign-caused outage
(F-REBOOT, 9m47s on demo-hp) stated plainly in the final-state section.
Tail-of-campaign additions after the Phase D revert (both re-injections declared):
- fault 18 (delete a snapshot mid restore-test): detection PASS, and it ROOT-CAUSES
F-LEAK — a failed restore-test cannot destroy its own scratch guest (403,
missing VM.Allocate; the agent token is pool-scoped and a failed restore never
joins the felhom pool)
- fault 11 (guest reboot mid-backup): new finding F-REBOOT — the backup succeeds
but the guest never comes back; ~9m47s outage until a manual pct start
Two evidence corrections, both self-inflicted tooling errors:
- pgrep -cf <pattern> matches its own ssh command line, which invalidated fault
11's first two injections and put one unsound line in fault 9 (withdrawn; that
finding stands on the controller's own job state)
- ep0 runs Etc/UTC, so its 03:30 prune fires at 05:30 CEST — nearly misread as a
broken prune job
Nine findings now, still two HIGH. Fleet healthy.
Unattended 10h run against demo-felhom, demo-hp and ep0. No production code
changed; findings recorded and ranked, not fixed inline.
8 findings, 2 HIGH — both in the system's ability to report that a backup did
NOT happen:
- F-CRIT-1: an app failing to restart after a quiesce never alarms (invariant
I1 in main.go:1213 is false for the failed-restart path)
- F-CRIT-2: a failed offsite backup leaves a phantom snapshot that resets the
tier's freshness clock (NewestArchiveTime has no completeness check)
Retires several never-validated items, including R-87 (first restic restore
round-trip, byte-verified), the full R-88 backoff ladder, age_state=absent,
and the crash-recovery unquiesce under a real SIGKILL.
peti-felhom untouched; ep0 rollback copy intact; fleet healthy at end.
Supervised runbook execution. No code change, no version bump.
felhom-offsite moved from ep0's 40 GB root disk (/srv/pbs-felhom) to a
dedicated 100 GB Hetzner Cloud Volume (/mnt/pbs-datastore, ext4 -m 0,
by-id fstab, relatime). Datastore NAME unchanged, so the PBS-DR
descriptors, per-box storage ids, ACLs and namespaces are untouched.
Capacity: 37.2 GB -> 98 GB total, 28.9% -> 13% used, headroom to the 80%
warn 19 GB -> ~65 GB. This CLEARS the R-82 Phase 0 P0.3 STOP. Per-tenant
encryption still precludes cross-customer dedup, so the slope is
unchanged - the volume buys runway, not a better cost model.
Verified: byte totals and chunk counts identical (9748), 7/7 snapshots
across all three namespaces, backup:backup ownership, clean itemised
dry-run, full verify job TASK OK with 0 errors, and a restore round-trip
(source_tier pbs, pass true, mount_parity ok, clean teardown).
Nothing deleted - the original 13 GB stays at /srv/pbs-felhom as the
rollback until a new weekly backup lands. GC deliberately not run.
Three findings recorded:
- the `scratch` datastore points at a non-existent path (pre-existing;
now logs ENOENT every start) - operator decision
- the runbook's S6 guard test proves the wrong proposition:
RequiresMountsFor re-mounts rather than refusing, so the test only
bites when the device is genuinely unavailable (re-run that way, and
the refusal was observed) - amendment recommended
- S11: storage box u629193 has no live backup path, BUT ep0 carries an
enabled sshfs mount unit against it that must be removed before the
box is deleted
Deviations: the volume arrived pre-formatted and mounted; S8 ran on
demo-felhom rather than demo-hp (no SSH key for demo-hp); the window was
contended by a stale in-memory 10-minute restore-test cadence whose
config had already been reverted on disk.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018ARoadHBf8rHoscfiqeVZn
Read-only gate. No code, no backup triggered, no config changed.
P0.1 weekly CONFIRMED: the only 7-day-exposed state is the non-SMB half of
settings.json (storage_paths, app_backup toggles, notification prefs,
password_hash, launcher token) — none catastrophic, all recoverable. The two
items that would have overturned it, encryption.key and the offbox
credentials, are STABLE files unchanged since first boot, so a 7-day-old copy
is byte-identical. Everything referentially coupled to app state is carried
daily by Tier-1/2/3, and Tier-3 offsite was verified running and ok on both
production boxes. CONDITIONAL: drill-r50 has offsite:null, so the verdict does
not hold there.
P0.2 RESOLVED, benign: PBS returns HTTP 200 with zeroed usage to the
namespace-scoped token (DatastoreBackup, not Datastore.Audit), so PVE prints
0/0/0. Ground truth via the hub ep0 df op: 28.9% of 37.2 GB. Writes work —
the 07-18 snapshot is owned by this token, 9.74 GB, verify ok.
P0.3 STOP: 37.2 GB datastore, 10.8 GB used, 80% alert at 29.8 GB. Encrypted
per-tenant backups do not dedup across customers. Current 3 boxes project to
~15-21 GB weekly keep-3; each additional customer costs ~5-10 GB, so the 80%
alert fires at roughly the SECOND additional customer — inside the alpha
horizon. Needs a ruling on datastore size / retention before recurring writes.
Read-only diagnostic. No fix applied.
Verdict: FALSE ALARM for demo-felhom + demo-hp, true-but-premature for
drill-r50. Fired ONCE (2026-07-26 03:00 UTC), not nightly. No real
external customer notified.
Root cause: felhom-agent/internal/backup/store.go is in-memory by design;
an agent restart empties the host-report 'backups' array until the next
backup runs. The R-50 island migration restarted the fleet on 2026-07-25
12:44 UTC, opening an ~18h blind window that the 03:00 UTC deadline check
fell into. Hub's assessBackupFreshness reads the empty array as 'no
backup exists'.
Separate real finding: the PBS/offsite-DR tier has no schedule at all
(local_backup_target=local, no jobs.cfg) — demo-felhom has one snapshot
from 07-18, demo-hp has zero since pbs_dr was applied 07-21.
Read-only diagnostic; nothing restarted, reconfigured or deployed.
CAUSE (H2, supported): the island migration rewrote bootstrap.json
(169.254.253.1:8443) but the controller reads controller.yaml, which still
holds the pre-island LAN address on both boxes -- mtimes 2026-07-21 and
2026-07-23, i.e. days BEFORE the migration. ensureLocalAPI
(bootstrap.go:254) returns early when local_api.endpoint is non-empty, so it
fills a MISSING endpoint and never reconciles a STALE one. The agent no longer
binds the LAN address, hence "connection refused".
H1 refuted: every island leg is healthy on both hosts -- agent active and bound
169.254.253.1:8443, vmbr9 up, guest eth1 up, TCP from the guest to the island
is OPEN. The island works; nothing is using it.
H3 refuted: hub-persisted events put the first alarm at 2026-07-25 12:44/12:46,
~17.5h before the v0.172.0 deploy. Today's re-fires are restart artifacts --
channel state is in-memory by design.
H4 refuted: felhom-agent is active on both hosts.
Also: the logged "up->down" is an ARTIFACT -- the debounce branch seeds an
unseeded state to "up", so the channel was born-down and has never been up.
Symptom B confirmed: EffectiveProtected gates cloudflared on a token predicate
but samba on smb.Enabled alone, so sharing-without-password reports FAIL for a
deliberately-undeployed stack. The function's own doc comment asserts the parity
that is broken. Caused by leg 4 of the R-75 task. Blast radius: zero emails so
far, but demo-hp flipped ok->fail and the health_critical path has reached the
CUSTOMER channel historically. The English issue strings are whole-surface, not
a one-off.
Fixes described, none applied.
Verdict GO, with one mandatory constraint.
Phase 0 HOLDS: the customer-facing path set is fully derivable from data the
controller already parses (ParseComposeClassifiableBinds), and
ValidateBackupSpec's refusal set already covers the path-safety class
data_paths: needs -- so the annotation-only design introduces no new
filesystem-write primitive. No catalog folder is reachable only via env
indirection; zero templates use long `type: bind` syntax.
Found off-brief: the derivation is ALREADY LIVE at deploy time
(stacks/manager.go:183 ensureUserdataMounts, "the deploy belt"), proven by a
clean two-box control -- media/podcasts exists on demo-felhom where
audiobookshelf is deployed, not on demo-hp, and is in no skeleton.
P0 GO: derived set 14 dirs vs hardcoded 14 (+media/podcasts, -documents); all
three hand-verified anchors match. P1 GO: 4 sources incl. nested + accented
names all index ready. P2 GO: deep-link template constructible in Go from
(sourceName, relPath) alone; login redirect preserves the target. P3 GO for the
feature, with a pre-existing setgid-chain break recorded. P4 GO: userdata/import
is shareable, guard unchanged, no live share created. P5: before-state inventory
captured read-only on both boxes. P6 GO-conditional: the naive derivation
produced 20 distinct outputs from 20 identical runs -- a guaranteed FileBrowser
force-recreate loop -- fixed by one sort. P7: 12 sources safe server-side.
No production code, no version bump, no live mutation.
- A4: scratch guest provisioned from golden on the island-configured drill came
up with net1 automatically (zero manual edits) — the v0.96.0 provisioning path
proven live. Spike method caveat CLOSED.
- REPORT: Phase A done + vouched 0.96.0; B0/B1/A4 done; B2/B3 await operator go.
Provisioned nested-PVE drill 'drill-r50' (qm300 on demo-hp) via the v1.25.0
nested-vm ISO through the real day-0, then ran the R-50 empirical spike:
- vmbr9 portless island bridge + guest island NIC hot-add (LAN undisturbed)
- F1 replay money shot: LAN move survives on the island; LAN-literal bind
reproduces the 2026-07-20 daemon-exit bug verbatim
- dnsmasq trap confirmed live + lan_resolver.host_ip fix proven
- pin address-independent (leaf SHA-256 unchanged, HTTP 200 over island)
- survival matrix: agent/guest/host-cold-reboot all return on the island
Docs: SPIKE verdict BLOCKED->GO, ROADMAP R-50 SPIKED->GO, nodes.md drill VM,
REPORT overwrite.
Read-only probes on demo-felhom: both disks (system SSD /dev/sda, USB /dev/sdb) report
PASSED via the exact allowlisted 'smartctl -a -j <dev>'. Card shows Nincs adat because the
agent never reads: 'local' is a dir on LVM pve-root (backing='' + smartDeviceFor has no
dm/LVM branch), and the USB is surfaced via the non-enriched driveTargets.Known union path.
-d sat NOT needed (bridge passes SMART through; sudoers grants only -a -j). Graded fix
directions B(low-risk, USB) > A(system SSD, medium) > C(reject). STOP — no fix implemented.
Virgin-ISO nested drill closed the train: dead-NIC install baked the
fallback (incl. the dead default gateway), the R-59 screen painted
(capture committed beside the spike doc), the cable move healed +
registered at the hub in 23s unaided, and the build's rootpw file
matched the installed box's shadow hash. R-59 SHIPPED with the recorded
deviation (first-boot gate; installer-initrd abort out of scope by
operator ack). R-60 SHIPPED (spike + drill cited; F-P9 route-flush fix
included). R-61 slice 1 SHIPPED. New R-62 row (hub delete-dialog
cosmetics, XS). Capability map: new PROVEN-LIVE row (nested != metal,
said so). Cleanup verified: felhom-pve interfaces byte-identical,
bridge/VMs/ISO removed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
Nested VM 9310 on felhom-pve (throwaway vmbr9 rig): re-point
bridge-ports -> ifreload (0.36s, raises the port itself) -> dhclient
lease 3.3s -> hub HTTPS 302 in 160ms; a dead candidate costs exactly
its timeout bound (rc=124 at 20s). Bonus finding: the auto-installer
picks the NIC that leases and bakes the lease as STATIC — so wrong-NIC
boxes come in two variants (192.168.100.2 fallback bake vs
truth-changed-after-install), and the gate must trigger on
hub-unreachable, never on the fallback signature. Eight design inputs
(F-P1..F-P8) recorded for the Part-3 gate. STOP-0 discharged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
Capture disproved addendum 1's guess: macOS DOES send a correct NBNS query and
IS answered correctly in 140us; it simply never acts on it. Records the proven
per-client matrix and keeps sidebar discovery explicitly unproven.
The v0.150.0 'orphaned dhclient' cleanup killed guest 9201's DHCP client, not a
host one: LXC guests share the host PID namespace, so the missing eth0, pidfile
and lease file were all true ON THE HOST and all irrelevant. Lease expired ~80
min later, guest lost its address, tunnel/hub/catalog/agent all went dark.
Restored; records the one-command ownership check (/proc/<pid>/cgroup) and
leaves the unsupervised-dhclient risk open for a ruling with R-50.
Also records against R-45 the lesson the loop taught: a phase a client answers
with a one-shot action must be an EDGE the registry serves once, never a level
synthesised into the phase channel.
VERDICT. H1 confirmed: the reconstitution never ran. /backup/offbox/reconstitute
was never hit (reconstituted/safety-dump/replay line counts all 0); the operator
clicked the old missing-only button, which merged 34 files and left the DB
untouched. H2 rejected (the full path never executed; staging used the correct
snapshot 49e7cb46). H3 rejected (no replay happened; DB was genuinely asset:0).
R-44 is exonerated and working: 49e7cb46 carries offsite_run_id 20260719T102851Z
and its dump probes to asset:11 / user:1 — a properly coherent pair, against
round 1's asset:0 / user:0.
H4 (NEW, the more important result). Phase-3 recovery ran the real sequence and
the v0.148.0 path executed correctly — safety dump, stop, start, replay — then
the replay ABORTED: relation "clip_index" already exists. Proven to the second:
10:58:25 controller begins replay
10:58:33 immich-server "Reindexing clip_index" -> "Reindexed clip_index"
10:58:35 dump's CREATE INDEX clip_index fails, exit 3
The replay races the app's own schema repair, because reconstitution starts the
stack before replaying (ImportDump needs a live container). The photos survived
only because pg_dump emits COPY data before CREATE INDEX — luck, not design. The
same start-then-replay shape is on the LOCAL path, so it is a class defect.
Recovery: PARTIAL and STOPPED as instructed. All 11 assets are back and visible
by every DB criterion (status=active, deletedAt null, all files resolve), but the
operation reported failure and immich reports schema drift. Not clean acceptance
evidence; no second attempt made; safety dump read only.
Findings -> R-47 (the replay race), R-48 (restore controls separable only by
layout — the proven cause of this incident), R-49 (1.1 GB of a 1.2 GB immich
backup is ML model cache + a postgres data-dir tar duplicating the .sql dump +
immich's own nightly dump; recorded, not changed).
RULINGS LANDED.
- 00-capability-map.md:61 -> PARTIAL, scope-corrected: 6D's destruction hit the
FILE TREE only; the DB survived in its named volume, so "end-to-end from
offsite alone" overclaimed scope.
- The 704.6 MiB "discrepancy" -> CLOSED, not a defect: it was immich's own
Tarhely widget, never a controller page. Same for round 2's 650MiB->1.4GiB.
- customer-restore row: partial evidence recorded, NOT flipped.
Diagnosis only — no code, label or layout changes; no forget/prune.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
ROADMAP R-43/R-44 → SHIPPED with what actually landed:
- R-43: ReconstituteFromOffsite — safety dump → stop → files overwritten to
the snapshot version → start → the snapshot's own dump replayed from the
SCRATCH unit (the live unit is still never overwritten). Invariants:
nothing is ever deleted, and the undo is verified on disk before the act.
- R-44: every run, manual AND nightly, refreshes dumps + units BEFORE the
capture, so each snapshot is a coherent {DB@T, files@T} pair stamped with
offsite_run_id + dumps_at. Nightly ordering is now structural rather than a
coincidence of two scheduler entries at 02:30 and 04:15.
DIAG doc gains a "fixed in v0.148.0" header pointing at both.
Capability map deliberately NOT flipped: the DB half now exists but is PARTIAL,
not PROVEN-LIVE — shipped and deployed with no live acceptance behind it. The
customer-restore row stays MISSING, now blocked only on the evidence run rather
than on missing capability, and R-3 stays DRAFT.
Still open: the 00-capability-map.md:61 ruling — did CAMPAIGN-6D's "immich
end-to-end from offsite alone" exercise the DB half, or only the file half?
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
Viktor deleted 11 immich photos to test offsite restore; both place runs
merged 0 files and flashed success. Two-layered root cause:
- A UI delete means TRASH, not deletion — no file left the disk, so there
was genuinely nothing missing to restore. The test proved nothing.
- A valid test would also have failed: no offsite path loads a database.
All three buttons are file-only (offbox_restore.go:171-232, :346-421).
Files return; the library stays empty. The only DB replay is
RestoreFromRecoveryUnit, reachable solely from the LOCAL restore panel.
Compounding: the manual push never refreshes dumps (offbox_handlers.go:
203-227) — today's unit dump is from 02:30 and probes to asset:0 / user:0 /
album:0, predating both the admin user and the photos. Had a full restore
loaded it, it would have written an empty DB over the live one, destroying
the trashed rows that were the only surviving recovery path.
Answer to "can a customer trust same-day offsite?" for a DB-indexed app:
no — files come back, content does not.
No product code changed. Photos left in trash (recovery not wanted).
Capability map: customer-restore row stays MISSING by finding, not by
absence of attempt; offsite-restore row flagged scope-contested pending
a ruling on whether CAMPAIGN-6D exercised the DB half.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
Fills the gap flagged in §8.1. Upstream notes for bookstack, immich, nextcloud,
n8n, grafana, vikunja, privatebin, radarr, romm; plus the ones this sweep found
EMPIRICALLY on a real deploy (outline PGSSLMODE, tandoor port+ALLOWED_HOSTS,
zipline DATABASE_URL/healthcheck rename, homebox tag prefix + pepper, wger
DJANGO_DB_* + :8000) -- which are the more useful half, being what actually
happens rather than what a changelog remembered to mention.
Four majors (calcom, claper, gramps-web, onlyoffice) deployed clean but have no
retrieved upstream one-liner; recorded as not-retrieved, not fabricated.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
An offsite run after the trio's fresh redeploy shows the snapshot TAGS are stable
across uninstall -> volume-destroy -> reinstall: calibre-web 243ad08c -> 9698ba74
and immich 6df12205 -> ea511336 both keep their exact tag sets, so a
'restic --tag <app>' query still spans the reinstall boundary and pre-uninstall
restore points stay addressable.
Side effect: the standing 'kimaradtak: bookstack' offsite warning is now CLEARED.
bookstack had no offsite snapshot because it had no tier-1 backup to push; the
fresh redeploy produced one, so its first-ever offsite snapshot landed (run ok,
1m13s, 6 snapshots, 211.6 MB, last_warning None).
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Adds the CAMPAIGN 7 audit doc (53/53 apps deployed and validated on the demo box
through the real controller endpoints; 45 pass, 13 template fixes, ranked
findings) and the uninstall-semantics map row it proves out: remove requires stop
first, named docker volumes are ALWAYS destroyed including the DB, HDD data and
backups survive unless explicitly ticked, and the per-app offsite toggle outlives
the app while tier-2 config is cleared.
ROADMAP +3:
R-40 the update path cannot express a multi-hop major upgrade (nextcloud
forbids skipping majors; the template moved 31 -> 34)
R-41 no standing 'does every template still deploy?' check -- the sweep was
the first thing to try, and found 5 apps that had NEVER been deployable
R-42 ruling needed on DB/cache sidecar majors (deliberately not bumped)
CLAUDE.md: REPORT.md is overwritten, so a second concurrent session in this repo
writes REPORT-<topic>.md instead -- applied to this run's own report.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Human Explorer leg exposed the split: FELHOM-SPIKE renders (WSD PASS) but the
double-click fails 0x80070035 — flat name resolves by no path (DNS/LLMNR/NetBIOS
all silent; disable netbios=yes). By-IP mount works => SMB is healthy, the gap is
name resolution. Fix verified live: adding nmbd (NetBIOS) => nbtstat lists
FELHOM-SPIKE, ping resolves, \FELHOM-SPIKE\spike-share mounts by name. R-7 must
ship smbd+wsdd+nmbd (+avahi/.local), not wsdd alone.
Verdict: appliance guest is LAN-bridged; multicast discovery works only in the
guest netns (guest-direct or docker --network host) — default bridge is deaf to
LAN multicast. Real samba+wsdd on host-net: Windows 11 ProbeMatch + 445 + SMB
round-trip PASS; SSDP MediaServer:1 reaches LAN clients. R-7 => host-network
LAN-bound; R-8 Jellyfin-DLNA plausible if host-network. ROADMAP R-6 -> spiked,
R-7/R-8 unblocked. S4.4 Explorer render pending human.
Empirical spike (demo arc, docs-only, no product code). Measures the two hub->box
'sync now' transports. Verdict: (b) long-poll = primary for user config (hub-only,
reaches controller, no ep0/WG/agent coupling; binding ceiling = twin 60s nginx
proxy_read_timeout + hub WriteTimeout, both config lifts); (a) ep0-relayed poke
= agent-plane nudge (~0.42s, zero ep0/box infra change, EKEYREJECTED-confined).
Viktor Q2 answered: hub.felhom.eu is DNS-only, no Cloudflare. All live mutations
reverted; arc re-verified green.
admin
Claude Code