Commit Graph

317 Commits

Author SHA1 Message Date
admin 6dab0f6a2c docs: ISO train v1.25.0 — REPORT (belt/apt/R-63/gate/vault live), F9 resolved, R-63 shipped, nodes belt+vault, F8 checklist; critical golden<floor finding
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR
2026-07-23 15:48:01 +02:00
admin b620435afe ISO train v1.25.0 code: OOB belt default appliance leg (F9) + apt no-subscription (Part2) + R-63 Lat2 console font + R-71 golden>=floor build gate
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR
2026-07-23 15:38:14 +02:00
admin 527d81cf70 docs: v0.72.0 ship report + live validation, CONTEXT rulings, DIAG-f10 R-70/R-71 status annotations
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR
2026-07-23 13:05:03 +02:00
admin 1133aade73 hub v0.72.0 — R-70 + R-71c: offsite delivery-state detector, card, stuck event, R-39(a)-guarded self-heal restage
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR
2026-07-23 12:59:04 +02:00
admin c801cee647 F10 offsite leg resolved on demo-hp: day-0 update-vs-bridge race diagnosed, designed-path repair + tier round-trip proven; R-70/R-71 minted
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR
2026-07-23 12:21:05 +02:00
admin c766c8af82 hub v0.71.0: paired recovery mails (F11), prefs seeding at claim + empty-email no-clobber (F12), priority headers + operator test leg (F14-light) 2026-07-22 20:57:29 +02:00
admin 5b35023574 docs(audits): power-outage recovery audit 2026-07-22 — power cut confirmed, full self-heal in 3m15s, dead-man's-switch fired on schedule; F8-F13 2026-07-22 19:57:44 +02:00
admin e57dd8b0e8 docs: R-67 SHIPPED (coupled to R-64) + NAS doc Browsing (FileBrowser) section
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
2026-07-22 14:18:30 +02:00
admin 8ecab3cd64 docs: R-66 shipped (ROADMAP + NAS naming caveat); mint R-64 pairing-blessed + R-65 buddy-box replication
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
2026-07-22 13:57:20 +02:00
admin a12c6f9730 docs(v1.24.0): R-59/R-60/R-61 SHIPPED — CHANGELOG, README, ROADMAP (+R-62), runbook, capability map, drill evidence, REPORT
Virgin-ISO nested drill closed the train: dead-NIC install baked the
fallback (incl. the dead default gateway), the R-59 screen painted
(capture committed beside the spike doc), the cable move healed +
registered at the hub in 23s unaided, and the build's rootpw file
matched the installed box's shadow hash. R-59 SHIPPED with the recorded
deviation (first-boot gate; installer-initrd abort out of scope by
operator ack). R-60 SHIPPED (spike + drill cited; F-P9 route-flush fix
included). R-61 slice 1 SHIPPED. New R-62 row (hub delete-dialog
cosmetics, XS). Capability map: new PROVEN-LIVE row (nested != metal,
said so). Cleanup verified: felhom-pve interfaces byte-identical,
bridge/VMs/ISO removed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
2026-07-22 11:45:55 +02:00
admin c043ba9abe docs(spike): R-59/R-60 Phase-0 — the NIC sweep mechanics are GO, measured on a nested wrong-NIC install
Nested VM 9310 on felhom-pve (throwaway vmbr9 rig): re-point
bridge-ports -> ifreload (0.36s, raises the port itself) -> dhclient
lease 3.3s -> hub HTTPS 302 in 160ms; a dead candidate costs exactly
its timeout bound (rc=124 at 20s). Bonus finding: the auto-installer
picks the NIC that leases and bakes the lease as STATIC — so wrong-NIC
boxes come in two variants (192.168.100.2 fallback bake vs
truth-changed-after-install), and the gate must trigger on
hub-unreachable, never on the fallback signature. Eight design inputs
(F-P1..F-P8) recorded for the Part-3 gate. STOP-0 discharged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
2026-07-22 10:50:43 +02:00
admin f4c2c4151b hub v0.70.1: the ghost customer's Delete button must exist (Danger-zone render gate split) 2026-07-22 09:23:37 +02:00
admin 2cf37f263b docs(pilot): agent 0.93.0 publish train record + onboarding §A5 gate discharged 2026-07-22 09:03:39 +02:00
admin 05dfaa1a10 docs(runbooks): onboarding draft v3 -> v4 — post-ship refresh (R-36/R-39 workarounds deleted, HP t740 second datapoint)
Per the 2026-07-21 refresh brief: R-39 interim blocks (B4/E1) and the R-36
manual-Save block (C4) deleted — both shipped and proven live; freemail.hu
gate proven (R-4 COMPLETE); golden/floor-lift note now cites two shapes
(rehearsal + virgin HP t740 day-0 lift 0.153.0->0.156.0); A3 loader table
per operations/nodes.md (N100=mkimage/SB-off per record, HP t740=shim/SB
ENABLED); B2 multi-NIC cabled-port gotcha (R-59/R-60 pending); new A5 gate
(agent >=0.93.0 deployed box-side before the first escrow ceremony); D
offboarding pointer to §G (R-25b). DRAFT status and the C7 graduation gate
unchanged. ROADMAP R-25b pointer follows the rename.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
2026-07-22 08:14:50 +02:00
admin 9b3381be0a feat(hub): v0.70.0 — a deleted customer actually disappears (residue leg + ghost cleanup)
Found validating v0.69.0 against the live hub. demo-vm-felhom was deleted
on 07-18 and was still on the Customers list AND still raising offsite_stale
(10 events, latest 07-21 17:34, operator email at 19:34) — because
GetCustomers() is report-derived and no lifecycle tier ever deleted a report.

New leg 3 (residue), before the record purge: reports, app_telemetry,
app_log_tails, log_tail_requests, customer_notifications, plus the
credential-bearing appliance_registrations and selfbind_tokens. Audit
(events, notification_log) and F-14 provenance still survive.

Ghost customers are now deletable: 404 means "nothing here", not "no config
row". With no config row the offsite descriptor is unknowable, so the Hetzner
and descriptor legs record skipped_no_config rather than a bare "skipped".

Two more red-proofs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS
2026-07-21 20:28:06 +02:00
admin 61dbd870c3 feat(hub): v0.69.0 — customer DELETE is the guided full-teardown cascade (R-25b)
POST /configs/{id}/delete now runs hosts -> RESET -> purge behind three
acknowledgements, a typed customer-id, a stale-preview check and the
ONLINE-host refusal (every gate before any write, so a refusal has zero
side effects). The shallow handleConfigDelete is gone.

Two invariants are asserted, not just commented: ruling 3 is preserved by
construction (leg 2 never sees a host row) and retained escrow custody is
purged exactly once, in leg 3 (leg 2 runs with purgeEscrow=false).

handleCustomerReset's committed half was extracted as commitCustomerReset;
the standalone RESET path is byte-identical to v0.68.1 and its suite is
untouched. Five red-proofs run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS
2026-07-21 19:31:48 +02:00
admin 8ec45f3b42 docs: node inventory for the two-host fleet; demo-hp on the tailnet
New documentation/operations/nodes.md: HP t740 hardware, disks (SanDisk system
SSD 182195804614; Toshiba 1TB NVMe 58BS11AFT8MQ PRESENT AND UNENROLLED, still
NTFS, do not touch), the five-NIC map and the trap that cost the first install,
and the access path - no SSH key is baked, auth is the hub-vaulted G1
break-glass password.

tailscale.md gains demo-hp, the operator-lab-exception warning so a future
product-shape audit does not conclude the product ships tailscale, and the
--accept-dns evidence: the join omitted the flag, MagicDNS rewrote
/etc/resolv.conf, and it was reverted. Harmless at the vacation site, would
have bitten silently at home where split-horizon matters.

OPEN: key expiry still enabled on demo-hp (2027-01-17) - needs an admin-console
toggle or an API token; a pre-auth key cannot do it.
2026-07-21 18:51:05 +02:00
admin 460cc73bad docs: R-59/R-60/R-61 from the HP install + second-hardware pairing proof
R-59 no-DHCP install must hard-abort (it baked 192.168.100.2 static and
completed - a box that can never call home). R-60 first-boot NIC sweep
self-heal. R-61 the baked root password must be knowable; a fixed well-known
password is explicitly rejected.

Positive evidence same-session: R-21 slice C PROVEN on a SECOND, virgin board
(HP t740) - and the shim loader booted with Secure Boot ENABLED, retiring the
assumption that Felhom installs need SB off. Fresh-box floor lift
0.153.0 -> 0.156.0 during day-0 cited on the publish-train row.

ISO README gains the t740 five-NIC trap: the 4-port igb card gets no lease,
the onboard r8169 port does.
2026-07-21 18:44:54 +02:00
admin 2143ae2d8d docs: R-58 assisted disk-picker install mode + HP mini ISO profile notes
R-58 records the operator ruling (2026-07-21) with the argument verbatim: the
installer should list available storage devices, excluding the install media,
and let one be selected. Third ISO mode alongside unattended-serial and
match-nothing-safety; unattended stays the appliance/factory mode. Slice 1 is
the abort-screen candidate table, same enumeration code, and it collapses the
two-boot dance on its own. Matters most for BYO/reinstall, where the serial is
unknown and a wrong guess is destructive.

ISO README gains the HP section: shim proven on this board by the safety boot,
the uncommitted-armed-profile pattern, verify-from-inside-the-ISO, and a
pointer to the prior-LVM abort that is the one likely failure on a
second-hand disk.
2026-07-21 18:00:21 +02:00
admin 3b41a982fe docs: R-55 PROVEN LIVE, new R-57 (app lifecycle), R-56 plumbing note, capability map
R-55's reboot leg ran operator-present on 9201: immich UI-stopped -> stayed
stopped across pct reboot, calibre-web recreated, zero alerts, ~15s.

R-57 records the lifecycle mechanism with the operator's abandoned-app
requirements verbatim and plant-it as the motivating case, including why the
retired/ directory move was wrong and the v0.158.1 pointer-receiver defect.
2026-07-21 16:35:28 +02:00
admin af8a7a58a4 docs+scripts: TASK-E parts 4-7 - operator one-pager, rulings, hub build.sh adoption
PROMPT-TEMPLATE: standard 'For the operator' plain-language section, mandatory
for M+ tasks and anything with a STOP.

ROADMAP rulings (operator, 2026-07-21): R-25b full-teardown cascade with three
acks + typed name (M-sized, spec to follow, no longer blocks R-3); R-11 channel
= direct Messenger, doc is the architect's; R-42 option (a), sidecars follow the
app; R-17 delete the archive - spike-lite found NO tooling verb targets it, so
it is an operator console action; R-4 complete (freemail.hu verified).
R-55 + R-41 slice 1 marked shipped; new R-56 (app difficulty classification -
the constructive half of the glance ruling).

scripts/build-hub.sh v1.23.0: the hub build script was outside any repo. Adopted
verbatim + versioned; the build-dir path is now a symlink to it.

felhom-testing skill: the ~1/5 recovery-code 'known flake' is retired - it was a
real defect the test was correctly detecting.
2026-07-21 15:32:56 +02:00
admin d4ac52d9e0 docs(roadmap): R-54 confirmed hub-side; record the in-memory damping limitation 2026-07-21 13:39:45 +02:00
admin befaa795e0 docs(roadmap): R-54 negative leg completed — 30 healthy cycles, zero heals 2026-07-21 13:17:36 +02:00
admin 3d28862db3 docs: STOP-1 + STOP-2 evidence — all three legs PROVEN LIVE; new R-55
Dead primary: degraded in 13 s, exactly one app_start_failed, banner rendered and
self-cleared. Boot orphan: recovered in one attempt with zero alerts. Dead dhclient:
detected in 57 s on process liveness while the lease was still live, healed 120 s after the
kill — the tunnel never dropped, so the outage was prevented rather than observed.

P1 answered as a by-product: bookstack StartedAt == the moment bootrecon StartStack
returned, so unless-stopped did NOT resurrect it. F5 hypothesis confirmed.

New R-55, surfaced by the leg designed to prove the opposite: the boot bind gate recreates
and STARTS every deployed drive-backed app unconditionally, so a customer Stop does not
survive a reboot for those apps. Predates R-52 and does not implicate it, but it narrows
R-52's practical scope and needs a ruling.
2026-07-21 12:56:39 +02:00
admin 907e5ce65c docs(TASK-D): R-51/R-52 SHIPPED + new R-54 row; capability-map row; seam-discipline rider
R-51's roadmap diagnosis is corrected at the source: aggregation returned StateRunning
("partial") for a running/stopped mix, so the stack read RUNNING and IsDownState was never
consulted about  at all — the constraint that row protects was never in tension
with the fix.

New R-54 row closes the INCIDENT-guest-dhclient-killed-2026-07-20 §5 OPEN RISK, and records
the design fact that makes it work: liveness of the DHCP client is itself a probe, because
the damage is timed and the address outlives its cause by 1-2 hours. The static-guest leg is
deliberately deferred to R-50.

New capability-map row is IMPLEMENTED, not PROVEN-LIVE: one leg is live (the watchdog's
healthy cycle on felhom-pve), the three that matter are destructive and operator-present and
have not run.

PROMPT-TEMPLATE §10 gains the seam-discipline row, including that a strings.Contains source
assertion is NOT sufficient — a commented-out call still contains the string.
2026-07-21 12:40:00 +02:00
admin f761f69d46 docs: R-39 CLOSED — STOP-2 proven live; DR-tier row -> PROVEN-LIVE
The operator pressed Re-issue PBS credentials and the chain closed in 13 seconds. The
identical click on 2026-07-18 did nothing at all.

  hub  08:39:31Z  fresh mint, generation 0 -> 1; descriptor gains secret_generation: 1
                  (token_id + fingerprint BYTE-IDENTICAL — the invisible re-key shape)
  agent 10:39:34  felhom-pbs-apply read felhom-pbs      (leg b: the impossible read)
  agent 10:39:38  ERROR REJECTED ... applied and DEAD, previous_state=applied
                                                        (leg c: the R-39 state, loud)
  hub  08:39:45Z  consumed_at stamped
  agent 10:39:45  one-time token secret consumed        (leg a: NO short-circuit)
  agent 10:39:45  reconcile (set-only, no --server)
  agent 10:39:47  pbsdr: converged state=applied

Corroboration: marker hash moved to afbb3b41… (it was byte-identical to the pre-reissue
marker in the failure); secret mtime 2026-07-18 -> 2026-07-21 10:39:45; new credential
probes 200; three consecutive reports trace applied -> auth_failed -> applied; ZERO
self-heal escalations, one mint, one consume, no consumed-failed.json — the box healed
through the descriptor path before the damper was ever needed.

Recorded for future runbooks: the operator first pressed the OFFSITE re-issue (two
distinct Re-issue actions exist). Harmless to PBS-DR, but it rotated the restic password
and correctly marked the escrow STALE, so the ceremony had to be re-run. Name the surface
explicitly next time.
2026-07-21 10:52:52 +02:00
admin 54a4644721 docs: R-39 fleet fix SHIPPED (hub 0.68.0 + agent 0.91.2); R-50b(a) SHIPPED; (b)/(c) open
R-39's three legs are closed and deployed: the hub stamps a monotonic secret_generation
so a re-key finally moves the descriptor hash; the wrapper gains a narrow read verb so
the non-root agent can read the credential it writes; and ProbeAuth turns a 401 into a
loud auth_failed the existing damper escalates to a fresh mint. Plus a consumed_at
honesty gauge for the applied-but-never-consumed disagreement.

Recorded in the R-39 row, because both are the kind of thing a future reader needs:

- A load-bearing fact the spec did not flag, checked rather than trusted: Apply bails out
  if the storage status probe ERRORS and adopt converges without consuming when the
  storage reads active, so the fix depended on PVE's 401 behaviour. PVE's storage_info
  wraps activation in eval{} and leaves active=0, so a 401 returns HTTP 200 with
  active:0 — never an API error. The chain is sound by proof, not inference.

- A defect I shipped and caught: v0.91.0 built the probe seam and main.go never wired it,
  so the leg was inert while every test passed. Same class as controller v0.154.0 the day
  before. Fixed in v0.91.1 (artifact superseded, not overwritten); v0.91.2 made a healthy
  probe observable so "no auth_failed" can never again be confused with "never probed".

The DR-tier capability row is deliberately NOT upgraded to PROVEN-LIVE: the decisive
evidence is STOP-2, the operator pressing Re-issue and the box converging where the
identical click did nothing on 2026-07-18.

R-50b(a) shipped — wrapper sha256 in the manifest + agent reporting + host drift surface,
with unknown-on-either-side reading as quiet rather than drift. (b)/(c) remain open: the
wrapper is still fetched unversioned from raw/branch/main.
2026-07-21 10:24:11 +02:00
admin 11ead4be0e docs: R-23(a) restart leg BANKED; "round-trips in seconds" PARTIAL -> PROVEN-LIVE
The operator moved the global floor to a version the box did NOT run (0.153.0 ->
v0.154.0) and the managed self-update fired exactly once:

  06:57:13Z  UpdateState pending, initiated_by=auto-floor
  06:57:17Z  agent: controller swap requested 0.153.0 -> 0.154.0
  06:57:21Z  container restarted
  06:57:29Z  agent: new controller healthy      (16 s save -> healthy)

Over a 39-minute window: swap requests 1, agent-driven bootstrap restarts 1,
rollbacks 0, container RestartCount 0. VerifyStartup confirmed on the next boot;
the following periodic check logged "Current version 0.154.0 is up to date" —
the at/above-floor branch correctly doing nothing.

The 2026-07-20 attempt proved nothing because it targeted an already-running
version; that was the whole reason this leg stayed open.

Disclosed in both rows: a hand-deploy of v0.155.0 at 07:17:10 falls inside the
observation window and is what StartedAt shows afterwards. It never goes through
SwapController, so the swap-count assertions hold across the full window — and it
incidentally re-confirmed the at/above-floor branch (0.155.0 running against a
0.154.0 floor -> updater did nothing).

R-23(b) (cosmetic Waiter "recovered" log timing) remains open.
2026-07-21 09:30:50 +02:00
admin c35da9de6a docs(roadmap): R-48 SHIPPED (ctrl v0.154.0); R-39 cheap half closed in-field; new R-50b
R-48 — the offsite restore controls collapse to one „Visszaállítás…" entry per app plus
a per-app wizard with three described intent cards. Shipped in controller v0.154.0
(3a9d744). Live click-through still pending the operator's floor save.

R-39 — the planned v0.90.1 artifact publish was CANCELLED as a false signal (operator
ruling 2026-07-21). 9596d5a changes zero non-test Go files; its own message says "the Go
binary is unchanged". The fix is the felhom-pbs-apply wrapper, which felhom-pve has
carried since 2026-07-18 and which every new install fetches from raw/branch/main
regardless of binary version. Publishing would have delivered no behaviour change and
advertised a versioned fix the artifact channel never carried.

R-50b (new) — that stop surfaced the real defect: a root-owned privileged host artifact
is delivered unversioned from main, absent from the Day-0 manifest, so the fleet has no
way to answer which wrapper a given host is running.
2026-07-21 08:54:03 +02:00
admin ce8c5393f1 docs(capability-map): offsite restore cell PARTIAL -> PROVEN-LIVE (prose/enum mismatch)
Row 61's status cell still read "PARTIAL — scope corrected 2026-07-19" while the note
body already ended with the destructive drill of 2026-07-20 and the words "the row now
earns PROVEN-LIVE": 40 files placed after the operator deleted the photos in immich's own
UI and emptied the trash, 1 DB dump replayed rc-0, 11 assets active, no schema drift,
timeline confirmed — taken through the customer-facing buttons, not endpoint shortcuts.

Cell-only correction; the full note history is retained verbatim.
2026-07-21 08:31:08 +02:00
admin fb0b8c1d42 docs: C6 destructive drill PASSED; R-23(a) mostly banked
Capability map — the destroy-then-recover drill ran through the customer UI:
photos deleted, TRASH EMPTIED, full files+database restore. 40 files placed
against 6 in the earlier non-destructive run, 1 DB dump replayed rc-0, 11
assets active, no drift, timeline confirmed. That is the proof the 6D
downgrade asked for, so the offsite-restore row earns PROVEN-LIVE. The
customer-restore row records the honest residual: an operator ran it, so the
row's literal 'a customer, not the operator' wording still owes one pass.

ROADMAP R-23(a) — the STOP-2 floor save released the held wait in the SAME
SECOND (hub 18:56:27 CEST = controller 16:56:27Z), out-of-cycle report 2s
later, generation advanced 0 -> 1. Still open: the self-restart single-fire
leg, since the floor was set to a version the box already ran.

Trap recorded: the wake is logx.Debugf, so it is invisible in docker logs at
INFO and lives only in the debug ring.
2026-07-20 19:23:11 +02:00
admin 0a399e3c02 docs(capability-map): offsite-restore stays PARTIAL — the 6D scope lesson applies
The operator confirmed the immich timeline renders correctly after the
2026-07-20 reconstitute, which was the last named residual. But that run
restored over an already-good state (11 assets before, 11 after), so it
proves the H4 window is closed and the operation is clean — it is NOT a
destroy-then-recover drill.

Flipping the row on it would repeat the 6D overclaim verbatim: 'immich
end-to-end from offsite alone' must not be claimed by a run in which
nothing was ever lost. Row stays PARTIAL pending a real destructive drill.
2026-07-20 19:07:20 +02:00
admin 9c8505649a docs: R-47 live-validated + golden 0.153.0 published
ROADMAP R-47 and the capability map's offsite-restore row now carry the live
evidence: a supervised reconstitute of immich from snapshot 49e7cb46 — the
same snapshot that aborted in round 2 — ran stop -> DB-service-only start ->
replay rc-0 -> full start, with no 'already exists', a SUCCESS outcome, and
immich's own 'No schema drift detected' twice where round 2 left it
reporting drift.

The offsite-restore row's residual for a PROVEN-LIVE flip is now only the
immich timeline screenshot (no browser on DooPlex).

Golden 0.153.0 baked and published the same day, sha256 15fdd191f3c660a6...,
first golden carrying all four infra images.
2026-07-20 18:04:02 +02:00
admin db3378bbf7 docs: R-47 shipped in controller v0.153.0 — H4 window closed on both restore paths
ROADMAP: R-47 collapsed to a shipped one-liner naming both paths (offsite
ReconstituteFromOffsite and local RestoreFromRecoveryUnit), the DB-only
window mechanism, the fail-closed rule, and the no-agent-coupling
declaration (MinAgent stays 0.90.0).

Capability map: the offsite-restore PARTIAL row and the customer-restore
MISSING row both note that H4 is closed in controller v0.153.0 and that they
now await only one clean acceptance run.

Also corrects the stale "closes in v0.149" wording in BOTH rows — v0.149.0
was the F3 dashboard BackupStatus fix; R-47 shipped in v0.153.0.
2026-07-20 17:26:17 +02:00
admin 1d1d60a737 docs: DIAG addendum 2 — S-3 fixed (samba 1.1.0 mDNS + v0.152.0 copy); Finder sidebar stays OPEN
Capture disproved addendum 1's guess: macOS DOES send a correct NBNS query and
IS answered correctly in 140us; it simply never acts on it. Records the proven
per-client matrix and keeps sidebar discovery explicitly unproven.
2026-07-20 13:40:55 +02:00
admin a7d9837976 docs: INCIDENT — guest 9201 dhclient killed as a host orphan; tunnel down ~1h15m
The v0.150.0 'orphaned dhclient' cleanup killed guest 9201's DHCP client, not a
host one: LXC guests share the host PID namespace, so the missing eth0, pidfile
and lease file were all true ON THE HOST and all irrelevant. Lease expired ~80
min later, guest lost its address, tunnel/hub/catalog/agent all went dark.

Restored; records the one-command ownership check (/proc/<pid>/cgroup) and
leaves the unsupervised-dhclient risk open for a ruling with R-50.
2026-07-20 12:09:01 +02:00
admin eaa217373f docs: DIAG-sharing addendum — S-1/S-2/S-4-core/S-5 shipped in controller v0.151.0; S-3 stays open
Also records against R-45 the lesson the loop taught: a phase a client answers
with a one-shot action must be an EDGE the registry serves once, never a level
synthesised into the phase channel.
2026-07-20 10:49:35 +02:00
admin 0c4886958e docs: DIAG — Megosztás reload loop root-caused (b5d78d1 phase coercion) + SMB endpoint is the guest, not .162
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nhk3eBHT8Mg5L8c2aj57aU
2026-07-20 10:11:58 +02:00
admin 1e1181939b docs(roadmap): R-53 shipped in controller v0.150.0; correct R-50's TLS-pin note (leaf-DER SHA-256, SAN never checked) 2026-07-20 09:42:34 +02:00
admin 30fd9d3ecd docs(audits): remote-ops remediation — F1 mitigated, F2 closed, F3 fixed (v0.149.0), Immich resolved; R-50..R-53 minted 2026-07-20 09:09:25 +02:00
admin 648ee67910 docs(audits): vacation remote-ops — access change + demo-node health diagnosis
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nhk3eBHT8Mg5L8c2aj57aU
2026-07-20 08:07:49 +02:00
admin 83c47eaf23 docs(ops): Tailscale makes the N100 (felhom-pve) location-independent
felhom-pve joins the tailnet as a host package (100.70.170.35, expiry
disabled); DooPlex already advertised 192.168.0.0/24 via its GitOps k3s
tailscale pod (100.107.87.53). ssh felhom-pve now targets the tailnet IP
(direct-over-LAN at home, tunnel when away); felhom-pve-lan = LAN fallback.
--accept-dns=false on the host. Host converted static->DHCP (reservation
keeps .162 at home). Measured: --accept-routes hairpins the local subnet
at home -> travel-only opt-in. PBS is offsite (own WireGuard tunnel), needs
none of this. Recorded not fixed: controller<->agent plane hard-pinned to
192.168.0.162 (agent listen_addr + guest bootstrap.json) -> not yet portable
off-LAN. No code changed. Full doc: documentation/operations/tailscale.md

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XMForrE4c1wZxd9LukxYVt
2026-07-19 13:31:09 +02:00
admin 97249099d6 docs: DIAG round 2 — H1 confirmed, H4 found; two standing rulings landed
VERDICT. H1 confirmed: the reconstitution never ran. /backup/offbox/reconstitute
was never hit (reconstituted/safety-dump/replay line counts all 0); the operator
clicked the old missing-only button, which merged 34 files and left the DB
untouched. H2 rejected (the full path never executed; staging used the correct
snapshot 49e7cb46). H3 rejected (no replay happened; DB was genuinely asset:0).

R-44 is exonerated and working: 49e7cb46 carries offsite_run_id 20260719T102851Z
and its dump probes to asset:11 / user:1 — a properly coherent pair, against
round 1's asset:0 / user:0.

H4 (NEW, the more important result). Phase-3 recovery ran the real sequence and
the v0.148.0 path executed correctly — safety dump, stop, start, replay — then
the replay ABORTED: relation "clip_index" already exists. Proven to the second:
  10:58:25 controller begins replay
  10:58:33 immich-server "Reindexing clip_index" -> "Reindexed clip_index"
  10:58:35 dump's CREATE INDEX clip_index fails, exit 3
The replay races the app's own schema repair, because reconstitution starts the
stack before replaying (ImportDump needs a live container). The photos survived
only because pg_dump emits COPY data before CREATE INDEX — luck, not design. The
same start-then-replay shape is on the LOCAL path, so it is a class defect.

Recovery: PARTIAL and STOPPED as instructed. All 11 assets are back and visible
by every DB criterion (status=active, deletedAt null, all files resolve), but the
operation reported failure and immich reports schema drift. Not clean acceptance
evidence; no second attempt made; safety dump read only.

Findings -> R-47 (the replay race), R-48 (restore controls separable only by
layout — the proven cause of this incident), R-49 (1.1 GB of a 1.2 GB immich
backup is ML model cache + a postgres data-dir tar duplicating the .sql dump +
immich's own nightly dump; recorded, not changed).

RULINGS LANDED.
  - 00-capability-map.md:61 -> PARTIAL, scope-corrected: 6D's destruction hit the
    FILE TREE only; the DB survived in its named volume, so "end-to-end from
    offsite alone" overclaimed scope.
  - The 704.6 MiB "discrepancy" -> CLOSED, not a defect: it was immich's own
    Tarhely widget, never a controller page. Same for round 2's 650MiB->1.4GiB.
  - customer-restore row: partial evidence recorded, NOT flipped.

Diagnosis only — no code, label or layout changes; no forget/prune.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
2026-07-19 13:04:58 +02:00
admin 29b2b8ad7c docs: DooPlex-local commands in PROMPT-TEMPLATE and the manual-build runbook 2026-07-19 12:27:44 +02:00
admin 238472b8ed docs: R-43 + R-44 shipped in controller v0.148.0 (live acceptance still pending)
ROADMAP R-43/R-44 → SHIPPED with what actually landed:
  - R-43: ReconstituteFromOffsite — safety dump → stop → files overwritten to
    the snapshot version → start → the snapshot's own dump replayed from the
    SCRATCH unit (the live unit is still never overwritten). Invariants:
    nothing is ever deleted, and the undo is verified on disk before the act.
  - R-44: every run, manual AND nightly, refreshes dumps + units BEFORE the
    capture, so each snapshot is a coherent {DB@T, files@T} pair stamped with
    offsite_run_id + dumps_at. Nightly ordering is now structural rather than a
    coincidence of two scheduler entries at 02:30 and 04:15.

DIAG doc gains a "fixed in v0.148.0" header pointing at both.

Capability map deliberately NOT flipped: the DB half now exists but is PARTIAL,
not PROVEN-LIVE — shipped and deployed with no live acceptance behind it. The
customer-restore row stays MISSING, now blocked only on the evidence run rather
than on missing capability, and R-3 stays DRAFT.

Still open: the 00-capability-map.md:61 ruling — did CAMPAIGN-6D's "immich
end-to-end from offsite alone" exercise the DB half, or only the file half?

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
2026-07-19 12:27:11 +02:00
admin fa8c0d5972 docs(workspace): record the migrated project-memory location 2026-07-19 12:22:07 +02:00
admin ee48a48288 docs: migrate workflow to DooPlex-local execution 2026-07-19 12:16:52 +02:00
admin 13af252907 docs: scripts v1.22.0 + website grid CHANGELOGs, ROADMAP R-38 shipped, R-45/R-46, REPORT
R-38 flips to SHIPPED with the live-canary evidence. Two new items registered
from tonight's work: R-45 (unified async-job feedback — five one-off progress
surfaces now exist and agree on nothing; the two lessons any framework must
encode are "probe the terminal state, never infer it" and "a source reporting
nothing is normal, not broken") and R-46 (verification copies are now visible
but still cannot be browsed and never expire — the invisible-result gap and the
disk-lifecycle gap, designed together because they are the same object).

Pre-invite checklist gains the golden note: build-golden v2.1.0 makes goldens
>= 0.147.x carry all four infra images; no rebuild for this alone.

REPORT-<topic>.md per the parallel-session rule — another session is writing in
this clone tonight, so the shared REPORT.md is left alone.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
2026-07-19 10:00:10 +02:00
admin f3d1ba3062 docs: DIAGNOSE immich offsite restore — findings + ROADMAP R-43/R-44
Viktor deleted 11 immich photos to test offsite restore; both place runs
merged 0 files and flashed success. Two-layered root cause:

- A UI delete means TRASH, not deletion — no file left the disk, so there
  was genuinely nothing missing to restore. The test proved nothing.
- A valid test would also have failed: no offsite path loads a database.
  All three buttons are file-only (offbox_restore.go:171-232, :346-421).
  Files return; the library stays empty. The only DB replay is
  RestoreFromRecoveryUnit, reachable solely from the LOCAL restore panel.

Compounding: the manual push never refreshes dumps (offbox_handlers.go:
203-227) — today's unit dump is from 02:30 and probes to asset:0 / user:0 /
album:0, predating both the admin user and the photos. Had a full restore
loaded it, it would have written an empty DB over the live one, destroying
the trashed rows that were the only surviving recovery path.

Answer to "can a customer trust same-day offsite?" for a DB-indexed app:
no — files come back, content does not.

No product code changed. Photos left in trash (recovery not wanted).
Capability map: customer-restore row stays MISSING by finding, not by
absence of attempt; offsite-restore row flagged scope-contested pending
a ruling on whether CAMPAIGN-6D exercised the DB half.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
2026-07-19 08:47:34 +02:00
admin 3e2078be8a CAMPAIGN 7 doc: MAJOR breaking-note table (§6.3)
Fills the gap flagged in §8.1. Upstream notes for bookstack, immich, nextcloud,
n8n, grafana, vikunja, privatebin, radarr, romm; plus the ones this sweep found
EMPIRICALLY on a real deploy (outline PGSSLMODE, tandoor port+ALLOWED_HOSTS,
zipline DATABASE_URL/healthcheck rename, homebox tag prefix + pepper, wger
DJANGO_DB_* + :8000) -- which are the more useful half, being what actually
happens rather than what a changelog remembered to mention.

Four majors (calcom, claper, gramps-web, onlyoffice) deployed clean but have no
retrieved upstream one-liner; recorded as not-retrieved, not fabricated.

Campaign 7 catalog sweep.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
2026-07-19 03:30:50 +02:00