admin 36ae3b3191 CHAOS NIGHT round 6: I stopped a LEG, and the box carried on by itself ...

Correction to my own account, recorded where the wrong version stood. I wrote
"I stopped the backup". What I stopped was its LOCAL leg: the task list shows
that job ending 23:59:45 with status "job errors" - my pkill - while a second
vzdump was already running, streaming encrypted to ep0
(--repository felhom@pbs!tester-1@...:felhom-offsite --ns tester-1).

The arithmetic that forced the intervention still stands: the local leg was
writing a ~29GB source into a filesystem with 3.6GB free, falling at ~16MB/s,
which gave under four minutes before / filled and the nested PVE wedged - the
Phase 0 failure one level up. But the off-site leg needs no local space at all,
so the box's design copes with exactly the problem I thought I was rescuing it
from. It is still counted as an intervention: I reached in and killed a job.

The box then recovered unaided: 22 containers at 22:00:52Z, 26 at 22:01:13Z,
and / went from 2539MB free back to 7573MB once the partial archive was removed.
The older completed backup was left untouched.

Round 6's valuable half stands: during the backup 4 of 26 containers were up,
every app returned 404 through the public route, and NO alarm fired - the
suppression of the backup's own stack stops held.

And the waiting rule for the off-site leg is fixed in advance: round 7 does not
start while it runs, unless it is still running at 00:45, in which case round 7
proceeds and records that it cut an in-flight backup - labelled as that, not as
a clean internet-cut round.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

2026-09-17 00:02:46 +02:00

..

DRILL-backup-truth-2026-08-21/evidence

DRILL closeout: the scheduled cycle agrees, the abandonment sweep watched firing, R-366

2026-08-22 05:25:30 +02:00

DRILL-cooldown-grain-2026-08-23

docs(hub v0.108.0): the delivery grain, the cooldown ruling, and gate 11's first subject

2026-08-23 14:12:23 +02:00

DRILL-r102-tier2-unit-2026-08-31

R-102 + R-103 CLOSED (controller v0.229.0) — architecture, register, STATUS, drill evidence

2026-08-31 12:21:52 +02:00

DRILL-r329-r386-2026-08-23

docs(R-329/R-386/R-387): the severity contract, the intent ruling, and Part 5 recorded

2026-08-23 12:03:49 +02:00

DRILL-r356-hot-only-restore-2026-08-22

R-356 docs: correct R-107 in the architecture, record the design, refresh STATUS, compress the register

2026-08-22 13:26:10 +02:00

DRILL-r356b-driveless-db-restore-2026-08-22

DRILL R-356b: the off-site restore for a driveless app that HAS a database

2026-08-22 16:33:45 +02:00

DRILL-r361-2026-08-22

R-361 docs: the [FACT], the negative that cancelled Part 2, R-383/R-384, golden 0.221.1

2026-08-23 00:33:12 +02:00

DRILL-r379-rollback-2026-08-22

R-379/R-380 docs: the failure ladder, the drill record, register housekeeping

2026-08-22 18:40:22 +02:00

DRILL-r384-dead-db-alarm-2026-08-23

R-385: make an UNRECORDED golden fail the currency gate; file R-386; own the alarm ladder

2026-08-23 07:59:52 +02:00

DRILL-r403-tier2-delete-2026-08-31

R-403 CLOSED (controller v0.230.0), R-404 filed as a decision for Viktor

2026-08-31 14:39:29 +02:00

DRILL-soak-2026-08-31

soak drill COMPLETE: 7 phases, 4 findings, the observer found the worst one (R-414)

2026-09-01 06:07:13 +02:00

evidence-agent-transport-leak-2026-08-20

agent 0.130.0 published and vouched; R-347 closed, R-349 + R-350 filed

2026-08-20 12:54:54 +02:00

evidence-backup-promise-2026-09-16

the volunteer's own path verified: felhom.eu/letoltes serves 1.28.0

2026-09-16 20:27:25 +02:00

evidence-bignight-2026-09-14

BIGNIGHT teardown complete: VM destroyed, host deleted, ep0 peer gone, customer and its ep0 data kept

2026-09-15 01:05:05 +02:00

evidence-chaos-night-2026-09-17

CHAOS NIGHT round 6: I stopped a LEG, and the box carried on by itself

2026-09-17 00:02:46 +02:00

evidence-doorstep-walk-1270-2026-09-14

doorstep teardown complete: host tester-1-8603a2 deleted, ep0 peer gone, customer kept; ep0 DR data retained for ruling

2026-09-14 18:52:09 +02:00

evidence-drill-0243-2026-09-16

drill 0.243.0 complete: 0 interventions, the connect e-mail proven, NOT ready for a volunteer

2026-09-16 14:27:04 +02:00

evidence-drill-fresh-install-0242-2026-09-14

drill 0242: teardown complete in three layers; R-501 filed

2026-09-14 16:40:59 +02:00

evidence-drill-r95-recovery-2026-09-01

DRILL R-95: the recovery route does not exist — stopped before the destructive phase

2026-09-01 16:55:50 +02:00

evidence-drill-retained-key-2026-08-12

DRILL: the retained key works, and the customer cannot reach it

2026-08-12 17:41:56 +02:00

evidence-ep0-established-connections-2026-08-20

SPIKE ep0 connections: CI green by run id (360/237, 19672e685)

2026-08-20 10:42:50 +02:00

evidence-ep0-fd-2026-08-18

INCIDENT + registers: ep0's PBS proxy served nobody for 9.5h (R-336..R-338)

2026-08-18 06:09:38 +02:00

evidence-ep0-pbs-upgrade-2026-08-18

RUNBOOK ep0: PBS 4.2.2-1 -> 4.2.5-1, slope unchanged as predicted (R-341)

2026-08-18 12:26:53 +02:00

evidence-p1fixes-2026-09-15

REPORT + STATUS: P1 fixes — supervisor proven incl. crash-loop resume and hub events; publish; rows

2026-09-15 11:45:21 +02:00

evidence-r316-2026-08-13

REPORT: installer v1.28.0 published and verified live; evidence, and the Part 1 logs I lost again

2026-08-13 08:19:41 +02:00

evidence-r341-plus7d-2026-08-30

R-341 check taken, and the golden-currency bypass declared

2026-08-30 18:49:53 +02:00

evidence-r353-r360-live-2026-08-30

Register + architecture for controller v0.226.0 (R-353/357/358/360/396), and R-395 fixed

2026-08-30 19:39:41 +02:00

evidence-recovery-code-2026-09-16

R-543 closed: the household is asked for the recovery code (controller v0.245.0)

2026-09-16 21:20:51 +02:00

evidence-rehearsal-2026-08-09

Rehearsal 2026-08-09 COMPLETE: data BYTE-IDENTICAL, journey needs a shell twice

2026-08-09 12:35:26 +02:00

evidence-spike-app-update-2026-09-01

SPIKE: what an app update actually does, and which other paths do it too

2026-09-01 21:35:32 +02:00

evidence-spike-restic-restore-2026-08-31

R-87 SPIKE: measured, do not build it as written (R-407..R-409 filed)

2026-08-31 15:59:09 +02:00

fixtures

docs(audits): first genuinely failing disk — the SMART PASSED trap

2026-08-14 07:57:07 +02:00

nightly-2026-09-13-adventurelog

evening: rules file in place; R-483 CLOSED (catalog, operator-confirmed); R-479 CLOSED (controller v0.241.0, proven live); R-481 blocked on the one-host-one-customer model; R-491 filed

2026-09-13 21:57:47 +02:00

nightly-2026-09-13b-bentopdf

second night: scratch guest 9202 built (R-481 CLOSED, persists); controller v0.242.0 delivered (R-487 R-491 R-490 R-476 R-456 CLOSED, R-489 re-scoped); R-492 filed; rotation restarted from bentopdf; morning note

2026-09-13 23:06:21 +02:00

r459-close-2026-09-13

R-459 CLOSED (MariaDB converts itself, proven by harness + live), golden 0.236.0 (R-467), the golden waiver (R-468)

2026-09-13 10:14:37 +02:00

r459-spike-2026-09-06

SPIKE R-459: the skipped MariaDB conversion is stable, and the trade it implied does not exist

2026-09-06 17:42:19 +02:00

R411-R414-2026-09-01

R-411/408/407, R-414, R-412a leg 1, R-410, R-406 CLOSED; determination + live evidence

2026-09-01 10:36:54 +02:00

R442-2026-09-13

R-442 CLOSED (controller v0.236.0): "delete my data too" deletes it or refuses; R-465..467 opened

2026-09-13 09:07:08 +02:00

rulings-r472-r475-2026-09-13

docs: rulings 7 and 8 shipped and proven live (R-470/R-472/R-475 CLOSED); R-477..R-480 opened

2026-09-13 17:54:23 +02:00

slice4-2026-09-13

Slice 4 shipped (R-448/R-443/R-439 CLOSED, proven live); R-472..R-476; the floor-between-bakes claim corrected

2026-09-13 12:30:17 +02:00

upgrade-spike-2026-09-06/evidence

SPIKE: an upgrade test that runs again — and a real defect in our own bookstack template

2026-09-06 11:48:57 +02:00

v0240-2026-09-13

R-452 CLOSED: the catalog-since gate (hook-enforced); 09 §8.2 limitation lifted; STATUS note updated

2026-09-13 19:42:42 +02:00

v0241-2026-09-13

evening: rules file in place; R-483 CLOSED (catalog, operator-confirmed); R-479 CLOSED (controller v0.241.0, proven live); R-481 blocked on the one-host-one-customer model; R-491 filed

2026-09-13 21:57:47 +02:00

v0242-2026-09-14

second night: scratch guest 9202 built (R-481 CLOSED, persists); controller v0.242.0 delivered (R-487 R-491 R-490 R-476 R-456 CLOSED, R-489 re-scoped); R-492 filed; rotation restarted from bentopdf; morning note

2026-09-13 23:06:21 +02:00

AUDIT-blast-radius-hostroot-localapi-2026-07-02.md

docs: AUDIT — host-root + local-API blast radius (non-Proxmox-token axes)

2026-07-02 10:43:19 +02:00

AUDIT-gate-decoys-2026-09-01.md

the decoy sweep: 29 gates read, 16 fooled, 10 fixed - and a gate that refuses the next one (R-421)

2026-09-01 12:39:45 +02:00

AUDIT-hub-gui-2026-06-30.md

docs(audit): hub GUI vs current architecture findings (2026-06-30)

2026-06-30 20:13:48 +02:00

AUDIT-nas-cwa-rca-2026-07-11.md

docs: Q1c GREEN — reboot survival automatic since agent 0.84.0 (feature doc + audit §7 + CONTEXT + REPORT)

2026-07-11 21:25:14 +02:00

AUDIT-power-outage-recovery-2026-07-22.md

docs: ISO train v1.25.0 — REPORT (belt/apt/R-63/gate/vault live), F9 resolved, R-63 shipped, nodes belt+vault, F8 checklist; critical golden<floor finding

2026-07-23 15:48:01 +02:00

AUDIT-vacation-remote-ops-2026-07-20.md

docs(audits): remote-ops remediation — F1 mitigated, F2 closed, F3 fixed (v0.149.0), Immich resolved; R-50..R-53 minted

2026-07-20 09:09:25 +02:00

B3-storage-lifecycle-2026-06-15.md

…

BIGNIGHT-household-month-2026-09-14.md

BIGNIGHT teardown complete: VM destroyed, host deleted, ep0 peer gone, customer and its ep0 data kept

2026-09-15 01:05:05 +02:00

bughunt-reconcile-2026-06-13.md

…

CAMPAIGN-3-2026-07-11.md

CAMPAIGN-3: morning RCA — host loss = F12 CRITICAL automount ordering cycle (hardware exonerated); box recovered to defined state

2026-07-12 06:55:40 +02:00

CAMPAIGN-4-2026-07-13.md

@

2026-07-14 00:45:20 +02:00

CAMPAIGN-5-2026-07-14.md

@

2026-07-14 10:12:21 +02:00

CAMPAIGN-6A-2026-07-14.md

@

2026-07-14 12:17:26 +02:00

CAMPAIGN-6B-2026-07-14-PROMPT.md

@

2026-07-14 12:17:26 +02:00

CAMPAIGN-6B-2026-07-14.md

CAMPAIGN-6B: note credential fully reverted at cleanup

2026-07-14 13:32:14 +02:00

CAMPAIGN-6C-2026-07-14-PROMPT.md

CAMPAIGN-6B: note credential fully reverted at cleanup

2026-07-14 13:32:14 +02:00

CAMPAIGN-6C-2026-07-14.md

CAMPAIGN-6C: v0.131.0 F-S2/F-S3 live acceptance (4P-1..4P-5 all PASS) + F-6C-1

2026-07-14 20:01:13 +02:00

CAMPAIGN-6D-2026-07-15.md

docs: 6D audit — tag campaign6 autofs finding as track-only (operator reboot)

2026-07-15 19:48:43 +02:00

CAMPAIGN-6E-2026-07-15.md

docs(audit): CAMPAIGN-6E — P-TIER2 deep-4 + P4-DEEP timing (all 8 legs PASS)

2026-07-15 21:11:39 +02:00

CAMPAIGN-7-catalog-sweep-2026-07-19.md

CAMPAIGN 7 doc: MAJOR breaking-note table (§6.3)

2026-07-19 03:30:50 +02:00

CAMPAIGN-8-backup-restore-2026-07-27.md

F-LEAK: record all three attempts; the ACL fix is consumed by use, the fenced destroy is the fix

2026-07-28 11:32:45 +02:00

CAMPAIGN-9-restore-proof-2026-07-28.md

C9-F1 (honesty half) + C9-F2 shipped in controller v0.183.0; C9-F1b and C9-F4 filed

2026-07-28 19:08:25 +02:00

CAMPAIGN-10-closeout-2026-08-02.md

Campaign 10 closeout Part 2: teardown complete — five layers, each verified gone

2026-08-02 10:11:21 +02:00

CAMPAIGN-10-two-storage-soak-2026-07-31.md

Campaign 10: M-band RTO measured — RTO ~= 40s + 26.9s/GB, and a capacity ceiling that matters more

2026-08-02 08:31:51 +02:00

CAMPAIGN-11-recovery-journey-2026-08-05.md

R-201 re-walk: the data PASSES again, the journey still FAILS — two dead ends, down from four

2026-08-06 12:18:29 +02:00

CAMPAIGN-12-class-sweep-2026-08-08.md

CAMPAIGN 12 — the class sweep: golden 0.208.0 baked (awaiting vouch), R-256..R-263 filed, gating ranked

2026-08-07 23:07:42 +02:00

CAMPAIGN-nomercy-2026-07-06.md

docs(audit): closure appendix — F2 closed, F4-UX fixed, healthcheck class fixed

2026-07-06 20:43:52 +02:00

CAMPAIGN-offsite-overnight-2026-07-10.md

campaign: offsite overnight soak 2026-07-10 — PASS + 1 HIGH finding (crash-mid-prune stale lock)

2026-07-10 04:32:32 +02:00

CAMPAIGN-pool-effects-2026-07-01.md

docs: F3 provisioning runbook (vmid from full host list) + campaign F1/F2 resolution banner

2026-07-02 10:27:17 +02:00

D5-drive-alone-restore-2026-07-30.md

D5 SHIPPED: Tier-1/2 restore no longer depends on the whole-guest tier

2026-07-30 16:58:06 +02:00

deep-sweep-2026-06-13.md

…

DIAG-agent-channel-2026-07-26.md

DIAG: agent-channel banner is a stale controller.yaml endpoint, not island fallout

2026-07-26 08:51:45 +02:00

DIAG-backup-missed-2026-07-26.md

docs(audits): R-80 DIAG — expected_backup_missed root-caused to in-memory agent backup store

2026-07-26 09:59:35 +02:00

DIAG-bootrecon-drive-absent-2026-08-02.md

docs: R-157 A / R-170 / R-171 closed — boot recovery finished

2026-08-02 20:38:21 +02:00

DIAG-f10-demo-hp-offsite-2026-07-23.md

docs(R-71): SHIPPED (a)+(c) — the apply-bridge settle-gate (controller v0.162.0)

2026-07-24 07:49:11 +02:00

DIAG-immich-restore-2026-07-19.md

docs: DIAG round 2 — H1 confirmed, H4 found; two standing rulings landed

2026-07-19 13:04:58 +02:00

DIAG-immich-restore-round2-2026-07-19.md

docs: DIAG round 2 — H1 confirmed, H4 found; two standing rulings landed

2026-07-19 13:04:58 +02:00

DIAG-r116-disks-payload-2026-07-30.md

docs: write down which boxes are disposable (target selection by blast radius)

2026-07-30 08:13:58 +02:00

DIAG-sharing-2026-07-20.md

docs: DIAG addendum 2 — S-3 fixed (samba 1.1.0 mDNS + v0.152.0 copy); Finder sidebar stays OPEN

2026-07-20 13:40:55 +02:00

DIAG-smart-passed-trap-2026-08-14.md

docs(audits): first genuinely failing disk — the SMART PASSED trap

2026-08-14 07:57:07 +02:00

DIAGNOSE-drive-bind-after-reprovision-2026-06-23.md

docs(audit): DIAGNOSE re-enrolled drives not binding after 9201 re-provision

2026-06-23 17:15:30 +02:00

DIAGNOSE-offbox-repo-orphaned-2026-07-17.md

docs: offsite continuity Part B REPORT/CONTEXT; DIAGNOSE incident→fix pointer; capability map (offsite reinstall-continuity + escrow retention); ROADMAP R-25 (TOCTOU hardening) + R-26 (guided old-history recovery)

2026-07-17 10:55:17 +02:00

DISPOSITION-ia-finding2-systemdisks-2026-07-13.md

audits: disposition — backups-IA finding 2 (legacy-boot SystemDisks dead wizard) RESOLVED by felhom-agent v0.87.0 (slaves-walk, operator ruling 2026-07-13); live evidence drill replay + felhom-pve byte-identical regression; remaining: supervised wizard leg + Day-0 manifest vouch

2026-07-13 13:21:14 +02:00

DOORSTEP-walk-1270-2026-09-14.md

doorstep teardown complete: host tester-1-8603a2 deleted, ep0 peer gone, customer kept; ep0 DR data retained for ruling

2026-09-14 18:52:09 +02:00

DRILL-appdata-restore-2026-07-04.md

docs(audit): app-data restore drill 2026-07-04 (guest 9201)

2026-07-04 22:53:07 +02:00

DRILL-chaos-night-2026-09-17.md

CHAOS NIGHT rounds 4-5: the box repairs its own tunnel, and survives docker dying

2026-09-16 23:56:26 +02:00

DRILL-day0-cleanroom-2026-07-03.md

docs: golden 0.98.3 live — D.1b retired, drill B1/B5 fixed, backlog note resolved + evidence doc

2026-07-03 18:38:28 +02:00

DRILL-day0-take2-2026-07-12.md

docs: hub v0.53.0 closing bundle - CHANGELOG + CONTEXT (both operator rulings verbatim) + take-two F-14 disposition (SHIPPED, gated on escrow-acked deletion provenance)

2026-07-13 15:02:44 +02:00

DRILL-day0-vm-2026-07-12.md

docs: DR-tier-by-default batch — decisions 1-5 in CONTEXT, day0 ACL advice reversed, ceremony decision-4 policy block, WG disclosure (tester agreement + GO-LIVE D3), drill F-disposition table, REPORT

2026-07-12 20:43:25 +02:00

DRILL-fresh-install-0242-2026-09-14.md

drill: a stranger's first hour on 0.242.0 — 1 intervention, not ready for a volunteer

2026-09-14 16:17:47 +02:00

DRILL-golden-098-2026-07-03.md

docs: golden 0.98.3 live — D.1b retired, drill B1/B5 fixed, backlog note resolved + evidence doc

2026-07-03 18:38:28 +02:00

DRILL-prove-fixes-0243-2026-09-16.md

drill 0.243.0 complete: 0 interventions, the connect e-mail proven, NOT ready for a volunteer

2026-09-16 14:27:04 +02:00

DRILL-r201-night-run-2026-08-04.md

R-201 PASSED: a customer's file survived a machine rebuild and came back byte-identical

2026-08-04 23:18:47 +02:00

DRILL-r201-offsite-recovery-2026-08-04.md

R-203 shipped: the app and its backup agree, and 'ok' means it — R-201 unblocked

2026-08-04 18:52:14 +02:00

DRILL-retained-key-2026-08-12.md

Correct the drill's wall clock to the measured end time (17:45, not the estimated 17:55)

2026-08-12 17:45:23 +02:00

E2D-fresh-vm-2026-07-29.md

Record work, banked first: shrink the E-2d row, create the missing capability-map rows

2026-07-29 23:34:06 +02:00

FINDING-app-email-rollout-2026-06-29.md

docs: FINDING — app-email rollout mechanism gaps (calcom/nextcloud/immich)

2026-06-29 10:32:56 +02:00

host-addresses-visible-2026-07-31.md

docs: host-addresses audit + capability-map row + REPORT (agent 0.119.0 / hub 0.85.0)

2026-07-31 08:55:53 +02:00

INCIDENT-ep0-pbs-fd-exhaustion-2026-08-18.md

RUNBOOK ep0: PBS 4.2.2-1 -> 4.2.5-1, slope unchanged as predicted (R-341)

2026-08-18 12:26:53 +02:00

INCIDENT-guest-dhclient-killed-2026-07-20.md

docs: INCIDENT — guest 9201 dhclient killed as a host orphan; tunnel down ~1h15m

2026-07-20 12:09:01 +02:00

LEDGER-instruction-closeout-2026-08-06.md

docs: close out the instruction arc — t740 corrected on evidence, registers, ledger, S-37

2026-08-06 11:49:22 +02:00

LEDGER-instruction-trim-2026-08-06.md

docs+gate: instruction files cannot silently regrow (R-229)

2026-08-06 09:38:52 +02:00

LEDGER-instruction-trim-part2-2026-08-06.md

docs: backup verified IN the repository — snapshot b587f775, 118/118 + 40/40 (R-229)

2026-08-06 11:22:34 +02:00

live-drive-findings-2026-06-14.md

…

live-drive-fixspec-2026-06-14.md

…

R106-R109-recipe-completeness-2026-07-30.md

R-106/R-109 audit + registers: shipped at agent 0.118.1, plus R-125

2026-07-30 13:28:10 +02:00

R108-network-app-namespace-2026-07-30.md

R-108 CLOSED — D5's precondition is met (controller v0.187.0)

2026-07-30 14:21:44 +02:00

R116-PART5-2026-07-30.md

R-116 Part 5 FAILED: the fix shipped, C5 still fails, mechanism NOT isolated

2026-07-30 07:15:31 +02:00

R116-v0116-2026-07-30.md

docs(R-116): CLOSED — proven live; capability row F to PROVEN-LIVE; R-120 filed

2026-07-30 09:34:08 +02:00

R117-v0117-2026-07-30.md

docs(R-117): CLOSED — proven live on demo-hp; R-121 filed for agent-on-box drift

2026-07-30 12:42:26 +02:00

R120-golden-rebake-2026-07-30.md

docs(R-120): CLOSED on both halves — golden current, and the class has a gate that refuses

2026-07-30 10:48:16 +02:00

R-63-console-font-drill-2026-07-23.png

drill: R-63 ő/ű captured live on the KVM HP (+ pairing + rootpw); HP SVM-enable fix; evidence PNG

2026-07-23 20:46:27 +02:00

RECON-dooplex-backup-2026-08-06.md

RECON: what DooPlex's own backup actually protects (R-232) — read-only, nothing changed

2026-08-06 12:16:38 +02:00

RECON-offsite-dr-chain-2026-08-04.md

RECON: trace the offsite DR chain link by link — it does not join up (R-198..R-201)

2026-08-04 12:16:04 +02:00

RECON-subdomain-onboarding-2026-07-31.md

the golden gate reads a fact not a name (R-410); R-133's collision resolved (R-406, R-416)

2026-09-01 10:19:55 +02:00

REHEARSAL-byo-reinstall-2026-08-09.md

Rehearsal 2026-08-09 COMPLETE: data BYTE-IDENTICAL, journey needs a shell twice

2026-08-09 12:35:26 +02:00

REPORT-DRILL-backup-truth-2026-08-21.md

R-354 + R-355 CLOSED, proven live; golden 0.218.0 baked; R-367 filed

2026-08-22 10:11:46 +02:00

REPORT-ep0-pbs-upgrade-2026-08-18.md

DRILL 2026-08-21: the off-site restore never replays named volumes (R-354..R-365)

2026-08-21 23:30:27 +02:00

REPORT-mis-called-defect-and-sweep-2026-08-22.md

One register, enforced by a gate; closed work compressed into siblings (R-376..R-378)

2026-08-22 12:13:54 +02:00

REPORT-r316-installer-v1.28.0-2026-08-13.md

hub v0.104.0: the guest network gets a reader (R-319), and the hub half of the naming (R-295)

2026-08-13 10:50:12 +02:00

REPORT-small-debts-and-first-reader-2026-08-13.md

REPORT + CONTEXT: the third name, the second door, and a number that answered a different question

2026-08-13 16:01:45 +02:00

REPORT-v0.218.0-r354-r355-2026-08-22.md

Correct the placement mis-framing, and file what we wrote down and never filed (R-368..R-375)

2026-08-22 11:18:08 +02:00

RERUN-p1p3-2026-07-06.md

docs(audit): closure appendix — F2 closed, F4-UX fixed, healthcheck class fixed

2026-07-06 20:43:52 +02:00

RUNBOOK-usb-enrollment-3bfix-2026-07-01.md

docs: RUNBOOK result — USB enrollment on pool-scoped ACL (residual CLOSED)

2026-07-01 13:52:44 +02:00

SESSION-C-2026-07-29.md

Correct the Session C audit: the union row is DEDUPED AWAY, not written false

2026-07-29 23:51:17 +02:00

SPIKE-a1-pool-membership-read-2026-07-03.md

docs: SPIKE — A1 pool-membership read for the stale-lock reaper

2026-07-03 13:00:00 +02:00

SPIKE-agent-pin-mismatch-2026-06-29.md

spike(pin-mismatch): §9 Outcome — R1 executed, controller↔agent service restored

2026-06-29 15:08:58 +02:00

SPIKE-agent-selfupdate-2026-07-05.md

docs(audit): SPIKE — agent self-update rollback mechanics (systemd 257 semantics on felhom-pve)

2026-07-05 14:45:58 +02:00

SPIKE-app-update-2026-09-01.md

SPIKE: what an app update actually does, and which other paths do it too

2026-09-01 21:35:32 +02:00

SPIKE-backup-classification-2026-07-14.md

docs(audit): backup-classification spike — coupling model VALIDATED (Immich DB-only restore broken-not-empty); offsite=unit-only live-proven; NEW F-S1 (tier-2 copies no userdata) + F-S2 (paperless-ngx appdata skipped); round-trip verdict: exclusion scoping

2026-07-14 16:39:55 +02:00

SPIKE-baremetal-iso-2026-07-16.md

docs(spike): bare-metal Felhom ISO — R-21 spiked (all mechanisms GREEN on nested virt)

2026-07-16 11:29:03 +02:00

SPIKE-catalog-data-paths-2026-07-26.md

SPIKE: catalog-derived userdata skeleton + import surfaces (2026-07-26)

2026-07-26 06:59:46 +02:00

SPIKE-ci-runner-2026-08-02.md

docs: R-168 SHIPPED, R-29 CLOSED on the demonstrated alarm, R-169 minted

2026-08-02 16:35:34 +02:00

SPIKE-connectivity-wireguard-2026-07-03.md

docs: R-182 closed, R-90 closed on measurement, R-86 unblocked, ep0 record corrected

2026-08-03 14:00:54 +02:00

SPIKE-controller-agent-channel-health-2026-06-29.md

docs: felhom.eu REPORT (HostLeafChecker) + hub README + self-health arc complete (spike doc)

2026-06-29 23:32:12 +02:00

SPIKE-controller-escrow-2026-07-13.md

audits: SPIKE controller-driven escrow ceremony — daemon-context mechanics ALL GO (drill VM, agent 0.87.0): PTY re-key no-TTY 3/3, sudoers refusal matrix 5/5, R pipe round-trip proven, env_reset clean, --upload hub-verified ~2.4s; premise correction: capability probes are list-mode

2026-07-13 17:15:52 +02:00

SPIKE-controllerswap-narrow-grants-2026-06-29.md

docs: controller-swap narrow-grant Option A IMPLEMENTED (agent v0.45.0) — spike + multidrive pointers resolved

2026-06-29 19:46:17 +02:00

SPIKE-day0-firstboot-handshake-2026-06-26.md

REPORT + Day-0 doc: BUNDLE slice (hub v0.16.0 artifact manifest + host-install v1.1.0 self-install)

2026-06-28 09:03:34 +02:00

SPIKE-decommission-migration-2026-06-14.md

…

SPIKE-dooplex-buildcache-2026-08-05.md

docs: P6 EXECUTED — containerd store moved to SSD2; reboot deferred (R-209, R-209a)

2026-08-05 10:27:40 +02:00

SPIKE-dr-bindmount-source-2026-07-07.md

docs: campaign-2 R1/R2 root-cause correction + DR bind-mount source spike

2026-07-07 19:08:03 +02:00

SPIKE-dr-recipe-2026-06-16.md

docs(audit): SPIKE — Phase 2 DR recipe + storage diagnosis (report-only)

2026-06-16 14:18:39 +02:00

SPIKE-drive-enrollment-2026-07-01.md

docs: spike status — Impl-2b DONE (raw enrollment live: ctrl v0.95.0 + agent v0.58.0)

2026-07-01 21:27:48 +02:00

SPIKE-ep0-established-connections-2026-08-20.md

agent 0.130.0 published and vouched; R-347 closed, R-349 + R-350 filed

2026-08-20 12:54:54 +02:00

SPIKE-ep0-storagebox-2026-07-09.md

docs(audit): ep0 Hetzner Storage Box SSHFS + PBS-datastore feasibility spike

2026-07-09 09:48:57 +02:00

SPIKE-felhom-sshd-2026-07-05.md

docs(audit): dedicated felhom-sshd instance spike (2026-07-05) — GO

2026-07-05 18:18:37 +02:00

SPIKE-firstboot-nic-sweep-2026-07-22-screen.png

docs(v1.24.0): R-59/R-60/R-61 SHIPPED — CHANGELOG, README, ROADMAP (+R-62), runbook, capability map, drill evidence, REPORT

2026-07-22 11:45:55 +02:00

SPIKE-firstboot-nic-sweep-2026-07-22.md

docs(v1.24.0): R-59/R-60/R-61 SHIPPED — CHANGELOG, README, ROADMAP (+R-62), runbook, capability map, drill evidence, REPORT

2026-07-22 11:45:55 +02:00

SPIKE-hetzner-api-provisioning-2026-07-09.md

docs(audit): Hetzner API provisioning spike — both offsite models FIT

2026-07-09 17:46:43 +02:00

SPIKE-immediate-sync-transport-2026-07-16.md

docs(spike): immediate-sync transport — poke-relay vs long-poll, both GO/complementary

2026-07-16 20:00:01 +02:00

SPIKE-infra-backup-2026-06-15.md

docs(audit): SPIKE — characterize Infra Backup (revive vs retire)

2026-06-16 10:04:55 +02:00

SPIKE-intermediary-mount-2026-06-15.md

docs(spike): stable intermediary mount — swap drive underneath (C1-immune, no-restart)

2026-06-15 15:26:37 +02:00

SPIKE-island-bridge-2026-07-25.md

R-50: A4 fresh-provision validated (agent auto-net1 live); spike caveat closed

2026-07-25 14:36:56 +02:00

SPIKE-lan-discovery-2026-07-18.md

spike(R-6): human Explorer leg PASSED (nmbd) + stack torn down, report-clean

2026-07-18 10:50:14 +02:00

SPIKE-multidrive-mutual-exclusion-2026-06-29.md

docs: controller-swap narrow-grant Option A IMPLEMENTED (agent v0.45.0) — spike + multidrive pointers resolved

2026-06-29 19:46:17 +02:00

SPIKE-nas-dsm-2026-07-11.md

audit(Q1c): NAS automount trigger does NOT survive a guest reboot (empty-dir FAIL) + follow-up

2026-07-11 15:43:26 +02:00

SPIKE-nas-storage-2026-06-29.md

spike: NAS network storage (media automount + restic backup target) — findings

2026-06-29 11:44:40 +02:00

SPIKE-nas-verify-2026-07-11.md

docs(audit): record SPIKE-nas-verify findings-doc commit hash (b57f6c1)

2026-07-11 08:20:22 +02:00

SPIKE-offsite-credential-recovery-2026-08-04.md

RECON: trace the offsite DR chain link by link — it does not join up (R-198..R-201)

2026-08-04 12:16:04 +02:00

SPIKE-oob-wg-operator-peer-2026-07-05.md

docs(audit): OOB-over-WG operator-peer spike (2026-07-05) — GO

2026-07-05 17:01:35 +02:00

SPIKE-pbs-tier-provisioning-2026-07-10.md

spike: PBS DR tier provisioning SLICE 0 — all 3 mechanisms settled (token-create impossible -> pinned sudoers; ep0 tenancy op-set + stdout secret transport proven; encryption-key autogen births K, pvesm remove deletes it)

2026-07-10 20:12:26 +02:00

SPIKE-pbsdr-selfheal-2026-07-15.md

docs: SPIKE PBS-DR self-heal — root cause = consumed secret, not missing descriptor

2026-07-15 17:53:42 +02:00

SPIKE-pool-scoped-acl-2026-07-01.md

docs: REPORT + spike-doc status for host-install v1.6.0 (pool-scoped ACL, live-migrated)

2026-07-01 13:02:16 +02:00

SPIKE-r82-phase0-2026-07-26.md

docs(ep0): relocate the felhom-offsite PBS datastore onto the 100 GB volume

2026-07-27 09:36:11 +02:00

SPIKE-r95-offsite-delete-2026-09-01.md

SPIKE R-95: the safety net cannot be seen from the box - and that RAISES the urgency

2026-09-01 13:55:35 +02:00

SPIKE-r117-bind-liveness-2026-07-30.md

docs(R-117): the hang case measured — an I/O probe turns a wedged drive into an unkillable agent

2026-07-30 11:49:59 +02:00

SPIKE-r165-mp1-merge-2026-08-02.md

docs: R-167 + R-158 CLOSED, R-165 SPIKED, R-174..R-177 filed

2026-08-02 23:56:16 +02:00

SPIKE-r165-phase0-2026-08-03.md

audits: R-165 Phase 0 — P1 and P2 measured, nothing changed

2026-08-03 06:34:26 +02:00

SPIKE-r241-recovery-offer-2026-08-07.md

SPIKE R-241 — ruled: a MINTING defect, not a screen-predicate defect

2026-08-07 08:53:16 +02:00

SPIKE-r459-mariadb-upgrade-2026-09-06.md

SPIKE R-459: the skipped MariaDB conversion is stable, and the trade it implied does not exist

2026-09-06 17:42:19 +02:00

SPIKE-raw-drive-enroll-2026-06-15.md

…

SPIKE-recovery-unit-space-2026-08-02.md

SPIKE: recovery-unit space — the ceiling is real on mp1, overflow is clean, but silent (R-158)

2026-08-02 09:07:09 +02:00

SPIKE-restic-password-custody-2026-07-09.md

docs(audit): fork-4 restic-password custody spike — VALIDATED (escrow-under-R)

2026-07-09 14:15:01 +02:00

SPIKE-restic-restore-test-2026-08-31.md

R-87 SPIKE: measured, do not build it as written (R-407..R-409 filed)

2026-08-31 15:59:09 +02:00

SPIKE-restic-snapshot-shape-2026-07-14.md

SPIKE restic snapshot shape: record findings-doc commit hash 0771d58

2026-07-14 21:18:36 +02:00

SPIKE-smart-coverage-2026-07-25.md

docs: SPIKE — SMART coverage on real hw (why the card shows Nincs adat); findings-only

2026-07-25 07:56:02 +02:00

SPIKE-smtp-app-relay-2026-06-28.md

docs(audit): SPIKE findings — on-box SMTP app-relay (apps → shim → hub → Resend)

2026-06-28 19:27:07 +02:00

SPIKE-storagebox-restic-direct-2026-07-09.md

docs(audit): restic→Storage Box direct-from-residential spike (FIT/IPv4)

2026-07-09 11:05:36 +02:00

SPIKE-universal-iso-2-2026-07-31.md

SPIKE 2 complete: locked root closes the PVE web UI; before-network gives a measured zero window

2026-07-31 13:57:21 +02:00

SPIKE-universal-iso-3-2026-07-31.md

hub v0.104.0: the guest network gets a reader (R-319), and the hub half of the naming (R-295)

2026-08-13 10:50:12 +02:00

SPIKE-universal-iso-4-2026-07-31.md

SPIKE 4: a .deb in the ISO DOES deliver on an interactive install

2026-07-31 15:35:07 +02:00

SPIKE-universal-iso-2026-07-31.md

SPIKE: a universal ISO needs a different disk strategy and a locked root

2026-07-31 12:12:35 +02:00

SPIKE-upgrade-test-2026-09-06.md

SPIKE: an upgrade test that runs again — and a real defect in our own bookstack template

2026-09-06 11:48:57 +02:00

SPIKE-userdata-layout-2026-06-14.md

…

storage-lifecycle-acceptance-2026-06-15.md

docs: M3 live re-verification addendum (closes E14/E15 residual) + two controller UI fixes

2026-06-15 21:53:10 +02:00

tester-gate-golden-0.188.0-2026-07-31.md

Tester gate: golden re-baked to 0.188.0, fresh-install proof PASSED — a fresh box is safe to hand to a tester

2026-07-31 08:27:36 +02:00

TESTRUN-fullstack-2026-06-29.md

audit: §D supervised reboot done — F2-a + F2-b red-proofs PASS (findings closed)

2026-06-30 10:04:59 +02:00

VALIDATION-fork4-custody-2026-07-09.md

docs(audit): fork-4 custody VALIDATED end-to-end live (supervised)

2026-07-09 16:31:58 +02:00

VALIDATION-newbox-restore-2026-07-10.md

validation: new-box restore round-trip PASS (9/9 sha-identical) + old-archive retirement decision

2026-07-10 09:29:22 +02:00

VALIDATION-offbox-storagebox-2026-07-09.md

docs(audit): offbox v0.104.0 VALIDATED live against the Storage Box (supervised)

2026-07-09 13:17:52 +02:00

VALIDATION-offsite-provisioning-e2e-2026-07-09.md

validation record: offsite provisioning live e2e PASS (2026-07-09) + CONTEXT/REPORT

2026-07-09 22:04:44 +02:00