admin 3abd25e681 CHAOS NIGHT round 7: a ten-minute outage falls between two reports ...

Measured from the box's own log rather than recalled from documentation:
  21:53:41Z Registered periodic job: hub-report (every 15m0s)
  21:53:49Z Hub report pushed successfully (30535 bytes)
  22:08:43Z Hub report pushed successfully (15934 bytes)  - exactly 15m later

The block runs ~22:10Z to ~22:20Z and the next report is due ~22:23:43Z, after
it lifts. So the box never attempts a push while cut off: nothing was tried,
nothing failed, nothing was lost.

The honest verdict for the question I wanted this round to answer - are alarms
raised while the hub is unreachable retried and then silently dropped? - is NOT
EXERCISED, not "passed". Recorded that way.

It is still a finding of its own: a ten-minute internet outage is invisible to
the fleet view because the box had nothing due to say, and the hub's staleness
threshold (30 minutes) is set well beyond it. The two mechanisms agree.

Rounds 8 and 9 are also ten-minute cuts at ~25-minute spacing against a
15-minute cycle, so one will very likely contain a scheduled report and exercise
the drop behaviour properly. They are NOT re-timed to make that happen -
re-timing a round to get a better result is choosing the night after the fact.

Also captured while blocked: internet unreachable from the box, LAN reachable,
26 containers up, free space unmoved, disk guard silent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

2026-09-17 00:15:11 +02:00

..

DRILL-backup-truth-2026-08-21/evidence

DRILL closeout: the scheduled cycle agrees, the abandonment sweep watched firing, R-366

2026-08-22 05:25:30 +02:00

DRILL-cooldown-grain-2026-08-23

docs(hub v0.108.0): the delivery grain, the cooldown ruling, and gate 11's first subject

2026-08-23 14:12:23 +02:00

DRILL-r102-tier2-unit-2026-08-31

R-102 + R-103 CLOSED (controller v0.229.0) — architecture, register, STATUS, drill evidence

2026-08-31 12:21:52 +02:00

DRILL-r329-r386-2026-08-23

docs(R-329/R-386/R-387): the severity contract, the intent ruling, and Part 5 recorded

2026-08-23 12:03:49 +02:00

DRILL-r356-hot-only-restore-2026-08-22

R-356 docs: correct R-107 in the architecture, record the design, refresh STATUS, compress the register

2026-08-22 13:26:10 +02:00

DRILL-r356b-driveless-db-restore-2026-08-22

DRILL R-356b: the off-site restore for a driveless app that HAS a database

2026-08-22 16:33:45 +02:00

DRILL-r361-2026-08-22

R-361 docs: the [FACT], the negative that cancelled Part 2, R-383/R-384, golden 0.221.1

2026-08-23 00:33:12 +02:00

DRILL-r379-rollback-2026-08-22

R-379/R-380 docs: the failure ladder, the drill record, register housekeeping

2026-08-22 18:40:22 +02:00

DRILL-r384-dead-db-alarm-2026-08-23

R-385: make an UNRECORDED golden fail the currency gate; file R-386; own the alarm ladder

2026-08-23 07:59:52 +02:00

DRILL-r403-tier2-delete-2026-08-31

R-403 CLOSED (controller v0.230.0), R-404 filed as a decision for Viktor

2026-08-31 14:39:29 +02:00

DRILL-soak-2026-08-31

soak drill COMPLETE: 7 phases, 4 findings, the observer found the worst one (R-414)

2026-09-01 06:07:13 +02:00

evidence-agent-transport-leak-2026-08-20

agent 0.130.0 published and vouched; R-347 closed, R-349 + R-350 filed

2026-08-20 12:54:54 +02:00

evidence-backup-promise-2026-09-16

the volunteer's own path verified: felhom.eu/letoltes serves 1.28.0

2026-09-16 20:27:25 +02:00

evidence-bignight-2026-09-14

BIGNIGHT teardown complete: VM destroyed, host deleted, ep0 peer gone, customer and its ep0 data kept

2026-09-15 01:05:05 +02:00

evidence-chaos-night-2026-09-17

CHAOS NIGHT round 7: a ten-minute outage falls between two reports

2026-09-17 00:15:11 +02:00

evidence-doorstep-walk-1270-2026-09-14

doorstep teardown complete: host tester-1-8603a2 deleted, ep0 peer gone, customer kept; ep0 DR data retained for ruling

2026-09-14 18:52:09 +02:00

evidence-drill-0243-2026-09-16

drill 0.243.0 complete: 0 interventions, the connect e-mail proven, NOT ready for a volunteer

2026-09-16 14:27:04 +02:00

evidence-drill-fresh-install-0242-2026-09-14

drill 0242: teardown complete in three layers; R-501 filed

2026-09-14 16:40:59 +02:00

evidence-drill-r95-recovery-2026-09-01

DRILL R-95: the recovery route does not exist — stopped before the destructive phase

2026-09-01 16:55:50 +02:00

evidence-drill-retained-key-2026-08-12

DRILL: the retained key works, and the customer cannot reach it

2026-08-12 17:41:56 +02:00

evidence-ep0-established-connections-2026-08-20

SPIKE ep0 connections: CI green by run id (360/237, 19672e685)

2026-08-20 10:42:50 +02:00

evidence-ep0-fd-2026-08-18

INCIDENT + registers: ep0's PBS proxy served nobody for 9.5h (R-336..R-338)

2026-08-18 06:09:38 +02:00

evidence-ep0-pbs-upgrade-2026-08-18

RUNBOOK ep0: PBS 4.2.2-1 -> 4.2.5-1, slope unchanged as predicted (R-341)

2026-08-18 12:26:53 +02:00

evidence-p1fixes-2026-09-15

REPORT + STATUS: P1 fixes — supervisor proven incl. crash-loop resume and hub events; publish; rows

2026-09-15 11:45:21 +02:00

evidence-r316-2026-08-13

REPORT: installer v1.28.0 published and verified live; evidence, and the Part 1 logs I lost again

2026-08-13 08:19:41 +02:00

evidence-r341-plus7d-2026-08-30

R-341 check taken, and the golden-currency bypass declared

2026-08-30 18:49:53 +02:00

evidence-r353-r360-live-2026-08-30

Register + architecture for controller v0.226.0 (R-353/357/358/360/396), and R-395 fixed

2026-08-30 19:39:41 +02:00

evidence-recovery-code-2026-09-16

R-543 closed: the household is asked for the recovery code (controller v0.245.0)

2026-09-16 21:20:51 +02:00

evidence-rehearsal-2026-08-09

Rehearsal 2026-08-09 COMPLETE: data BYTE-IDENTICAL, journey needs a shell twice

2026-08-09 12:35:26 +02:00

evidence-spike-app-update-2026-09-01

SPIKE: what an app update actually does, and which other paths do it too

2026-09-01 21:35:32 +02:00

evidence-spike-restic-restore-2026-08-31

R-87 SPIKE: measured, do not build it as written (R-407..R-409 filed)

2026-08-31 15:59:09 +02:00

fixtures

docs(audits): first genuinely failing disk — the SMART PASSED trap

2026-08-14 07:57:07 +02:00

nightly-2026-09-13-adventurelog

evening: rules file in place; R-483 CLOSED (catalog, operator-confirmed); R-479 CLOSED (controller v0.241.0, proven live); R-481 blocked on the one-host-one-customer model; R-491 filed

2026-09-13 21:57:47 +02:00

nightly-2026-09-13b-bentopdf

second night: scratch guest 9202 built (R-481 CLOSED, persists); controller v0.242.0 delivered (R-487 R-491 R-490 R-476 R-456 CLOSED, R-489 re-scoped); R-492 filed; rotation restarted from bentopdf; morning note

2026-09-13 23:06:21 +02:00

r459-close-2026-09-13

R-459 CLOSED (MariaDB converts itself, proven by harness + live), golden 0.236.0 (R-467), the golden waiver (R-468)

2026-09-13 10:14:37 +02:00

r459-spike-2026-09-06

SPIKE R-459: the skipped MariaDB conversion is stable, and the trade it implied does not exist

2026-09-06 17:42:19 +02:00

R411-R414-2026-09-01

R-411/408/407, R-414, R-412a leg 1, R-410, R-406 CLOSED; determination + live evidence

2026-09-01 10:36:54 +02:00

R442-2026-09-13

R-442 CLOSED (controller v0.236.0): "delete my data too" deletes it or refuses; R-465..467 opened

2026-09-13 09:07:08 +02:00

rulings-r472-r475-2026-09-13

docs: rulings 7 and 8 shipped and proven live (R-470/R-472/R-475 CLOSED); R-477..R-480 opened

2026-09-13 17:54:23 +02:00

slice4-2026-09-13

Slice 4 shipped (R-448/R-443/R-439 CLOSED, proven live); R-472..R-476; the floor-between-bakes claim corrected

2026-09-13 12:30:17 +02:00

upgrade-spike-2026-09-06/evidence

SPIKE: an upgrade test that runs again — and a real defect in our own bookstack template

2026-09-06 11:48:57 +02:00

v0240-2026-09-13

R-452 CLOSED: the catalog-since gate (hook-enforced); 09 §8.2 limitation lifted; STATUS note updated

2026-09-13 19:42:42 +02:00

v0241-2026-09-13

evening: rules file in place; R-483 CLOSED (catalog, operator-confirmed); R-479 CLOSED (controller v0.241.0, proven live); R-481 blocked on the one-host-one-customer model; R-491 filed

2026-09-13 21:57:47 +02:00

v0242-2026-09-14

second night: scratch guest 9202 built (R-481 CLOSED, persists); controller v0.242.0 delivered (R-487 R-491 R-490 R-476 R-456 CLOSED, R-489 re-scoped); R-492 filed; rotation restarted from bentopdf; morning note

2026-09-13 23:06:21 +02:00

AUDIT-blast-radius-hostroot-localapi-2026-07-02.md

docs: AUDIT — host-root + local-API blast radius (non-Proxmox-token axes)

2026-07-02 10:43:19 +02:00

AUDIT-gate-decoys-2026-09-01.md

the decoy sweep: 29 gates read, 16 fooled, 10 fixed - and a gate that refuses the next one (R-421)

2026-09-01 12:39:45 +02:00

AUDIT-hub-gui-2026-06-30.md

docs(audit): hub GUI vs current architecture findings (2026-06-30)

2026-06-30 20:13:48 +02:00

AUDIT-nas-cwa-rca-2026-07-11.md

docs: Q1c GREEN — reboot survival automatic since agent 0.84.0 (feature doc + audit §7 + CONTEXT + REPORT)

2026-07-11 21:25:14 +02:00

AUDIT-power-outage-recovery-2026-07-22.md

docs: ISO train v1.25.0 — REPORT (belt/apt/R-63/gate/vault live), F9 resolved, R-63 shipped, nodes belt+vault, F8 checklist; critical golden<floor finding

2026-07-23 15:48:01 +02:00

AUDIT-vacation-remote-ops-2026-07-20.md

docs(audits): remote-ops remediation — F1 mitigated, F2 closed, F3 fixed (v0.149.0), Immich resolved; R-50..R-53 minted

2026-07-20 09:09:25 +02:00

B3-storage-lifecycle-2026-06-15.md

docs: B3 live storage-lifecycle exercise + feature sweep (2026-06-15)

2026-06-15 12:39:44 +02:00

BIGNIGHT-household-month-2026-09-14.md

BIGNIGHT teardown complete: VM destroyed, host deleted, ep0 peer gone, customer and its ep0 data kept

2026-09-15 01:05:05 +02:00

bughunt-reconcile-2026-06-13.md

…

CAMPAIGN-3-2026-07-11.md

CAMPAIGN-3: morning RCA — host loss = F12 CRITICAL automount ordering cycle (hardware exonerated); box recovered to defined state

2026-07-12 06:55:40 +02:00

CAMPAIGN-4-2026-07-13.md

@

2026-07-14 00:45:20 +02:00

CAMPAIGN-5-2026-07-14.md

@

2026-07-14 10:12:21 +02:00

CAMPAIGN-6A-2026-07-14.md

@

2026-07-14 12:17:26 +02:00

CAMPAIGN-6B-2026-07-14-PROMPT.md

@

2026-07-14 12:17:26 +02:00

CAMPAIGN-6B-2026-07-14.md

CAMPAIGN-6B: note credential fully reverted at cleanup

2026-07-14 13:32:14 +02:00

CAMPAIGN-6C-2026-07-14-PROMPT.md

CAMPAIGN-6B: note credential fully reverted at cleanup

2026-07-14 13:32:14 +02:00

CAMPAIGN-6C-2026-07-14.md

CAMPAIGN-6C: v0.131.0 F-S2/F-S3 live acceptance (4P-1..4P-5 all PASS) + F-6C-1

2026-07-14 20:01:13 +02:00

CAMPAIGN-6D-2026-07-15.md

docs: 6D audit — tag campaign6 autofs finding as track-only (operator reboot)

2026-07-15 19:48:43 +02:00

CAMPAIGN-6E-2026-07-15.md

docs(audit): CAMPAIGN-6E — P-TIER2 deep-4 + P4-DEEP timing (all 8 legs PASS)

2026-07-15 21:11:39 +02:00

CAMPAIGN-7-catalog-sweep-2026-07-19.md

CAMPAIGN 7 doc: MAJOR breaking-note table (§6.3)

2026-07-19 03:30:50 +02:00

CAMPAIGN-8-backup-restore-2026-07-27.md

F-LEAK: record all three attempts; the ACL fix is consumed by use, the fenced destroy is the fix

2026-07-28 11:32:45 +02:00

CAMPAIGN-9-restore-proof-2026-07-28.md

C9-F1 (honesty half) + C9-F2 shipped in controller v0.183.0; C9-F1b and C9-F4 filed

2026-07-28 19:08:25 +02:00

CAMPAIGN-10-closeout-2026-08-02.md

Campaign 10 closeout Part 2: teardown complete — five layers, each verified gone

2026-08-02 10:11:21 +02:00

CAMPAIGN-10-two-storage-soak-2026-07-31.md

Campaign 10: M-band RTO measured — RTO ~= 40s + 26.9s/GB, and a capacity ceiling that matters more

2026-08-02 08:31:51 +02:00

CAMPAIGN-11-recovery-journey-2026-08-05.md

R-201 re-walk: the data PASSES again, the journey still FAILS — two dead ends, down from four

2026-08-06 12:18:29 +02:00

CAMPAIGN-12-class-sweep-2026-08-08.md

CAMPAIGN 12 — the class sweep: golden 0.208.0 baked (awaiting vouch), R-256..R-263 filed, gating ranked

2026-08-07 23:07:42 +02:00

CAMPAIGN-nomercy-2026-07-06.md

docs(audit): closure appendix — F2 closed, F4-UX fixed, healthcheck class fixed

2026-07-06 20:43:52 +02:00

CAMPAIGN-offsite-overnight-2026-07-10.md

campaign: offsite overnight soak 2026-07-10 — PASS + 1 HIGH finding (crash-mid-prune stale lock)

2026-07-10 04:32:32 +02:00

CAMPAIGN-pool-effects-2026-07-01.md

docs: F3 provisioning runbook (vmid from full host list) + campaign F1/F2 resolution banner

2026-07-02 10:27:17 +02:00

D5-drive-alone-restore-2026-07-30.md

D5 SHIPPED: Tier-1/2 restore no longer depends on the whole-guest tier

2026-07-30 16:58:06 +02:00

deep-sweep-2026-06-13.md

docs(audits): AGENT-001 -> MERGED + DEPLOYED (agent v0.30.0)

2026-06-13 22:32:23 +02:00

DIAG-agent-channel-2026-07-26.md

DIAG: agent-channel banner is a stale controller.yaml endpoint, not island fallout

2026-07-26 08:51:45 +02:00

DIAG-backup-missed-2026-07-26.md

docs(audits): R-80 DIAG — expected_backup_missed root-caused to in-memory agent backup store

2026-07-26 09:59:35 +02:00

DIAG-bootrecon-drive-absent-2026-08-02.md

docs: R-157 A / R-170 / R-171 closed — boot recovery finished

2026-08-02 20:38:21 +02:00

DIAG-f10-demo-hp-offsite-2026-07-23.md

docs(R-71): SHIPPED (a)+(c) — the apply-bridge settle-gate (controller v0.162.0)

2026-07-24 07:49:11 +02:00

DIAG-immich-restore-2026-07-19.md

docs: DIAG round 2 — H1 confirmed, H4 found; two standing rulings landed

2026-07-19 13:04:58 +02:00

DIAG-immich-restore-round2-2026-07-19.md

docs: DIAG round 2 — H1 confirmed, H4 found; two standing rulings landed

2026-07-19 13:04:58 +02:00

DIAG-r116-disks-payload-2026-07-30.md

docs: write down which boxes are disposable (target selection by blast radius)

2026-07-30 08:13:58 +02:00

DIAG-sharing-2026-07-20.md

docs: DIAG addendum 2 — S-3 fixed (samba 1.1.0 mDNS + v0.152.0 copy); Finder sidebar stays OPEN

2026-07-20 13:40:55 +02:00

DIAG-smart-passed-trap-2026-08-14.md

docs(audits): first genuinely failing disk — the SMART PASSED trap

2026-08-14 07:57:07 +02:00

DIAGNOSE-drive-bind-after-reprovision-2026-06-23.md

docs(audit): DIAGNOSE re-enrolled drives not binding after 9201 re-provision

2026-06-23 17:15:30 +02:00

DIAGNOSE-offbox-repo-orphaned-2026-07-17.md

docs: offsite continuity Part B REPORT/CONTEXT; DIAGNOSE incident→fix pointer; capability map (offsite reinstall-continuity + escrow retention); ROADMAP R-25 (TOCTOU hardening) + R-26 (guided old-history recovery)

2026-07-17 10:55:17 +02:00

DISPOSITION-ia-finding2-systemdisks-2026-07-13.md

audits: disposition — backups-IA finding 2 (legacy-boot SystemDisks dead wizard) RESOLVED by felhom-agent v0.87.0 (slaves-walk, operator ruling 2026-07-13); live evidence drill replay + felhom-pve byte-identical regression; remaining: supervised wizard leg + Day-0 manifest vouch

2026-07-13 13:21:14 +02:00

DOORSTEP-walk-1270-2026-09-14.md

doorstep teardown complete: host tester-1-8603a2 deleted, ep0 peer gone, customer kept; ep0 DR data retained for ruling

2026-09-14 18:52:09 +02:00

DRILL-appdata-restore-2026-07-04.md

docs(audit): app-data restore drill 2026-07-04 (guest 9201)

2026-07-04 22:53:07 +02:00

DRILL-chaos-night-2026-09-17.md

CHAOS NIGHT round 6: the local backup tier cannot fit, and the box says so properly

2026-09-17 00:11:37 +02:00

DRILL-day0-cleanroom-2026-07-03.md

docs: golden 0.98.3 live — D.1b retired, drill B1/B5 fixed, backlog note resolved + evidence doc

2026-07-03 18:38:28 +02:00

DRILL-day0-take2-2026-07-12.md

docs: hub v0.53.0 closing bundle - CHANGELOG + CONTEXT (both operator rulings verbatim) + take-two F-14 disposition (SHIPPED, gated on escrow-acked deletion provenance)

2026-07-13 15:02:44 +02:00

DRILL-day0-vm-2026-07-12.md

docs: DR-tier-by-default batch — decisions 1-5 in CONTEXT, day0 ACL advice reversed, ceremony decision-4 policy block, WG disclosure (tester agreement + GO-LIVE D3), drill F-disposition table, REPORT

2026-07-12 20:43:25 +02:00

DRILL-fresh-install-0242-2026-09-14.md

drill: a stranger's first hour on 0.242.0 — 1 intervention, not ready for a volunteer

2026-09-14 16:17:47 +02:00

DRILL-golden-098-2026-07-03.md

docs: golden 0.98.3 live — D.1b retired, drill B1/B5 fixed, backlog note resolved + evidence doc

2026-07-03 18:38:28 +02:00

DRILL-prove-fixes-0243-2026-09-16.md

drill 0.243.0 complete: 0 interventions, the connect e-mail proven, NOT ready for a volunteer

2026-09-16 14:27:04 +02:00

DRILL-r201-night-run-2026-08-04.md

R-201 PASSED: a customer's file survived a machine rebuild and came back byte-identical

2026-08-04 23:18:47 +02:00

DRILL-r201-offsite-recovery-2026-08-04.md

R-203 shipped: the app and its backup agree, and 'ok' means it — R-201 unblocked

2026-08-04 18:52:14 +02:00

DRILL-retained-key-2026-08-12.md

Correct the drill's wall clock to the measured end time (17:45, not the estimated 17:55)

2026-08-12 17:45:23 +02:00

E2D-fresh-vm-2026-07-29.md

Record work, banked first: shrink the E-2d row, create the missing capability-map rows

2026-07-29 23:34:06 +02:00

FINDING-app-email-rollout-2026-06-29.md

docs: FINDING — app-email rollout mechanism gaps (calcom/nextcloud/immich)

2026-06-29 10:32:56 +02:00

host-addresses-visible-2026-07-31.md

docs: host-addresses audit + capability-map row + REPORT (agent 0.119.0 / hub 0.85.0)

2026-07-31 08:55:53 +02:00

INCIDENT-ep0-pbs-fd-exhaustion-2026-08-18.md

RUNBOOK ep0: PBS 4.2.2-1 -> 4.2.5-1, slope unchanged as predicted (R-341)

2026-08-18 12:26:53 +02:00

INCIDENT-guest-dhclient-killed-2026-07-20.md

docs: INCIDENT — guest 9201 dhclient killed as a host orphan; tunnel down ~1h15m

2026-07-20 12:09:01 +02:00

LEDGER-instruction-closeout-2026-08-06.md

docs: close out the instruction arc — t740 corrected on evidence, registers, ledger, S-37

2026-08-06 11:49:22 +02:00

LEDGER-instruction-trim-2026-08-06.md

docs+gate: instruction files cannot silently regrow (R-229)

2026-08-06 09:38:52 +02:00

LEDGER-instruction-trim-part2-2026-08-06.md

docs: backup verified IN the repository — snapshot b587f775, 118/118 + 40/40 (R-229)

2026-08-06 11:22:34 +02:00

live-drive-findings-2026-06-14.md

docs(audits/backlog): preserve live-drive findings+fixspec and M18/M19 fix-plans

2026-06-14 11:00:26 +02:00

live-drive-fixspec-2026-06-14.md

docs: F9/F20-BUG2/F20-BUG3 FIXED (agent v0.31.0, live-validated); queue golden controller-tag follow-up

2026-06-14 15:43:07 +02:00

R106-R109-recipe-completeness-2026-07-30.md

R-106/R-109 audit + registers: shipped at agent 0.118.1, plus R-125

2026-07-30 13:28:10 +02:00

R108-network-app-namespace-2026-07-30.md

R-108 CLOSED — D5's precondition is met (controller v0.187.0)

2026-07-30 14:21:44 +02:00

R116-PART5-2026-07-30.md

R-116 Part 5 FAILED: the fix shipped, C5 still fails, mechanism NOT isolated

2026-07-30 07:15:31 +02:00

R116-v0116-2026-07-30.md

docs(R-116): CLOSED — proven live; capability row F to PROVEN-LIVE; R-120 filed

2026-07-30 09:34:08 +02:00

R117-v0117-2026-07-30.md

docs(R-117): CLOSED — proven live on demo-hp; R-121 filed for agent-on-box drift

2026-07-30 12:42:26 +02:00

R120-golden-rebake-2026-07-30.md

docs(R-120): CLOSED on both halves — golden current, and the class has a gate that refuses

2026-07-30 10:48:16 +02:00

R-63-console-font-drill-2026-07-23.png

drill: R-63 ő/ű captured live on the KVM HP (+ pairing + rootpw); HP SVM-enable fix; evidence PNG

2026-07-23 20:46:27 +02:00

RECON-dooplex-backup-2026-08-06.md

RECON: what DooPlex's own backup actually protects (R-232) — read-only, nothing changed

2026-08-06 12:16:38 +02:00

RECON-offsite-dr-chain-2026-08-04.md

RECON: trace the offsite DR chain link by link — it does not join up (R-198..R-201)

2026-08-04 12:16:04 +02:00

RECON-subdomain-onboarding-2026-07-31.md

the golden gate reads a fact not a name (R-410); R-133's collision resolved (R-406, R-416)

2026-09-01 10:19:55 +02:00

REHEARSAL-byo-reinstall-2026-08-09.md

Rehearsal 2026-08-09 COMPLETE: data BYTE-IDENTICAL, journey needs a shell twice

2026-08-09 12:35:26 +02:00

REPORT-DRILL-backup-truth-2026-08-21.md

R-354 + R-355 CLOSED, proven live; golden 0.218.0 baked; R-367 filed

2026-08-22 10:11:46 +02:00

REPORT-ep0-pbs-upgrade-2026-08-18.md

DRILL 2026-08-21: the off-site restore never replays named volumes (R-354..R-365)

2026-08-21 23:30:27 +02:00

REPORT-mis-called-defect-and-sweep-2026-08-22.md

One register, enforced by a gate; closed work compressed into siblings (R-376..R-378)

2026-08-22 12:13:54 +02:00

REPORT-r316-installer-v1.28.0-2026-08-13.md

hub v0.104.0: the guest network gets a reader (R-319), and the hub half of the naming (R-295)

2026-08-13 10:50:12 +02:00

REPORT-small-debts-and-first-reader-2026-08-13.md

REPORT + CONTEXT: the third name, the second door, and a number that answered a different question

2026-08-13 16:01:45 +02:00

REPORT-v0.218.0-r354-r355-2026-08-22.md

Correct the placement mis-framing, and file what we wrote down and never filed (R-368..R-375)

2026-08-22 11:18:08 +02:00

RERUN-p1p3-2026-07-06.md

docs(audit): closure appendix — F2 closed, F4-UX fixed, healthcheck class fixed

2026-07-06 20:43:52 +02:00

RUNBOOK-usb-enrollment-3bfix-2026-07-01.md

docs: RUNBOOK result — USB enrollment on pool-scoped ACL (residual CLOSED)

2026-07-01 13:52:44 +02:00

SESSION-C-2026-07-29.md

Correct the Session C audit: the union row is DEDUPED AWAY, not written false

2026-07-29 23:51:17 +02:00

SPIKE-a1-pool-membership-read-2026-07-03.md

docs: SPIKE — A1 pool-membership read for the stale-lock reaper

2026-07-03 13:00:00 +02:00

SPIKE-agent-pin-mismatch-2026-06-29.md

spike(pin-mismatch): §9 Outcome — R1 executed, controller↔agent service restored

2026-06-29 15:08:58 +02:00

SPIKE-agent-selfupdate-2026-07-05.md

docs(audit): SPIKE — agent self-update rollback mechanics (systemd 257 semantics on felhom-pve)

2026-07-05 14:45:58 +02:00

SPIKE-app-update-2026-09-01.md

SPIKE: what an app update actually does, and which other paths do it too

2026-09-01 21:35:32 +02:00

SPIKE-backup-classification-2026-07-14.md

docs(audit): backup-classification spike — coupling model VALIDATED (Immich DB-only restore broken-not-empty); offsite=unit-only live-proven; NEW F-S1 (tier-2 copies no userdata) + F-S2 (paperless-ngx appdata skipped); round-trip verdict: exclusion scoping

2026-07-14 16:39:55 +02:00

SPIKE-baremetal-iso-2026-07-16.md

docs(spike): bare-metal Felhom ISO — R-21 spiked (all mechanisms GREEN on nested virt)

2026-07-16 11:29:03 +02:00

SPIKE-catalog-data-paths-2026-07-26.md

SPIKE: catalog-derived userdata skeleton + import surfaces (2026-07-26)

2026-07-26 06:59:46 +02:00

SPIKE-ci-runner-2026-08-02.md

docs: R-168 SHIPPED, R-29 CLOSED on the demonstrated alarm, R-169 minted

2026-08-02 16:35:34 +02:00

SPIKE-connectivity-wireguard-2026-07-03.md

docs: R-182 closed, R-90 closed on measurement, R-86 unblocked, ep0 record corrected

2026-08-03 14:00:54 +02:00

SPIKE-controller-agent-channel-health-2026-06-29.md

docs: felhom.eu REPORT (HostLeafChecker) + hub README + self-health arc complete (spike doc)

2026-06-29 23:32:12 +02:00

SPIKE-controller-escrow-2026-07-13.md

audits: SPIKE controller-driven escrow ceremony — daemon-context mechanics ALL GO (drill VM, agent 0.87.0): PTY re-key no-TTY 3/3, sudoers refusal matrix 5/5, R pipe round-trip proven, env_reset clean, --upload hub-verified ~2.4s; premise correction: capability probes are list-mode

2026-07-13 17:15:52 +02:00

SPIKE-controllerswap-narrow-grants-2026-06-29.md

docs: controller-swap narrow-grant Option A IMPLEMENTED (agent v0.45.0) — spike + multidrive pointers resolved

2026-06-29 19:46:17 +02:00

SPIKE-day0-firstboot-handshake-2026-06-26.md

REPORT + Day-0 doc: BUNDLE slice (hub v0.16.0 artifact manifest + host-install v1.1.0 self-install)

2026-06-28 09:03:34 +02:00

SPIKE-decommission-migration-2026-06-14.md

docs(spike): bake in locked fork decisions (B0 gate cleared)

2026-06-14 18:14:08 +02:00

SPIKE-dooplex-buildcache-2026-08-05.md

docs: P6 EXECUTED — containerd store moved to SSD2; reboot deferred (R-209, R-209a)

2026-08-05 10:27:40 +02:00

SPIKE-dr-bindmount-source-2026-07-07.md

docs: campaign-2 R1/R2 root-cause correction + DR bind-mount source spike

2026-07-07 19:08:03 +02:00

SPIKE-dr-recipe-2026-06-16.md

docs(audit): SPIKE — Phase 2 DR recipe + storage diagnosis (report-only)

2026-06-16 14:18:39 +02:00

SPIKE-drive-enrollment-2026-07-01.md

docs: spike status — Impl-2b DONE (raw enrollment live: ctrl v0.95.0 + agent v0.58.0)

2026-07-01 21:27:48 +02:00

SPIKE-ep0-established-connections-2026-08-20.md

agent 0.130.0 published and vouched; R-347 closed, R-349 + R-350 filed

2026-08-20 12:54:54 +02:00

SPIKE-ep0-storagebox-2026-07-09.md

docs(audit): ep0 Hetzner Storage Box SSHFS + PBS-datastore feasibility spike

2026-07-09 09:48:57 +02:00

SPIKE-felhom-sshd-2026-07-05.md

docs(audit): dedicated felhom-sshd instance spike (2026-07-05) — GO

2026-07-05 18:18:37 +02:00

SPIKE-firstboot-nic-sweep-2026-07-22-screen.png

docs(v1.24.0): R-59/R-60/R-61 SHIPPED — CHANGELOG, README, ROADMAP (+R-62), runbook, capability map, drill evidence, REPORT

2026-07-22 11:45:55 +02:00

SPIKE-firstboot-nic-sweep-2026-07-22.md

docs(v1.24.0): R-59/R-60/R-61 SHIPPED — CHANGELOG, README, ROADMAP (+R-62), runbook, capability map, drill evidence, REPORT

2026-07-22 11:45:55 +02:00

SPIKE-hetzner-api-provisioning-2026-07-09.md

docs(audit): Hetzner API provisioning spike — both offsite models FIT

2026-07-09 17:46:43 +02:00

SPIKE-immediate-sync-transport-2026-07-16.md

docs(spike): immediate-sync transport — poke-relay vs long-poll, both GO/complementary

2026-07-16 20:00:01 +02:00

SPIKE-infra-backup-2026-06-15.md

docs(audit): SPIKE — characterize Infra Backup (revive vs retire)

2026-06-16 10:04:55 +02:00

SPIKE-intermediary-mount-2026-06-15.md

docs(spike): stable intermediary mount — swap drive underneath (C1-immune, no-restart)

2026-06-15 15:26:37 +02:00

SPIKE-island-bridge-2026-07-25.md

R-50: A4 fresh-provision validated (agent auto-net1 live); spike caveat closed

2026-07-25 14:36:56 +02:00

SPIKE-lan-discovery-2026-07-18.md

spike(R-6): human Explorer leg PASSED (nmbd) + stack torn down, report-clean

2026-07-18 10:50:14 +02:00

SPIKE-multidrive-mutual-exclusion-2026-06-29.md

docs: controller-swap narrow-grant Option A IMPLEMENTED (agent v0.45.0) — spike + multidrive pointers resolved

2026-06-29 19:46:17 +02:00

SPIKE-nas-dsm-2026-07-11.md

audit(Q1c): NAS automount trigger does NOT survive a guest reboot (empty-dir FAIL) + follow-up

2026-07-11 15:43:26 +02:00

SPIKE-nas-storage-2026-06-29.md

spike: NAS network storage (media automount + restic backup target) — findings

2026-06-29 11:44:40 +02:00

SPIKE-nas-verify-2026-07-11.md

docs(audit): record SPIKE-nas-verify findings-doc commit hash (b57f6c1)

2026-07-11 08:20:22 +02:00

SPIKE-offsite-credential-recovery-2026-08-04.md

RECON: trace the offsite DR chain link by link — it does not join up (R-198..R-201)

2026-08-04 12:16:04 +02:00

SPIKE-oob-wg-operator-peer-2026-07-05.md

docs(audit): OOB-over-WG operator-peer spike (2026-07-05) — GO

2026-07-05 17:01:35 +02:00

SPIKE-pbs-tier-provisioning-2026-07-10.md

spike: PBS DR tier provisioning SLICE 0 — all 3 mechanisms settled (token-create impossible -> pinned sudoers; ep0 tenancy op-set + stdout secret transport proven; encryption-key autogen births K, pvesm remove deletes it)

2026-07-10 20:12:26 +02:00

SPIKE-pbsdr-selfheal-2026-07-15.md

docs: SPIKE PBS-DR self-heal — root cause = consumed secret, not missing descriptor

2026-07-15 17:53:42 +02:00

SPIKE-pool-scoped-acl-2026-07-01.md

docs: REPORT + spike-doc status for host-install v1.6.0 (pool-scoped ACL, live-migrated)

2026-07-01 13:02:16 +02:00

SPIKE-r82-phase0-2026-07-26.md

docs(ep0): relocate the felhom-offsite PBS datastore onto the 100 GB volume

2026-07-27 09:36:11 +02:00

SPIKE-r95-offsite-delete-2026-09-01.md

SPIKE R-95: the safety net cannot be seen from the box - and that RAISES the urgency

2026-09-01 13:55:35 +02:00

SPIKE-r117-bind-liveness-2026-07-30.md

docs(R-117): the hang case measured — an I/O probe turns a wedged drive into an unkillable agent

2026-07-30 11:49:59 +02:00

SPIKE-r165-mp1-merge-2026-08-02.md

docs: R-167 + R-158 CLOSED, R-165 SPIKED, R-174..R-177 filed

2026-08-02 23:56:16 +02:00

SPIKE-r165-phase0-2026-08-03.md

audits: R-165 Phase 0 — P1 and P2 measured, nothing changed

2026-08-03 06:34:26 +02:00

SPIKE-r241-recovery-offer-2026-08-07.md

SPIKE R-241 — ruled: a MINTING defect, not a screen-predicate defect

2026-08-07 08:53:16 +02:00

SPIKE-r459-mariadb-upgrade-2026-09-06.md

SPIKE R-459: the skipped MariaDB conversion is stable, and the trade it implied does not exist

2026-09-06 17:42:19 +02:00

SPIKE-raw-drive-enroll-2026-06-15.md

docs: SPIKE raw-drive plug-and-enroll (GATE, 2026-06-15)

2026-06-15 10:57:33 +02:00

SPIKE-recovery-unit-space-2026-08-02.md

SPIKE: recovery-unit space — the ceiling is real on mp1, overflow is clean, but silent (R-158)

2026-08-02 09:07:09 +02:00

SPIKE-restic-password-custody-2026-07-09.md

docs(audit): fork-4 restic-password custody spike — VALIDATED (escrow-under-R)

2026-07-09 14:15:01 +02:00

SPIKE-restic-restore-test-2026-08-31.md

R-87 SPIKE: measured, do not build it as written (R-407..R-409 filed)

2026-08-31 15:59:09 +02:00

SPIKE-restic-snapshot-shape-2026-07-14.md

SPIKE restic snapshot shape: record findings-doc commit hash 0771d58

2026-07-14 21:18:36 +02:00

SPIKE-smart-coverage-2026-07-25.md

docs: SPIKE — SMART coverage on real hw (why the card shows Nincs adat); findings-only

2026-07-25 07:56:02 +02:00

SPIKE-smtp-app-relay-2026-06-28.md

docs(audit): SPIKE findings — on-box SMTP app-relay (apps → shim → hub → Resend)

2026-06-28 19:27:07 +02:00

SPIKE-storagebox-restic-direct-2026-07-09.md

docs(audit): restic→Storage Box direct-from-residential spike (FIT/IPv4)

2026-07-09 11:05:36 +02:00

SPIKE-universal-iso-2-2026-07-31.md

SPIKE 2 complete: locked root closes the PVE web UI; before-network gives a measured zero window

2026-07-31 13:57:21 +02:00

SPIKE-universal-iso-3-2026-07-31.md

hub v0.104.0: the guest network gets a reader (R-319), and the hub half of the naming (R-295)

2026-08-13 10:50:12 +02:00

SPIKE-universal-iso-4-2026-07-31.md

SPIKE 4: a .deb in the ISO DOES deliver on an interactive install

2026-07-31 15:35:07 +02:00

SPIKE-universal-iso-2026-07-31.md

SPIKE: a universal ISO needs a different disk strategy and a locked root

2026-07-31 12:12:35 +02:00

SPIKE-upgrade-test-2026-09-06.md

SPIKE: an upgrade test that runs again — and a real defect in our own bookstack template

2026-09-06 11:48:57 +02:00

SPIKE-userdata-layout-2026-06-14.md

docs(spike): userdata/ layout + shared-storage ownership mapping (report-only gate)

2026-06-14 21:15:24 +02:00

storage-lifecycle-acceptance-2026-06-15.md

docs: M3 live re-verification addendum (closes E14/E15 residual) + two controller UI fixes

2026-06-15 21:53:10 +02:00

tester-gate-golden-0.188.0-2026-07-31.md

Tester gate: golden re-baked to 0.188.0, fresh-install proof PASSED — a fresh box is safe to hand to a tester

2026-07-31 08:27:36 +02:00

TESTRUN-fullstack-2026-06-29.md

audit: §D supervised reboot done — F2-a + F2-b red-proofs PASS (findings closed)

2026-06-30 10:04:59 +02:00

VALIDATION-fork4-custody-2026-07-09.md

docs(audit): fork-4 custody VALIDATED end-to-end live (supervised)

2026-07-09 16:31:58 +02:00

VALIDATION-newbox-restore-2026-07-10.md

validation: new-box restore round-trip PASS (9/9 sha-identical) + old-archive retirement decision

2026-07-10 09:29:22 +02:00

VALIDATION-offbox-storagebox-2026-07-09.md

docs(audit): offbox v0.104.0 VALIDATED live against the Storage Box (supervised)

2026-07-09 13:17:52 +02:00

VALIDATION-offsite-provisioning-e2e-2026-07-09.md

validation record: offsite provisioning live e2e PASS (2026-07-09) + CONTEXT/REPORT

2026-07-09 22:04:44 +02:00