admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

DRILL new household on golden 0.282.0: 0 interventions; ready for a first real tester on a new record
gates / gates (push) Successful in 28s
Details

Browse Source 
Audit doc, STATUS one sentence, capability-map first-hour row (walk re-proven, day-one
off-site sentence narrowed: R-720/R-726/R-727), teardown in four layers, R-600 measured again.
Secret scan over all audits: 0 hits, positive control 1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
authentik Default Admin
2026-09-30 09:25:17 +02:00
parent be8f79e2f9
commit cdf974d186
9 changed files with 306 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT-drill-new-household-2026-09-30.md
+20
View File
@@ -0,0 +1,20 @@
# REPORT — 2026-09-29/30: DRILL — a new household's first day on golden 0.282.0
The full record is `documentation/audits/DRILL-new-household-2026-09-30.md` (evidence beside it). This file is the
session summary only.
- **Interventions: 0.** Ready for a first real tester: **yes, on a new customer record** — after the guide fix
(R-722) and with the off-site-per-app decision (R-720) in view.
- **Golden 0.282.0** baked, round-trip verified, vouched (agent 0.137.0, min_agent 0.131.0); R-120 negative control
refused 0.276.0. Record `documentation/tests/golden-0.282.0-2026-09-29/`.
- **Operator ruling during the run:** customer `tester-1` instead of a new drill record → no Cloudflare items were
created; the customer is kept; only the drill host was deleted.
- **Rows:** opened R-719 … R-727 (six P2, three P3); closed R-505; measured again R-718, R-600. Register **350 → 359**.
- **Night one:** database dump ran; second-drive copy not configured; off-site copy skipped (orphaned repository of
an earlier box, R-726; and apps are off-site OFF by default, R-720); whole-guest tiers not due; restore test failed
on an earlier box's archive (R-727).
- **Teardown:** VM 340 destroyed, ISO removed, storages back to their starting sizes; host `tester-1-693e79` deleted
with the escrow acknowledgement; ep0's WireGuard peer gone 48 s later by itself; three tester-1 whole-guest
archives on ep0 listed and left for the operator. Demo boxes' guests, `drill-r50`, DooPlex beyond bake/vouch/hub
pages: untouched.
- **No product code changed. No `--no-verify`.**
STATUS.md
+14 -13
View File
@@ -1,22 +1,23 @@
# STATUS — what works, what's broken, what's next
**Updated 2026-09-29 evening. Both demo boxes run controller 0.282.0 and host agent 0.137.0. Hub 0.125.0. New installs get golden 0.276.0 with agent 0.137.0.**
**Ready for a first real tester: YES, on a new customer record — a fresh box went from the download to restored apps with no help at all; fix the volunteer guide first, and decide whether their apps go off-site by themselves (today they do not).**
**Decisions today** (yours, recorded): wanderer stays, with its warning, until it is closed. Apps installed before the sign-up rule get a "close sign-up now" button.
**Updated 2026-09-30 morning. Both demo boxes run controller 0.282.0 and host agent 0.137.0. Hub 0.125.0. New installs get golden 0.282.0 with agent 0.137.0 (baked and vouched last night).**
**What I did, and it worked.**
- **Sign-up now has two locks.** 9 apps have their own "no sign-up" setting. The box switches it on after your first account exists. The address block stays as the second lock.
- **The block resists address tricks now.** I tried 113 tricks on 11 apps (capital letters, extra slashes and similar). None got in. Two tricks worked before I fixed them.
- **"Close sign-up now" works.** I pressed it on the HP box's adventurelog and opengist, and on the N100's opengist. Strangers can no longer sign up there. adventurelog restarted once for about 30 seconds.
- **wanderer is closed too.** It cannot have the gate. You make your account, then press "close sign-up now".
- **ghost, home-assistant and gramps-web now open their gate by themselves** after the setup, like 9 other apps.
- **A new catalog check** stops a wrong "setup done" check from reaching the catalog.
- **A fresh box, walked like a volunteer, needed no help.** Download, install on the Hungarian keyboard, the e-mailed link, the dashboard through the internet, the recovery code, three apps, a family member, backup, restore, delete, a power cut, typos, a phone. Last walk needed one intervention; this one needed none.
- **The new safety parts held on a fresh box.** Strangers saw only the gate page. The gate opened by itself after the household's setup. Sign-up stayed closed. The random first password worked, and the old default password was refused.
**What is not done.**
- **opengist and wishlist** keep their own "no sign-up" setting inside their database. The box cannot reach it yet, so they have the address block only. It held against every trick.
- **The "close sign-up now" card does not say the app restarts.** Small text fix, written down.
**What broke.**
- **No off-site copy on night one.** Every app starts with its off-site copy switched off, and nothing tells the household. On top of that, tester-1's old off-site store belonged to an earlier box, so the box skipped it all night. The household's data stayed safe on the box.
- **The night's restore test tried an old box's backup** and failed on its key. The page shows a bare ✗ and names the wrong place.
- **A Stop pressed during a backup was undone by that backup.**
- **The volunteer guide is out of date in five places.** For example, it still gives BookStack's old default password, which is now refused.
**Rows.** 3 closed, 2 opened. The list went from 353 to 355 rows.
**Rows.** 9 opened, 1 closed. The list went from 350 to 359 rows.
**What needs you.**
1. **D4, the image copies, the Peti leftovers:** unchanged. If you do nothing, nothing changes.
1. **Decide: should apps go off-site by themselves?** Option A: yes, every app is switched on when the customer has off-site (my pick — that is what the guide already promises). Option B: no, the guide tells the household to switch each app on. If you do nothing, a new household's apps have no off-site copy.
2. **Approve the guide fix** (five lines, I write them). If you do nothing, a volunteer tries BookStack's old password and fails.
3. **Three whole-machine backups of deleted drill boxes are still in tester-1's space on ep0** (two from 16 September, one from last night). I only looked; I did not touch them. The old ones make the restore test fail. If you do nothing, every new tester-1 box repeats that failure. Say "remove them" and I will do it with before/after checks.
4. **D4, the image copies, the Peti leftovers:** unchanged. If you do nothing, nothing changes.
documentation/architecture/00-capability-map.md
+1 -1
View File
File diff suppressed because one or more lines are too long
documentation/audits/DRILL-new-household-2026-09-30.md
+149
View File
@@ -0,0 +1,149 @@
# DRILL — a new household's first day on golden 0.282.0 (2026-09-29/30)
**Interventions a volunteer could not have made: 0.**
**Ready for a first real tester: YES, on a new customer record — nothing stopped the walk, from the download to a
restored app; but send the guide only after its five stale lines are fixed (R-722), and know that their apps get NO
off-site copy until someone switches each one on (R-720).**
Evidence: `evidence-drill-new-household-2026-09-30/` — `journal.md` (every observable, in order), `golden/`,
`phase0/`, `phase1/`, `phase2/`, `screens/` (codes blacked out), `box-logs-phase1/`, `box-logs-final/`, `teardown/`.
Golden record: `../tests/golden-0.282.0-2026-09-29/`. Architecture read first: `01` §5 (who may reach an app), `07`
§6.5–§6.6, `09` §3 decisions 11–49, `FIRST-ADMIN.md`, the first-hour row of `00-capability-map.md`.
---
## Claims in the brief that turned out wrong (or right), named first
| claim | verdict |
|---|---|
| The box lands on 0.282.0 from the golden | **RIGHT.** Controller 0.282.0 + agent 0.137.0, the vouched set; `selfupdate: Current version 0.282.0 is up to date`; no self-update. |
| A one-level-deeper app address (`<app>.<sub>.felhom.eu`) fails TLS at the edge | **CONSISTENT, NOT MEASURED END TO END.** Every working customer domain is its own zone and the edge serves `*.enkisfelhom.hu, enkisfelhom.hu` only; `felhom.eu`'s own names are DNS-only. A proxied deeper name would get `*.felhom.eu`, which cannot match. No record was made to prove it (no Cloudflare key; the operator ruled the drill onto an existing zone). |
| The gate never re-gates a set-up app after a kept-data reinstall | **THE PREMISE DID NOT HOLD for the app walked.** A volume-only app has no „A megőrzött adataimat használom / Tiszta lappal kezdem" choice — the volume is always deleted and only the backup is kept. The reinstall came back **empty and gated** (correct); after the household restored its backup the data returned and **the gate opened by itself 10 s later**. The household never sees a set-up app gated over its own data. The drive-kept path was not walked (no data drive was set up). |
| A fresh customer's box runs an off-site leg on night one | **WRONG, twice.** The leg ran and **copied nothing it could keep**: (1) apps start with off-site OFF and nothing tells the household to switch them on (**R-720**); (2) on this customer, who had a box before, the repository was found **orphaned** and every run skips until the household presses a reset nobody pointed them to (**R-726**). |
| The claim code is one-shot | **RIGHT.** The spent setup code → „Hibás vagy lejárt kód"; the bind link survived one typo and then bound. |
## 1. What held, in one table
| # | step | result | time |
|---|---|---|---|
| 0.1 | golden 0.282.0 baked, round-trip verified, vouched; R-120 refused 0.276.0 after | **PASS** | bake 7 min |
| 0.2 | operator's part | customer `tester-1` (operator ruling); tunnel already on the record; passphrase from the page; **„Send self-bind link" pressed** (R-719) | — |
| 1 | download from `felhom.eu/letoltes` | **PASS** — 1 705 324 544 B, sha == page == release record | 46 s |
| 2 | install (guide answers, **Hungarian keyboard kept**) | **PASS** — two screens differ from the guide (pre-selected disk; focus on „Previous") | ≤ 7 m 27 s copying |
| 3 | first boot → **the mailed link** → self-bind page | **PASS** — registered 31 s after power-on; one typo refused; bound; console turns to „a doboz össze van kötve" | bind → controller 2 m 51 s |
| 4 | „ready", versions | **PASS** — 0.282.0 / 0.137.0 from the golden | — |
| 5 | dashboard **through the public tunnel**; recovery code | **PASS** — no LAN detour (R-505 closed); code ready 12 min after the claim, made in 5 s, shown once, 410 after | power-on → claimed 27 m 51 s |
| 6 | three apps: random first password / gated with a probe / gated with a sign-up lock | **PASS** — 65 s / 23 s / 12 s; the stranger saw only the gate page; „Kész" before the setup refused 409; gates opened (probe ~12 s; press) | — |
| 7 | use for ~10 min; a family member through the 15-minute window | **PASS** — BookStack page + attachment, Vikunja project + tasks + file (sha equal); stranger 403 before and after the window, family member in | — |
| 8 | backups page, „Mentés most" | **PASS** — true dates on both pages, no customer alarm; every app reads „3. mentés Kikapcsolva" (R-720) | 19 s |
| 9 | „Naprakész" | **TRUE** — installed == catalog; no newer tested step offered | — |
| 10 | remove keeping data, reinstall | **PASS with a defect** — a Stop during the first whole-guest backup was undone by that backup (**R-721**); no kept-data choice for a volume app; restore brought the data back and the gate lifted itself | — |
| 11 | restore the password app | **PASS** — page + attachment byte-identical, the shown password works, same version | 26 s |
| 12 | remove with „delete my data too" | **PASS** — volume, backup, restore points gone; front door 404; „Nincs megőrzött adat" | — |
| 13 | status pages as a week-old household | small disagreements (**R-724**) | — |
| A1 | power cut (60 s dark) | **PASS** — same versions, gates kept their state, data equal, **no alarm**; one re-login | +123 s to dashboard |
| A2 | typos: pairing code on the bind page; the setup code on „Elfelejtett jelszó" | **PASS** — both refused in Hungarian, the right code accepted right after; spent code refused | — |
| A3 | a phone opens a gated app first | **PASS** — Hungarian gate page → „Bejelentkezés" → dashboard login → straight back into the app; a native app's call gets English JSON (R-725) | — |
## 2. The operator's part — every step the volunteer depends on
1. **The customer:** `tester-1` (operator ruling during the run, instead of a new drill record) — domain
`enkicsifelhom.hu`, `tester1@felhom.eu`, off-site shared 100 GB ON, DR tier ON.
2. **The domain and tunnel:** already on the record (R-505, 2026-09-14). No Cloudflare API token exists in the
operator's credentials file; nothing in Cloudflare was created or changed.
3. **The owner passphrase:** read from the customer page's reveal field (5 words), handed over out of band.
4. **„Send self-bind link" — pressed at 18:54:51 UTC.** The guide says no press is needed; for a customer whose last
link expired (2026-09-24) nothing re-sends one when a new box registers (**R-719**).
Everything after that was the volunteer's own: both mails arrived in the tester's mailbox and were followed.
## 3. The first night, leg by leg (box time CEST; UTC in brackets)
| leg | what happened | why |
|---|---|---|
| database dump 02:30 (00:30) | **RAN, OK** — 1 database, 3 volume dumps, 15 s; recovery units for all three apps | — |
| second-drive copy 03:30 (01:30) | **not run** — „nincs másik fizikai meghajtó — a 2. mentéshez 2. meghajtó szükséges" | not configured: no second drive assigned |
| off-site copy 04:15 (02:15) | **SKIPPED** — `offsite repo ORPHANED … runs will skip until reset`; household timeline + operator mail | the customer's previous box left its repository under a destroyed key (**R-726**); and only BookStack was switched on at all (**R-720**) |
| app update leg (after off-site) | **RAN** — `done=0 undone=0 held=0 failed=0` | nothing newer in the catalog |
| whole-guest backup (agent) | **not run tonight** — no new archive on either storage | not due: local tier ran 21:27 CEST (24 h cadence), off-site PBS tier 21:37 CEST (7-day cadence) |
| restore test (agent) 03:52 (01:52) | **FAILED** — picked a 2026-09-16 archive of an earlier drill box in the same namespace, `wrong key`; operator mail; the household sees „✗ Visszaállítás ellenőrizve — Helyi tároló (local)" | **R-727** |
| off-site proof / integrity 05:30 / 06:00 | checked nothing — the repository is orphaned (said so, not called a failure) | follows R-726 |
**So a new household's box does not fully protect itself on night one.** The local layers did (database, volumes,
whole-guest on the evening before); the off-site app copy did not, for two separate reasons, and the restore test
proved nothing.
## 4. Findings
| row | rank | one line |
|---|---|---|
| **R-719** | P2 | An existing customer's new box gets no fresh connect link — the operator must press „Send self-bind link" |
| **R-720** | P2 | Apps start with off-site OFF and nothing tells the household to switch them on — night one copies nothing off-site |
| **R-721** | P2 | A household's Stop during a whole-guest backup is undone by that backup's resume |
| **R-722** | P2 | The volunteer guide is stale in five places (BookStack's old default login is now refused, among them) |
| **R-726** | P2 | A customer with an earlier box: the off-site repository is orphaned on night one; the fix is an unannounced button |
| **R-727** | P2 | The restore test picks a previous box's archive, fails on its key; the page blames the local tier |
| R-723 | P3 | Two operator mails on day one that describe nothing wrong (`node_recovered`, `backup_tier_skipped`) |
| R-724 | P3 | Status pages disagree (schedule, raw timestamp, LAN unknown, „0 órája", R-500's UTC) |
| R-725 | P3 | Copy slips (bind page „a beállításkor", formal wizard, stray console glyph, English JSON to phone apps) |
| R-718 | P3 | measured again: the gate-open press restarts an app with its own switch (2 s 404) unannounced |
**Closed by this walk:** **R-505** — the box-side tunnel hop, proven end to end on a fresh box. **Held:** R-214
(console after bind), R-496/R-497 (console text). **Still reproducing:** R-498 (`wiki.DOMAIN`), R-500 (UTC).
**Register 350 → 359 rows.**
**No P1.** Nothing stopped a volunteer; every stop above is either an operator press (R-719) or a backup promise
not yet kept (R-720, R-726, R-727).
## 5. Harness substitutions — not interventions, and what each hides
| | what | consequence |
|---|---|---|
| H3 | auto-reboot unticked, ISO detached by `qm set` | the "stick still in" reboot not exercised |
| H4 | the Terminal-UI entry chosen over the graphical default | the graphical installer not exercised |
| H5 | a nested VM with three virtual disks | disk choice and drive health not realistic („0 °C", „Nincs adat") |
| H6 | no browser: every screen driven through the endpoint the page calls | script-rendered state not observed |
**Gone since 2026-09-14:** H1 (no mailbox — the tester1@ mailbox was read through the Gmail connector, both customer
mails followed) and H2 (US keyboard — keystrokes sent as Hungarian-layout scancodes, `@` as AltGr+V).
**Harness slips, recorded:** two joined form tokens on the first claim („Érvénytelen űrlap", no code judged); a POST
without `Origin` and a `-X POST` carried across a redirect on the phone login (two 403s, both the harness); a hub-page
search for the MAC that could never match; `pkill -f` ended my own shell once (redone).
## 6. Teardown — four layers
See `evidence-drill-new-household-2026-09-30/teardown/`.
1. **Machine:** VM 340 destroyed `--purge` 06:48 UTC; `qm list` → only 9202 (and 9201 as a CT) remained.
2. **Host (demo-hp):** the drill ISO removed (the three older ISOs were there before and stay); `nvme-scratch`
back to 77.0 GB used, `local` to 24.07 GB; guests 9201 and 9202 running throughout; `local-lvm` never used.
3. **Hub:** host `tester-1-693e79` deleted with the escrow acknowledgement (key → retained custody). **Customer
`tester-1` KEPT, not reset** (operator ruling); its events are append-only and stay.
4. **Outside the hub:** Cloudflare — nothing created, nothing removed (the tunnel is tester-1's own). **ep0
(read-only):** see §6b.
Untouched, and said: both demo boxes' guests, `drill-r50`, DooPlex beyond the bake / vouch / hub pages.
### 6a. Teardown, measured
| layer | UTC | result |
|---|---|---|
| **1. machine** | 06:48:25 | `qm destroy 340 --purge` → `qm list` shows only 9202; `/mnt/hdd_1/images/340` gone |
| **2. host** | 06:48:3x | drill ISO removed; `nvme-scratch` used 89.3 → **77.0 GB** (76.98 GB before the VM existed); `local` 25.7 → **24.07 GB** (24.06 before); 9201 and 9202 running before and after |
| **3. hub** | 07:23:03 | the host was „ONLINE" to the hub until 45 min after its last report (the configured stale threshold, R-549) — 26 refusals `409 Host is ONLINE`, none changing anything; then `host deleted: tester-1-693e79` with the escrow acknowledgement (key demoted to retained custody — the log's „escrow deleted: true" wording is R-544). Host page 404; hosts list back to the three it had before. **Customer `tester-1` KEPT (200).** The delete sent the connect mail by itself (`self-bind link auto-minted … on host delete`) — **R-509's trigger proven again**; a valid 7-day link now sits in the tester mailbox with no box behind it |
| **4. outside the hub** | 07:23:51 | **ep0, read-only:** `wgsync: pushed 4 peers` → `10.77.0.5/32` gone, 48 s after the delete; peers 5 → 4; the four others unchanged. **No manual step was needed (R-600 did not reproduce on a host delete).** Cloudflare: nothing created, nothing removed |
### 6b. Left on ep0 — listed, not touched, operator asked
`/mnt/pbs-datastore/ns/tester-1/ct/9201/`: `2026-09-16T17:27:32Z`, `2026-09-16T21:59:54Z` (earlier drill boxes, their
keys destroyed on acknowledged deletes) and `2026-09-29T19:37:07Z` (this drill's box, key now in retained custody).
The older two are what the restore test tripped on (R-727). **Per the brief, these are listed and left; STATUS asks
the operator.** The customer's Storage Box sub-account (311327) holds the orphaned restic repository of an earlier
box; this drill's box wrote nothing to it (every run skipped) — also left.
### 6c. Local secrets
The session's 0600 files (passphrase, pairing and setup codes, dashboard and app passwords, recovery code, the
vaulted break-glass credential, the bind token) are shredded at the end of this session. The break-glass credential
belonged to the deleted host.
documentation/audits/evidence-drill-new-household-2026-09-30/teardown/before.txt
+44
View File
@@ -0,0 +1,44 @@
## 2026-09-30T06:48:13Z BEFORE — demo-hp
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
340 drill-nh0930-new-household running 8192 200.00 3897073
VMID Status Lock Name
9201 running demo-hp
9202 running demo-hp-scratch
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
felhom-pbs pbs active 0 0 0 0.00%
local dir active 40453376 25735408 12630852 63.62%
local-lvm lvmthin active 56487936 33943600 22544335 60.09%
nvme-scratch dir active 983379700 89289708 844063380 9.08%
total 6661448
drwxr-xr-x 2 root root 4096 Sep 29 20:53 .
drwxr-xr-x 4 root root 4096 Aug 21 17:42 ..
-rw-r--r-- 1 root root 1705324544 Sep 29 20:54 drill-nh0930-felhom-installer-1.29.0-pve9.2-1.iso
-rw-r--r-- 1 root root 1705322496 Sep 16 10:56 felhom-installer-1.27.1-pve9.2-1.iso
-rw-r--r-- 1 root root 1705322496 Sep 16 17:15 felhom-installer-1.28.0-pve9.2-1.iso
-rw-r--r-- 1 root root 1705324544 Sep 18 20:42 felhom-installer-1.29.0-pve9.2-1.iso
340
9202
## hub
/hosts/demo-felhom-8363b5
/hosts/demo-hp-bb76ea
/hosts/drill-r50-0a4f9a
/hosts/tester-1-693e79
## ep0 (read-only)
wg0 10.77.0.2/32
wg0 10.77.0.250/32
wg0 10.77.0.4/32
wg0 10.77.0.3/32
wg0 10.77.0.5/32
peers=5
demo-felhom
demo-hp
tester-1
/mnt/pbs-datastore/ns/tester-1:
/mnt/pbs-datastore/ns/tester-1/ct:
/mnt/pbs-datastore/ns/tester-1/ct/9201:
2026-09-16T17:27:32Z
2026-09-16T21:59:54Z
2026-09-29T19:37:07Z
/mnt/pbs-datastore/ns/tester-1/ct/9201/2026-09-16T17:27:32Z:
/mnt/pbs-datastore/ns/tester-1/ct/9201/2026-09-16T21:59:54Z:
/mnt/pbs-datastore/ns/tester-1/ct/9201/2026-09-29T19:37:07Z:
documentation/audits/evidence-drill-new-household-2026-09-30/teardown/layer1-2.txt
+16
View File
@@ -0,0 +1,16 @@
## 2026-09-30T06:48:25Z layer 1 — machine
purging VM 340 from related configurations..
qm list:
9202
## layer 2 — host
felhom-installer-1.27.1-pve9.2-1.iso
felhom-installer-1.28.0-pve9.2-1.iso
felhom-installer-1.29.0-pve9.2-1.iso
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
felhom-pbs pbs active 0 0 0 0.00%
local dir active 40453376 24070056 14296204 59.50%
local-lvm lvmthin active 56487936 33943600 22544335 60.09%
nvme-scratch dir active 983379700 77026856 856326232 7.83%
VMID Status Lock Name
9201 running demo-hp
9202 running demo-hp-scratch
documentation/audits/evidence-drill-new-household-2026-09-30/teardown/layer3-hub.txt
+46
View File
@@ -0,0 +1,46 @@
## layer 3 — hub: delete host tester-1-693e79 with the escrow acknowledgement (retained custody); customer tester-1 is KEPT
{"deletable":false,"escrow_present":true,"guests":1,"log_bundles":0,"pbs_secret_present":true,"recovery_present":true,"reports":47,"status":"ok","wg_peer_bound":true}
06:48:54 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
06:49:54 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
06:50:54 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
06:51:55 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
06:52:55 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
06:53:55 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
06:54:56 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
06:55:56 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
06:56:56 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
06:57:56 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
06:58:57 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
06:59:57 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:00:57 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:01:57 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:02:58 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:03:58 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:04:58 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:05:58 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:06:59 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:07:59 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:08:59 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:09:59 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:11:00 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:12:00 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:13:00 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:14:00 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:15:01 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:16:01 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:17:01 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:18:01 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:19:02 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:20:02 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:21:02 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:22:03 → 409 Host is ONLINE — deletion is refused (a live agent would receive 401s permanently).
07:23:03 → 303
## 2026-09-30T07:23:21Z layer 3 AFTER
2026/09/30 09:23:03 [INFO] host deleted: tester-1-693e79 (escrow deleted: true)
2026/09/30 09:23:03 [INFO] self-bind link emailed to the registered address of tester-1
2026/09/30 09:23:03 [INFO] self-bind link (hash ……, valid 7 days) emailed to the registered address of tester-1
2026/09/30 09:23:03 [INFO] self-bind link auto-minted for tester-1 on host delete (the console banner's promised email now exists)
hosts list: /hosts/demo-felhom-8363b5 /hosts/demo-hp-bb76ea /hosts/drill-r50-0a4f9a
host page: 404
customer tester-1 page (kept): 200
documentation/audits/evidence-drill-new-household-2026-09-30/teardown/layer4-ep0.txt
+15
View File
@@ -0,0 +1,15 @@
## 2026-09-30T07:24:05Z layer 4 — ep0, read-only (host deleted 07:23:03)
wg0 10.77.0.2/32
wg0 10.77.0.250/32
wg0 10.77.0.4/32
wg0 10.77.0.3/32
peers=4
demo-felhom
demo-hp
tester-1
2026-09-16T17:27:32Z
2026-09-16T21:59:54Z
2026-09-29T19:37:07Z
owner
2026/09/30 09:18:51 [INFO] wgsync: pushed 5 peers to 167.233.158.164:22
2026/09/30 09:23:51 [INFO] wgsync: pushed 4 peers to 167.233.158.164:22
documentation/backlog/OPEN-ITEMS.md
+1 -1
View File
@@ -757,7 +757,7 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
| **R-597** | **[P2-MEDIUM] The setup code is three Hungarian words, inside an otherwise fully English e-mail, sent to a household the hub knows is English.** FOUND 2026-09-20 by the slice-6 drill. The mail is English end to end (slice 3 working); the code it carries was **`képző-szkítia-ásatás`** — 20 characters, 3 words, **5 of them outside ASCII** (ő, í, á×2, é). An English speaker must copy three words they cannot read, spell or say aloud, and type them into a box on a keyboard that has no ő. They can paste — until the day they read the code to someone over the telephone, which is precisely what a three-word code is FOR. **The same generator feeds the recovery code (10 words) and the owner passphrase (5 words)**, so the fault is one wordlist wide, not one mail wide: this walk saw the passphrase too and it is Hungarian. **Fix shape:** an English wordlist chosen per `customer.language`, with the same word count and the same entropy, and a test that pins BOTH lists' entropy and that no word in either needs a character outside the reader's keyboard. **Not a rename of the existing words** — a second list. **CLOSED 2026-09-21, hub v0.119.0.** **One third of the row was wrong: the recovery code was never Hungarian.** `felhom-agent` mints it (`internal/escrow`) from the **EFF large wordlist** and always has — ten English words, ≈129 bits. The hub does not own that secret and no row was opened for it: a second definition here is the drift `backupTargetAbsentText` already demonstrates across two repos. The two the hub DOES mint now follow the household: setup code 3 hu words (44.6 bits) → **4 en words (51.7)**, owner passphrase 5 hu (74.3) → **6 en (77.5)**, list and count chosen together by `RandomPassphraseFor(lang, use)` so a caller cannot pair an English list with a Hungarian count. **The floor is computed from the embedded lists at test time, not compared with a constant** — red-proofed at 3 English words (38.77 vs 44.56). Hungarian is byte-unchanged, and the list length is pinned so a swap cannot move it quietly. **The task's proposed "read it over the phone" filter was MEASURED and NOT adopted** — it removes 5270 of 7772 words (68%, 12.92 → 11.29 bits/word) and would make this list stricter than the one the product already uses for the code a household writes on paper during a disaster; what it reached for is kept as an assertion (`TestEnglishListIsTranscribable`: 3-9 lower-case ASCII letters, no digit, no separator). Decision recorded in source, **operator may reverse**. Also: **no claim mail ever stated a word count** — the only count wording was the bind page's passphrase hint, whose English half is now count-free. | **CLOSED 2026-09-21 — hub v0.119.0** |
| **R-598** | **[P2-MEDIUM] The Backup page's two protection warnings — the ones that say whether the household's files are safe — are Hungarian on an English dashboard.** FOUND 2026-09-20 by the slice-6 drill on a fresh box, and confirmed on the demo box. Of 73 lines on `/backups` exactly four are Hungarian: **„Csak egy másolat készül (nincs második meghajtó) — a 3-2-1 mentéshez csatlakoztasson egy második meghajtót vagy offsite tárolót"**, **„A rendszermentés jelenleg ugyanazon a lemezen van, mint a rendszer — így hibás fájlok ellen véd, lemezhiba ellen nem"**, and the two backup-target names **„Helyi tároló (local)"** and **„Biztonsági szerver – külön hardver (PBS)"**. They come from `internal/web/backup_handlers.go` (12 Hungarian literals) and `internal/web/backup_target_offer.go` (10) — again composed sentences handed to the page, the R-573/R-596 shape. **It matters more than its line count:** those two warnings are the only place the product tells a household that one copy on one disk is not protection, and the volunteer guide's §9 sends every tester to exactly this page to read exactly these two sentences. **Fix shape:** keys + args for both files, with the retrieval-promise gate run over the English (these sentences are about what a backup does and does not protect). **CLOSED 2026-09-21, controller v0.259.0.** The row's count of `backup_handlers.go` was 12; **nine are code and three are Hungarian inside COMMENTS**. The offer file's ten is right. `degradedMessageFor` now returns a **KEY** — the decision stays language-free and in one place, the words are chosen by the caller that knows the reader — and `buildTierViews` / `backupTargetLabel` / `loadGuestBackup` take the language the way `buildDataPathCards` already did. **The English is asserted to carry the same NEGATION the Hungarian does** ("protects against corrupted files, **but not** against a disk failure"); an English sentence that promised disk-failure protection would be worse than leaving it Hungarian. **Proven LIVE on guest 9201** for the two tier names ("Local storage (felhom-backup)", "Backup server – separate hardware (PBS)"); **the two warnings themselves were NOT walked live** — that box is healthy and a healthy box renders nothing by design, and producing the state would mean un-assigning a live backup target. They are covered by render tests through the real handler in both states. **An apostrophe cost a render:** the first English absent-drive sentence never matched because `html/template` escapes `'` to `&#39;` — caught by the test, not by review. | **CLOSED 2026-09-21 — controller v0.259.0; the two warnings proven by render test, not live** |
| **R-599** | **[P3-LOW] A drill's teardown is blocked for 30 minutes by design, and nothing says so.** FOUND 2026-09-20 tearing the slice-6 drill down. VM destroyed at 17:12Z; the hub then refused **both** `POST /configs/<id>/delete` (409, *host … is ONLINE*) and the host delete (`deletable:false`) — correctly, because an online host would receive permanent 401s. But "online" is not a liveness probe: it is a **report-staleness window**, and the window is **45 minutes** — `manifests/hub.yaml` sets `alerting.stale_threshold: "45m"`, which `hostStatus()` reads (`ok` under it, `stale` over, `down` at 2x). A machine that no longer exists therefore reads ONLINE for three quarters of an hour. **Measured the boring way, and worth recording:** this row first said 30 minutes, because `monitor/host_staleness.go`'s literal default says 30m — the DEPLOYED value is in the manifest, and the 409s kept coming after the half hour was up. Reading a default and calling it the live value is the same mistake in a smaller coat. **The consequence is not theoretical:** a session that destroys its VM and then tears down the hub side walks away believing the delete failed, or leaves the customer behind — and the 2026-09-14 drill's teardown had the same shape without recording this. **Fix shape (smallest first):** the 409 body says *how long* it will refuse ("the last report was N minutes ago; deletion opens at HH:MM"), and `runbooks/target-selection.md`'s drill section names the wait. A force flag is NOT proposed — the refusal is right, only silent about its own clock. | **READY — rank P3-LOW; owner: CC (hub)** |
| **R-600** | **[P2-MEDIUM] "Full teardown" is logged while the deleted box's WireGuard peer is still configured on ep0.** FOUND 2026-09-20 by the slice-6 drill's teardown, **measured on ep0 rather than inferred from the hub**. The customer delete cascade finished at 19:41:54 with `customer DELETE cascade COMPLETE for drill-en-0920 (journal #18) — full teardown`, and every hub-side row was gone (0 configs, 0 hosts, 17 residue rows purged, PBS tenancy deprovisioned, escrow demoted). **Three minutes later `wg show wg0 allowed-ips` on ep0 still listed `10.77.0.5/32`** — the drill box's peer — because `wgsync` pushes on its own cycle. **Watched to the end rather than assumed: the peer was gone by 17:47:46Z — it outlived the *full teardown* line by about 6 minutes.** (My first estimate said ~35, read off the gap between two log lines; the sync runs oftener and only LOGS when something changes. That is the second time in this session that a period inferred from two log lines was wrong — the other was the delete's own staleness window. **A period read off two log lines is not a measurement.**) **The 2026-09-14 drill's findings say "the teardown removes it through the host delete"; measured, the host delete removes the hub's RECORD and the peer goes on the next push.** The mechanism is not broken — it is asynchronous, and the log line claims a completeness it does not yet have. **Why it is P2 rather than P3:** a session that tears down, reads *full teardown*, and leaves is the normal case; the peer outlives it by minutes, and the third teardown layer is the one the workspace rules single out as the one that gets forgotten. Six minutes is short — but the session that reads *full teardown* and leaves has no way to know it is six and not six hours. **Fix shape (smallest first):** the cascade triggers a wgsync push before it logs COMPLETE, or the log line says what is still pending and when ("wg peer removal queued; next push in N min"). A session should not have to read ep0 to know whether a teardown finished. **2026-09-25 (Peti's retirement):** nothing to remove for `peti-felhom-86d37d` — its host was deleted 2026-07-15 and ep0's live `wg show` carries no peer beyond the demo boxes', drill-r50 and the operator OOB (`audits/retire-peti-2026-09-25/A1-ep0-before.txt`); whether that July delete removed a peer, or none existed, is not recorded. **-- 2026-09-28: the Day-0 test install's peer (`drill-g0276`, key `Ly0yjK…`, 10.77.0.5) was checked on ep0 the day after its customer DELETE: gone from `wg show`, absent from `/etc/wireguard/*.conf` (control: a live key found), no namespace or file names the customer — the hub's sync removed it; nothing was removed by hand.** `audits/logins-nvme-2026-09-28/E/`. | **READY - rank P2-MEDIUM; owner: CC (hub)** |
| **R-600** | **[P2-MEDIUM] "Full teardown" is logged while the deleted box's WireGuard peer is still configured on ep0.** FOUND 2026-09-20 by the slice-6 drill's teardown, **measured on ep0 rather than inferred from the hub**. The customer delete cascade finished at 19:41:54 with `customer DELETE cascade COMPLETE for drill-en-0920 (journal #18) — full teardown`, and every hub-side row was gone (0 configs, 0 hosts, 17 residue rows purged, PBS tenancy deprovisioned, escrow demoted). **Three minutes later `wg show wg0 allowed-ips` on ep0 still listed `10.77.0.5/32`** — the drill box's peer — because `wgsync` pushes on its own cycle. **Watched to the end rather than assumed: the peer was gone by 17:47:46Z — it outlived the *full teardown* line by about 6 minutes.** (My first estimate said ~35, read off the gap between two log lines; the sync runs oftener and only LOGS when something changes. That is the second time in this session that a period inferred from two log lines was wrong — the other was the delete's own staleness window. **A period read off two log lines is not a measurement.**) **The 2026-09-14 drill's findings say "the teardown removes it through the host delete"; measured, the host delete removes the hub's RECORD and the peer goes on the next push.** The mechanism is not broken — it is asynchronous, and the log line claims a completeness it does not yet have. **Why it is P2 rather than P3:** a session that tears down, reads *full teardown*, and leaves is the normal case; the peer outlives it by minutes, and the third teardown layer is the one the workspace rules single out as the one that gets forgotten. Six minutes is short — but the session that reads *full teardown* and leaves has no way to know it is six and not six hours. **Fix shape (smallest first):** the cascade triggers a wgsync push before it logs COMPLETE, or the log line says what is still pending and when ("wg peer removal queued; next push in N min"). A session should not have to read ep0 to know whether a teardown finished. **2026-09-25 (Peti's retirement):** nothing to remove for `peti-felhom-86d37d` — its host was deleted 2026-07-15 and ep0's live `wg show` carries no peer beyond the demo boxes', drill-r50 and the operator OOB (`audits/retire-peti-2026-09-25/A1-ep0-before.txt`); whether that July delete removed a peer, or none existed, is not recorded. **-- 2026-09-28: the Day-0 test install's peer (`drill-g0276`, key `Ly0yjK…`, 10.77.0.5) was checked on ep0 the day after its customer DELETE: gone from `wg show`, absent from `/etc/wireguard/*.conf` (control: a live key found), no namespace or file names the customer — the hub's sync removed it; nothing was removed by hand.** `audits/logins-nvme-2026-09-28/E/`. **MEASURED AGAIN 2026-09-30 (a HOST delete, not a customer delete; new-household drill):** the host was deleted 07:23:03Z and `wgsync: pushed 4 peers` at 07:23:51Z — `10.77.0.5/32` gone from ep0 48 s later, the other four peers unchanged (`audits/evidence-drill-new-household-2026-09-30/teardown/layer4-ep0.txt`). | **READY - rank P2-MEDIUM; owner: CC (hub)** |
| **R-601** | **[P2-MEDIUM] ~~demo-hp is unreachable~~ — WRONG, WITHDRAWN THE SAME DAY. The box was never down; MY ROUTES WERE.** Filed 2026-09-21 morning after `ssh demo-hp`, the hub-vaulted break-glass over the tailnet, `demo-hp-lan`, a ping and `ip neigh` on felhom-pve all failed, and `tailscale status` said *`demo-hp … offline, last seen 30d ago`*. **The operator looked at the hub and said it was ONLINE. It was**: it had reported 13 minutes earlier, and it has been up **4 weeks 2 days**. **Two stale facts, each enough on its own:** (1) `~/.ssh/config` sends `demo-hp` to the tailnet address `100.76.96.79`, and **tailscale is not installed on that box at all** (checked on it: no `tailscaled`, no `tailscale` binary) — so that entry is a dead peer from an earlier build and can never answer; (2) `demo-hp-lan` and `nodes.md` both say `192.168.0.87`, and the box is **statically** on **`192.168.0.104/24`**, bridge-port `nic0` (nodes.md says `enp2s0f0`). **The hub knew the right address the whole time** — every host report carries `addresses: [{iface: vmbr0, cidr: 192.168.0.104/24}, …]`. **What I actually did wrong, and it is the part worth keeping:** I ran `ip neigh` on felhom-pve, and `192.168.0.104 … STALE` was *in that output*, four lines above the `192.168.0.87 … FAILED` I quoted. I searched the output for the address I expected instead of reading it for the address that was there. **The standing rule says a "no access" claim must list what was tried; it does not say the list makes the claim true.** Six failed routes to a stale address are six failures of one assumption, not six pieces of evidence. **FIXED:** both `~/.ssh/config` entries repointed to `.104` (each carrying a comment saying why, including that there is no tailscale on this box), both verified live; `nodes.md` corrected. | **CLOSED 2026-09-21 — withdrawn, the claim was false; the routes are fixed** |
| **R-604** | **[P2-MEDIUM] A per-customer controller floor silently excludes that box from every global floor raise, and NOTHING says so — demo-hp missed four of them.** FOUND 2026-09-21 while raising the global floor to 0.259.0 at the operator's request. The raise logged `Global controller-version floor set to "0.259.0"` and then `managed floor SERVED for demo-felhom` — **and nothing at all for demo-hp**, which went on reporting every few minutes and stayed on 0.258.0. Cause: `customer_configs.min_controller_version` for demo-hp held **`0.243.0`**, a per-customer override that wins over the global. It is a **leftover from the 2026-09-16 drill**, whose golden was 0.243.0; **R-343 measured on 2026-08-18 that all five rows were EMPTY and recorded that as a safety property** — it stopped being true and nothing surfaced the change. demo-hp had therefore silently missed the raises to 0.253.0, 0.254.0, 0.257.0 and 0.259.0. **Why it is invisible rather than merely quiet:** `managed floor SERVED` fires **once per CHANGE** (`h.floorNotes`, `api/handler.go:600`), deliberately, because a box reports every few minutes — so a box whose override never changes is silent for ever, and its silence is indistinguishable from the silence of a box that already had the line. A session that raises the floor reads one SERVED line and reasonably concludes the fleet took it. **CLEARED** for demo-hp the same session (rollback line: POST `/customers/demo-hp/floor` with `min_controller_version=0.243.0`, `min_agent=0.131.0`); it then self-updated 0.258.0 → 0.259.0 in **under four minutes**, healthy, `settle-gate: GO — at/above floor 0.259.0`, and its claim page answers **"Wrong or expired code"** in English — the floor delivered the FIX, not a version string, to a box nobody hand-deployed. All five overrides are now empty. **Fix shape (smallest first):** the floor-raise page shows which customers carry an override and would NOT be moved, before the save; or the raise logs one line per customer naming the ones it skipped and why. A raise that quietly reaches half the fleet is worse than one that refuses. | **READY — rank P2-MEDIUM; owner: CC (hub)** |
| **R-605** | **[P3-LOW] A catalog gate that REFUSED TO RUN and a gate that ran and could not decide print the same word, so a reader cannot tell which happened.** FOUND 2026-09-21 while answering why the chaos night's update round could not run. On 2026-09-17 `check-image-resolvable` and `check-volume-persistence` both returned INCONCLUSIVE and the drawn `update` action was replaced with `use` (`audits/DRILL-chaos-night-2026-09-17.md:692-695`). **Neither script is defective — they behaved exactly as designed**, and both headers say why: a detector that cannot prove itself must refuse to report rather than guess (`check-image-resolvable.py` cites the 2026-07-21 incident where a Docker Hub throttle read as 24 of 65 pins falsely dead). **What is missing is the DISTINCTION.** `check-volume-persistence.py`'s `self_test` refuses to evaluate ANY app when it cannot build its canary image — a harness-level refusal — while `classify()` returns a per-app UNDETERMINED for an app that wrote nothing; `check-image-resolvable.py` likewise separates a harness-level canary failure (exit 2 at `check()` L180-183) from a per-pin throttle (L121-128). **`catalog_gates.py`'s VERDICT map collapses all of them into one `INCONCLUSIVE` label**, so the operator-facing summary cannot say whether the gate ran at all. **The cost is real and already paid:** no raw stdout of the 2026-09-17 run survives in either evidence directory, so the exact triggering path is INFERRED from the code plus the documented throttle precedent, not observed — a distinct summary line would have recorded it for free. **Fix shape:** have each gate's exit distinguish "the harness refused" from "the result is undetermined" (a third exit code, or a marker line the runner matches), and have `catalog_gates.py` print the two differently. **Ships with a decoy each way (R-421): a run whose canary fails must NOT read as a per-app undetermined, and vice versa.** Small. | **READY — rank P3-LOW; owner: CC (catalog)** |