diff --git a/REPORT.md b/REPORT.md index 3df55ae..0c682ef 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,86 +1,87 @@ -# REPORT — the record, the second promise, the removal chain (2026-08-12) +# REPORT — DRILL: the retained key, and the two fixes nobody had watched work (2026-08-12) -Commits `f76cbf0` (Part 0), `2389544` (spec + bake evidence), `125aec1` (R-300), `890a474` (STATUS). -Controller: `68f3e12` (R-299). All gates green in both repos; **`--no-verify` used nowhere** — the -currency gate refused a push mid-session and stayed refused until the bake made it true. +**Class:** drill (unattended, destructive on Tier 0) + spike for Phase C's first step +**Venues:** `drill-r50` (nested PVE on DooPlex), `demo-felhom` (guest 9201) — both Tier 0. +**`demo-hp` was never touched. `peti-felhom` was never contacted. No abandon countdown was started, +shortened or triggered.** +**Full record:** `documentation/audits/DRILL-retained-key-2026-08-12.md` -## Part 0 — the record now says what is true +--- -Re-read from the hub's own store, with a control (escrow query returns 1+1 for each demo box, 0+0 for -`drill-r50`). Confirmed: **3 hosts total, no peti row under any name**; 4 escrow rows hub-wide, all -demo; `host_deletions` id=1 `peti-felhom-86d37d` **2026-07-15 08:56:22**, `escrow_acked = 0`; last -report **2026-07-15 08:39:00 UTC**, controller 0.115.0, `offsite: {escrow_state: "pending", -snapshot_count: 0}`; local app-data repo empty; dashboard never claimed. The `PETI` row and `STATUS.md` -now state that the mitigation they named **does not exist** — and leave the parked/not-parked ruling -open, because that is the operator's and the fact does not need his opinion to be true. +## The answer to the question this drill existed to answer -**Contact since the deletion — answered without touching the machine.** No inbound row of any kind -after 2026-07-15 08:39; the only later rows are the hub's OWN alarms (`source = hub`: `node_stale` -09:09:32, `node_down` 09:39:32). No contact attempt, accepted or rejected, in the current hub pod's -logs (since 2026-08-09 17:26Z) — grep proven by **851 `demo-hp` hits against 0 for peti, 0 -unauthorized**. **The window 2026-07-15 → 2026-08-09 cannot be answered from records**: a report from a -deleted host 401s and is not persisted, and those logs are gone. +**(a) Is the old key kept? YES** — proven for the first time in the fleet's history. +**(b) Does the kept key open the old backups? YES** — three planted files, including a Hungarian +accented filename verified as raw bytes, restored **byte-identical** from a store the machine itself +could no longer open. +**(c) Can the customer get there through the product? NO — and they are told their correct code is +wrong.** -## Part 1/2 — the second promise (controller v0.212.0) +The brief said to be ready for the answer to be no, and our own records predicted retention would be +*"a box we fill and cannot open"*. **That was half right, and the wrong half was the one nobody had +checked.** The box opens. What does not exist is the door: `ListSupersededEscrow` +(`hub/internal/store/store.go:2841`) is the only reader of a retained key and has **zero production +callers**; the recovery path selects `FROM host_escrow` — the current row only. Asked with the very +code that had just opened the retained row by hand, the product answered *"the recovery code did not +open the sealed bundle — nothing was written"*. → **R-304, rank 1** -`backups_remote.html` line 98 — the **always-visible** half — still ended *„…visszaállíthatók -lehetnek"*. Replaced; the two accurate halves kept. +Consequences: the census answer **stands**; the countdown banner's promise is **true in substance, +false in practice**; the capability map's recovery claim **has been moved** with today's evidence. -**Why it survived, which is the useful part:** the spec called that line *"Accurate; keep"*, **and the -guard matched one INFLECTION** (`visszaállítható lehet`) that the plural does not contain. Guard -broadened to the stem `visszaállíthat`. Spec corrected in both places. +## What shipped -**Plant → convict → remove → pass:** planted the exact shipped plural → the stem guard **FAILED** and -quoted it back; the old singular guard **does not match that sentence at all** (`False`, shown as a -pure string fact, not a contaminated source grep); removed → 5/5 orphan-card tests pass. +**`installer-v1.27.0` published** — tag cut and **both** `--ref`s in `manifests/webpage.yaml` bumped +(sidecar line 327, init container line 372). Publication was earned: both faults were watched +happening first, from a machine reset to factory state. -Two instrument defects fixed on the way: the guard's failure message sliced rendered HTML at a **byte** -offset and cut Hungarian mid-character (now rune-safe); and a first pass at the bake's acceptance -markers returned a false `0` through shell quoting — re-run with `grep -F`, because a zero from a -broken instrument is not a measurement. +- **R-300 CLOSED** — pre-fix uninstall left dnsmasq `enabled`/`active` on `0.0.0.0:53`; the next byo + install refused, exit 1. Fixed path: recorded `not present before Felhom` → `stopping + disabling + it` → `:53 FREE` → preflight PASS. The owner's side proven too (record `yes` → left running). +- **R-297 CLOSED** — a stale `golden-0.98.3.tar.zst` planted as newest-by-filename; v1.25.0 took it + with no comparison and **the box came up on controller 0.98.3** against a vouched 0.213.0 — below + the floor and below v0.206.0 where the off-site recovery screen exists. Fixed path re-fetched and + sha-verified the vouched golden (landed 0.213.0); an operator-named stale archive was **refused**. -## Part 3 — the removal chain (R-300), CODE ONLY +## Findings opened — ceiling R-303 → R-310 -Confirmed at source: uninstall removes the snippet and **restarts** (`:1075-1082`), leaving the unit -enabled; the byo preflight then refuses on `:53`. +| # | Rank | What | +|---|---|---| +| **R-304** | **1** | Retained key has no product route; the correct old code is reported as wrong | +| **R-305** | 2 | The R-300 cleanup fires **once per machine** — the leftover returns on the second reinstall (proven, cycles 2/3) | +| **R-308** | 2 | Stored controller `PASSWORD` no longer opens demo-felhom (`Hibás jelszó`) — not the quoting trap | +| **R-306** | 3 | `--preflight-only` says *"no state written"* and writes `state.json` — with an ownership answer that can be wrong | +| **R-309** | 3 | The day-0 runbook says pushing publishes the installer; false since R-110 (measured: public URL served 1.25.0 while `main` had 1.27.0) | +| **R-310** | 4 | Duplicated sentence in the golden refusal; `--uninstall` needs a pty and `--force` does not bypass it | +| **R-307** | — | **Operator decision, deadline 2026-08-24** — see below | -**The prompt's framing needed one correction:** the installer does **not** install dnsmasq — the -**agent** does (`lanresolver.go:107`), conditionally, and `felhom-agent` is fenced this session. So -ownership is recorded at **preflight**, before anything is installed, which is the only moment it is a -fact — not a package mtime. At removal: Felhom's → stop+disable; the owner's → restart only; **no -record (every box in the field) → restart only, fail-safe, with the reason and the command logged.** -The refusal keeps its two routes and its promise, and gains the missing line naming our own leftover. +## What needs you -**NOT OBSERVED LIVE.** The `drill-r50` install→uninstall→install cycle was not run, so the wrong -outcome was never quoted and **no `installer-v1.27.0` tag is cut.** +**`demo-felhom` carries a live abandon countdown** — started 2026-08-10, **firing 2026-08-24**, for +`/home/felhom-repo.orphaned-20260810`. This drill did **not** start it and deliberately did **not** +cancel it. The brief's end state asked for no countdown anywhere; satisfying that means choosing: +**cancel it** (copy kept indefinitely, storage cost, no data risk) or **let it run** (copy deleted, +irreversibly). **Doing nothing selects deletion.** → **R-307** -## Golden 0.212.0 — baked, published, round-trip verified +## End state -`4b0a7dacc503c38732ed0a44949398639248c7fbd90758a1e4a047c21a7a15d8`, 656 611 277 B, served bytes -re-downloaded and hashed identical. All six markers counted with `grep -F`. Token never on a command -line; leak grep 0, believable because a planted-token control grepped 1. Drill VM reverted to `virgin`. -Evidence: `documentation/tests/golden-0.212.0-2026-08-12/`. +- **`demo-felhom`** — up, reporting, healthy, on the vouched pair; `repo_password` restored to the + original (`sha c60c8bc737a6b7c6…`), escrow re-sealed and uploaded, off-site repo reachable + (`restic snapshots` exit 0). Its recovery code was rotated by the final ceremony and + `R_DEMO-FELHOM` updated in place (prior file backed up alongside). Planted data removed; eight + secret-bearing files **shredded**. +- **`demo-hp`** — untouched, reporting. +- **`drill-r50`** — **reverted to snapshot `virgin`, powered off.** +- **Hub** — two new retained rows (the P1 and P2 blobs), deliberately kept as the fixture proving the + retention works. `drill-r50-0a4f9a` re-used, not duplicated: no new scratch customer. +- **Off-site** — only demo-felhom's own repository path touched, `backup` the only mutating verb used. + **No prune, no forget, no delete, no rename anywhere.** One snapshot added and deliberately left: + `6ea85413`, 66 KiB, tagged `drill-retained-key-20260812` — removable by ID if you want it gone. -## DROPPED — named plainly +## Honest gaps -- **The hub half of the naming (R-295)** — dropped first, exactly as the drop order allows. The four - hub surfaces were NOT enumerated at `file:line`; that enumeration is still owed. -- **The stale-golden observation (R-297)** — dropped second. Nothing published, which is the safe state. -- **And one that was NOT droppable: R-300's live cycle.** The code shipped; the demonstration did not. - Named here rather than shortened silently. - -## Observations, not acted on - -- **R-301** — the abandon countdown banner (`layout.html:143`) states the retired promise a third time - and un-hedged. It is probably TRUE where it renders, and it renders on every page; a rebuilt box can - have an active countdown while its store is orphaned. Not established: whether the two sentences name - the same bytes. Left alone deliberately — this session was fenced to the orphan card. -- The orphan card now says "we cannot determine / it depends on the key / write to us" **twice** once a - customer clicks through — reinforcement at the decision point rather than a contradiction, but worth - an eye if the card is revisited. - -## Deliberately out of scope - -The CI runs that fail with no log; the twenty facts the machines report that nothing reads; the nine -grey claims; the storage page's separate empty-list cause (R-298); and proving a *retained* key can -actually open an old store — the one thing the retention fix has never been shown to do. +- **The Phase A logs did not survive** the intermediate revert to `virgin`. Every quotation in the + audit is verbatim from the live run, but the raw files are gone. Procedural lesson, recorded. +- The planted data reached the store via `restic` directly, not the dashboard button, because of + R-308 — so the app-backup→unit→offsite chain went unexercised. Not what this drill measured. +- Wall clock **≈ 1 h 13 min** against a 4–5 h envelope. Nothing was dropped; Phase C ran concurrently + with Phase B on a different machine. diff --git a/STATUS.md b/STATUS.md index 221fb70..a3871c2 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,6 +1,6 @@ # STATUS — what works, what's broken, what's next -**Updated 2026-08-12 (evening).** +**Updated 2026-08-12 (late evening — retained-key drill).** > **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority; this page restates > part of it in plain words, and **nothing may exist only here**. **Items, not paragraphs. One screen.** @@ -8,12 +8,15 @@ ## Waiting on you -- **Vouch golden 0.213.0** — three fields together: golden **0.213.0**, agent **0.128.0**, min agent - **0.127.0**. Both are already verified downloadable and selectable. -- **Raise the floor to 0.213.0** afterwards if you want the fix on the existing machines. The floor is - **0.212.0** (raised 2026-08-12; both demo boxes confirmed on it), and boxes update to the *floor*, - never to the newest — so until you raise it, v0.213.0 reaches newly installed machines only. One - field, reversible; the R-216 hold does not apply, since min agent 0.127.0 is below the vouched 0.128.0. +*(The golden-vouch and floor-raise asks that stood here are **done** — the hub reads golden **0.213.0**, +floor **0.213.0**, agent **0.128.0**. Checked live, not assumed.)* + +- **R-307 — a countdown is already running on `demo-felhom`**, started 10 Aug, **firing 24 August**, + after which that set-aside off-site copy is deleted for good. Today's drill did not start it and + deliberately did not cancel it. Cancel it (copy kept, costs storage) or let it run (copy deleted). + **Doing nothing chooses deletion.** +- **R-304 — decide what we promise about old backups.** The retention works, but only an operator can + use it; the customer is told their correct code is wrong. Either build the door, or stop promising. - **R-303 — one coherence decision**, ranked low: a box can still raise the "we cannot open your new backups" card while counting down to deleting the old ones. The two no longer contradict each other, but the state is odd and the wrong fix would hide a real second fault. @@ -36,6 +39,9 @@ Both demo machines are home, healthy and reporting on the approved pair. Off-sit fifth deliberately left, because it is true where it renders. - **One name per secret, box side** (R-295): the dashboard code is „Beállító kód" everywhere; „Visszaállító kód" is retired. It collided with the escrow „Helyreállítási kód" and cost a real code. +- **Both installer fixes are now PUBLISHED** as `installer-v1.27.0` (R-297 + R-300). Each fault was + watched happening first, on a machine reset to factory state: the old installer really did build a + machine on a base image from July, and our own uninstall really did block our own next install. ## Broken, or knowingly incomplete @@ -44,18 +50,19 @@ Both demo machines are home, healthy and reporting on the approved pair. Off-sit everything on it is lost.** First act of the visit: copy the ~3.6 GB off before anything is reinstalled — it is currently the only copy in existence. Whether it stays parked is your call and is deliberately left open. -- **Two installer fixes are written but NOT published** — pushing publishes nothing, and no tag is cut: - - **R-297** — an install could silently reuse an old base image lying on the machine, including one - too old to run the recovery screen. - - **R-300** — our own uninstall left `dnsmasq` holding `:53`, so our own next install refused and - blamed the household's network. - Both are unpublished for the same reason: **neither fault has been watched happening.** One session on - `drill-r50` covers both, and that is the right order. +- **Kept backups can be opened — but not by the customer** (R-304). We now know for certain the old key + is kept and **does** open the old backups: three planted files came back byte-for-byte from a store the + machine itself could no longer read. But nothing in the product reads a kept key, so it takes an + operator with a shell. Worse, a customer who types their **correct** old code is told it *"did not open + the sealed bundle"*. The keeping is real; the promise that they can use it is not. +- **The dnsmasq fix helps a machine once** (R-305). On a machine that never had Felhom it works. On the + second reinstall the leftover comes back, because the package is never removed — so the machine looks, + to our own installer, as if the household had installed it. Watched happening the same afternoon. - **The hub half of the naming is undone** (R-295 PARTIAL): the emails still use the retired name and send people to a page a rebuilt machine does not show. - **The storage page has its own separate reason for showing an empty list** (R-298), untouched. ## Working on next -The `drill-r50` session that unblocks both installer fixes; then the hub naming; then the -2026-08-09 batch (R-279 … R-292) which is still untriaged against everything since. +Your two decisions above (R-307's deadline is the hard one); then R-304's shape; then the hub naming; +then the 2026-08-09 batch (R-279 … R-292), still untriaged against everything since. diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 00c072f..f3b4eb7 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -36,7 +36,7 @@ | Box survives a wrong-NIC install: hub-unreachable first boot → legible Hungarian console screen (NIC table + remedy) + NIC sweep self-heal (bounded DHCP + hub probe per NIC, success-only persist), and the baked root password is operator-knowable (`.rootpw.txt`) | scripts v1.24.0 (`scripts/iso/felhom-bootstrap.sh` `network_gate`/`sweep_nics`, `build-felhom-iso.sh` rootpw emission) | **PROVEN-LIVE (nested drill — nested ≠ metal: metal proof rides the next real multi-NIC install)** | `audits/SPIKE-firstboot-nic-sweep-2026-07-22.md` — dead-NIC install from the virgin v1.24.0 ISO baked the 192.168.100.2 fallback (WITH a dead default gateway), the R-59 screen painted on the console (screendump captured), and after the cable move the box swept to the working NIC, re-leased and **self-registered at the hub unaided in under a minute**; the drill also caught + fixed the stale-fallback-route trap (flush before the bounded dhclient) and verified the emitted rootpw against the installed box's shadow hash | R-59 ships as a first-boot gate, not an install-time abort (recorded deviation — the fallback is the auto-installer's own, initrd hook out of scope); sweep is structurally first-boot-only (`state.json` gate + unit done-flag condition); a box past install-start gets the screen but its interfaces are never touched | | Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** (drill VM) | `DRILL-day0-vm-2026-07-12` §10/F-4 (gate ON via real edge; claimed, code consumed) | Never executed by a non-Viktor human → R-3. **Deliverability (R-4), gmail half DONE 2026-07-18:** the rehearsal's claim email was the first sent under the tightened DMARC `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu remains Viktor's open half.** (Dropped mis-cited `CAMPAIGN-4` F-C — that is the escrow-claim 502, not password claim) | | Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** (real customer-zero bind on metal, 2026-07-18) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md`:** operator minted + emailed the link 16:28:55 (7-day TTL, expiry 2026-07-25 recorded); **the customer bound their own box at 16:29:55 with `attempts=0`, `locked=0`** — `appliance_bound` carries source **`customer_selfbind`**, and the credential was delivered **26 s later** with no operator action. Hub-side lifecycle in `hub-state.txt` (`selfbind_tokens` mint→email→consume). Prior unit evidence: hub v0.66.0 (`web/selfbind.go`, `store/selfbind.go`; Scenarios A–F + F1/F2; 4 red-proofs verified red — THE TRAP `/bind/` exemption, no-oracle, lockout, single-active); GC verdict §3 (no appliance GC → TTL stands alone) | R-27 **slice 1**. No appliance list ever rendered; wrong code == wrong passphrase (one generic failure); 5-attempt lockout → call support; expiry falls back to operator-bind. **Live first-run DONE 2026-07-18** (rehearsal; the console banner rendered on the real ISO). **R-27b** (controller second-box dismissable prompt) deferred; **multi-box-per-link** = repeated operator sends | -| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention) — ⚠ the claim as written was FALSE for the offsite tier for two months; **FIXED in hub v0.93.0 (2026-08-04), and the row below states what ships TODAY:** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`) — and **since v0.93.0 the retained row carries `identity_blob` as well as the K-escrow blob**, so a previous passphrase does now stay recoverable with the recovery code that sealed it. **What was wrong until v0.93.0, recorded because it is the ninth entry in `CLAUDE.md`'s comment-vs-code table and the first that was also customer-facing copy:** `host_escrow_superseded` had no `identity_blob` column and `demoteCurrentEscrowTx` did not copy one, so what survived a supersession was the **PBS datastore key only** — never the restic repository password, which lives in `identity_blob`. The destroying act was the **escrow ceremony** a rebuilt box asks its customer to run. Measured live 2026-08-04, before the fix: both current rows held `blob`=383 B **and** `identity_blob`=572 B; both retained rows held `blob`=383 B only. → **R-198** (SHIPPED), evidence `audits/RECON-offsite-dr-chain-2026-08-04.md` §7. **THREE SCOPE LIMITS THIS ROW MUST NOT BE READ PAST.** (1) **Nothing was backfilled and nothing could be** — rows superseded before v0.93.0 were written without the blob and their source rows are already overwritten; both demo boxes' pre-2026-08-04 repository passwords are gone permanently. (2) **A retained key is not a restore — but as of 2026-08-04 evening it IS a recovered key.** See the row below. (3) The customer-facing orphan card still promises recoverability unconditionally (R-202, gate hit, card untouched). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test | +| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention) — ⚠ the claim as written was FALSE for the offsite tier for two months; **FIXED in hub v0.93.0 (2026-08-04), and the row below states what ships TODAY:** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`) — and **since v0.93.0 the retained row carries `identity_blob` as well as the K-escrow blob**, so a previous passphrase does now stay recoverable with the recovery code that sealed it — **OPERATOR-ONLY, and the customer-facing half of that sentence is FALSE (R-304, drill 2026-08-12).** The retention was exercised end-to-end for the first time that day and **it works**: the retained row carried the identity blob byte-identically (`sha256 a10032341c8584ed…`), the old recovery code unsealed it, and three planted files — including a Hungarian accented filename verified as raw bytes — restored **byte-identical** from a store the box itself could no longer open (negative control first: `Fatal: wrong password or no key found`). **What does not exist is the door.** `ListSupersededEscrow` (`hub/internal/store/store.go:2841`) is the only reader of a retained `identity_blob` and has **zero production callers**; the product's recovery path (`POST /escrow/recover-offsite-password` → `GetHostDRBundle`, `store.go:3152`) selects `FROM host_escrow` — the CURRENT row only. Asked with the code that demonstrably opens the retained row, the product answers **"the recovery code did not open the sealed bundle"**. So: recoverable by an operator with SQLite, `age` and a shell; **not recoverable by the customer, who is told their correct code is wrong.** **What was wrong until v0.93.0, recorded because it is the ninth entry in `CLAUDE.md`'s comment-vs-code table and the first that was also customer-facing copy:** `host_escrow_superseded` had no `identity_blob` column and `demoteCurrentEscrowTx` did not copy one, so what survived a supersession was the **PBS datastore key only** — never the restic repository password, which lives in `identity_blob`. The destroying act was the **escrow ceremony** a rebuilt box asks its customer to run. Measured live 2026-08-04, before the fix: both current rows held `blob`=383 B **and** `identity_blob`=572 B; both retained rows held `blob`=383 B only. → **R-198** (SHIPPED), evidence `audits/RECON-offsite-dr-chain-2026-08-04.md` §7. **THREE SCOPE LIMITS THIS ROW MUST NOT BE READ PAST.** (1) **Nothing was backfilled and nothing could be** — rows superseded before v0.93.0 were written without the blob and their source rows are already overwritten; both demo boxes' pre-2026-08-04 repository passwords are gone permanently. (2) **A retained key is not a restore — but as of 2026-08-04 evening it IS a recovered key.** See the row below. (3) The customer-facing orphan card still promises recoverability unconditionally (R-202, gate hit, card untouched). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test | | **The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code** | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE (2026-08-04)** | On demo-felhom, through the real endpoints end to end: the box fetched its own sealed blob from the hub with its own per-host credential (hub log: *escrow blob SERVED … 572 opaque bytes, self_scope=true*), the agent unsealed it with the customer's recovery code, and the extracted repository password's sha256 was **byte-identical** to the one on disk — `c60c8bc737a6…`, which is ALSO the hash the hub had independently stored, so three sources agree. Five minutes earlier the same path with a WRONG code failed closed at age's KDF with nothing written, which proves links 6 and 7 ran independently of the success. R was searched for afterwards and found in 0 log lines and 0 files, with a positive control confirming the search would have found it. Evidence: per-repo CHANGELOGs; `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 links 6–8 | **WHAT THIS ROW DOES NOT CLAIM, stated because the previous over-claim here was struck out four hours earlier.** It covers the KEY, not the DATA. **R-199 BACK-POINTER (omitted when this row was written): the recovery chain's link inventory and the per-link status live in `audits/RECON-offsite-dr-chain-2026-08-04.md` §3; links 1–8 are walked, 9–11 are not.** **Re-confirmed 2026-08-04 evening by an attempt to prove the DATA half:** the R-201 drill was prepared on demo-hp and **halted before the wipe** — the sentinel file was not in the off-site snapshot (R-203), so the wipe would have destroyed it and proven nothing. **No file has still ever been restored from an off-site backup after a wipe** (`audits/DRILL-r201-offsite-recovery-2026-08-04.md`). The install half of the chain (controller v0.196.0 `--recover-offsite-install`, R-200) is likewise unit-proven only — it has never run against a live recovery. A recovered password has never been **installed** (the diagnostic compares and refuses to write, by design), no existing repository has ever been **reopened** under one, and **no file has ever been restored** from an off-site history via a recovered key. Links 9–11 of the chain are open (R-200's remaining half, R-201). The proof also used a box whose local key still exists — the rebuilt-box case, where there is nothing to compare against, is exactly what the drill covers and it has not run || DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction); and `pbs.ProbeAuth` turns a 401 into a loud `auth_failed` that the existing `pbsdrheal` damper escalates to a fresh mint. Plus a consumed_at honesty gauge for the disagreement no single tier can see (box says `applied`, hub's staged secret never consumed). Proven live on felhom-pve: the agent read its credential through the wrapper (`rc=0`) and probed successfully (`credential probe OK storage=felhom-pbs`). **STOP-2 RAN 2026-07-21 AND THE CHAIN CLOSED — 13 SECONDS, operator click to converged.** The operator pressed **Re-issue PBS credentials**; the identical click on 2026-07-18 did nothing at all. Full chain (hub UTC / host CEST = UTC+2): `08:39:31Z` hub mints a fresh secret, **generation 0 → 1**, and the descriptor gains `"secret_generation": 1` — with `token_id` and `fingerprint` **byte-identical**, i.e. exactly the re-key shape that used to be invisible → `10:39:34` the agent READS its credential through the wrapper (leg b — the read that was impossible until v0.91.0) → `10:39:38` **`ERROR pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD` `previous_state=applied`** (leg c: the exact R-39 failure state, detected out loud for the first time ever) → `10:39:45` **`one-time token secret consumed`** `secret_len=36` (leg a: **NO short-circuit** — this is the line that never appeared on 2026-07-18) → `10:39:45` `felhom-pbs-apply reconcile` (the set-only wrapper, no `--server`) → `10:39:47` **`pbsdr: converged state=applied`**. Corroboration: the agent marker hash moved to `afbb3b41…` (it was byte-identical to the pre-reissue marker in the failure); `consumed_at` stamped `08:39:45Z`; the on-disk secret's mtime moved `2026-07-18 20:28:52` → `2026-07-21 10:39:45`; a live probe with the NEW credential returns **200**; three consecutive hub reports trace the whole state machine `applied → auth_failed → applied`; and **zero** `pbsdr_selfheal` escalations fired — the box healed through the descriptor path before the damper was ever needed, with exactly ONE mint and ONE consume and no `consumed-failed.json`. **Row upgraded to PROVEN-LIVE (2026-07-21).** Evidence: `felhom-agent/REPORT.md` (2026-07-21). | diff --git a/documentation/audits/DRILL-retained-key-2026-08-12.md b/documentation/audits/DRILL-retained-key-2026-08-12.md new file mode 100644 index 0000000..d573860 --- /dev/null +++ b/documentation/audits/DRILL-retained-key-2026-08-12.md @@ -0,0 +1,514 @@ +# DRILL — the retained key, and the two fixes nobody had watched work + +**Date:** 2026-08-12 · **Class:** drill (unattended, destructive on Tier 0; spike for Phase C's first step) +**Venues:** `drill-r50` (nested PVE on DooPlex) · `demo-felhom` (guest 9201 on felhom-pve) — both Tier 0 +**Baselines re-confirmed live on arrival, not assumed:** controller **0.213.0** on both demo boxes +(`pct exec 9201 -- docker ps`), agent **0.128.0** (hub `hosts`), hub **0.102.0** (deployment image), +register ceiling **R-303** (grep), `felhom.eu` clean at `fbe1155f` and level with `origin/main`. + +--- + +## 1. The three verdicts, first, in plain language + +> **(a) Is the material retained? — YES.** Proven for the first time in the fleet's history. +> +> **(b) Does the retained material, once unsealed, yield a password that actually opens the old +> store? — YES.** Three planted files came back byte-identical from a store the box itself could no +> longer open, including the Hungarian accented filename. +> +> **(c) Does the customer have a route to (b) through the product? — NO. And the failure is worse +> than absence:** the customer's old recovery code — the one that demonstrably opens their old +> backups — is rejected by the product with *"the recovery code did not open the sealed bundle"*. +> They are told their correct code is wrong, and given no next step. + +**The headline the operator asked for.** The brief said to be ready for the answer to be no, and our +own records predicted the retention would be *"a box we fill and cannot open"*. That prediction was +**half right, and the wrong half was the one nobody checked.** The box can be opened — the crypto, +the retention and the copy are all sound. What does not exist is the door: no code path reads a +retained row, so the only way in is an operator with SQLite, `age` and a shell. Everything I did in +§4 a customer cannot do. + +That distinction matters for the three pieces of work resting on this: + +| Rests on | Verdict | +|---|---| +| The census answer ("nobody else is exposed") | **Stands.** It was a statement about retention, and retention works. | +| The countdown banner's promise (old backups stay recoverable) | **True in substance, false in practice.** The bytes are recoverable; the customer cannot recover them. The promise is made to someone who has no way to act on it. | +| The capability map's recovery claims | **Must move.** Any claim of the form "the customer can recover the old history with their recovery code" is false today. | + +--- + +## 2. §7.0 — the probe, run first and read-only + +Run before anything was installed or planted, exactly as the order required. + +**Q1 — does any host today hold a superseded record with key material present? NO. Zero, fleet-wide.** + +``` +host_id kfp16 blob_len identity_blob superseded_at +demo-hp-bb76ea 3f:4f:65:c0:d8:f 383 NULL 2026-08-04 07:15:36 +demo-felhom-8363b5 7e:a6:af:f7:ea:6 383 NULL 2026-08-04 07:20:08 +``` + +Both are the pre-R-198 rows written hours before the fix. `SELECT COUNT(*) … WHERE identity_blob IS +NOT NULL` returned **0**. The retention as fixed had never once been exercised. + +*Instrument:* hub SQLite copied WITH `hub.db-wal` + `-shm` (the standing gotcha). Freshness proved by +a **positive observable** — newest `host_reports` row 61 s old — not by "the query returned no error". +`PRAGMA integrity_check` = ok. + +**Q2 — what act produces a retaining supersession? Two, both read from source:** + +- `hub/internal/store/store.go:2807` — `SaveHostEscrow` demotes when the newly sealed + `restic_pw_sha256` **differs** from the stored one. +- `hub/internal/store/store.go:2636` — `DeleteHost` demotes the current row unconditionally. + +**Q3 — is there any code path that USES a retained record to recover a password? NO.** + +The product's recovery route is real and wired (R-199) but reaches the **current** row only: + +``` +controller → agent POST /escrow/recover-offsite-password + → escrow.OffsiteKeyRecoverer.Fetch felhom-agent/cmd/felhom-agent/main.go:1757 + → hubClient.FetchIdentityEscrow + → hub GET identity-escrow handler hub/internal/api/handler.go:1325 + → store.GetHostDRBundle hub/internal/store/store.go:3152 + SELECT blob, identity_blob, directive_json FROM host_escrow WHERE host_id = ? +``` + +`ListSupersededEscrow` (`store.go:2841`) is the **sole** reader of a retained `identity_blob`. It has +**zero production callers** — five call sites, all in `_test.go`. `CountSupersededEscrow` is used +twice and only to render a *count* (`api/handler.go:1209`, `web/hosts.go:572`). + +**Feasibility verdict: producible.** Phase C proceeded. + +--- + +## 3. Phase C — producing the state + +### 3.1 The planted dataset + +Planted in `/mnt/sys_drive/felhom-data/userdata/drill-20260812` on `demo-felhom`. + +| file | sha256 | bytes | +|---|---|---| +| `payload.bin` | `6863824a67bf6cb636defca3cbddc1d58092bcdda3af939224dfef4663c15359` | 65536 | +| `plain-ascii.txt` | `e294e0ee5b8fd131dc0fa71f5cf1253dc39473241aa090be36404a568285c83a` | 56 | +| `árvíztűrő-tükörfúrógép.txt` | `a1957a87104c63be22b34e4f937e700edcdc925da5984bfcca407df4989f7190` | 37 | + +**The accented filename never crossed a shell as text** (rule 6). It was built from explicit bytes and +verified as hex, both as intended and as listed on disk — identical: + +``` +c3 a1 72 76 c3 ad 7a 74 c5 b1 72 c5 91 2d 74 c3 bc 6b c3 b6 72 66 c3 ba 72 c3 b3 67 c3 a9 70 2e 74 78 74 +``` + +### 3.2 Positive control on the comparator — done BEFORE it was relied on + +One byte of a copy of `payload.bin` flipped at offset 40000: + +``` +unaltered f5de179b5ca941645819a879187db953db5c973d4c1a07b2a3ba96d3d79bfb6c +altered 515a65d96326bd847a87ea35c9fb5e6afebc9499ec0c1cd80ca3531b22cb2324 +payload.bin: FAILED · sha256sum: WARNING: 1 computed checksum did NOT match +``` + +The comparator convicts. The copy was discarded and the original re-hashed unchanged. + +### 3.3 The store, written under P1 + +`restic backup` into demo-felhom's **own** repository path +(`sftp:u629488-sub1@…your-storagebox.de:/home/felhom-repo`) → snapshot **`6ea85413`**, confirmed by +identity and by **listing its contents**, not by the success line: + +``` +6ea85413 2026-08-12 15:15:33 demo-felhom felhom-offbox,drill-retained-key-20260812 + …/drill-20260812/payload.bin + …/drill-20260812/plain-ascii.txt + …/drill-20260812/M-CM-!rvM-CM--ztM-EM-1rM-EM-^Q-tM-CM-32G), data +46G (->70G, ONE volume) +[STEP] 4/8 host enrollment (POST /host-enroll) +[OK] host REUSED (idempotent — existing credential) +[INFO] host_id: drill-r50-0a4f9a (api_key captured, not logged) +[STEP] 4b/8 break-glass credential (root@pam console password → hub vault) +[OK] root@pam password set + vaulted to the hub (retrieve via the operator /admin path; never logged here) +[WARN] NOTE: the root@pam password just CHANGED — the old one now fails at the PVE web GUI (:8006). +[WARN] Retrieve the new one at hub → host page (vaulted recovery credential). +[STEP] 5/8 agent install (fetch + verify + install) +[INFO] disabled pve-enterprise.sources (Enabled: no) +[INFO] disabled ceph.sources (Enabled: no) +[INFO] added pve-no-subscription.sources (suite=trixie) +[OK] apt repos aligned to no-subscription (changed; apt-get update OK) +[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged) +[INFO] manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.213.0 +[INFO] fetching agent binary v0.128.0 from Gitea … +[OK] verified sha256 c6eba73bf9b9ad69… matches the hub manifest +[OK] installed /usr/local/bin/felhom-agent (felhom-agent 0.128.0) +[INFO] installing the 'sudo' package (required for the non-root agent model) … +[OK] sudo installed (Sudo version 1.9.16p2) +[INFO] installing the 'age' package (escrow ceremony identity-wrap dependency) … +[OK] age installed (1.2.1) +[OK] created service user felhom-agent +[OK] added felhom-agent to systemd-journal (unprivileged journal read for NAS verify) +[OK] installed /usr/local/sbin/felhom-mkfs-guarded (0755, the guarded mkfs path) +[OK] installed /usr/local/sbin/felhom-selfupdate-guarded (0755, the guarded A/B binary-swap path) +[OK] installed /usr/local/sbin/felhom-pbs-apply (0755, the guarded PBS-DR apply path) +[OK] installed /usr/local/sbin/felhom-backup-target-apply (0755, the guarded backup-target path) +[OK] installed /etc/sudoers.d/felhom-agent (0440, visudo-validated) +[OK] installed /etc/systemd/system/felhom-agent.service + enabled (started in step 6 after config) +[OK] installed self-update rollback unit + start-limit drop-in (auto-rollback armed) +[OK] installed break-glass layers 1+2 (tmpfiles /run/sshd + agent-independent watchdog timer) +[OK] installed OOB felhom-sshd instance + static belt (agent renders config + fills sets once oob.enabled) +[STEP] 6/8 agent config + service +[WARN] backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive. +[WARN] It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard. +[INFO] creating island bridge vmbr9 (portless, 169.254.253.1/30) +[OK] vmbr9 up: 169.254.253.1/30 +[INFO] R-50 island ON: local_api=169.254.253.1:8443 (vmbr9); guest net1=169.254.253.2/30; lan_resolver.host_ip=10.0.2.15 +[INFO] node=drill-pve local_api=169.254.253.1:8443 tls_fp=F7:CC:33:E6:CA:F5… +[OK] wrote /etc/felhom-agent/agent.json (0600 felhom-agent) +[OK] agent --selftest (read-only) passed +[OK] felhom-agent service active (non-root felhom-agent reads the config OK) +[STEP] 7/8 golden archive +[SKIP] using local golden: local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst +[STEP] 8/8 provision guest 120 +[SKIP] pool felhom already exists +=== felhom-agent 0.128.0 selftest=provision (vmid=120 customer=drill-r50 hostname=drill-r50) === + --- front half: bring-up (provision) local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst → vmid 120 --- +time=2026-08-12T17:18:55.237+02:00 level=INFO msg="bring-up: pool membership re-asserted" vmid=120 pool=felhom + [OK] front half: vmid 120 up (boot+running) in 1m20s; MAC=BC:24:11:6A:90:26 + --- back half: mint per-guest token + populate bootstrap config mount --- +time=2026-08-12T17:19:01.173+02:00 level=INFO msg="provision: back-half complete" vmid=120 mount=mp9 guest_path=/etc/felhom-bootstrap endpoint=169.254.253.1:8443 + [OK] back half: bootstrap mount mp9 → /etc/felhom-bootstrap on vmid 120 (host dir /var/lib/felhom-agent/guests/120/bootstrap) + local-api endpoint 169.254.253.1:8443 · leaf fp e4cba31879281d094ea2ab492e412a9aa380854b99fd5792134658f7f0971dc5 · token: minted (not printed) +=== selftest=provision OK — guest 120 provisioned + bootstrap-mounted (KEPT) === + next: reboot the guest → the golden's baked controller-bootstrap unit deploys the controller, + which PULLS its controller.yaml from the hub (retrieval passphrase) and merges in this local_api. +[OK] provision completed +[INFO] rebooting guest 120 so the baked controller-bootstrap unit picks up the mount +[STEP] verify +[OK] pct status: running +[OK] onboot: 1 + mp0: local-lvm:vm-120-disk-1,mp=/var/lib/docker,backup=1,size=62G + mp1: local-lvm:vm-120-disk-2,mp=/mnt/sys_drive,backup=1,size=8G + mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives + rootfs: local-lvm:vm-120-disk-0,size=32G +[OK] pool: guest 120 is a member of felhom +[OK] acl: FelhomAgentBase@/ present (user+token) +[OK] acl: FelhomAgentGuest@/pool/felhom present (user+token) +[OK] acl: FelhomAgentStore@/storage/local present (user+token) +[OK] acl: FelhomAgentStore@/storage/local-lvm present (user+token) +[OK] acl: FelhomAgentStore@/storage/felhom-pbs present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990000 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990001 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990002 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990003 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990004 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990005 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990006 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990007 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990008 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990009 present (user+token) +[OK] authz signers: 2 (operator-signed self-update armed) +[OK] controller: Up 22 seconds (healthy) (after ~0s) +[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.98.3 +[WARN] cloudflared not visible yet +[INFO] (confirm in the hub UI that host drill-r50-0a4f9a reports guest 120) + +[OK] Day-0 provision SUCCESS — vmid=120 host_id=drill-r50-0a4f9a customer=drill-r50 golden=local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst +[INFO] root@pam was rotated + vaulted at step 4b — retrieve at hub → host page (the old GUI password no longer works). diff --git a/documentation/audits/evidence-drill-retained-key-2026-08-12/B2-uninstall.log b/documentation/audits/evidence-drill-retained-key-2026-08-12/B2-uninstall.log new file mode 100644 index 0000000..f01b2a8 --- /dev/null +++ b/documentation/audits/evidence-drill-retained-key-2026-08-12/B2-uninstall.log @@ -0,0 +1,790 @@ + +[INFO] felhom-host-install v1.25.0 — mode=uninstall + +[STEP] UNINSTALL — local host teardown +[INFO] vmid 120 looks like a Felhom guest (has the /etc/felhom-bootstrap mount) +[INFO] pct config 120: + arch: amd64 + cores: 2 + features: nesting=1,keyctl=1 + hookscript: local:snippets/felhom-guest-hook.sh + hostname: drill-r50 + memory: 4096 + mp0: local-lvm:vm-120-disk-1,mp=/var/lib/docker,backup=1,size=62G + mp1: local-lvm:vm-120-disk-2,mp=/mnt/sys_drive,backup=1,size=8G + mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives + mp9: /var/lib/felhom-agent/guests/120/bootstrap,mp=/etc/felhom-bootstrap,ro=1 + net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:6A:90:26,ip=dhcp,type=veth + net1: name=eth1,bridge=vmbr9,hwaddr=BC:24:11:89:76:A8,ip=169.254.253.2/30,type=veth + onboot: 1 + ostype: debian + rootfs: local-lvm:vm-120-disk-0,size=32G + swap: 512 + unprivileged: 1 +Type the vmid (120) to confirm PERMANENT destruction: perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). + Logical volume "vm-120-disk-0" successfully removed. + Logical volume "vm-120-disk-1" successfully removed. + Logical volume "vm-120-disk-2" successfully removed. +[OK] guest 120 destroyed +[STEP] host-level removal +Removed '/etc/systemd/system/multi-user.target.wants/felhom-agent.service'. +Removed '/etc/systemd/system/timers.target.wants/felhom-mgmt-watchdog.timer'. +Removed '/etc/systemd/system/multi-user.target.wants/felhom-sshd.service'. +Removed '/etc/systemd/system/multi-user.target.wants/felhom-oob-nft.service'. +The unit files have no installation config (WantedBy=, RequiredBy=, UpheldBy=, +Also=, or Alias= settings in the [Install] section, and DefaultInstance= for +template units). This means they are not meant to be enabled or disabled using systemctl. + +Possible reasons for having these kinds of units are: +• A unit may be statically enabled by being symlinked from another unit's + .wants/, .requires/, or .upholds/ directory. +• A unit's purpose may be to act as a helper for some other unit which has + a requirement dependency on it. +• A unit may be started when needed via activation (socket, path, timer, + D-Bus, udev, scripted systemctl call, ...). +• In case of template units, the unit is meant to be enabled with some + instance name specified. +Removed '/etc/systemd/system/multi-user.target.wants/felhom-shared-parent.service'. +Removed '/etc/systemd/system/pve-guests.service.wants/felhom-shared-parent.service'. +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +[SKIP] old broad role FelhomAgent already absent +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +[SKIP] golden vzdump left in place (pass --remove-golden to remove) + +[STEP] kept vs wiped — read before pulling drives or closing the customer out + WIPED (this run): + - guest 120 (container + its OS/Docker/user-data volumes) + - the felhom-agent: binary, unit, sudoers, config (+ its .bak backups), state dir, service user + - self-update artifacts: guarded wrapper, A/B slots (.prev/.new.*), rollback unit, start-limit drop-in + - break-glass watchdog + OOB artifacts (where present); guest-hook snippet; dnsmasq snippets; the mkfs + pbs-apply wrappers + - pveum: the Felhom roles/user/token/scoped ACL; the emptied felhom pool + - the install state file + KEPT (lives on deliberately — remove/rotate these out-of-band if the customer is leaving): + - the enrolled drives + ALL data under /mnt/felhom-drives — unmounted only, NEVER wiped; + the drives are physically removable now. + - the hub host/customer record + report history (operator UI / DB) + - the escrow blob in the hub, if one exists (operator UI) + - the hub-vaulted root@pam recovery credential — the box KEEPS the password step 4b set; rotate it if the box leaves Felhom management + +[OK] UNINSTALL complete — removed: guest 120, the felhom-agent (unit/sudoers/binary/state/config+baks/user + selfupdate-artifacts/shared-parent/mkfs-wrapper/pbs-apply-wrapper/hook-snippet/dnsmasq-snippets), the pveum role/user/token/ACL, the felhom pool, and /var/lib/felhom-install/state.json. +[INFO] golden vzdump: left in place (--remove-golden to remove). +[INFO] NOTE: the 'sudo' and 'dnsmasq' packages were left installed (system packages); the host record still exists in the hub — remove it there if desired. diff --git a/documentation/audits/evidence-drill-retained-key-2026-08-12/B3-explicit-stale.log b/documentation/audits/evidence-drill-retained-key-2026-08-12/B3-explicit-stale.log new file mode 100644 index 0000000..2b6df69 --- /dev/null +++ b/documentation/audits/evidence-drill-retained-key-2026-08-12/B3-explicit-stale.log @@ -0,0 +1,767 @@ + +[INFO] felhom-host-install v1.27.0 — mode=appliance customer=drill-r50 vmid=120 + +[STEP] 1/8 pre-flight +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve) +[INFO] node: drill-pve (auto) +[INFO] agent config: /etc/felhom-agent/agent.json +[INFO] agent: not installed yet — will be fetched + installed in step 5/8 +[INFO] local-lvm free: ~75 GiB +[WARN] local-lvm free ~75 GiB < hard min 120 GiB +[INFO] free RAM: ~6119 MiB +[INFO] existing guests on this host: 0 (pct+qm) +[INFO] archive storage 'local' present +[INFO] acl storage 'felhom-pbs' not present yet — expected: the PBS-DR tier creates it; the grant is pre-positioned deliberately +[INFO] dnsmasq: already installed BEFORE Felhom — recorded; uninstall will not touch it +[INFO] hub reachable (https://hub.felhom.eu) +[OK] customer 'drill-r50' exists + passphrase valid +[INFO] golden (local): local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst +[OK] pre-flight passed +[STEP] 2/8 Proxmox API token +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +[OK] token minted (secret captured, not logged) +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). +[OK] scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm felhom-pbs]) +[SKIP] old broad role FelhomAgent already absent +[STEP] 3/8 compute volume grows +[INFO] auto-computed from ~75 GiB free (ONE volume since R-165) +[INFO] grows: rootfs +0G (->32G), data +46G (->70G, ONE volume) +[STEP] 4/8 host enrollment (POST /host-enroll) +[OK] host REUSED (idempotent — existing credential) +[INFO] host_id: drill-r50-0a4f9a (api_key captured, not logged) +[STEP] 4b/8 break-glass credential (root@pam console password → hub vault) +[OK] root@pam password set + vaulted to the hub (retrieve via the operator /admin path; never logged here) +[WARN] NOTE: the root@pam password just CHANGED — the old one now fails at the PVE web GUI (:8006). +[WARN] Retrieve the new one at hub → host page (vaulted recovery credential). +[STEP] 5/8 agent install (fetch + verify + install) +[OK] apt repos aligned to no-subscription (already aligned; apt-get update OK) +[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged) +[INFO] manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.213.0 +[INFO] fetching agent binary v0.128.0 from Gitea … +[OK] verified sha256 c6eba73bf9b9ad69… matches the hub manifest +[OK] installed /usr/local/bin/felhom-agent (felhom-agent 0.128.0) +[OK] created service user felhom-agent +[OK] added felhom-agent to systemd-journal (unprivileged journal read for NAS verify) +[OK] installed /usr/local/sbin/felhom-mkfs-guarded (0755, the guarded mkfs path) +[OK] installed /usr/local/sbin/felhom-selfupdate-guarded (0755, the guarded A/B binary-swap path) +[OK] installed /usr/local/sbin/felhom-pbs-apply (0755, the guarded PBS-DR apply path) +[OK] installed /usr/local/sbin/felhom-backup-target-apply (0755, the guarded backup-target path) +[OK] installed /etc/sudoers.d/felhom-agent (0440, visudo-validated) +[OK] installed /etc/systemd/system/felhom-agent.service + enabled (started in step 6 after config) +[OK] installed self-update rollback unit + start-limit drop-in (auto-rollback armed) +[OK] installed break-glass layers 1+2 (tmpfiles /run/sshd + agent-independent watchdog timer) +[OK] installed OOB felhom-sshd instance + static belt (agent renders config + fills sets once oob.enabled) +[STEP] 6/8 agent config + service +[WARN] backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive. +[WARN] It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard. +[INFO] island bridge vmbr9 already present — leaving it +[INFO] R-50 island ON: local_api=169.254.253.1:8443 (vmbr9); guest net1=169.254.253.2/30; lan_resolver.host_ip=10.0.2.15 +[INFO] node=drill-pve local_api=169.254.253.1:8443 tls_fp=F7:CC:33:E6:CA:F5… +[OK] wrote /etc/felhom-agent/agent.json (0600 felhom-agent) +[OK] agent --selftest (read-only) passed +[OK] felhom-agent service active (non-root felhom-agent reads the config OK) +[STEP] 7/8 golden archive +[ERROR] refusing the golden you named (local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst): it is controller 0.98.3, but the vouched golden is 0.213.0. + The vouched golden is 0.213.0. Either pass the archive that matches it, + or re-run with --force-gitea-golden to fetch the vouched one from Gitea. diff --git a/documentation/audits/evidence-drill-retained-key-2026-08-12/B4-autodiscover-stale.log b/documentation/audits/evidence-drill-retained-key-2026-08-12/B4-autodiscover-stale.log new file mode 100644 index 0000000..dd23b17 --- /dev/null +++ b/documentation/audits/evidence-drill-retained-key-2026-08-12/B4-autodiscover-stale.log @@ -0,0 +1,114 @@ + +[INFO] felhom-host-install v1.27.0 — mode=appliance customer=drill-r50 vmid=120 + +[STEP] 1/8 pre-flight +[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve) +[INFO] node: drill-pve (auto) +[INFO] agent config: /etc/felhom-agent/agent.json +[INFO] agent (existing): felhom-agent 0.128.0 +[INFO] local-lvm free: ~75 GiB +[WARN] local-lvm free ~75 GiB < hard min 120 GiB +[INFO] free RAM: ~6231 MiB +[INFO] existing guests on this host: 0 (pct+qm) +[INFO] archive storage 'local' present +[INFO] acl storage 'felhom-pbs' not present yet — expected: the PBS-DR tier creates it; the grant is pre-positioned deliberately +[INFO] dnsmasq: already installed BEFORE Felhom — recorded; uninstall will not touch it +[INFO] hub reachable (https://hub.felhom.eu) +[OK] customer 'drill-r50' exists + passphrase valid +[INFO] golden (local): local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst +[OK] pre-flight passed +[STEP] 2/8 Proxmox API token +[SKIP] pool felhom already exists +[INFO] user felhom-agent@pve exists +[INFO] existing agent config has a token — testing it (read-only --selftest) +[OK] existing token authenticates — REUSING (no rotation) +[INFO] role FelhomAgentBase exists — ensuring exact privileges +[INFO] role FelhomAgentGuest exists — ensuring exact privileges +[INFO] role FelhomAgentStore exists — ensuring exact privileges +[OK] scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm felhom-pbs]) +[SKIP] old broad role FelhomAgent already absent +[STEP] 3/8 compute volume grows +[INFO] auto-computed from ~75 GiB free (ONE volume since R-165) +[INFO] grows: rootfs +0G (->32G), data +46G (->70G, ONE volume) +[STEP] 4/8 host enrollment (POST /host-enroll) +[OK] host REUSED (idempotent — existing credential) +[INFO] host_id: drill-r50-0a4f9a (api_key captured, not logged) +[STEP] 4b/8 break-glass credential (root@pam console password → hub vault) +[SKIP] recovery credential already vaulted (use --rotate-recovery to regenerate) +[STEP] 5/8 agent install (fetch + verify + install) +[OK] apt repos aligned to no-subscription (already aligned; apt-get update OK) +[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged) +[INFO] manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.213.0 +[SKIP] agent v0.128.0 already installed + service active — skipping binary install +[INFO] service user felhom-agent exists +[INFO] felhom-agent already in systemd-journal +[OK] installed /usr/local/sbin/felhom-mkfs-guarded (0755, the guarded mkfs path) +[OK] installed /usr/local/sbin/felhom-selfupdate-guarded (0755, the guarded A/B binary-swap path) +[OK] installed /usr/local/sbin/felhom-pbs-apply (0755, the guarded PBS-DR apply path) +[OK] installed /usr/local/sbin/felhom-backup-target-apply (0755, the guarded backup-target path) +[OK] installed /etc/sudoers.d/felhom-agent (0440, visudo-validated) +[OK] installed /etc/systemd/system/felhom-agent.service + enabled (started in step 6 after config) +[OK] installed self-update rollback unit + start-limit drop-in (auto-rollback armed) +[OK] installed break-glass layers 1+2 (tmpfiles /run/sshd + agent-independent watchdog timer) +[OK] installed OOB felhom-sshd instance + static belt (agent renders config + fills sets once oob.enabled) +[STEP] 6/8 agent config + service +[WARN] backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive. +[WARN] It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard. +[INFO] island bridge vmbr9 already present — leaving it +[INFO] R-50 island ON: local_api=169.254.253.1:8443 (vmbr9); guest net1=169.254.253.2/30; lan_resolver.host_ip=10.0.2.15 +[INFO] node=drill-pve local_api=169.254.253.1:8443 tls_fp=F7:CC:33:E6:CA:F5… +[OK] wrote /etc/felhom-agent/agent.json (0600 felhom-agent) +[OK] agent --selftest (read-only) passed +[OK] felhom-agent service active (non-root felhom-agent reads the config OK) +[STEP] 7/8 golden archive +[WARN] ignoring the local golden local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst — it is controller 0.98.3, but the vouched golden is 0.213.0 +[WARN] fetching the vouched golden instead (this is what the manifest is for) +[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged) +[INFO] fetching golden v0.213.0 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_08_12-17_31_32.tar.zst +[OK] verified sha256 77429bfc7e39ffc8… matches the hub manifest +[OK] golden imported + verified: local:backup/vzdump-lxc-9100-2026_08_12-17_31_32.tar.zst +[STEP] 8/8 provision guest 120 +[SKIP] pool felhom already exists +=== felhom-agent 0.128.0 selftest=provision (vmid=120 customer=drill-r50 hostname=drill-r50) === + --- front half: bring-up (provision) local:backup/vzdump-lxc-9100-2026_08_12-17_31_32.tar.zst → vmid 120 --- +time=2026-08-12T17:32:57.812+02:00 level=INFO msg="bring-up: pool membership re-asserted" vmid=120 pool=felhom + [OK] front half: vmid 120 up (boot+running) in 1m10s; MAC=BC:24:11:D3:D7:8C + --- back half: mint per-guest token + populate bootstrap config mount --- +time=2026-08-12T17:33:03.626+02:00 level=INFO msg="provision: back-half complete" vmid=120 mount=mp9 guest_path=/etc/felhom-bootstrap endpoint=169.254.253.1:8443 + [OK] back half: bootstrap mount mp9 → /etc/felhom-bootstrap on vmid 120 (host dir /var/lib/felhom-agent/guests/120/bootstrap) + local-api endpoint 169.254.253.1:8443 · leaf fp 0c7b12eec0ab8c4f8e879dba1af0f7203ef82aea92b023d543b7632b865ead58 · token: minted (not printed) +=== selftest=provision OK — guest 120 provisioned + bootstrap-mounted (KEPT) === + next: reboot the guest → the golden's baked controller-bootstrap unit deploys the controller, + which PULLS its controller.yaml from the hub (retrieval passphrase) and merges in this local_api. +[OK] provision completed +[INFO] rebooting guest 120 so the baked controller-bootstrap unit picks up the mount +[STEP] verify +[OK] pct status: running +[OK] onboot: 1 + mp0: local-lvm:vm-120-disk-1,mp=/var/lib/felhom,backup=1,size=70G + mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives + rootfs: local-lvm:vm-120-disk-0,size=32G +[OK] pool: guest 120 is a member of felhom +[OK] acl: FelhomAgentBase@/ present (user+token) +[OK] acl: FelhomAgentGuest@/pool/felhom present (user+token) +[OK] acl: FelhomAgentStore@/storage/local present (user+token) +[OK] acl: FelhomAgentStore@/storage/local-lvm present (user+token) +[OK] acl: FelhomAgentStore@/storage/felhom-pbs present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990000 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990001 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990002 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990003 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990004 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990005 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990006 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990007 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990008 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990009 present (user+token) +[OK] authz signers: 2 (operator-signed self-update armed) +[OK] controller: Up 24 seconds (healthy) (after ~0s) +[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.213.0 +[WARN] cloudflared not visible yet +[INFO] (confirm in the hub UI that host drill-r50-0a4f9a reports guest 120) + +[OK] Day-0 provision SUCCESS — vmid=120 host_id=drill-r50-0a4f9a customer=drill-r50 golden=local:backup/vzdump-lxc-9100-2026_08_12-17_31_32.tar.zst +[INFO] root@pam was rotated + vaulted at step 4b — retrieve at hub → host page (the old GUI password no longer works). diff --git a/documentation/audits/evidence-drill-retained-key-2026-08-12/Z-final-state.txt b/documentation/audits/evidence-drill-retained-key-2026-08-12/Z-final-state.txt new file mode 100644 index 0000000..59f8119 --- /dev/null +++ b/documentation/audits/evidence-drill-retained-key-2026-08-12/Z-final-state.txt @@ -0,0 +1,15 @@ +=== final drill-r50 state 2026-08-12T15:39:28Z === +VMID Status Lock Name +120 running drill-r50 +--- controller --- +gitea.dooplex.hu/admin/felhom-controller:0.213.0 +--- goldens on the box --- +Volid Format Type Size VMID +local:backup/vzdump-lxc-120-2026_08_12-17_24_19.tar.zst tar.zst backup 612587653 120 +local:backup/vzdump-lxc-9100-2026_08_12-17_31_32.tar.zst tar.zst backup 656657008 9100 +local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst tar.zst backup 612002238 9100 +--- dnsmasq --- +enabled +active +udp UNCONN 0 0 127.0.0.1:53 0.0.0.0:* users:(("dnsmasq",pid=15124,fd=6)) +udp UNCONN 0 0 10.0.2.15:53 0.0.0.0:* users:(("dnsmasq",pid=15124,fd=4)) diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 7425b4a..90e311e 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -597,10 +597,17 @@ unrelated finding, which is why the sweep's proposals were renumbered to R-159 class (an image `VOLUME` at an unmounted path) is still live — `immich-server` has one today. | **R-295** | **One name per secret — CONTROLLER HALF SHIPPED.** The claim page called the SAME three-word dashboard code „Beállító kód" on the first-time branch and „Visszaállító kód" on the reset branch, while the TEN-word escrow code is „Helyreállítási kód". Two near-homographs for two different secrets; the collision cost a real code. „Visszaállító kód" is **retired** in the controller (`claim.html` label/subtitle/button, `claim.go` print-reset-code + lockout strings); the name is now constant and the SENTENCE changes. Naming only — pinned by `TestResetCode_StillAcceptedOnTheSetupPage`. **HUB HALF NOT DONE (Part 4a, dropped per the session's own drop order):** the hub's send button „Visszaállító kód küldése", the mail subject „Jelszó-visszaállítási kód", its body „Visszaállító kód:", and the mail sending the customer to an „Elfelejtett jelszó" page while a rebuilt box actually serves „A szerver beállítása" | **PARTIAL — controller shipped v0.211.0; hub half OPEN (S)** | R-294 | Apply the same ruling in `felhom.eu/hub`, and make the mail name the page the machine is actually showing | CC | | **R-296** | **The orphan card's OTHER sentence makes the same promise, and the spec says it is fine.** `SPEC-orphan-card-copy-2026-08-10.md` §1 lists `backups_remote.html:98` as *"Accurate; keep"*. It is not: it ends *„A meglévő mentések nem sérültek — a hozzájuk tartozó helyreállítási kóddal később **visszaállíthatók lehetnek**."* — the identical unevaluable promise in a different conjugation, which the spec's own regression guard (`visszaállítható lehet`, singular) does not match. Line 101 was fixed; line 98 was left **deliberately unchanged**: the instruction was not to improvise Hungarian at the customer | **CLOSED — shipped in controller v0.212.0 (R-299); verified: the sentence at backups_remote.html:98 was replaced and the stem guard covers it** | R-294 | Needs a wording decision, then the same treatment as line 101 and an assertion covering BOTH conjugations | operator + CC | -| **R-297** | **An install took whatever golden was lying around.** Step 7 of the host installer short-circuited on any local archive with no version compare, no digest and no warning, so the manifest's sha256 — vouching from a different trust root than the code host — was consulted only on the fetch path. Discovery is newest-by-filename: correct by recency, never by verification. A box reinstalled from a stale archive can come back below the version where the off-site recovery screen exists, and below the update floor. Three goldens on `demo-hp` carry controller 0.153.0 and 0.210.0; nothing compared either to the vouched version. **FIXED in `felhom-host-install.sh` v1.26.0** (digest first, then the baked `/etc/felhom-controller-image` tag; auto-discovered mismatch re-fetches, an operator-named mismatch REFUSES; an unreadable manifest refuses). **NOT PUBLISHED and NOT YET OBSERVED FAILING** — the pre-fix stale install was never watched on `drill-r50`, so the `installer-v1.26.0` tag is deliberately not cut | **SHIPPED TO main, UNPUBLISHED (M)** | — | Observe a fresh install taking a stale local golden on `drill-r50` BEFORE publishing; then cut `installer-v1.26.0` and bump both `--ref`s in `webpage.yaml` | CC | +| **R-297** | **An install took whatever golden was lying around.** Step 7 of the host installer short-circuited on any local archive with no version compare, no digest and no warning, so the manifest's sha256 — vouching from a different trust root than the code host — was consulted only on the fetch path. Discovery is newest-by-filename: correct by recency, never by verification. A box reinstalled from a stale archive can come back below the version where the off-site recovery screen exists, and below the update floor. Three goldens on `demo-hp` carry controller 0.153.0 and 0.210.0; nothing compared either to the vouched version. **FIXED in `felhom-host-install.sh` v1.26.0** (digest first, then the baked `/etc/felhom-controller-image` tag; auto-discovered mismatch re-fetches, an operator-named mismatch REFUSES; an unreadable manifest refuses). **OBSERVED LIVE 2026-08-12** on `drill-r50` from `virgin`: a genuinely stale `golden-0.98.3.tar.zst` planted as the newest-by-filename `vzdump-lxc-9100-…-23_59_59.tar.zst`; v1.25.0 logged `manifest: … golden v0.213.0` at step 5 and `[SKIP] using local golden` at step 7, and **the box came up on controller 0.98.3** — confirmed from both `docker ps` and the in-guest `/etc/felhom-controller-image`. That is below the floor (0.213.0) AND below v0.206.0, where the off-site recovery screen exists: the row's sharp end was real. Both fixed shapes then watched — auto-discovered → `ignoring the local golden … it is controller 0.98.3, but the vouched golden is 0.213.0` → re-fetched and sha-verified → landed **0.213.0**; operator-named → **refused**, exit 1 | **CLOSED — observed live + PUBLISHED as `installer-v1.27.0` (both refs bumped)** | — | — | CC | | **R-298** | **The `/storage` page's unregistered list is filtered by `role==='user-data'`, so a drive that is also the backup target can never be registered from it.** `storage.html:363` routes anything not `user-data` into the read-only protected group with NO actions. On the rebuilt `demo-hp` the NVMe is deliberately BOTH the user-data drive and the `felhom-backup` target (`/etc/pve/storage.cfg`: `dir: felhom-backup` → `/mnt/nvme-1tb`), so it renders locked. **This is the SECOND reason that page was empty** during the reinstall rehearsal, independent of R-280's candidate-source defect, and R-280's fix does not touch it — attaching is non-destructive, so the format-wizard protection is the wrong gate for a REGISTER action | **READY (S) — NEW 2026-08-10** | R-280 | Split the role gate: `user-data` keeps destructive actions; any mounted role may be REGISTERED | CC | | **R-299** | **The orphan card's OTHER sentence made the same unevaluable promise, and the spec called it accurate.** `backups_remote.html` line 98 — the **always-visible** half of the card, where the paragraph fixed in v0.211.0 renders only after a click — ended *„a hozzájuk tartozó helyreállítási kóddal később **visszaállíthatók lehetnek**"*. It survived for two independent reasons: `SPEC-orphan-card-copy-2026-08-10.md` §1 listed it as *"Accurate; keep"*, and **the regression guard matched one INFLECTION** (`visszaállítható lehet`, singular) which the plural does not contain. **A guard matching one inflection of a Hungarian verb guards one sentence, not the claim.** Guard broadened to the stem `visszaállíthat`; proven by planting the shipped plural (stem guard convicts, old singular guard does not match it at all). Spec corrected in both places. The two accurate halves kept | **CLOSED — controller v0.212.0** | R-294 | — | CC | -| **R-300** | **Our own uninstall left the thing that makes our own reinstall refuse.** `--uninstall` removed the dnsmasq snippet and RESTARTED the daemon, leaving it enabled and unconstrained on `0.0.0.0:53`; the next byo install's preflight then hard-refuses with *"a resolver is already bound to :53"* and **the customer goes debugging a home network that was never at fault**. Ownership is now RECORDED at preflight (`dnsmasq_preexisting`) — the only moment it is a fact, since the package is installed by the AGENT (`lanresolver.go:107`), not the installer — and honoured at removal: Felhom's → stop+disable; the owner's → restart only; **no record (every box already in the field) → restart only, fail-safe, with the reason and the exact command logged**. The preflight refusal keeps its two routes and its promise not to touch DNS on a host we do not own, and gains the missing line: when the bound resolver is dnsmasq, it says it looks like ours and names the command. **NOT OBSERVED LIVE — the install→uninstall→install cycle on `drill-r50` was not run**, so `installer-v1.27.0` is NOT cut | **SHIPPED TO main, UNPUBLISHED + UNVERIFIED LIVE (M)** | — | Run the cycle on `drill-r50` (quote the second install refusing before the fix, passing after), together with R-297's stale-golden observation — one drill session covers both; only then publish | CC | +| **R-300** | **Our own uninstall left the thing that makes our own reinstall refuse.** `--uninstall` removed the dnsmasq snippet and RESTARTED the daemon, leaving it enabled and unconstrained on `0.0.0.0:53`; the next byo install's preflight then hard-refuses with *"a resolver is already bound to :53"* and **the customer goes debugging a home network that was never at fault**. Ownership is now RECORDED at preflight (`dnsmasq_preexisting`) — the only moment it is a fact, since the package is installed by the AGENT (`lanresolver.go:107`), not the installer — and honoured at removal: Felhom's → stop+disable; the owner's → restart only; **no record (every box already in the field) → restart only, fail-safe, with the reason and the exact command logged**. The preflight refusal keeps its two routes and its promise not to touch DNS on a host we do not own, and gains the missing line: when the bound resolver is dnsmasq, it says it looks like ours and names the command. **OBSERVED LIVE 2026-08-12** on `drill-r50`. Chain re-established from `virgin` (dnsmasq **not-installed**, `:53` **free**) → appliance install → the **agent** installed it, snippet-constrained to `10.0.2.15:53`+`127.0.0.1:53` → pre-fix uninstall → `enabled`/`active` on `0.0.0.0:53`+`[::]:53` → pre-fix byo install **refused, exit 1**. Fixed path from `virgin`: recorded `not present before Felhom` → uninstall logged `stopping + disabling it` → `:53 FREE` → byo preflight **PASS**. Owner's side also proven: with dnsmasq pre-installed the record read `yes` and the uninstall left it running. **But see R-305 — the cleanup fires only once per machine** | **CLOSED — observed live + PUBLISHED as `installer-v1.27.0` (both refs bumped)** | — | — | CC | | **R-301** | **The abandon countdown banner makes the retired promise a third time, and as a flat statement.** `controller/internal/web/templates/layout.html:143`: *"A korábbi távoli mentéseidet N nap múlva véglegesen töröljük, a kérésed szerint. **Addig még visszaszerezheted őket a helyreállítási kóddal.**"* Unlike the orphan card this one is not hedged at all. **It is probably TRUE where it renders** — it is gated on `backupMgr.AbandonStatus().Active`, i.e. the customer chose to abandon a recovery offer that exists — which is why it was NOT changed (this session was fenced to the orphan card). **But it renders in `layout.html`, i.e. on EVERY page, and a rebuilt box can have an active abandon countdown WHILE its remote store is orphaned**, so a customer can read *"we cannot determine whether these can be opened"* on the backups page and *"until then you can still retrieve them with your recovery code"* in the banner above it, about copies they have no way to tell apart. **Not established:** whether the two sentences refer to the same bytes. The spec anticipated this in its §4 note | **CLOSED — premise CONFIRMED and fixed in controller v0.213.0 (R-302)** | R-299 | Establish whether the two can co-render for one customer and whether they name the same copies; if so, make the banner name WHICH copy | operator + CC | | **R-302** | **The abandon banner promised retrieval it could not see was still true — fixed by PINNING a fingerprint at the decision.** The countdown strip said *„Addig még visszaszerezheted őket a helyreállítási kóddal"* unconditionally, on every page, and R-301 established it can co-render with the orphan card correctly saying we cannot tell. **THE OBVIOUS CONDITION WAS REJECTED AND HERE IS WHY, so nobody re-proposes it:** the proxy *"does the hub hold a key different from the one I use?"* asks about the WRONG key — the set-aside copies were written under an OLDER key the box no longer has, which is why they were set aside — so on a twice-rebuilt box it answers "promise it" about copies no key on file can open. **Demonstrated, not argued:** under the proxy both Scenario B (package replaced) and Scenario D (legacy countdown) flip back to promising. Instead `startAbandonCountdown` pins `AbandonPinnedEscrowKeySHA256` — the hub's escrow key fingerprint AS CACHED AT THE DECISION — written once, never refreshed; the banner then asks only *"is the hub still holding that same package?"*. **⚠ THE PIN IS A RECORDED ASSUMPTION:** nothing on the box records which key wrote the set-aside copies, so it presumes the package held at the decision is that one — true in the ordinary rebuilt-box story, NOT provable, wrong on a twice-rebuilt box. Recorded in the field comment so it can be narrowed rather than hardening into a fact. Empty is not a match on either side; a countdown started before v0.213.0 carries no pin and takes the cautious branch (deliberately NOT backfilled). **A template sweep found a FOURTH instance** (the backups-page abandon block — same condition applied, since fixing the strip and not the page would leave them contradicting) **and a FIFTH** (the abandon confirmation screen, deliberately LEFT: it renders at the moment of the decision, where the promise is true by construction because that is the package about to be pinned) | **CLOSED — controller v0.213.0** | R-301 | — | CC | | **R-303** | **`markOrphaned` has no guard against an active abandon countdown — the co-render is made HARMLESS, not IMPOSSIBLE.** `ensureOffboxRepo` calls `markOrphaned()` for a claimed box (`offbox.go:804`) with no check on `AbandonAt`, so a later run finding the FRESH store unopenable re-raises the orphan card while the countdown runs. R-302 ensures the two surfaces no longer contradict each other in that state, but the state itself is still reachable and is arguably incoherent: a box counting down to deleting its old history while simultaneously reporting its NEW history is unopenable is in trouble in two ways at once and says so in two separate cards. **Ranked LOW deliberately** — it is a coherence question, not a correctness one, and the wrong fix (suppressing the orphan card during a countdown) would hide a real second fault | **READY (S) — NEW 2026-08-12** | R-302 | Decide whether a countdown should suppress, defer or annotate the orphan card — and beware hiding a genuine second failure | operator + CC | +| **R-304** | **The retained escrow key works, and the customer is told their correct code is wrong.** DRILL 2026-08-12 answered the three questions separately, on `demo-felhom`, with planted data. **(a) retention: WORKS** — the first retained row in fleet history to carry material (`host_escrow_superseded` id 11, `identity_blob` 572 B), byte-identical (`sha256 a10032341c8584ed…`) to the pre-supersession `host_escrow` row. **(b) the material opens the old store: YES** — unsealed with the OLD recovery code it yielded a password byte-identical to the pre-change one (`sha c60c8bc737a6b7c6…`), and restored three planted files **byte-identical** from a store the box itself could no longer open (negative control first: `Fatal: wrong password or no key found`), **including a Hungarian accented filename verified as raw bytes**. **(c) the customer's route: DOES NOT EXIST, and misinforms.** `ListSupersededEscrow` (`store.go:2841`) is the only reader of a retained `identity_blob` and has **zero production callers** — five call sites, all `_test.go`; the product path (`POST /escrow/recover-offsite-password` → `FetchIdentityEscrow` → `GetHostDRBundle`, `store.go:3152`) selects `FROM host_escrow` — the CURRENT row only. Asked for the old password with the code that demonstrably opens the retained row, the product answered **"the recovery code did not open the sealed bundle — nothing was written"**. **This is the R-224 class again**: there an unreachable hub was reported as a bad code; here a VALID code for retained history is reported as a bad code, and the customer's attempt ends there. **Consequence:** the census answer stands (it was about retention); the countdown banner's promise is true in substance and false in practice; **any capability-map claim that the customer can recover the old history with their recovery code is false today and must move** | **READY (L) — NEW 2026-08-12, RANK 1** | R-198, R-199, R-224, R-241 | Decide the shape: serve retained rows on the recovery path (needs a "which package?" choice — a customer may have several), or stop promising retrieval anywhere the customer cannot perform it. **Until one of those, the honest position is that retention is an operator-only capability.** At minimum, the refusal must stop asserting the code is wrong when the hub simply never looked | operator + CC | +| **R-305** | **The R-300 cleanup fires exactly once per machine, and the second reinstall hits the original wall.** PROVEN on `drill-r50` 2026-08-12, not deduced. The uninstall stops+disables dnsmasq but deliberately never **purges** the package, so on every later install `dpkg-query` reports it installed, preflight records `dnsmasq_preexisting=yes`, the agent's `EnsureDnsmasq` re-enables it, and the uninstall then only *restarts* it — leaving it `enabled`/`active` on `0.0.0.0:53` exactly as before the fix. Cycle 1: `no` → disabled → `:53 FREE` → next install PASSES. Cycle 2: `yes` → left running → **cycle-3 byo preflight FAILED, exit 1**. The improved refusal does fire and names the command, so the customer is not stranded — but they must run by hand a cleanup Felhom could have run itself, on a box Felhom does own | **READY (M) — NEW 2026-08-12, RANK 2** | R-300 | Distinguish "the package pre-dates Felhom" from "a previous **Felhom** install left it" — e.g. keep the ownership record outside `state.json` (which the uninstall deletes), or record Felhom-installed-ness where the agent installs it. **Do not fix by purging on hosts we may not own** | CC | +| **R-306** | **`--preflight-only` says "no state written" and writes state — with an answer that can be wrong.** `_state_put` short-circuits on `DRY_RUN` only (`felhom-host-install.sh:418`), so a preflight-only run creates `/var/lib/felhom-install/state.json`. Observed live: after a run whose banner read `PRE-FLIGHT PASS (mode=byo) — no state written, no install step executed`, the file existed containing `{"completed": [], "dnsmasq_preexisting": "yes"}`. Both the banner and the flag's own comment at line 226 assert the opposite. **The harm is not the file, it is the value**: the runbook recommends preflight-only first, then the same command without the flag, so on a box carrying a Felhom leftover the wrong ownership answer is baked in before the real install begins | **READY (S) — NEW 2026-08-12, RANK 3** | R-300, R-305 | Either make `_state_put` a no-op under `PREFLIGHT_ONLY` (and record ownership at install instead), or correct both claims. A comment asserting an invariant needs a test pinning it | CC | +| **R-307** | **`demo-felhom` carries a LIVE abandon countdown that this drill did not start — and the end state says there should be none.** `settings.json`: `abandon_started_at 2026-08-10T08:06:31Z`, `abandon_at 2026-08-24T08:06:31Z`, `abandon_repo_path /home/felhom-repo.orphaned-20260810`. The drill's fence forbade starting, shortening or triggering a countdown, and none was; but its required end state was *"no abandon countdown anywhere"*, and one exists. **The two ways to satisfy that are not equivalent and the choice is the operator's:** cancel it (the orphaned repository is kept indefinitely — storage cost, no data risk) or let it run (on **2026-08-24** that repository is deleted, **irreversibly**). **Doing nothing selects deletion by default.** Deliberately not resolved by CC | **BLOCKED ON OPERATOR — NEW 2026-08-12** | R-301, R-302, R-303 | Decide before 2026-08-24 | operator | +| **R-308** | **The stored controller password no longer opens `demo-felhom`.** `PASSWORD` in `~/.config/credentials` on DooPlex returns `Hibás jelszó` against `https://127.0.0.1:443/login` with `Host: felhom.demo-felhom.eu`. **NOT the known quoting trap** — the surrounding quotes were stripped before use and the stored value is unquoted (16 bytes for 15 chars), so this is a genuinely stale credential, not a mis-read. Cost during the drill: the dashboard was unreachable headlessly, so the offsite push had to be made with `restic` directly instead of the page's own button, which left the app-backup→unit→offsite chain unexercised. `R_DEMO-FELHOM` **was** correct and has since been rotated by this session's final ceremony (updated in place; prior file backed up alongside) | **READY (S) — NEW 2026-08-12, RANK 2** | — | Recover or reset the demo-felhom dashboard password and re-store it; note which boxes share one and which do not | operator | +| **R-309** | **The day-0 runbook says pushing the installer publishes it. It has not since R-110.** `documentation/runbooks/day0-install.md` §C.1: *"it is always the current `main` script. There is no release tag, no staging copy and no version selector; pushing `scripts/felhom-host-install.sh` publishes it."* R-110 moved `/scripts/` to the **installer tag's** tree — `manifests/webpage.yaml` carries **two** `--ref=installer-v…` pins (lines 327, 372) and `hostinstall_gates.py` gate 6 enforces that they name a tag. Confirmed live 2026-08-12: `https://felhom.eu/scripts/felhom-host-install.sh` served `SCRIPT_VERSION="1.25.0"` while `main` had `1.27.0`, three and a half hours after the push. **The runbook's version is the dangerous one to believe in both directions** — it would have an operator think a pushed fix is live when it is not, and think a pushed mistake is live when it is not | **READY (S) — NEW 2026-08-12, RANK 3** | R-110 | Correct §C.1 to name the tag-and-manifest publication step (as performed today) | CC | +| **R-310** | **Two small edges on the installer, neither costing more than a moment.** (1) The R-297 operator-named refusal states the vouched version twice in consecutive sentences (*"…but the vouched golden is 0.213.0. The vouched golden is 0.213.0."*). (2) `--uninstall` reads its typed vmid confirmation from `/dev/tty` and `--force` deliberately does **not** bypass it, so teardown cannot be scripted without a pty — correct for an irreversible destroy, but undocumented; it surfaces as `line 891: /dev/tty: No such device or address` and an rc=1 that looks like a failure rather than a refusal to proceed unattended | **READY (S) — NEW 2026-08-12, RANK 4** | R-297 | Drop the duplicated sentence; add one runbook line naming the pty requirement | CC | diff --git a/manifests/webpage.yaml b/manifests/webpage.yaml index 1da6f10..a49b496 100644 --- a/manifests/webpage.yaml +++ b/manifests/webpage.yaml @@ -324,7 +324,7 @@ spec: image: registry.k8s.io/git-sync/git-sync:v4.4.0 args: - --repo=https://gitea.dooplex.hu/admin/felhom.eu.git - - --ref=installer-v1.25.0 + - --ref=installer-v1.27.0 - --root=/git-scripts - --link=current - --period=30s @@ -369,7 +369,7 @@ spec: image: registry.k8s.io/git-sync/git-sync:v4.4.0 args: - --repo=https://gitea.dooplex.hu/admin/felhom.eu.git - - --ref=installer-v1.25.0 + - --ref=installer-v1.27.0 - --root=/git-scripts - --link=current - --one-time diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index 85f53d0..08386a3 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,3 +1,28 @@ +## PUBLISHED — `installer-v1.27.0` (2026-08-12) + +**v1.26.0 (R-297) and v1.27.0 (R-300) were both written, pushed and deliberately left unpublished +until their faults had been watched happening. Both were watched on `drill-r50` on 2026-08-12 +(`documentation/audits/DRILL-retained-key-2026-08-12.md`), so the tag is now cut and both `--ref`s in +`manifests/webpage.yaml` moved to it** — the sidecar and the init container. No `installer-v1.26.0` +tag exists or is needed: 1.27.0's tree carries both fixes. + +**What was observed before publishing, from a `virgin` machine each time:** + +- R-300 — pre-fix uninstall left dnsmasq `enabled`/`active` on `0.0.0.0:53`; the next byo install + refused, exit 1. Fixed path: recorded `not present before Felhom` → `stopping + disabling it` → + `:53 FREE` → preflight PASS. The owner's side was proven too (record `yes` → left running). +- R-297 — a stale `golden-0.98.3.tar.zst` planted as the newest-by-filename archive; v1.25.0 took it + with no comparison and **the box came up on controller 0.98.3** against a vouched 0.213.0 — below + the update floor and below v0.206.0 where the off-site recovery screen exists. Fixed path: the + auto-discovered archive was rejected and the vouched one re-fetched and sha-verified (landed + 0.213.0); an operator-named stale archive was **refused**, exit 1. + +**Known limit of the R-300 fix, filed as R-305:** it fires only on a machine that never had Felhom. +The package is never purged, so every later install records `dnsmasq_preexisting=yes` and the +leftover returns on the second reinstall — proven on cycle 2/3 the same day. + +--- + ## felhom-host-install.sh v1.27.0 — our removal no longer blocks our reinstall (2026-08-12, R-300) **Felhom's own uninstall left the thing that makes Felhom's own next install refuse.** `--uninstall`