171 lines
9.6 KiB
Markdown
171 lines
9.6 KiB
Markdown
# REPORT — The removal that works once, and a status page that says what it is asking for (2026-08-13)
|
|
|
|
**Shipped:** `installer-v1.28.0`, published and verified against the live URL.
|
|
**Register:** R-305 CLOSED (by R-316), **R-316 / R-317 / R-318 opened**; ceiling R-315 → **R-318**.
|
|
**Venue:** `drill-r50` only. Neither demo box reinstalled. `peti-felhom` not contacted. Nothing deleted.
|
|
|
|
---
|
|
|
|
## 1. Cycle 3, before and after
|
|
|
|
**Before — on the PUBLISHED v1.27.0, from `virgin`, exit 1:**
|
|
|
|
```
|
|
[ERROR] a resolver is already bound to :53 on this host:
|
|
udp UNCONN 0 0 0.0.0.0:53 … users:(("dnsmasq",pid=7076,fd=4)) …
|
|
[ERROR] a resolver is already bound to :53 on this host — Felhom needs the guest reachable by name on your LAN.
|
|
Stop or reconfigure that resolver, OR point your LAN DNS at the guest's address, then re-run.
|
|
(Felhom does NOT touch DNS services on a host it does not own — this is a refusal, not a change.)
|
|
THIS LOOKS LIKE OURS. A previous Felhom install leaves the dnsmasq PACKAGE installed and its unit
|
|
enabled (only our config snippet is removed), and unconstrained it binds 0.0.0.0:53 — which is what
|
|
this gate is seeing. If this host had no dnsmasq before Felhom, clear it with:
|
|
systemctl disable --now dnsmasq
|
|
Then re-run this installer. If dnsmasq is YOURS, leave it and use one of the two routes above.
|
|
[ERROR] PRE-FLIGHT FAIL (exit 1) — fix the finding above and re-run
|
|
```
|
|
|
|
**After — v1.28.0, three fresh cycles from `virgin`, exit 0:**
|
|
|
|
```
|
|
:53 now: 0
|
|
CYCLE 3 rc=0
|
|
[INFO] host DNS (:53): free
|
|
[OK] pre-flight passed
|
|
[OK] PRE-FLIGHT PASS (mode=byo) — no state written, no install step executed
|
|
```
|
|
|
|
The three cycles were run on bytes **sha256-identical** to what the live URL now serves
|
|
(`cca1dedd9b7c152c…`, compared three ways: live URL, tested file, repo main).
|
|
|
|
## 2. The mechanism, at `file:line`
|
|
|
|
| | |
|
|
|---|---|
|
|
| ownership **recorded** | `felhom-host-install.sh:1718` — preflight, `dpkg-query -W -f='${Status}' dnsmasq \| grep -q "install ok installed"` |
|
|
| ownership **read** | `:1097` (`_state_get dnsmasq_preexisting`) at uninstall |
|
|
| state file **deleted** | `:1268` — *after* the read. **The order was already correct.** |
|
|
| Felhom's dnsmasq snippets removed | `:1162`, in the loop just above the ownership decision |
|
|
|
|
**Why cycle 2 concludes "pre-existing": package presence alone.** The preflight asks dpkg one question
|
|
and nothing else — it does **not** consult the absence of a record, and it cannot, because the record
|
|
was deleted with the state file. v1.27.0 stopped the unit and left the package, so the answer stayed
|
|
`yes` and our own package became "the household's" one cycle later.
|
|
|
|
**What the first uninstall did and did not remove:** snippets — removed. Unit — stopped + disabled.
|
|
State file (and with it the record) — removed. **Package — left.** That last one is the whole defect.
|
|
|
|
## 3. What v1.28.0 changes
|
|
|
|
When the record says we installed it, the uninstall **removes the package** as well as stopping the
|
|
unit. Order unchanged: read the record → act → delete the state.
|
|
|
|
**Two packages are recorded, not one.** `dnsmasq` ships the systemd unit; **`dnsmasq-base` ships
|
|
`/usr/sbin/dnsmasq`** (`dpkg -S`, measured on the box). Separately installable, so each is recorded at
|
|
preflight and taken back only if we added it.
|
|
|
|
**Guard rails:** ownership **read, never inferred**; the dependency check is an **`apt-get -s purge`
|
|
simulation** that proceeds only if the removal set is a subset of ours, else stop+disable **naming the
|
|
blocking package**; never interactive; **never fatal** — a wedged apt is recorded and restated in the
|
|
closing NOTE; and the success is **re-queried** rather than read off apt's exit code.
|
|
|
|
## 4. Scenarios and red-proofs
|
|
|
|
| Scenario | Result |
|
|
|---|---|
|
|
| **A** three cycles, fixed | install 3 **PASSES**, `:53` free at every uninstall |
|
|
| **B** resolver pre-dates Felhom | **untouched** — `install ok installed`, unit active, both packages recorded `yes` |
|
|
| **C** something depends on it | **not purged**; log named `household-dns-thing`; stop+disable; `:53` free; dependent survived |
|
|
| **D** no ownership record | **untouched**, reason logged, exact command named |
|
|
|
|
**Red-proofs — mutation asserted applied before each run:**
|
|
|
|
| Mutation | Outcome |
|
|
|---|---|
|
|
| remove the purge call | cycle 2 records `yes`; **cycle 3 refuses, exit 1, in those exact words** |
|
|
| remove the ownership check | **the household's resolver is PURGED** (`unknown ok not-installed`) |
|
|
| infer ownership when no record | **the guess is taken** — a field box loses its own DNS |
|
|
|
|
**One honest note on red-proof B.** Its first run appeared to pass *for the wrong reason*: apt failed
|
|
with `dpkg was interrupted` (a broken state my own earlier manual package juggling left), so the
|
|
resolver survived by accident rather than by the guard. I repaired dpkg and re-ran, and it then failed
|
|
as required. Worth stating twice over: the mutation looked like a passing guard, and **my own fix's
|
|
apt-failure path was incidentally observed doing exactly what it should** — logging, continuing, naming
|
|
the command.
|
|
|
|
## 5. The machines already in the field
|
|
|
|
**Measured, not reasoned.** A field box is one with the package installed and **no record** — scenario
|
|
D. Its uninstall leaves the resolver running with the reason logged, and its next install refuses with
|
|
the message quoted in §1.
|
|
|
|
**Is there an honest durable marker? No, and none can be invented.** The Felhom `/etc/dnsmasq.d/
|
|
felhom-*.conf` snippets are deleted by the uninstall's own loop *before* the ownership decision; the
|
|
state file carrying the record is deleted at `:1268`; nothing under `/etc/felhom*` survives.
|
|
`/var/log/dpkg.log` does record the install and is a **timestamp** — refused by the standing rule as a
|
|
heuristic dressed as a fact. → **R-318**
|
|
|
|
**The preflight message, judged as a customer would:** it states the finding, keeps its two routes and
|
|
its written promise not to touch DNS on a host we do not own, and adds the one thing that gets a person
|
|
moving — *"THIS LOOKS LIKE OURS"* plus one exact command. It hedges correctly (*looks like*). Its
|
|
weakness is that it asks them *"did this host have dnsmasq before Felhom?"* — precisely the question we
|
|
can no longer answer for them. **That is a mechanism, not a rule: the command is on the screen at the
|
|
moment it is needed.**
|
|
|
|
## 6. The defect I nearly shipped
|
|
|
|
The agent decides whether to install dnsmasq with `os.Stat("/usr/sbin/dnsmasq")`
|
|
(`felhom-agent/internal/lanresolver/lanresolver.go:105`) — but that path belongs to **`dnsmasq-base`**,
|
|
while the unit comes from **`dnsmasq`**. Purging only `dnsmasq` would leave the binary, so the next
|
|
install would skip the apt step and then fail to enable a unit that is gone — **a silent resolver where
|
|
today there is at least a visible refusal.** That is why ownership is recorded per package.
|
|
|
|
It remains reachable where `dnsmasq-base` pre-dated Felhom (we correctly keep it). **Pre-existing, not
|
|
introduced here**, filed as **R-317**, and the uninstall now says so out loud rather than leaving it to
|
|
be found from a resolver that never came up. Fixing it is a one-line agent change, deliberately **not**
|
|
made here to keep this session to one repo.
|
|
|
|
## 7. Publication
|
|
|
|
`installer-v1.28.0` tagged; **both** `--ref`s in `manifests/webpage.yaml` moved (sidecar 327, init 372);
|
|
ArgoCD synced to `823cd29`; live deployment carries the new ref.
|
|
|
|
```
|
|
live URL serves: SCRIPT_VERSION="1.28.0" (was 1.27.0)
|
|
_dnsmasq_purge_owned present in the served file: 3
|
|
```
|
|
|
|
**Pushing publishes nothing here** — `/scripts/` follows the tag. The runbook still says otherwise;
|
|
**R-309 remains open and is not forgotten**, deliberately out of scope.
|
|
|
|
## 8. Teardown — four layers
|
|
|
|
| Layer | State |
|
|
|---|---|
|
|
| **machine** | `drill-r50`: 0 guests, no `/var/lib/felhom-install`, dnsmasq `not-installed` at the end |
|
|
| **host** | `drill-r50` **reverted to snapshot `virgin`, powered off**; `qemu.pid` removed. DooPlex was drill host only |
|
|
| **hub** | `drill-r50-0a4f9a` **RETAINED** — pre-existing since 2026-07-25, re-used not duplicated. **No new host or customer row was created this session.** Nothing deleted |
|
|
| **off-site** | **Not contacted at all** — no restic, sftp or endpoint call was made. `demo-felhom`: `last_status=ok`, `escrow=escrowed`, **no abandon countdown** |
|
|
|
|
Both demo boxes untouched and healthy: controller **0.214.0**, agent **0.129.0**, `Up … (healthy)`.
|
|
|
|
## 9. Evidence, and a repeated miss
|
|
|
|
Logs at `documentation/audits/evidence-r316-2026-08-13/`: the fixed cycles (`F1`, `F2`, `F3`), all four
|
|
scenarios (`B`, `C`, `D`), and every red-proof (`RPA*`, `RPB*`, `RPD*`).
|
|
|
|
**The Part 1 logs did not survive.** They were on the drill VM's disk and were destroyed by the revert
|
|
to `virgin` between Part 1 and Part 2 — **the same mistake as Tuesday, in the same place.** The §1
|
|
quotation is verbatim from the live run as read at the time, and `RPA1/RPA2/RPA3.log` are an
|
|
independent reproduction of the identical three-cycle failure, retained. Recorded rather than glossed;
|
|
the fix is procedural and I have now got it wrong twice.
|
|
|
|
## 10. Observations — noticed, not acted on
|
|
|
|
- **`dnsmasq-base` is flagged "automatically installed and no longer required"** after `dnsmasq` goes.
|
|
We deliberately do not `autoremove` — that would be a blast radius nobody asked for.
|
|
- **`apt-get -s purge` exits 0 even when it prints `E: dpkg was interrupted`.** The simulation output is
|
|
the signal, never the exit code — which is why the guard parses `Remv`/`Purg` lines rather than
|
|
trusting `$?`. Another entry for the exit-codes-that-lie class.
|
|
- The drill VM's `pveam` index is stale on `virgin` and needs `pveam update` before listing templates —
|
|
already recorded yesterday, hit again today in passing.
|