docs(report): TASK-H - demo-hp on the tailnet, node inventory, R-59/60/61
This commit is contained in:
@@ -1,132 +1,132 @@
|
|||||||
# REPORT — TASK-G: arm the HP install ISO + assisted-picker filing (2026-07-21)
|
# REPORT — TASK-H: HP node access + docs + today's findings (2026-07-21, evening)
|
||||||
|
|
||||||
Baseline `0073691`, clean tree; ISO tooling unchanged since v1.22.0, so this is a **rebuild at
|
Baseline `59226ed`, clean tree. No product code touched; the N100 and guest 9201 were not modified.
|
||||||
v1.22.0** with a new profile name (`hp-demo`), not a version bump.
|
|
||||||
|
|
||||||
| Part | Status |
|
| Part | Status |
|
||||||
|---|---|
|
|---|---|
|
||||||
| 1 — arm the ISO | **DONE**, filter verified from inside the built ISO |
|
| 1 — tailscale on demo-hp | **DONE**, one item open (key expiry — needs an admin-console toggle) |
|
||||||
| 2 — write it (gated) | **DONE**, byte-proven, and re-verified from the written stick |
|
| 2 — inventory + docs | **DONE** — `documentation/operations/nodes.md` |
|
||||||
| 3 — filing | **DONE** — **R-58** |
|
| 3 — findings filed | **DONE** — **R-59, R-60, R-61** + positive evidence recorded |
|
||||||
|
|
||||||
## Part 1 — the armed ISO
|
## Part 1 — tailscale on demo-hp
|
||||||
|
|
||||||
Built from an **uncommitted** working profile, `/mnt/5_hdd/felhom.eu/felhom-iso/hp-demo.profile`
|
`tailscale 1.98.9` (Debian trixie apt repo, host package — same shape as `felhom-pve`).
|
||||||
(mode 0600, the `n100-demo.profile` precedent — a real target serial never goes in git). It is the
|
|
||||||
committed `profiles/hp.profile` with exactly one change: the placeholder serial replaced.
|
|
||||||
|
|
||||||
```
|
```
|
||||||
felhom-pve-9.2-1-v1.22.0-hp-demo-generic.iso
|
demo-hp 100.76.96.79 online direct path 37.191.56.193:45127
|
||||||
sha256 de693f622c4c628d75eade691b2e8cb008175231f3ed97e29535e61de12bbd0b
|
|
||||||
bytes 1 705 338 880
|
|
||||||
loader shim (stock MS-signed chain; Secure Boot OK) secret-bearing: no
|
|
||||||
```
|
```
|
||||||
|
|
||||||
`validate-answer` gated on its **output text**, not `$?` — log shows
|
Verified **from DooPlex over the tailnet, no jumphost**: ping ~40 ms, `ssh root@100.76.96.79` works,
|
||||||
`The answer file was parsed successfully, no errors found!` through `gate_validate_answer`.
|
and the path is **direct**, not a DERP relay. `~/.ssh/config` gains `demo-hp` (tailnet) and
|
||||||
|
`demo-hp-lan` (`192.168.0.87` via `ProxyJump felhom-pve`) as the away-fallback.
|
||||||
|
|
||||||
### The filter, extracted from INSIDE the built ISO
|
The pre-auth key was passed as `--authkey=file:<path>` and shredded immediately, so it never entered
|
||||||
|
the box's process list.
|
||||||
|
|
||||||
Not read from the profile — from the artifact, via `osirrox`, comments stripped:
|
### OPEN — key expiry is NOT disabled
|
||||||
|
|
||||||
```
|
`demo-hp` expires **2027-01-17**; `felhom-pve` and `dooplex` both have expiry disabled, so this is
|
||||||
[disk-setup]
|
the one place it diverges from the fleet convention. **I could not do it from here:** disabling
|
||||||
filesystem = "ext4"
|
per-device key expiry is an admin-console toggle or an API call, and the supplied `tskey-auth-…`
|
||||||
filter-match = "all"
|
pre-auth key cannot drive the API (verified — `GET /api/v2/tailnet/-/devices` returns **401**).
|
||||||
filter.ID_SERIAL_SHORT = "182195804614"
|
|
||||||
```
|
|
||||||
|
|
||||||
| assertion | result |
|
**One click when convenient:** Tailscale admin → Machines → `demo-hp` → *Disable key expiry*. Or
|
||||||
|---|---|
|
hand me a `tskey-api-…` token and I will do it. Verify with:
|
||||||
| pins the SanDisk X600 128GB (`sda`) `182195804614` | 1 occurrence |
|
`ssh demo-hp 'tailscale status --json' | python3 -c "import sys,json;print(json.load(sys.stdin)['Self'].get('KeyExpiry'))"` → should print `None`.
|
||||||
| NVMe `58BS11AFT8MQ` present anywhere in the answer file | **0** |
|
|
||||||
| `REPLACE-WITH-TARGET-SSD-SERIAL` placeholder remaining | **0** |
|
|
||||||
| `filter.*` directives | exactly **1** |
|
|
||||||
| `disk-list =` directive | **0** |
|
|
||||||
|
|
||||||
`filter-match = "all"` with a single `ID_SERIAL_SHORT` is an exact-one-disk selector. **The 1TB NVMe
|
### A rule I broke, then fixed
|
||||||
is outside the filter** and is not touched by the installer; it joins later through the normal
|
|
||||||
Tárhely flow.
|
|
||||||
|
|
||||||
*Method note:* my first sanity check grepped the whole profile and reported the NVMe serial as
|
The join omitted `--accept-dns=false`, and MagicDNS immediately rewrote `/etc/resolv.conf` to
|
||||||
present — it appears in a **comment** documenting what is excluded. Re-verified by sourcing the
|
`nameserver 100.100.100.100` — exactly what `operations/tailscale.md` forbids for a host node.
|
||||||
profile and inspecting the consumed `FELHOM_DISK_SETUP` variable, then again from inside the ISO,
|
|
||||||
which is what actually ships. Grepping a file is not the same as reading what a tool consumes.
|
|
||||||
|
|
||||||
## Part 2 — the write
|
Nothing broke *at the vacation site*: there is no pi-hole there, and both `gitea.dooplex.hu` and
|
||||||
|
`hub.felhom.eu` resolve publicly anyway. **The damage would have surfaced silently when the box comes
|
||||||
|
home**, where split-horizon is what makes `gitea.dooplex.hu` resolve to `192.168.0.180` locally — a
|
||||||
|
class of failure that looks like "the network is slow" rather than "DNS is wrong".
|
||||||
|
`tailscale set --accept-dns=false` restored `nameserver 192.168.0.1`; tailnet and agent unaffected.
|
||||||
|
The doc now carries the evidence and the instruction to pass the flag **at join time**.
|
||||||
|
|
||||||
Same procedure as TASK-F, no shortcuts. Full evidence block on the Samsung stick
|
## Part 2 — the node inventory
|
||||||
(`/dev/disk/by-id/usb-Samsung_Flash_Drive_0375123010001586-0:0` → `/dev/sdc`, 119.5 GiB):
|
|
||||||
|
|
||||||
| check | result |
|
**`documentation/operations/nodes.md`** (new). The fleet is now two hosts, both agent 0.92.1:
|
||||||
|---|---|
|
`demo-felhom-8363b5` (N100) and `demo-hp-bb76ea` (HP t740). Both are at the vacation site and travel
|
||||||
| `ID_BUS=usb` | usb |
|
home ~2026-08-02.
|
||||||
| `removable=1` | 1 |
|
|
||||||
| `findmnt` (device **and** every partition) | empty |
|
|
||||||
| `/etc/fstab` | 0 lines |
|
|
||||||
| LVM PV | 0 (only `/dev/sda3` is a PV) |
|
|
||||||
| ZFS | 0 pools on the host |
|
|
||||||
| guest configs referencing it | 0 |
|
|
||||||
| open handles | 0 |
|
|
||||||
|
|
||||||
Destroyed: the `hp-generic` **safety** ISO written in TASK-F. Excluded: `/dev/sdb`, the ADATA
|
**demo-hp:** HP t740 Thin Client, s/n `8CN944035T`, AMI M42 v01.10 (11/11/2020), Ryzen Embedded
|
||||||
HD710 PRO — mounted at `/mnt/hdd_1` and bind-mounted into guest 9201, i.e. **live customer data**;
|
V1756B (8 threads), **30 GiB RAM**, PVE 9.2.2 as node `felhom-host`, guest 9201 `demo-hp` running,
|
||||||
it reports `ID_BUS=ata` and `removable=0`, so it fails both required positive checks despite a
|
WireGuard `10.77.0.3/32` up.
|
||||||
`usb-…` by-id name. Operator confirmed the printed device; identity was **re-asserted at write
|
|
||||||
time** in case of re-enumeration since the confirm.
|
|
||||||
|
|
||||||
```
|
| device | serial | role |
|
||||||
device sha256: de693f622c4c628d75eade691b2e8cb008175231f3ed97e29535e61de12bbd0b
|
|---|---|---|
|
||||||
iso sha256: de693f622c4c628d75eade691b2e8cb008175231f3ed97e29535e61de12bbd0b
|
| `sda` SanDisk X600 128GB | `182195804614` | system disk — PVE + LVM + guest volumes |
|
||||||
RESULT: MATCH - byte-proven
|
| `nvme0n1` Toshiba KXG50 1024GB | `58BS11AFT8MQ` | **PRESENT AND UNENROLLED — do not touch** |
|
||||||
```
|
|
||||||
|
|
||||||
`dd … oflag=direct conv=fsync`, then the ISO's exact byte-length read back from the device and
|
The NVMe still holds its previous **NTFS** partition, is unmounted, and is in no LVM PV and no ZFS
|
||||||
hashed. Customer data drive re-checked after the write: still mounted at both paths.
|
pool — the exact-serial filter did its job on hardware it had never seen. It joins later through the
|
||||||
|
**Tárhely flow, never the installer**.
|
||||||
|
|
||||||
**Belt-and-braces:** the stick was then mounted read-only and the filter read back off the physical
|
**NIC map** (documented in both `nodes.md` and `scripts/iso/README.md`): `enp1s0f0–f3` are the
|
||||||
device — `filter.ID_SERIAL_SHORT = "182195804614"`, NVMe serial absent. The profile is what I
|
4-port `igb` card with **no carrier and no DHCP** at this site; **`enp2s0f0`** (`r8169`, MAC
|
||||||
intended, the ISO is what I built, and the stick is what you will boot; all three now agree.
|
`7c:d3:0a:77:d9:76`) is the onboard port that works and is now `vmbr0`'s bridge-port. MACs for all
|
||||||
|
five are in the doc.
|
||||||
|
|
||||||
Staged ISO copies removed from `felhom-pve:/root/` (71G free).
|
**Loader/firmware finding:** the box installed with the **shim** loader and **Secure Boot ENABLED**
|
||||||
|
(`mokutil --sb-state` → `SecureBoot enabled`). That retires an assumption — SB-off was an
|
||||||
|
N100-firmware workaround, not a Felhom requirement.
|
||||||
|
|
||||||
## Part 3 — R-58, assisted disk-picker install mode
|
**Access, and why it is awkward:** no operator SSH key is baked (the HP profile deliberately left
|
||||||
|
`FELHOM_ROOT_SSH_KEY` blank), so authentication is the **G1 break-glass root password vaulted in the
|
||||||
|
hub** (`host_recovery` row `demo-hp-bb76ea`, set 2026-07-21 16:24 UTC). I retrieved it by streaming
|
||||||
|
`/data/hub.db` out of the hub pod, extracting one field to a 0600 file, and **shredding the copy
|
||||||
|
immediately** — that DB holds every host's secret. Recipe is in the doc. This lockout is R-61.
|
||||||
|
|
||||||
Filed with your argument verbatim. Shape: a third ISO mode beside unattended-serial and
|
**The operator-lab exception is documented prominently**, in both `nodes.md` and `tailscale.md`:
|
||||||
match-nothing-safety — enumerate candidates with size/model/serial, exclude the install media, take
|
demo-hp is *customer-shaped* but tailscale is **not** part of that shape. Real customer boxes get the
|
||||||
a selection plus a confirm, proceed. Unattended+serial stays the appliance/factory mode.
|
WireGuard tunnel and the H1 OOB path and nothing else. A future product-shape audit finding tailscale
|
||||||
|
here must not conclude the product ships it.
|
||||||
|
|
||||||
**Slice 1 is the abort screen**, deliberately, because it is the same enumeration code and it
|
## Part 3 — findings filed
|
||||||
collapses the two-boot dance on its own: on filter-no-match, print the candidate table plus the
|
|
||||||
one-line hint naming which serial to put in the profile. Today the fail-safe is correct but mute —
|
|
||||||
which is exactly what made this task need a hand-read serial between two boots.
|
|
||||||
|
|
||||||
It matters most for the **BYO / reinstall** flow: a customer's existing hardware, or a rebuild of a
|
**TASK-G Part 3 verified as already filed** — R-58 (assisted disk-picker) exists and includes the
|
||||||
box whose disk layout nobody recorded, is precisely where the serial is unknown and a wrong guess is
|
abort-screen candidate-table slice. Nothing to complete.
|
||||||
destructive.
|
|
||||||
|
|
||||||
Also documented `scripts/iso/README.md` § **HP mini profile**: shim proven on this board by the
|
New rows:
|
||||||
safety boot, the uncommitted-armed-profile pattern, verify-from-inside-the-ISO, and the prior-LVM
|
|
||||||
caveat below.
|
|
||||||
|
|
||||||
## For the operator — boot instruction
|
- **R-59 [P1] — a no-DHCP install must HARD-ABORT.** Instead it baked `192.168.100.2` as a *static*
|
||||||
|
`vmbr0` address and completed: the install "succeeded", the box looked finished, and it could never
|
||||||
|
call home. The worst silent onboarding failure shape there is. The philosophy already exists one
|
||||||
|
layer over — the disk filter refuses loudly and touches nothing when it cannot identify its target;
|
||||||
|
networking should do the same, with the candidate NIC table on screen (same grammar as R-58 slice 1).
|
||||||
|
- **R-60 [P2] — first-boot NIC sweep self-heal.** Hub unreachable ⇒ DHCP across every carrier-bearing
|
||||||
|
NIC before settling. Today's repair was a human moving one cable; a sweep would have healed it
|
||||||
|
unaided. Scoped to first boot and the hub-unreachable condition only — a running box must never
|
||||||
|
re-shuffle its own networking.
|
||||||
|
- **R-61 [P1] — the baked root password must be knowable.** The ISO mints a throwaway hash per build
|
||||||
|
and discards the plaintext, so nobody can reach the console of a box they just installed. Slice 1:
|
||||||
|
emit it into the build REPORT + operator cheat-sheet alongside the sha256. **A fixed well-known
|
||||||
|
password is explicitly rejected** (operator ruling) — a pre-pairing box sits on a stranger's LAN.
|
||||||
|
|
||||||
**Stick → HP, boot it, walk away.** It formats the SanDisk `sda`, installs, and reboots into day-0.
|
Positive evidence, same session:
|
||||||
Secure Boot stays **ON** (this is the shim loader, proven on this board). Nothing else to press.
|
|
||||||
|
|
||||||
**One thing that could stop it, worth knowing before you walk away.** The Proxmox auto-installer does
|
- **R-21 slice C is no longer a one-board result.** The row and the capability-map ISO row now read
|
||||||
not force-clear a pre-existing **LVM PV** signature. If that SanDisk carries a previous
|
**PROVEN-LIVE on TWO different boards** (N100 2026-07-18; HP t740 2026-07-21) — virgin hardware,
|
||||||
Linux/Proxmox install, the install aborts at *"cannot init physical volume"* — safely, touching
|
one pass, self-registration → operator bind → day-0 → running guest + agent check-in. The Secure
|
||||||
nothing. A previous Windows/NTFS install is fine. If you see that message: wipe the target
|
Boot finding is recorded there too.
|
||||||
(`blkdiscard -f /dev/sda` from any live environment) and boot **the same stick** again — the ISO does
|
- **Fresh-box floor lift**: the new box came up on golden **0.153.0** and self-updated to the fleet
|
||||||
not need rebuilding. This is documented behaviour (spike S2b), not a defect in this build.
|
floor **0.156.0** during day-0, unattended. Cited on the publish-train capability-map row — the
|
||||||
|
floor mechanism works on first contact, not only on boxes with history.
|
||||||
|
- **The t740 five-NIC trap** is recorded as a board gotcha in `scripts/iso/README.md`.
|
||||||
|
|
||||||
## Remaining (unchanged from TASK-F unless noted)
|
## Observations
|
||||||
|
|
||||||
- **Agent v0.93.0 is source-only** — committed, unbuilt, unpublished; rides the next agent train.
|
1. **The break-glass path works, and it is also the argument for R-61.** Reaching a box whose root
|
||||||
- **R-17** — deleting the `u629193-sub1` archive is an operator **console** action.
|
password was never known required a working hub, a working network, and operator tooling — at
|
||||||
- **The full catalog resolvability table** is still owed (needs one `docker login` on DooPlex).
|
exactly the moment the reason you want the console is usually that one of those is broken.
|
||||||
- **wanderer** was re-pinned to a 3-service upstream shape and has never been deployed.
|
2. **`apt update` fails on this box** against the PVE **enterprise** repos (401, no subscription) —
|
||||||
- **`hidden` lifecycle state** has no catalog user yet; only `abandoned` is exercised end to end.
|
pre-existing from the install, unrelated to today. I worked around it with a list-scoped
|
||||||
- **After the HP comes up:** bind it on the hub and run day-0; the 1TB NVMe is added afterwards
|
`apt-get update` rather than editing the box's repo config. Worth deciding whether `host-install`
|
||||||
through the Tárhely flow, never through the installer.
|
should switch fresh boxes to the no-subscription repo; left alone deliberately.
|
||||||
|
3. **A heredoc silently ate a piped secret.** `printf … | ssh host 'bash -s' <<'EOF'` sends the
|
||||||
|
*heredoc* as stdin, so the piped key never arrives and `read` consumes script text instead. The
|
||||||
|
working shape is the command as an argument, with stdin free for the secret — then stage it as a
|
||||||
|
file and use `--authkey=file:`. Worth remembering next time a credential has to cross an SSH hop.
|
||||||
|
|||||||
Reference in New Issue
Block a user