admin
e34b614e5b
gates / gates (push) Successful in 7s
R-182 CLOSED (controller v0.194.0 + hub v0.90.0/.1), proven live on demo-hp. The hub's notification_log for the run reads: two per-app failures RECORDED, one digest SENT naming both, and the customer channel SKIPPED with operator_only. Against the measured previous behaviour — two failures, one email naming one app, one leaving no trace anywhere. Scenario D proved itself on an event I had not planned: disk_critical alarmed on two filesystems, the second was collapsed by the cooldown, and that collapse is now visible WITH ITS KEY. Yesterday it would have left nothing at all. A gap the spec did not anticipate is recorded with its fix: the per-app event also fires from the periodic sweep, outside any run, so making it record-only would have created a NEW silence. The sweep emits a digest too, with no run_id, so it stays under the ordinary hourly cooldown. ep0: MEASURED on the box — 7757 MB (8 GB), 4 vCPU, and the 4 GiB swapfile SURVIVED the resize and is active (checked, because a resize is a stop/start). The 40 GB local disk is UNCHANGED, so no disk figure was touched anywhere. Five documents corrected — three of which the task's list did not name, found by searching. Two audit/evidence documents ANNOTATED, body untouched: they record what was true when written and that is their value. R-90 CLOSED. R-86 unblocked and re-ranked, stated honestly: 8 GB is comfortable, not unbounded — the original OOM was a 14.46 GB restore — so the restore-test cadence should still be paced, just not by fear of the endpoint. target-selection.md's "D-d did not name ep0 either way" is deliberately left standing. It is the operator's question, not CC's. STATUS.md 127 -> 83 lines, items rather than sentences.
157 lines
9.7 KiB
Markdown
157 lines
9.7 KiB
Markdown
# Campaign 10 — Phase A journal (2026-07-31)
|
||
|
||
All times CEST unless marked Z.
|
||
|
||
## Venue
|
||
|
||
| | |
|
||
|---|---|
|
||
| Host | `demo-hp` (HP t740), Tier 0, the designated drill/build host |
|
||
| VM | **311 `c10-appliance`** — q35 / OVMF (`pre-enrolled-keys=0`), 4 cores, 8 GB, cpu=host |
|
||
| Disks | `scsi0` 200 G (system) · `scsi1` 50 G · `scsi2` 50 G — all qcow2 on `c10-scratch` |
|
||
| Storage | **`c10-scratch`**, `dir` at **`/mnt/nvme-1tb` — the mount ROOT**, `content=images`. Root chosen deliberately: a storage at a *subdirectory* sets `exactMount=false` → `reachable=false` → `StorageStateDisconnected` (`felhom-agent/internal/storage/observe.go:336`), which would emit `storage_disconnected` for demo-hp for the whole run — the exact signal I1/I2 test. Coexists with `felhom-backup` on the same path; that storage was **not modified**. |
|
||
| Guest network | demo-hp's vmbr0 is `192.168.0.87/24`, a **different physical LAN** from DooPlex's own 192.168.0.0/24. Reached via `ssh -J demo-hp`. |
|
||
| Console | no browser on DooPlex → `qm monitor screendump` → PPM → PNG, read visually; keystrokes via `sendkey` |
|
||
| Untouched | `drill-r50` (VM 300), guest 9201 on both demo boxes, `local-lvm`, both backup targets |
|
||
|
||
## Baselines — every value read fresh at the start of this run
|
||
|
||
| What | Value | How read |
|
||
|---|---|---|
|
||
| controller `main` | **v0.188.0** | `CHANGELOG.md` top; `HEAD`=`origin/main`=`4115e88`, tree clean |
|
||
| **golden's baked controller** | **0.188.0** (sha `7353d8be…`) | hub `/configuration`, *selected* option |
|
||
| agent published + vouched | **0.119.0** (sha `e37aca82…`) | hub `/configuration`, *selected* option |
|
||
| agent on both demo boxes | 0.119.0 | `felhom-agent --version` |
|
||
| agent `main` | v0.119.0 | `felhom-agent/CHANGELOG.md` |
|
||
| hub | **0.86.0** | live `deploy/hub` image tag == `hub/CHANGELOG.md` top |
|
||
| ISO | **1.26.1**, sha256 `f3cc86d5…`, 1 705 322 496 B | live round trip from `https://iso.felhom.eu/` |
|
||
| `felhom.eu` HEAD | `e9a74a0`, clean | `git rev-parse` |
|
||
| managed floor | v0.156.0 (DB override); env fallback v0.120.0 | hub `/configuration` |
|
||
|
||
**Golden vs `main`: NOT behind — both 0.188.0.** The deliberate decision the brief asks for is therefore
|
||
vacuous this run: the campaign tests what is shipping *and* what is current, with no divergence.
|
||
|
||
**Two brief assumptions were stale and were corrected by reading fresh:**
|
||
|
||
1. The brief was written against ISO **v1.25.0**; `iso.felhom.eu` began serving **1.26.1** at 18:37 Z
|
||
the same day — ~90 minutes before this run started.
|
||
2. The brief (and `CLAUDE.md`, `target-selection.md:41-42`) say demo-hp has **no baked SSH key**.
|
||
`ssh -o BatchMode=yes demo-hp` authenticated **by key, first try**. Already filed as **R-129** by the
|
||
2026-07-31 tester-gate run — **not re-filed** (register grep per the fences).
|
||
|
||
## A1 — build the appliance from the published ISO
|
||
|
||
**1.26.1 is a PUBLIC RELEASE image and cannot auto-install.** Verified against the ISO bytes on
|
||
demo-hp, not inferred from the manifest:
|
||
|
||
| Probe | Hits |
|
||
|---|---|
|
||
| `auto-installer-mode.toml` | **0** |
|
||
| `answer.toml` | **0** |
|
||
| `proxmox-auto-install` | **0** |
|
||
| `felhom-bootstrap_1.26.1_all.deb` | 1 |
|
||
| Felhom grub menuentries | exactly **2**, both interactive |
|
||
|
||
Manifest agrees: `mode: release … NO answer.toml, NO baked credential, interactive disk selection`,
|
||
`root-password: NONE — the installer prompts`, `loader: shim (stock MS-signed chain)`.
|
||
|
||
Driven blind through the **Terminal UI** entry (the graphical entry is not drivable — the QEMU monitor's
|
||
mouse events do not move the guest cursor, per `REPORT-iso-release.md`). Screendump before every
|
||
keystroke.
|
||
|
||
**Traps hit and handled, each verified rather than assumed:**
|
||
|
||
- **GRUB default is the graphical entry.** The first attempt auto-booted it because the `down` keystroke
|
||
landed after the 15 s countdown expired. Reset, and sent `down` inside the window in one round trip.
|
||
- **Guest keymap was Hungarian**, but `sendkey` emits US scancodes — every symbol afterwards (notably
|
||
`@`, which is AltGr+V on a Hungarian layout) would have been silently mangled, *including the root
|
||
password*, locking the box out. Switched to **U.S. English** before typing anything. Confirmed
|
||
positively: `c10@felhom.eu` rendered with a correct `@` and `.`.
|
||
- **Disk-selection safety:** only `scsi0` was attached during the install, so the installer offered
|
||
exactly one target (`/dev/sda`, 200 GiB) and could not reach a data drive. The two 50 G data drives
|
||
were attached afterwards.
|
||
- **Boot-order trap** (`REPORT-iso-release.md`, recorded twice there): `--boot` was set in its **own**
|
||
`qm set` after the disks existed, and verified from `qm config` (`boot: order=ide2;scsi0`). After the
|
||
install, the order was flipped to `order=scsi0` — again verified from `qm config`, not from the screen.
|
||
|
||
**Real completion signal** (not "an artifact changed"): 4.2 GB actually written to
|
||
`vm-311-disk-1.qcow2`, the installed system booted on its own, and the console showed the PVE banner at
|
||
`https://192.168.0.105:8006/`.
|
||
|
||
**Positive observables — all three present:**
|
||
|
||
| Observable | Evidence |
|
||
|---|---|
|
||
| package delivered | `ii felhom-bootstrap 1.26.1 all` on the box |
|
||
| unit fired at first boot | `systemctl is-enabled felhom-bootstrap` = `enabled`; `is-active` = `activating` |
|
||
| registers as an unclaimed appliance | hub `/hosts`: uuid `55ab3776-…` (matches VM smbios), pairing code **FD6-A6S**, MAC `bc:24:11:46:25:fe`, 3 SSH host keys, hardware read |
|
||
|
||
**A1 — PASS.**
|
||
|
||
## A2 — claim
|
||
|
||
Customer created: **`c10-soak`** — "Campaign 10 two-storage soak (SCRATCH - delete at teardown)",
|
||
domain `c10.felhom.eu`, email `nagyfenyvesi.viktor@gmail.com`. Deliberately **not** `drill-r50`,
|
||
`sess-*`, or any existing record.
|
||
|
||
Appliance 19 bound to it. Hub log, verbatim:
|
||
|
||
22:50:03 appliance 19 BOUND to customer c10-soak (mode=appliance) — delivery staged for its next poll
|
||
22:50:12 appliance credentials DELIVERED once to appliance 19 (passphrase withheld)
|
||
22:50:19 [claim] claim code (gen 1) emailed to the registered address of c10-soak
|
||
22:50:49 host enrolled: c10-soak-8a405e (customer c10-soak)
|
||
22:50:49 vaulted break-glass recovery credential for host c10-soak-8a405e (user=root@pam, secret 32 chars)
|
||
22:50:51 Artifact manifest served for customer c10-soak (agent=0.119.0 golden=0.188.0)
|
||
22:51:13 wg registered: host=c10-soak-8a405e ip=10.77.0.6/32 gen=1 sync=ok
|
||
22:51:14 host-report from c10-soak-8a405e (0 guests, 2 storage targets, ...)
|
||
22:51:14 DR-recipe host-half stored for customer c10-soak (host c10-soak-8a405e, v1)
|
||
|
||
**The installer's root password stops working at 22:50:49** — day-0 replaces it with a hub-vaulted
|
||
managed credential (G1 `host_recovery`). Retrieved via `POST /hosts/c10-soak-8a405e/reveal-recovery-credential`
|
||
→ `root@pam`, 32 chars, stored file→file at `~/.config/campaign10/host-recovery.json` (0600), never printed.
|
||
|
||
Box state after day-0:
|
||
|
||
felhom-agent 0.119.0 (== published+vouched)
|
||
/etc/felhom-controller-image : …/felhom-controller:0.188.0 (== golden == main)
|
||
guest 9201 c10-soak running
|
||
docker: controller 0.188.0 (healthy), traefik v3.6.7, filebrowser 1.3.3-stable (healthy)
|
||
block devices: sda 200G, sdb 50G, sdc 50G
|
||
|
||
Claim gate, the positive discriminator, read over the path the UI uses (traefik on 443, `Host:` header
|
||
mandatory — the controller does **not** listen on 8080; 80 is a 301 to HTTPS):
|
||
|
||
GET https://127.0.0.1/api/apps Host: felhom.c10.felhom.eu
|
||
→ {"ok":false,"error":"dashboard not yet claimed"}
|
||
|
||
**A2 — BLOCKED on the emailed claim code (R-119, confirmed live again).** The code exists only in the
|
||
customer's mailbox; there is no operator-side route.
|
||
|
||
## Scope constraint found in Phase A — Tier 3 cannot be isolated, so it will not run
|
||
|
||
The brief's §2 requires the campaign's PBS to be *a dedicated datastore and namespace on DooPlex*. That
|
||
is not reachable, for two reasons that are each **already-recorded deliberate positions**, not new defects:
|
||
|
||
1. **Offsite hard-requires the DR tier.** `hub/internal/web/configs.go:1300` refuses offsite without it:
|
||
*"Offsite backup requires the DR tier — enable it first (the escrow ceremony depends on the PBS key)"*.
|
||
Closed by policy 2026-07-12 (drill F-6, DR-tier-by-default decisions 3/4).
|
||
2. **The DR tier only ever provisions on ep0.** Peer allocation and endpoint sync use the lowest
|
||
`endpoint_id` only; per-endpoint allocation is an explicitly deferred arc (`hub/README.md:260`).
|
||
|
||
So the only two configurations are: *DR tier on* → the campaign's PBS traffic lands on **ep0**, which is
|
||
Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90 — **note added 2026-08-03: ep0 has since been rescaled to a CX33 with 8 GB RAM; this journal records what was true when it was written and is deliberately not revised**) and fenced by §3 — or *DR tier off* →
|
||
no Tier 3 at all. **Chosen: DR tier OFF, offsite OFF**, which is the only option §3 permits.
|
||
|
||
Consequence, stated plainly rather than discovered later: the campaign touches **neither ep0 nor the
|
||
Storage Box at any point** — a stronger isolation than the brief asked for, obtained by not running the
|
||
tier. What is lost: all Tier-3 operations in B1, **I8** entirely, and the Tier-3 rows of the RTO/RPO
|
||
table. Everything else — the two-drive core (R-112/R-113/R-114/R-116/R-117), D5, I1–I7 and I9–I11 — is
|
||
unaffected, because those live on the drives and the local/whole-guest tiers.
|
||
|
||
**No `R-n` minted**: per the fences, the register was grepped first and both halves are already recorded
|
||
(F-6 policy close; the `hub/README.md:260` deferral). This is a campaign constraint, not a finding.
|
||
|
||
## Isolation gate — §2
|
||
|
||
See `../isolation/pbs-denial.txt` and `../isolation/restic-denial.txt`. Summary in the main audit.
|