Files

T

gates / gates (push) Successful in 7s

Details

docs: R-182 closed, R-90 closed on measurement, R-86 unblocked, ep0 record corrected

R-182 CLOSED (controller v0.194.0 + hub v0.90.0/.1), proven live on demo-hp. The
hub's notification_log for the run reads: two per-app failures RECORDED, one
digest SENT naming both, and the customer channel SKIPPED with operator_only.
Against the measured previous behaviour — two failures, one email naming one
app, one leaving no trace anywhere.

Scenario D proved itself on an event I had not planned: disk_critical alarmed on
two filesystems, the second was collapsed by the cooldown, and that collapse is
now visible WITH ITS KEY. Yesterday it would have left nothing at all.

A gap the spec did not anticipate is recorded with its fix: the per-app event
also fires from the periodic sweep, outside any run, so making it record-only
would have created a NEW silence. The sweep emits a digest too, with no run_id,
so it stays under the ordinary hourly cooldown.

ep0: MEASURED on the box — 7757 MB (8 GB), 4 vCPU, and the 4 GiB swapfile
SURVIVED the resize and is active (checked, because a resize is a stop/start).
The 40 GB local disk is UNCHANGED, so no disk figure was touched anywhere.

Five documents corrected — three of which the task's list did not name, found by
searching. Two audit/evidence documents ANNOTATED, body untouched: they record
what was true when written and that is their value.

R-90 CLOSED. R-86 unblocked and re-ranked, stated honestly: 8 GB is comfortable,
not unbounded — the original OOM was a 14.46 GB restore — so the restore-test
cadence should still be paced, just not by fear of the endpoint.

target-selection.md's "D-d did not name ep0 either way" is deliberately left
standing. It is the operator's question, not CC's.

STATUS.md 127 -> 83 lines, items rather than sentences.

2026-08-03 14:00:54 +02:00

9.7 KiB

Raw Blame History

Campaign 10 — Phase A journal (2026-07-31)

All times CEST unless marked Z.

Venue


Host	`demo-hp` (HP t740), Tier 0, the designated drill/build host
VM	311 `c10-appliance` — q35 / OVMF (`pre-enrolled-keys=0`), 4 cores, 8 GB, cpu=host
Disks	`scsi0` 200 G (system) · `scsi1` 50 G · `scsi2` 50 G — all qcow2 on `c10-scratch`
Storage	`c10-scratch`, `dir` at `/mnt/nvme-1tb` — the mount ROOT, `content=images`. Root chosen deliberately: a storage at a subdirectory sets `exactMount=false` → `reachable=false` → `StorageStateDisconnected` (`felhom-agent/internal/storage/observe.go:336`), which would emit `storage_disconnected` for demo-hp for the whole run — the exact signal I1/I2 test. Coexists with `felhom-backup` on the same path; that storage was not modified.
Guest network	demo-hp's vmbr0 is `192.168.0.87/24`, a different physical LAN from DooPlex's own 192.168.0.0/24. Reached via `ssh -J demo-hp`.
Console	no browser on DooPlex → `qm monitor screendump` → PPM → PNG, read visually; keystrokes via `sendkey`
Untouched	`drill-r50` (VM 300), guest 9201 on both demo boxes, `local-lvm`, both backup targets

Baselines — every value read fresh at the start of this run

What	Value	How read
controller `main`	v0.188.0	`CHANGELOG.md` top; `HEAD`=`origin/main`=`4115e88`, tree clean
golden's baked controller	0.188.0 (sha `7353d8be…`)	hub `/configuration`, selected option
agent published + vouched	0.119.0 (sha `e37aca82…`)	hub `/configuration`, selected option
agent on both demo boxes	0.119.0	`felhom-agent --version`
agent `main`	v0.119.0	`felhom-agent/CHANGELOG.md`
hub	0.86.0	live `deploy/hub` image tag == `hub/CHANGELOG.md` top
ISO	1.26.1, sha256 `f3cc86d5…`, 1 705 322 496 B	live round trip from `https://iso.felhom.eu/`
`felhom.eu` HEAD	`e9a74a0`, clean	`git rev-parse`
managed floor	v0.156.0 (DB override); env fallback v0.120.0	hub `/configuration`

Golden vs main: NOT behind — both 0.188.0. The deliberate decision the brief asks for is therefore vacuous this run: the campaign tests what is shipping and what is current, with no divergence.

Two brief assumptions were stale and were corrected by reading fresh:

The brief was written against ISO v1.25.0; iso.felhom.eu began serving 1.26.1 at 18:37 Z the same day — ~90 minutes before this run started.
The brief (and CLAUDE.md, target-selection.md:41-42) say demo-hp has no baked SSH key. ssh -o BatchMode=yes demo-hp authenticated by key, first try. Already filed as R-129 by the 2026-07-31 tester-gate run — not re-filed (register grep per the fences).

A1 — build the appliance from the published ISO

1.26.1 is a PUBLIC RELEASE image and cannot auto-install. Verified against the ISO bytes on demo-hp, not inferred from the manifest:

Probe	Hits
`auto-installer-mode.toml`	0
`answer.toml`	0
`proxmox-auto-install`	0
`felhom-bootstrap_1.26.1_all.deb`	1
Felhom grub menuentries	exactly 2, both interactive

Manifest agrees: mode: release … NO answer.toml, NO baked credential, interactive disk selection, root-password: NONE — the installer prompts, loader: shim (stock MS-signed chain).

Driven blind through the Terminal UI entry (the graphical entry is not drivable — the QEMU monitor's mouse events do not move the guest cursor, per REPORT-iso-release.md). Screendump before every keystroke.

Traps hit and handled, each verified rather than assumed:

GRUB default is the graphical entry. The first attempt auto-booted it because the down keystroke landed after the 15 s countdown expired. Reset, and sent down inside the window in one round trip.
Guest keymap was Hungarian, but sendkey emits US scancodes — every symbol afterwards (notably @, which is AltGr+V on a Hungarian layout) would have been silently mangled, including the root password, locking the box out. Switched to U.S. English before typing anything. Confirmed positively: c10@felhom.eu rendered with a correct @ and ..
Disk-selection safety: only scsi0 was attached during the install, so the installer offered exactly one target (/dev/sda, 200 GiB) and could not reach a data drive. The two 50 G data drives were attached afterwards.
Boot-order trap (REPORT-iso-release.md, recorded twice there): --boot was set in its own qm set after the disks existed, and verified from qm config (boot: order=ide2;scsi0). After the install, the order was flipped to order=scsi0 — again verified from qm config, not from the screen.

Real completion signal (not "an artifact changed"): 4.2 GB actually written to vm-311-disk-1.qcow2, the installed system booted on its own, and the console showed the PVE banner at https://192.168.0.105:8006/.

Positive observables — all three present:

Observable	Evidence
package delivered	`ii felhom-bootstrap 1.26.1 all` on the box
unit fired at first boot	`systemctl is-enabled felhom-bootstrap` = `enabled`; `is-active` = `activating`
registers as an unclaimed appliance	hub `/hosts`: uuid `55ab3776-…` (matches VM smbios), pairing code FD6-A6S, MAC `bc:24:11:46:25:fe`, 3 SSH host keys, hardware read

A1 — PASS.

A2 — claim

Customer created: c10-soak — "Campaign 10 two-storage soak (SCRATCH - delete at teardown)", domain c10.felhom.eu, email nagyfenyvesi.viktor@gmail.com. Deliberately not drill-r50, sess-*, or any existing record.

Appliance 19 bound to it. Hub log, verbatim:

22:50:03 appliance 19 BOUND to customer c10-soak (mode=appliance) — delivery staged for its next poll
22:50:12 appliance credentials DELIVERED once to appliance 19 (passphrase withheld)
22:50:19 [claim] claim code (gen 1) emailed to the registered address of c10-soak
22:50:49 host enrolled: c10-soak-8a405e (customer c10-soak)
22:50:49 vaulted break-glass recovery credential for host c10-soak-8a405e (user=root@pam, secret 32 chars)
22:50:51 Artifact manifest served for customer c10-soak (agent=0.119.0 golden=0.188.0)
22:51:13 wg registered: host=c10-soak-8a405e ip=10.77.0.6/32 gen=1 sync=ok
22:51:14 host-report from c10-soak-8a405e (0 guests, 2 storage targets, ...)
22:51:14 DR-recipe host-half stored for customer c10-soak (host c10-soak-8a405e, v1)

The installer's root password stops working at 22:50:49 — day-0 replaces it with a hub-vaulted managed credential (G1 host_recovery). Retrieved via POST /hosts/c10-soak-8a405e/reveal-recovery-credential → root@pam, 32 chars, stored file→file at ~/.config/campaign10/host-recovery.json (0600), never printed.

Box state after day-0:

felhom-agent 0.119.0                                     (== published+vouched)
/etc/felhom-controller-image : …/felhom-controller:0.188.0 (== golden == main)
guest 9201 c10-soak running
docker: controller 0.188.0 (healthy), traefik v3.6.7, filebrowser 1.3.3-stable (healthy)
block devices: sda 200G, sdb 50G, sdc 50G

Claim gate, the positive discriminator, read over the path the UI uses (traefik on 443, Host: header mandatory — the controller does not listen on 8080; 80 is a 301 to HTTPS):

GET https://127.0.0.1/api/apps  Host: felhom.c10.felhom.eu
→ {"ok":false,"error":"dashboard not yet claimed"}

A2 — BLOCKED on the emailed claim code (R-119, confirmed live again). The code exists only in the customer's mailbox; there is no operator-side route.

Scope constraint found in Phase A — Tier 3 cannot be isolated, so it will not run

The brief's §2 requires the campaign's PBS to be a dedicated datastore and namespace on DooPlex. That is not reachable, for two reasons that are each already-recorded deliberate positions, not new defects:

Offsite hard-requires the DR tier. hub/internal/web/configs.go:1300 refuses offsite without it: "Offsite backup requires the DR tier — enable it first (the escrow ceremony depends on the PBS key)". Closed by policy 2026-07-12 (drill F-6, DR-tier-by-default decisions 3/4).
The DR tier only ever provisions on ep0. Peer allocation and endpoint sync use the lowest endpoint_id only; per-endpoint allocation is an explicitly deferred arc (hub/README.md:260).

So the only two configurations are: DR tier on → the campaign's PBS traffic lands on ep0, which is Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90 — note added 2026-08-03: ep0 has since been rescaled to a CX33 with 8 GB RAM; this journal records what was true when it was written and is deliberately not revised) and fenced by §3 — or DR tier off → no Tier 3 at all. Chosen: DR tier OFF, offsite OFF, which is the only option §3 permits.

Consequence, stated plainly rather than discovered later: the campaign touches neither ep0 nor the Storage Box at any point — a stronger isolation than the brief asked for, obtained by not running the tier. What is lost: all Tier-3 operations in B1, I8 entirely, and the Tier-3 rows of the RTO/RPO table. Everything else — the two-drive core (R-112/R-113/R-114/R-116/R-117), D5, I1–I7 and I9–I11 — is unaffected, because those live on the drives and the local/whole-guest tiers.

No R-n minted: per the fences, the register was grepped first and both halves are already recorded (F-6 policy close; the hub/README.md:260 deferral). This is a campaign constraint, not a finding.

Isolation gate — §2

See ../isolation/pbs-denial.txt and ../isolation/restic-denial.txt. Summary in the main audit.

9.7 KiB Raw Blame History Unescape Escape