docs: REPORT + CONTEXT cover v0.128.1 + demo legacy dir-storage retirement (gates G1-G3, 4/4 verifications)

This commit is contained in:
2026-07-13 21:51:06 +02:00
parent d752f159b9
commit 3c9de42c20
2 changed files with 92 additions and 77 deletions
+13 -1
View File
@@ -7,7 +7,19 @@
>
> Ask Claude Code: "Please update CONTEXT.md with what we did today"
Last updated: 2026-07-13 night (v0.128.0browser .fab upload on /import)
Last updated: 2026-07-13 night (v0.128.1USB class-badge suppression + demo PVE dir-storage retirement)
> **2026-07-13 night — v0.128.1 + demo storage hygiene (ruling F5).** `classTag` suppresses the
> rotational class hint for `type==='usb'` (card already carries the USB tag; hub `ClassHint`
> UNCHANGED; pinned by `TestStorageTemplate_USBClassBadgeSuppressed` + red-proof). Host op on
> demo-felhom: the two pre-intermediary legacy `dir:` storages (`felhom-usb`, `felhom-flash`,
> content=Backup, is_mountpoint) RETIRED via `pvesm remove` after G1/G2/G3 gates all PASSED
> (agent-owned UUID .mount units; both `enrolled` in drive-intents.json; zero /etc/pve refs, empty
> dump/, no customer app on either drive). Post-removal: mounts+binds intact (marker round-trip
> through the guest), `GET /disks` shows both registry-sourced (role+durable-id intact, class
> absent), `pvesm status` clean. **The demo node now matches the fresh-install storage shape** —
> drives are registry+units-sourced only, no legacy PVE dir: storages. v0.128.1 LIVE on demo 9201
> (drill guest skipped — optional, no behavioral dependency; it runs 0.128.0).
> **2026-07-13 night — v0.128.0: CHUNKED BROWSER .FAB UPLOAD on /import (ruling F3: chunked).**
> Step-0 probe on the real tunnel PROVED the Cloudflare edge cap (120 MiB POST → edge 413 with
+79 -76
View File
@@ -1,92 +1,95 @@
# REPORT — v0.128.0: chunked browser .fab upload on /import (tunnel-proof)
# REPORT — demo storage hygiene: USB class-badge suppression (v0.128.1) + legacy PVE dir-storage retirement
**Date:** 2026-07-13 · **Version:** controller v0.128.0 (baseline v0.127.3 `59e4ca7` reconfirmed
against live Gitea; local == origin/main) · Commit `db16371` on `main` · MinAgent unchanged (no
agent coupling — the feature is controller-local fs + HTTP). Implements
TASK-fab-browser-upload (ruling F3 2026-07-13: chunked).
**Date:** 2026-07-13 · **Version:** controller v0.128.1 (commit `d752f15` on `main`) · Host op on
demo-felhom PVE (192.168.0.162), config-level only — no data moved or deleted.
**Sequencing note:** the TASK was numbered v0.127.3 and sequenced BEFORE the .fab upload task, but
v0.127.3 (escrow reveal copy) and v0.128.0 (upload) had already shipped — Part 1 ships as
**v0.128.1**; content unaffected. Baseline at start: `61c1391` (v0.128.0 REPORT), live Gitea ==
local.
## Step-0 probe — the Cloudflare edge cap, measured on the REAL tunnel (before any code)
## Part 1 — controller: suppress the class badge for USB drives (ruling F5)
| Probe | Result |
|---|---|
| POST 120 MiB (zeros) → `https://felhom.demo-felhom.eu/step0-cap-probe` | **HTTP 413 "Payload Too Large" from the edge** (`Server: cloudflare`, `CF-RAY a1aa78cd1f04324d-VIE`, cloudflare-branded HTML body). Rejected on the declared Content-Length: only ~1.1 MB left the client, 0.41 s. |
| POST 80 MiB, same target | **HTTP 302 → `/login?next=…`** — the controller's auth redirect, i.e. the origin was reached (edge passage PROVEN). 0.48 s; the origin answered before consuming the body (unauth redirect), so no full-upload throughput datum — the edge decision is header-based, which is the datum that matters. |
`storage.html` `classTag(d)`: `if(d.type==='usb') return '';` ahead of the class branches — one
guard covers both render sites (card badges L~324; metarow L~343, which additionally guards on
`d.class`). Non-USB storages keep the hint. **Hub-report `ClassHint` UNCHANGED** (UI-only
suppression). No existing test pinned the badge → new
`TestStorageTemplate_USBClassBadgeSuppressed` (renders the production template tree, asserts the
guard exists AND precedes the class branches). **Red-proof:** guard line removed → FAIL
`"classTag USB guard missing from the storage page"`; restored → PASS. Green gate: full module
build/vet/test clean; template gates (id/emoji/native-confirm) green.
Gotcha recorded: local DNS (192.168.0.250) overrides `felhom.demo-felhom.eu` to the LAN guest
(192.168.0.139), and curl's default `Expect: 100-continue` lets the server answer before the body
is sent — the honest probe needed `--resolve` onto Cloudflare's public IP (172.67.149.59) plus
`-H "Expect:"`. **Outcome: cap confirmed at ~100 MB → `uploadChunkBytes` = 64 MiB (the spec's
default), server-side per-request cap 96 MiB.**
**Deployed:** image `0.128.1` → demo guest 9201, `Up (healthy)`. Drill guest skipped (task:
optional, no behavioral dependency).
## What shipped
## Part 2 — host op: retire `felhom-usb` + `felhom-flash` PVE dir storages
- `controller/internal/web/handler_export_upload.go` (new) — `POST
/api/export/upload/{init,chunk,finalize,abort}` inside `ServeExportAPI` (inherits the main.go
`RequireAuth(CsrfProtect(...))` mount — nothing added at the mux). Single-flight slot
(netstorage-job shape); init sanitizes to a `[A-Za-z0-9._ -]` base name + mandatory `.fab`,
free-space gate (declared + 1 GiB margin, Hungarian error with both numbers); chunk requires
`offset == received` (else 409 + `received_bytes` echo) and streams via `io.Copy`; finalize =
exact declared size (else 422 + `.part` deleted) → fsync → atomic rename, collision → lowest-free
`"name (N).fab"` computed from the ORIGINAL name (never `(1)(1)`); abort deletes the `.part`.
Startup GC sweeps `*.part-*` on every registered drive (wired in `main.go` after
`SetAppExporter`); 15-min idle timer aborts server-side. No client-side sha256 — deliberate,
rationale in the file header (WebCrypto can't stream-hash multi-GB; `.fab` self-validates at
import).
- `controller/internal/appexport/estimate.go` — `diskFree` exported as `DiskFree` (REUSE.md row
added); web seam `uploadDiskFree` package var.
- `controller/internal/web/templates/app_import.html` — upload zone above the bundle list
(drag-and-drop + "Fájl kiválasztása", accept=".fab"), progress "Feltöltés: {pct}% ({done} /
{total} GB)" + "Megszakítás", spec'd Hungarian error strings; `File.slice` sequential loop with
one retry per chunk re-synced from the 409 echo; on success the page reloads (the EXISTING scan
renders the row — no second lister). Copy-by-hand hint now names BOTH paths. Gates green:
template_id, emoji, native_confirm, mojibake.
### Gate evidence (all PASS)
## Tests (all green; full module `go build && go vet && go test ./...` = 23 packages ok)
**G1 — mount provenance:** both mounts are agent-owned systemd units, independent of the PVE
entries:
- `/mnt/felhom-usb` = `/dev/sdc1` (ext4) via `mnt-felhom\x2dusb.mount` — "`Managed by
felhom-agent`", `What=/dev/disk/by-uuid/da9e7089-cf8e-4617-adcb-a377743fae00`,
`WantedBy=multi-user.target`.
- `/mnt/felhom-flash` = `/dev/sdd1` (ext4) via `mnt-felhom\x2dflash.mount` — same shape,
`What=/dev/disk/by-uuid/81a26531-62d8-408d-812f-a178b1d35310`.
- `/etc/fstab`: no entries for either. The PVE `storage.cfg` blocks carried `is_mountpoint 1`
(checker, not owner) and no `mkdir`.
| Scenario | Test | Result |
|---|---|---|
| A happy path (3 chunks byte-exact, existing scan lists it, no `.part` residue) | `TestFabUpload_HappyPath` | PASS |
| B offset replay → 409 + echo, disk unchanged, re-sync continues | `TestFabUpload_OffsetMismatchRejected` | PASS |
| C traversal neutered inside exports; ext/charset/hidden refused; oversize (faked statfs) → Hungarian msg with 3.0/2.0 GB; 2nd init → 409 | `TestFabUpload_SecurityGates` | PASS |
| C route wrap (unauth POST → 401 on all four paths via the main.go-shaped mount) | `TestFabUpload_MountInheritsAuth` | PASS |
| D finalize size mismatch → 422, `.part` deleted, slot freed | `TestFabUpload_FinalizeSizeMismatch` | PASS |
| E startup GC removes stray `.part`, keeps real bundles; idle-expiry deletes + frees slot | `TestFabUpload_GCAndIdleTimeout` | PASS |
| F collision → `app (1).fab`, re-run → `app (2).fab`, original bytes untouched | `TestFabUpload_CollisionLowestFreeSuffix` | PASS |
| abort deletes `.part` + frees slot | `TestFabUpload_Abort` | PASS |
**G2 — registry membership:** `/var/lib/felhom-agent/drive-intents.json` (read-only) lists both
UUIDs as `"enrolled"` (plus teszt_enroll's). Guest binds live under the drives tree: mp8
`/mnt/felhom-drives` (no `backup=1`); `/mnt/felhom-drives/felhom-{usb,flash}` are `[/felhom-data]`
subdir mounts of the same devices (Model A). Controller settings register both stable paths
(`Tárhely (felhom-usb)` / `Tárhely (felhom-flash)`).
**Red-proofs (§10, all run + reverted, `git diff` clean):**
- **C** — sanitize bypassed (raw client filename used): `TestFabUpload_SecurityGates` FAILED with
"sanitized upload must land inside the exports dir" (the file materialized at the traversal
target `{drive}/evil.fab`).
- **B** — offset check removed (append regardless): `TestFabUpload_OffsetMismatchRejected` FAILED
with "replay: got 200, want 409".
- **F** — naive overwrite (rename straight to the requested name):
`TestFabUpload_CollisionLowestFreeSuffix` FAILED with `first collision landed as "app.fab",
want "app (1).fab"`.
**G3 — zero references:** `grep -r 'felhom-usb\|felhom-flash' /etc/pve/` → only the two
`storage.cfg` blocks themselves + `/etc/pve/.rrd` metric rings (auto-generated stats, not config).
No jobs.cfg/vzdump/replication; both storages' `dump/` dirs EMPTY. No customer app on either
drive: the only stack referencing them is the protected infra FileBrowser, via the agent-owned
`/mnt/felhom-drives/...` bind chain — unaffected by the PVE entry. (Live re-verified, not from
the screenshot.)
## Deploy + live verification
### storage.cfg blocks removed (before-state, verbatim)
- Image `gitea.dooplex.hu/admin/felhom-controller:0.128.0`
(`sha256:4c943ea1…`) built on 180.
- **Demo guest 9201**: `Up … (healthy)` on 0.128.0; selfupdate log confirms "Current version
0.128.0 is up to date"; no ERROR/FATAL. **Drill guest** (nested PVE 192.168.0.152, guest 9201):
`Up … (healthy)` on 0.128.0.
- Route live-checked on the deployed box: unauthenticated `POST /api/export/upload/init` → 401
JSON both via LAN and **through the real Cloudflare edge** (`--resolve` onto 172.67.149.59) —
the new path is deployed and auth+CSRF-wrapped on the customer path.
```
dir: felhom-usb
path /mnt/felhom-usb
content backup
is_mountpoint 1
## NOT live-validated (Viktor's ~5-minute leg)
dir: felhom-flash
path /mnt/felhom-flash
content backup
is_mountpoint 1
```
The end-to-end multi-GB browser upload through the real tunnel needs a dashboard login (CC has no
customer session). Suggested test: export a real app to `.fab`, download it via the browser,
re-upload it on `/import`, import it — the full circle. Everything beneath the login was
exercised: edge cap probed, endpoints tested AF, deployed route answers through the tunnel.
### Removal + verification
Pre-removal marker files created under each drive's felhom-data namespace and confirmed readable
from the guest. Then `pvesm remove felhom-usb && pvesm remove felhom-flash` → OK. Verified in
order:
1. **Host mounts intact**, same sources: `/mnt/felhom-usb` = `/dev/sdc1`, `/mnt/felhom-flash` =
`/dev/sdd1`; drives-tree subdir mounts intact (`/dev/sdc1[/felhom-data]`,
`/dev/sdd1[/felhom-data]`).
2. **Guest binds intact**: both markers still readable from guest 9201 post-removal
(`retire-check-2026-07-13`); markers removed afterwards.
3. **Meghajtók data**: agent `GET /disks` (pinned local-API token, value never echoed) returns
both drives registry-sourced — `type: usb`, `role: user-data`, durable-ids intact,
`class: ""` (consistent with `teszt_enroll` and Part 1), mount+guest paths intact.
4. **storage.cfg clean**: zero `felhom-usb|felhom-flash` matches; `pvesm status` lists only
`local`, `local-lvm`, `felhom-pbs`, `felhom-offsite` — the untouchables all present and active.
Post-op: controller `Up (healthy)` on 0.128.1; agent journal (last 10 min) free of
error/warn/detach — no false detach from the registry-sourced watchdog.
## Observations
- The per-drive target picker was deliberately left out (v1 = default drive only, per §12); if a
customer keeps exports on a non-default drive, the upload still lands on the default one and the
scan shows both — acceptable, but a picker is the natural follow-up.
- `controller/cmd/controller` is matched by a gitignore rule — `main.go` changes need
`git add -f` (it IS tracked; add refuses the path without `-f`).
- Startup GC logs only when it removes something; the clean deploy logged nothing (expected).
- The guest-bind mounts are `[/felhom-data]` subdir mounts (Model A) — a marker written at
`<drive>/felhom-data/` on the host appears at the guest drive ROOT (`/mnt/felhom-drives/<n>/`).
First marker check looked one level too deep; corrected before the removal.
- `felhom-flash` still carries empty PVE-era skeleton dirs on-disk (`images/ private/ snippets/
template/ dump/`) from a past content=all era; harmless, deliberately NOT cleaned (task §12: no
touching drive contents).
- The `/etc/pve/.rrd` stats ring keeps historical series for the removed storages; it's
append-only telemetry and stops updating on its own — not a reference.
- The demo node now matches the fresh-install storage shape: no legacy dir: storages; drives are
registry+units-sourced only.