docs: REPORT + CONTEXT cover v0.128.1 + demo legacy dir-storage retirement (gates G1-G3, 4/4 verifications)
This commit is contained in:
+13
-1
@@ -7,7 +7,19 @@
|
||||
>
|
||||
> Ask Claude Code: "Please update CONTEXT.md with what we did today"
|
||||
|
||||
Last updated: 2026-07-13 night (v0.128.0 — browser .fab upload on /import)
|
||||
Last updated: 2026-07-13 night (v0.128.1 — USB class-badge suppression + demo PVE dir-storage retirement)
|
||||
|
||||
> **2026-07-13 night — v0.128.1 + demo storage hygiene (ruling F5).** `classTag` suppresses the
|
||||
> rotational class hint for `type==='usb'` (card already carries the USB tag; hub `ClassHint`
|
||||
> UNCHANGED; pinned by `TestStorageTemplate_USBClassBadgeSuppressed` + red-proof). Host op on
|
||||
> demo-felhom: the two pre-intermediary legacy `dir:` storages (`felhom-usb`, `felhom-flash`,
|
||||
> content=Backup, is_mountpoint) RETIRED via `pvesm remove` after G1/G2/G3 gates all PASSED
|
||||
> (agent-owned UUID .mount units; both `enrolled` in drive-intents.json; zero /etc/pve refs, empty
|
||||
> dump/, no customer app on either drive). Post-removal: mounts+binds intact (marker round-trip
|
||||
> through the guest), `GET /disks` shows both registry-sourced (role+durable-id intact, class
|
||||
> absent), `pvesm status` clean. **The demo node now matches the fresh-install storage shape** —
|
||||
> drives are registry+units-sourced only, no legacy PVE dir: storages. v0.128.1 LIVE on demo 9201
|
||||
> (drill guest skipped — optional, no behavioral dependency; it runs 0.128.0).
|
||||
|
||||
> **2026-07-13 night — v0.128.0: CHUNKED BROWSER .FAB UPLOAD on /import (ruling F3: chunked).**
|
||||
> Step-0 probe on the real tunnel PROVED the Cloudflare edge cap (120 MiB POST → edge 413 with
|
||||
|
||||
@@ -1,92 +1,95 @@
|
||||
# REPORT — v0.128.0: chunked browser .fab upload on /import (tunnel-proof)
|
||||
# REPORT — demo storage hygiene: USB class-badge suppression (v0.128.1) + legacy PVE dir-storage retirement
|
||||
|
||||
**Date:** 2026-07-13 · **Version:** controller v0.128.0 (baseline v0.127.3 `59e4ca7` reconfirmed
|
||||
against live Gitea; local == origin/main) · Commit `db16371` on `main` · MinAgent unchanged (no
|
||||
agent coupling — the feature is controller-local fs + HTTP). Implements
|
||||
TASK-fab-browser-upload (ruling F3 2026-07-13: chunked).
|
||||
**Date:** 2026-07-13 · **Version:** controller v0.128.1 (commit `d752f15` on `main`) · Host op on
|
||||
demo-felhom PVE (192.168.0.162), config-level only — no data moved or deleted.
|
||||
**Sequencing note:** the TASK was numbered v0.127.3 and sequenced BEFORE the .fab upload task, but
|
||||
v0.127.3 (escrow reveal copy) and v0.128.0 (upload) had already shipped — Part 1 ships as
|
||||
**v0.128.1**; content unaffected. Baseline at start: `61c1391` (v0.128.0 REPORT), live Gitea ==
|
||||
local.
|
||||
|
||||
## Step-0 probe — the Cloudflare edge cap, measured on the REAL tunnel (before any code)
|
||||
## Part 1 — controller: suppress the class badge for USB drives (ruling F5)
|
||||
|
||||
| Probe | Result |
|
||||
|---|---|
|
||||
| POST 120 MiB (zeros) → `https://felhom.demo-felhom.eu/step0-cap-probe` | **HTTP 413 "Payload Too Large" from the edge** (`Server: cloudflare`, `CF-RAY a1aa78cd1f04324d-VIE`, cloudflare-branded HTML body). Rejected on the declared Content-Length: only ~1.1 MB left the client, 0.41 s. |
|
||||
| POST 80 MiB, same target | **HTTP 302 → `/login?next=…`** — the controller's auth redirect, i.e. the origin was reached (edge passage PROVEN). 0.48 s; the origin answered before consuming the body (unauth redirect), so no full-upload throughput datum — the edge decision is header-based, which is the datum that matters. |
|
||||
`storage.html` `classTag(d)`: `if(d.type==='usb') return '';` ahead of the class branches — one
|
||||
guard covers both render sites (card badges L~324; metarow L~343, which additionally guards on
|
||||
`d.class`). Non-USB storages keep the hint. **Hub-report `ClassHint` UNCHANGED** (UI-only
|
||||
suppression). No existing test pinned the badge → new
|
||||
`TestStorageTemplate_USBClassBadgeSuppressed` (renders the production template tree, asserts the
|
||||
guard exists AND precedes the class branches). **Red-proof:** guard line removed → FAIL
|
||||
`"classTag USB guard missing from the storage page"`; restored → PASS. Green gate: full module
|
||||
build/vet/test clean; template gates (id/emoji/native-confirm) green.
|
||||
|
||||
Gotcha recorded: local DNS (192.168.0.250) overrides `felhom.demo-felhom.eu` to the LAN guest
|
||||
(192.168.0.139), and curl's default `Expect: 100-continue` lets the server answer before the body
|
||||
is sent — the honest probe needed `--resolve` onto Cloudflare's public IP (172.67.149.59) plus
|
||||
`-H "Expect:"`. **Outcome: cap confirmed at ~100 MB → `uploadChunkBytes` = 64 MiB (the spec's
|
||||
default), server-side per-request cap 96 MiB.**
|
||||
**Deployed:** image `0.128.1` → demo guest 9201, `Up (healthy)`. Drill guest skipped (task:
|
||||
optional, no behavioral dependency).
|
||||
|
||||
## What shipped
|
||||
## Part 2 — host op: retire `felhom-usb` + `felhom-flash` PVE dir storages
|
||||
|
||||
- `controller/internal/web/handler_export_upload.go` (new) — `POST
|
||||
/api/export/upload/{init,chunk,finalize,abort}` inside `ServeExportAPI` (inherits the main.go
|
||||
`RequireAuth(CsrfProtect(...))` mount — nothing added at the mux). Single-flight slot
|
||||
(netstorage-job shape); init sanitizes to a `[A-Za-z0-9._ -]` base name + mandatory `.fab`,
|
||||
free-space gate (declared + 1 GiB margin, Hungarian error with both numbers); chunk requires
|
||||
`offset == received` (else 409 + `received_bytes` echo) and streams via `io.Copy`; finalize =
|
||||
exact declared size (else 422 + `.part` deleted) → fsync → atomic rename, collision → lowest-free
|
||||
`"name (N).fab"` computed from the ORIGINAL name (never `(1)(1)`); abort deletes the `.part`.
|
||||
Startup GC sweeps `*.part-*` on every registered drive (wired in `main.go` after
|
||||
`SetAppExporter`); 15-min idle timer aborts server-side. No client-side sha256 — deliberate,
|
||||
rationale in the file header (WebCrypto can't stream-hash multi-GB; `.fab` self-validates at
|
||||
import).
|
||||
- `controller/internal/appexport/estimate.go` — `diskFree` exported as `DiskFree` (REUSE.md row
|
||||
added); web seam `uploadDiskFree` package var.
|
||||
- `controller/internal/web/templates/app_import.html` — upload zone above the bundle list
|
||||
(drag-and-drop + "Fájl kiválasztása", accept=".fab"), progress "Feltöltés: {pct}% ({done} /
|
||||
{total} GB)" + "Megszakítás", spec'd Hungarian error strings; `File.slice` sequential loop with
|
||||
one retry per chunk re-synced from the 409 echo; on success the page reloads (the EXISTING scan
|
||||
renders the row — no second lister). Copy-by-hand hint now names BOTH paths. Gates green:
|
||||
template_id, emoji, native_confirm, mojibake.
|
||||
### Gate evidence (all PASS)
|
||||
|
||||
## Tests (all green; full module `go build && go vet && go test ./...` = 23 packages ok)
|
||||
**G1 — mount provenance:** both mounts are agent-owned systemd units, independent of the PVE
|
||||
entries:
|
||||
- `/mnt/felhom-usb` = `/dev/sdc1` (ext4) via `mnt-felhom\x2dusb.mount` — "`Managed by
|
||||
felhom-agent`", `What=/dev/disk/by-uuid/da9e7089-cf8e-4617-adcb-a377743fae00`,
|
||||
`WantedBy=multi-user.target`.
|
||||
- `/mnt/felhom-flash` = `/dev/sdd1` (ext4) via `mnt-felhom\x2dflash.mount` — same shape,
|
||||
`What=/dev/disk/by-uuid/81a26531-62d8-408d-812f-a178b1d35310`.
|
||||
- `/etc/fstab`: no entries for either. The PVE `storage.cfg` blocks carried `is_mountpoint 1`
|
||||
(checker, not owner) and no `mkdir`.
|
||||
|
||||
| Scenario | Test | Result |
|
||||
|---|---|---|
|
||||
| A happy path (3 chunks byte-exact, existing scan lists it, no `.part` residue) | `TestFabUpload_HappyPath` | PASS |
|
||||
| B offset replay → 409 + echo, disk unchanged, re-sync continues | `TestFabUpload_OffsetMismatchRejected` | PASS |
|
||||
| C traversal neutered inside exports; ext/charset/hidden refused; oversize (faked statfs) → Hungarian msg with 3.0/2.0 GB; 2nd init → 409 | `TestFabUpload_SecurityGates` | PASS |
|
||||
| C route wrap (unauth POST → 401 on all four paths via the main.go-shaped mount) | `TestFabUpload_MountInheritsAuth` | PASS |
|
||||
| D finalize size mismatch → 422, `.part` deleted, slot freed | `TestFabUpload_FinalizeSizeMismatch` | PASS |
|
||||
| E startup GC removes stray `.part`, keeps real bundles; idle-expiry deletes + frees slot | `TestFabUpload_GCAndIdleTimeout` | PASS |
|
||||
| F collision → `app (1).fab`, re-run → `app (2).fab`, original bytes untouched | `TestFabUpload_CollisionLowestFreeSuffix` | PASS |
|
||||
| abort deletes `.part` + frees slot | `TestFabUpload_Abort` | PASS |
|
||||
**G2 — registry membership:** `/var/lib/felhom-agent/drive-intents.json` (read-only) lists both
|
||||
UUIDs as `"enrolled"` (plus teszt_enroll's). Guest binds live under the drives tree: mp8
|
||||
`/mnt/felhom-drives` (no `backup=1`); `/mnt/felhom-drives/felhom-{usb,flash}` are `[/felhom-data]`
|
||||
subdir mounts of the same devices (Model A). Controller settings register both stable paths
|
||||
(`Tárhely (felhom-usb)` / `Tárhely (felhom-flash)`).
|
||||
|
||||
**Red-proofs (§10, all run + reverted, `git diff` clean):**
|
||||
- **C** — sanitize bypassed (raw client filename used): `TestFabUpload_SecurityGates` FAILED with
|
||||
"sanitized upload must land inside the exports dir" (the file materialized at the traversal
|
||||
target `{drive}/evil.fab`).
|
||||
- **B** — offset check removed (append regardless): `TestFabUpload_OffsetMismatchRejected` FAILED
|
||||
with "replay: got 200, want 409".
|
||||
- **F** — naive overwrite (rename straight to the requested name):
|
||||
`TestFabUpload_CollisionLowestFreeSuffix` FAILED with `first collision landed as "app.fab",
|
||||
want "app (1).fab"`.
|
||||
**G3 — zero references:** `grep -r 'felhom-usb\|felhom-flash' /etc/pve/` → only the two
|
||||
`storage.cfg` blocks themselves + `/etc/pve/.rrd` metric rings (auto-generated stats, not config).
|
||||
No jobs.cfg/vzdump/replication; both storages' `dump/` dirs EMPTY. No customer app on either
|
||||
drive: the only stack referencing them is the protected infra FileBrowser, via the agent-owned
|
||||
`/mnt/felhom-drives/...` bind chain — unaffected by the PVE entry. (Live re-verified, not from
|
||||
the screenshot.)
|
||||
|
||||
## Deploy + live verification
|
||||
### storage.cfg blocks removed (before-state, verbatim)
|
||||
|
||||
- Image `gitea.dooplex.hu/admin/felhom-controller:0.128.0`
|
||||
(`sha256:4c943ea1…`) built on 180.
|
||||
- **Demo guest 9201**: `Up … (healthy)` on 0.128.0; selfupdate log confirms "Current version
|
||||
0.128.0 is up to date"; no ERROR/FATAL. **Drill guest** (nested PVE 192.168.0.152, guest 9201):
|
||||
`Up … (healthy)` on 0.128.0.
|
||||
- Route live-checked on the deployed box: unauthenticated `POST /api/export/upload/init` → 401
|
||||
JSON both via LAN and **through the real Cloudflare edge** (`--resolve` onto 172.67.149.59) —
|
||||
the new path is deployed and auth+CSRF-wrapped on the customer path.
|
||||
```
|
||||
dir: felhom-usb
|
||||
path /mnt/felhom-usb
|
||||
content backup
|
||||
is_mountpoint 1
|
||||
|
||||
## NOT live-validated (Viktor's ~5-minute leg)
|
||||
dir: felhom-flash
|
||||
path /mnt/felhom-flash
|
||||
content backup
|
||||
is_mountpoint 1
|
||||
```
|
||||
|
||||
The end-to-end multi-GB browser upload through the real tunnel needs a dashboard login (CC has no
|
||||
customer session). Suggested test: export a real app to `.fab`, download it via the browser,
|
||||
re-upload it on `/import`, import it — the full circle. Everything beneath the login was
|
||||
exercised: edge cap probed, endpoints tested A–F, deployed route answers through the tunnel.
|
||||
### Removal + verification
|
||||
|
||||
Pre-removal marker files created under each drive's felhom-data namespace and confirmed readable
|
||||
from the guest. Then `pvesm remove felhom-usb && pvesm remove felhom-flash` → OK. Verified in
|
||||
order:
|
||||
1. **Host mounts intact**, same sources: `/mnt/felhom-usb` = `/dev/sdc1`, `/mnt/felhom-flash` =
|
||||
`/dev/sdd1`; drives-tree subdir mounts intact (`/dev/sdc1[/felhom-data]`,
|
||||
`/dev/sdd1[/felhom-data]`).
|
||||
2. **Guest binds intact**: both markers still readable from guest 9201 post-removal
|
||||
(`retire-check-2026-07-13`); markers removed afterwards.
|
||||
3. **Meghajtók data**: agent `GET /disks` (pinned local-API token, value never echoed) returns
|
||||
both drives registry-sourced — `type: usb`, `role: user-data`, durable-ids intact,
|
||||
`class: ""` (consistent with `teszt_enroll` and Part 1), mount+guest paths intact.
|
||||
4. **storage.cfg clean**: zero `felhom-usb|felhom-flash` matches; `pvesm status` lists only
|
||||
`local`, `local-lvm`, `felhom-pbs`, `felhom-offsite` — the untouchables all present and active.
|
||||
|
||||
Post-op: controller `Up (healthy)` on 0.128.1; agent journal (last 10 min) free of
|
||||
error/warn/detach — no false detach from the registry-sourced watchdog.
|
||||
|
||||
## Observations
|
||||
|
||||
- The per-drive target picker was deliberately left out (v1 = default drive only, per §12); if a
|
||||
customer keeps exports on a non-default drive, the upload still lands on the default one and the
|
||||
scan shows both — acceptable, but a picker is the natural follow-up.
|
||||
- `controller/cmd/controller` is matched by a gitignore rule — `main.go` changes need
|
||||
`git add -f` (it IS tracked; add refuses the path without `-f`).
|
||||
- Startup GC logs only when it removes something; the clean deploy logged nothing (expected).
|
||||
- The guest-bind mounts are `[/felhom-data]` subdir mounts (Model A) — a marker written at
|
||||
`<drive>/felhom-data/` on the host appears at the guest drive ROOT (`/mnt/felhom-drives/<n>/`).
|
||||
First marker check looked one level too deep; corrected before the removal.
|
||||
- `felhom-flash` still carries empty PVE-era skeleton dirs on-disk (`images/ private/ snippets/
|
||||
template/ dump/`) from a past content=all era; harmless, deliberately NOT cleaned (task §12: no
|
||||
touching drive contents).
|
||||
- The `/etc/pve/.rrd` stats ring keeps historical series for the removed storages; it's
|
||||
append-only telemetry and stops updating on its own — not a reference.
|
||||
- The demo node now matches the fresh-install storage shape: no legacy dir: storages; drives are
|
||||
registry+units-sourced only.
|
||||
|
||||
Reference in New Issue
Block a user