docs: golden 0.98.3 live — D.1b retired, drill B1/B5 fixed, backlog note resolved + evidence doc
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
@@ -3,6 +3,15 @@
|
|||||||
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
|
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
|
||||||
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
|
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
|
||||||
|
|
||||||
|
- **2026-07-03 — GOLDEN 0.98.3 LIVE + VOUCHED; guide D.1b RETIRED (drill B1+B5 FIXED)** —
|
||||||
|
`build-golden.sh` v2.0.0 (mandatory controller tag + baked bootstrap `.path` unit); golden baked
|
||||||
|
with controller 0.98.3, clean-room validated (drill VM) through all four scenarios incl. the
|
||||||
|
published-artifact `--force-gitea-golden` path, published + operator-vouched (Day-0 manifest now:
|
||||||
|
agent 0.63.0 + golden 0.98.3). Fresh installs land CURRENT and self-manage — D.1b is a one-line
|
||||||
|
check; the prior "vouch agent 0.63.0" operator follow-up is DONE. Evidence:
|
||||||
|
`documentation/audits/DRILL-golden-098-2026-07-03.md`. NEW operator follow-ups: (1) SECURITY —
|
||||||
|
the customer-config `git.token` is a Gitea admin token with package-WRITE (proven during the
|
||||||
|
bake); scope down + rotate. (2) optionally raise the global controller floor to 0.98.3.
|
||||||
- **2026-07-03 — drill findings B2+B3 FIXED (agent v0.63.0, live on felhom-pve + published)** —
|
- **2026-07-03 — drill findings B2+B3 FIXED (agent v0.63.0, live on felhom-pve + published)** —
|
||||||
token reload-on-miss (no more fresh-install 401/manual restart) + guesthook snippets-dir mkdir.
|
token reload-on-miss (no more fresh-install 401/manual restart) + guesthook snippets-dir mkdir.
|
||||||
Guide D.1b restart step narrowed to "agent < v0.63.0"; drill ledger updated. **OPERATOR: vouch
|
Guide D.1b restart step narrowed to "agent < v0.63.0"; drill ledger updated. **OPERATOR: vouch
|
||||||
|
|||||||
@@ -4,75 +4,75 @@
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# Go-live package — Day-0 install guide, clean-room drill, uninstall proof (2026-07-03)
|
# REPORT — Golden rebuild 0.98.3 (drill B5 + B1) — docs half (2026-07-03)
|
||||||
|
|
||||||
|
Implementation half in `felhom-agent/REPORT.md` (build-golden.sh v2.0.0 @ `ceca355`). Full drill
|
||||||
|
evidence: **`documentation/audits/DRILL-golden-098-2026-07-03.md`** (A–D transcripts, unit states,
|
||||||
|
resolution-order + fetch/sha proofs, cleanup, observations). This repo's changes are docs-only.
|
||||||
|
|
||||||
## Baselines
|
## Baselines
|
||||||
|
|
||||||
| Repo | `main` before | After |
|
| Repo | Base → head |
|
||||||
|---|---|---|
|
|---|---|
|
||||||
| felhom.eu | `996526273a` (script v1.9.0) | this commit — script **v1.9.1** + `documentation/runbooks/day0-install.md` (NEW) + `documentation/audits/DRILL-day0-cleanroom-2026-07-03.md` (NEW) |
|
| felhom.eu | `2e33a8b` → this push (docs) |
|
||||||
| felhom-agent | `84f3f7ddb1` v0.62.0 | untouched (read-only; 3 findings recorded, not patched) |
|
| felhom-agent | `c9f963d` → `ceca355` (script + CHANGELOG) |
|
||||||
|
| felhom-controller | untouched; **0.98.3** reconfirmed current + pullable, and is the baked tag |
|
||||||
|
|
||||||
## What shipped
|
## What shipped (system-level)
|
||||||
|
|
||||||
1. **`documentation/runbooks/day0-install.md`** — the complete first-time deployment guide
|
- **Golden 0.98.3** — bakes controller 0.98.3 + the `felhom-controller-bootstrap.path` unit;
|
||||||
(Parts A–F: hub onboarding / box-prereq checklist (OQ-3) / the canonical shared-box install
|
published to Gitea (`felhom-golden/0.98.3/golden.tar.zst`, HTTP 201, round-trip sha
|
||||||
command / post-install verification incl. the mandatory one-time controller update D.1b /
|
`b9a02ef1b6f02b9b58babc4c6aad9cf6c053ebdfba116c78c8e7830de757fd01`) and **operator-vouched** in
|
||||||
uninstall + expected-remnants / troubleshooting from real drill hiccups). Header states the
|
the Day-0 manifest (now: agent 0.63.0 + golden 0.98.3 — verified via `/api/v1/artifacts`).
|
||||||
drill date/environment/versions. Every command was executed verbatim during the drill.
|
- **B5 dead:** clean-room Day-0 install (Scenario C, local golden; Scenario D, vouched Gitea
|
||||||
2. **Clean-room drill (Phase 0 + D0–D6)** — throwaway nested PVE 9.2.2 under QEMU/KVM on the build
|
fetch + sha verify) lands controller **0.98.3 on first boot**; `selfupdate/check` reports
|
||||||
server (unattended install via `proxmox-auto-install-assistant`, virgin qcow2 snapshot, slirp
|
up-to-date → the box self-manages; agent selftest clean; bentopdf deploys + answers 200.
|
||||||
NAT; the primary Phase-0 mechanism worked — no fallback needed). Full cycle proven: baseline
|
- **B1 dead:** isolated proof — service condition-failed + path `active (waiting)` on a mount-less
|
||||||
manifest → hub onboarding of throwaway customer `drill-1` → install (real
|
boot; `pct set -mp9 …` against the RUNNING guest started the controller in ~1 s, no reboot
|
||||||
`--force-gitea-golden` + `--cores 2 --memory 4096` + `--acl-storages "local local-lvm"`,
|
(`uptime -s` unchanged). Installer v1.9.1 reboot retained as belt (removal = recorded cleanup).
|
||||||
sha256-verified agent v0.62.0 + golden v0.85.1) → verify (selftest incl. `pve:pool-read`,
|
|
||||||
hub reporting, dashboard 200 behind traefik) → app deploy via the exact UI endpoint
|
|
||||||
(`bentopdf`, 200 at its own hostname) → uninstall → **residue diff** → reinstall
|
|
||||||
(host-enroll REUSED; fresh leaf on an empty box). Evidence doc:
|
|
||||||
`documentation/audits/DRILL-day0-cleanroom-2026-07-03.md`.
|
|
||||||
3. **`scripts/felhom-host-install.sh` v1.9.1** — drill-justified fixes, each re-verified live:
|
|
||||||
- header/version sync (v1.8.0 header vs 1.9.0 var) + keep-in-sync note;
|
|
||||||
- uninstall removes the five drill-found residue items (agent **config with the live hub
|
|
||||||
api_key**, shared-parent unit+script+`/mnt/felhom-drives`, guarded-mkfs wrapper, guest-hook
|
|
||||||
snippet, dnsmasq snippets) — v1.9.1 re-drill diff vs the virgin baseline shows **zero
|
|
||||||
Felhom-named leftovers**;
|
|
||||||
- **post-provision guest reboot + bounded verify wait (R6)** — without it the golden's
|
|
||||||
controller-bootstrap unit (boot-time `ConditionPathExists`) loses the race with the agent's
|
|
||||||
hot-plugged bootstrap mount on slower hardware and the controller never deploys; v1.9.1
|
|
||||||
reinstall brought the controller up with no manual intervention.
|
|
||||||
Gates: `bash -n`, `--dry-run` previews, live re-drill of both paths.
|
|
||||||
|
|
||||||
## OQ answers (detail in the drill doc §8)
|
## Docs changed (this repo)
|
||||||
|
|
||||||
- **OQ-1:** go live **local-backups-only** (`--acl-storages "local local-lvm"`) — PBS is LAN-only
|
- `documentation/runbooks/day0-install.md` — **D.1b retired** to a one-line `selfupdate/check`
|
||||||
until Headscale; retrofit later via `--rescope-acl`. Documented as the guide's standard.
|
verification; old procedure → Part F troubleshooting row keyed on "golden older than 0.86.0";
|
||||||
- **OQ-2 (evidence-based):** a fresh install lands controller **0.85.1** (golden-baked, pre-floor)
|
header versions line (script v1.9.1 / agent v0.63.0 / golden v0.98.3); A.3 drilled-known-good
|
||||||
and **never self-updates** — raising the floor does not help a fresh box. The guide's D.1b
|
pair + vouch-≥0.98.3 note; A.4 floor text rewritten + raise-floor recommendation.
|
||||||
(restart `felhom-agent` → trigger the settings-page update → verify) is the mandatory
|
- `documentation/audits/DRILL-day0-cleanroom-2026-07-03.md` — ledger **B1, B5 → FIXED**; R6
|
||||||
install-day step; drill-proven 0.85.1 → 0.98.3. Structural fix = rebuild+re-vouch the golden
|
belt-note.
|
||||||
(operator follow-up).
|
- `documentation/backlog/FOLLOWUP-golden-default-controller-tag.md` + `backlog/README.md` —
|
||||||
- **OQ-3:** Part B of the guide is the customer-confirmed pre-install checklist.
|
**RESOLVED** (M18/M19 convention: file kept + annotated, README entry marked FIXED; the note's
|
||||||
|
`:0.43.0` numbers were history — the live default had already rotted to `:0.85.1`, which is the
|
||||||
|
form of the problem the mandatory arg kills).
|
||||||
|
- NEW `documentation/audits/DRILL-golden-098-2026-07-03.md` — the evidence doc.
|
||||||
|
|
||||||
## Go/no-go for Peti: **GO**
|
## Key proofs (short form; transcripts in the evidence doc)
|
||||||
|
|
||||||
With three install-day items: serve v1.9.1 (this push), execute D.1b, create the real CF tunnel +
|
| Gate | Evidence |
|
||||||
git credentials in Part A. Full list + blockers ledger: drill doc §8–§9.
|
|---|---|
|
||||||
|
| B5 red-proof | no-arg `build-golden.sh` dies with usage, exit 1, before any `pct` op (run on Windows + in the drill VM) |
|
||||||
|
| Scenario A | `[golden] build-golden.sh v2.0.0 — baking controller …0.98.3`; vzdump log: mp0 AND mp1 **included**; guest 9100 destroyed |
|
||||||
|
| Scenario B | before: `ConditionPathExists … not met` + path `active (waiting)`; after mp9 hot-plug: service SUCCESS @ +1 s, container `Up (healthy)` 0.98.3, boot time unchanged |
|
||||||
|
| Scenario C | `[SKIP] using local golden: …18_01_21.tar.zst` (resolution order); first boot 0.98.3; `update_available:false`; hub rows agent 0.63.0 / controller 0.98.3; bentopdf 200 |
|
||||||
|
| Publish | pre-delete 404 → PUT **201** → round-trip GET sha **matches** |
|
||||||
|
| Scenario D | `fetching golden v0.98.3 from Gitea` → `verified sha256 b9a02ef1… matches the hub manifest` → SUCCESS; first boot 0.98.3; up-to-date |
|
||||||
|
| Cleanup | all 8 drill-1 hub tables at count **0**, demo-felhom + peti-felhom intact; drill VM reverted to `virgin` (kept); bake cred file removed |
|
||||||
|
|
||||||
## Recorded findings NOT fixed here (per the gap policy)
|
Secrets: registry read-cred via 0600 env file only; the bake script's in-guest
|
||||||
|
`docker logout + rm /root/.docker/config.json` line is present and ran before archiving; publish
|
||||||
|
used the build server's out-of-band Gitea admin credential; nothing committed.
|
||||||
|
|
||||||
- **B1** (agent/golden, LOW): product-side fix for the bootstrap-race (golden path unit or
|
## Observations / operator follow-ups
|
||||||
agent-side restart) — the installer reboot is a correct external fix.
|
|
||||||
- **B2** (agent, LOW): `guesthook.InstallSnippet` fails on fresh boxes (`/var/lib/vz/snippets`
|
|
||||||
missing; `install` can't create parents) → no pre-start self-heal hook, warn-only.
|
|
||||||
- **B3** (agent, MED, pre-existing): fresh-install local-API 401 until agent restart — new
|
|
||||||
consequence found: blocks the controller self-update swap.
|
|
||||||
- **B4** (hub, LOW): no host-delete path (UI or API); drill rows cleaned via direct SQL.
|
|
||||||
- **B5** (fleet, MED): golden bakes a pre-floor controller → D.1b needed on every fresh install
|
|
||||||
until the golden is rebuilt ≥ 0.86.0 and re-vouched.
|
|
||||||
|
|
||||||
## Cleanup
|
1. **SECURITY:** the customer-config `git.token` (held by every customer box) is a Gitea **admin**
|
||||||
|
token with **package-WRITE** — the bake proved it by successfully publishing with it. The
|
||||||
Hub `drill-1` rows fully deleted (all tables → 0; `demo-felhom` intact). Drill VM
|
manifest-sha chain protects installs from tampered artifacts, but the capability shouldn't exist
|
||||||
**virgin-snapshotted and stopped** (kept at `~/drill/` on 192.168.0.180, ~6.7 GiB, for future
|
customer-side: issue a scoped read-only account/token + rotate.
|
||||||
drills); evidence logs archived there. Nothing touched felhom-pve, guest 9201, or any real
|
2. `build-golden.sh`'s publish block auto-fires whenever `REGISTRY_*` is set (needed for the pull
|
||||||
customer/host record. No secrets in any committed file.
|
too) → it published BEFORE Scenario C; deleted (204) and re-published after the gate. Candidate
|
||||||
|
cleanup: a `GOLDEN_PUBLISH=1` opt-in flag.
|
||||||
|
3. The installer's post-provision reboot is now redundant (path unit wins first) — candidate
|
||||||
|
removal in a future installer version; kept per the task rules.
|
||||||
|
4. Recommended: raise the global controller floor to 0.98.3 (UI, 1 min) for drift protection.
|
||||||
|
5. Drill-environment note: launching the drill VM with `dhcpstart=10.0.2.30` (+ explicit
|
||||||
|
`hostfwd…-10.0.2.15:22`) eliminates the prior drill's slirp DHCP/IP-collision quirk — worth
|
||||||
|
using in every future drill.
|
||||||
|
|||||||
@@ -174,12 +174,12 @@ cores/RAM for the cap, root SSH, outbound reach incl. the three exact URLs+expec
|
|||||||
| ID | Where | Severity | What | Status |
|
| ID | Where | Severity | What | Status |
|
||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| R1–R5 | installer | — | uninstall residue (config w/ secrets, shared-parent unit+script+mount, mkfs wrapper, hook snippet, dnsmasq snippets) | **FIXED v1.9.1**, residue-diff-proven |
|
| R1–R5 | installer | — | uninstall residue (config w/ secrets, shared-parent unit+script+mount, mkfs wrapper, hook snippet, dnsmasq snippets) | **FIXED v1.9.1**, residue-diff-proven |
|
||||||
| R6 | installer | — | controller-bootstrap unit skipped: no post-provision reboot (agent hot-plugs the mount; unit condition is boot-time) | **FIXED v1.9.1** (reboot + bounded verify wait), D6b-proven |
|
| R6 | installer | — | controller-bootstrap unit skipped: no post-provision reboot (agent hot-plugs the mount; unit condition is boot-time) | **FIXED v1.9.1** (reboot + bounded verify wait), D6b-proven. NOTE (2026-07-03, golden-098 task): goldens ≥ 0.98.3 bake a `.path` unit that makes the reboot redundant — the reboot is RETAINED as a belt; its removal is a recorded candidate cleanup, not done |
|
||||||
| B1 | agent/golden | LOW | structural fix for R6 belongs in the product too (path unit in the golden, or agent-side restart after back-half) — installer reboot is a correct but external crutch | RECORDED (agent/golden follow-up) |
|
| B1 | agent/golden | LOW | structural fix for R6 belongs in the product too (path unit in the golden, or agent-side restart after back-half) — installer reboot is a correct but external crutch | **FIXED** (build-golden.sh v2.0.0, golden 0.98.3): baked `felhom-controller-bootstrap.path` starts the service on the bootstrap-mount hot-plug — isolated + full-install proven, `DRILL-golden-098-2026-07-03.md` §4/§5 |
|
||||||
| B2 | agent | LOW | `guesthook.InstallSnippet` fails on a fresh box: `/var/lib/vz/snippets` doesn't exist and `install` won't create it → no pre-start self-heal hook, warn-only (`install: cannot create regular file … No such file or directory`, agent journal 14:58:40) | **FIXED agent v0.63.0** (fenced `mkdir -p` precedes the install + the one sudoers grant; red-proofed — felhom-agent REPORT 2026-07-03) |
|
| B2 | agent | LOW | `guesthook.InstallSnippet` fails on a fresh box: `/var/lib/vz/snippets` doesn't exist and `install` won't create it → no pre-start self-heal hook, warn-only (`install: cannot create regular file … No such file or directory`, agent journal 14:58:40) | **FIXED agent v0.63.0** (fenced `mkdir -p` precedes the install + the one sudoers grant; red-proofed — felhom-agent REPORT 2026-07-03) |
|
||||||
| B3 | agent | MED | fresh-install local-API **401 until `systemctl restart felhom-agent`** — root cause: the daemon's TokenStore index is built once at open, while provisioning is a SEPARATE one-shot process minting into the shared file; NEW consequence found: it blocks the controller self-update agent-swap | **FIXED agent v0.63.0** (`Lookup` reload-on-miss, red-proofed + run on felhom-pve — felhom-agent REPORT 2026-07-03). Guide D.1b restart-step narrowed to "agent < v0.63.0" — applies until the Day-0 manifest vouches ≥ 0.63.0 |
|
| B3 | agent | MED | fresh-install local-API **401 until `systemctl restart felhom-agent`** — root cause: the daemon's TokenStore index is built once at open, while provisioning is a SEPARATE one-shot process minting into the shared file; NEW consequence found: it blocks the controller self-update agent-swap | **FIXED agent v0.63.0** (`Lookup` reload-on-miss, red-proofed + run on felhom-pve — felhom-agent REPORT 2026-07-03). Guide D.1b restart-step narrowed to "agent < v0.63.0" — applies until the Day-0 manifest vouches ≥ 0.63.0 |
|
||||||
| B4 | hub | LOW | **no host-delete path** (UI or API) — drill host rows removed via direct SQL (spike precedent); customer-delete exists but leaves hosts/reports/guests/events orphaned | RECORDED (candidate small hub follow-up) |
|
| B4 | hub | LOW | **no host-delete path** (UI or API) — drill host rows removed via direct SQL (spike precedent); customer-delete exists but leaves hosts/reports/guests/events orphaned | RECORDED (candidate small hub follow-up) |
|
||||||
| B5 | golden/fleet | MED | golden bakes a pre-floor controller (0.85.1) → every fresh install needs D.1b manually | RECORDED (operator follow-up: rebuild golden ≥ 0.86.0 + re-vouch in the Day-0 manifest UI) |
|
| B5 | golden/fleet | MED | golden bakes a pre-floor controller (0.85.1) → every fresh install needs D.1b manually | **FIXED** (build-golden.sh v2.0.0: mandatory controller tag; golden **0.98.3** baked, published, vouched): fresh install lands 0.98.3 on first boot, self-update reports up-to-date, guide D.1b retired — `DRILL-golden-098-2026-07-03.md` §5/§6 |
|
||||||
|
|
||||||
## 10. Cleanup confirmation
|
## 10. Cleanup confirmation
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,201 @@
|
|||||||
|
# DRILL — Golden rebuild 0.98.3: current-controller baseline (B5) + hot-plug path unit (B1) (2026-07-03)
|
||||||
|
|
||||||
|
**Class:** implementation + clean-room drill. Companion deliverables: `felhom-agent/configs/build-golden.sh`
|
||||||
|
**v2.0.0** (commit `ceca355`) and the golden archive **0.98.3** (published + operator-vouched).
|
||||||
|
Source findings: `DRILL-day0-cleanroom-2026-07-03.md` §9 **B5** (golden bakes a pre-floor controller →
|
||||||
|
mandatory manual D.1b on every fresh install) and **B1** (controller-bootstrap only fires at boot; the
|
||||||
|
hot-plugged config mount needed the installer's reboot crutch). Also closes the stale backlog note
|
||||||
|
`documentation/backlog/FOLLOWUP-golden-default-controller-tag.md`.
|
||||||
|
|
||||||
|
**Verdict (short):** both findings are FIXED in the product. A fresh Day-0 install now lands controller
|
||||||
|
**0.98.3** on first boot and self-manages from there (no D.1b), and the baked
|
||||||
|
`felhom-controller-bootstrap.path` unit starts the controller the moment the agent hot-plugs the
|
||||||
|
bootstrap mount — **no reboot needed** (the installer's v1.9.1 reboot is retained as a redundant belt).
|
||||||
|
Everything was proven clean-room on the drill VM BEFORE publish; the vouch is the only production change.
|
||||||
|
|
||||||
|
No secrets appear in this document. The bake's registry credential was passed via env from a 0600 file
|
||||||
|
on the build server; the script logs out + removes `/root/.docker/config.json` in the build guest before
|
||||||
|
archiving (`build-golden.sh` — verified present in the bake transcript path).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Environment & baselines
|
||||||
|
|
||||||
|
| Item | Value |
|
||||||
|
|---|---|
|
||||||
|
| Drill VM | the DRILL-day0-cleanroom nested PVE 9.2.2 on the build server (192.168.0.180), restored from the `virgin` qcow2 snapshot before the bake and again before each install scenario |
|
||||||
|
| VM launch quirk fixed | slirp DHCP pool moved with `dhcpstart=10.0.2.30` (+ explicit `hostfwd=…-10.0.2.15:22`) so nested guests can't grab the PVE host's static 10.0.2.15 — the prior drill's IP-collision quirk is fully avoided, no guest IP pinning needed |
|
||||||
|
| felhom-agent base | `c9f963d` → script commit `ceca355` (build-golden.sh v2.0.0; no Go change, no agent version bump) |
|
||||||
|
| felhom-controller | **v0.98.3** reconfirmed as the current released tag (CHANGELOG top entry; manifest fetch HTTP 200 with the read credential) — the tag baked |
|
||||||
|
| Hub manifest at start | agent **0.63.0** (the v0.63.0 REPORT's operator follow-up was already done) + golden **0.85.1** |
|
||||||
|
| Debian template | `debian-13-standard_13.1-2_amd64.tar.zst` (pveam-downloaded into the virgin VM — not present on a virgin box) |
|
||||||
|
| Throwaway hub customer | `drill-1` re-created store-identically (SQL insert matching `SaveCustomerConfig`: plaintext retrieval passphrase reused from the prior drill's 0600 file, `RandomHex(32)`-format api_key, demo customer's config_json with **fake** CF tunnel + CF API tokens, real Gitea read credential); probes: config 200 with passphrase / 401 wrong; deleted after (§7) |
|
||||||
|
|
||||||
|
## 2. Script changes (Phase 1) + red-proof
|
||||||
|
|
||||||
|
`build-golden.sh` v2.0.0 (see felhom-agent CHANGELOG for the full rationale):
|
||||||
|
|
||||||
|
1. **CONTROLLER_IMAGE (arg 6) mandatory** — no default; die-with-usage names the convention.
|
||||||
|
2. **`felhom-controller-bootstrap.path` baked + enabled** (`PathExists=/etc/felhom-bootstrap/bootstrap.json`,
|
||||||
|
`WantedBy=multi-user.target`); the service (oneshot/RemainAfterExit + ConditionPathExists) unchanged.
|
||||||
|
3. `GOLDEN_SCRIPT_VERSION=2.0.0` + a `[golden]` provenance line (script version + baked tag) in every transcript.
|
||||||
|
|
||||||
|
Gates: `bash -n` clean (Windows + in-VM after CRLF strip). **B5 red-proof:** the no-arg invocation dies
|
||||||
|
with usage, exit 1, before any `pct` op:
|
||||||
|
|
||||||
|
```text
|
||||||
|
[golden] FATAL: CONTROLLER_IMAGE (argument 6) is required — pass the released controller tag explicitly.
|
||||||
|
Usage: build-golden.sh [VMID] [TEMPLATE_VOLID] [ROOTFS_STORAGE] [ARCHIVE_STORAGE] [BRIDGE] CONTROLLER_IMAGE
|
||||||
|
e.g.: build-golden.sh 9100 local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst local-lvm local vmbr0 gitea.dooplex.hu/admin/felhom-controller:0.98.3
|
||||||
|
exit=1
|
||||||
|
```
|
||||||
|
|
||||||
|
(Re-run in the drill VM: rc=1 as well.)
|
||||||
|
|
||||||
|
## 3. Scenario A — bake integrity (PASS)
|
||||||
|
|
||||||
|
Bake on the virgin-restored drill VM as root@pam:
|
||||||
|
`build-golden-new.sh 9100 local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst local-lvm local vmbr0 gitea.dooplex.hu/admin/felhom-controller:0.98.3`
|
||||||
|
with `REGISTRY_USER/REGISTRY_TOKEN` from env (0600 file). Transcript (`bake-A.log`, archived at
|
||||||
|
`~/drill/` on the build server):
|
||||||
|
|
||||||
|
```text
|
||||||
|
[golden] build-golden.sh v2.0.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.98.3
|
||||||
|
…
|
||||||
|
[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) …
|
||||||
|
…
|
||||||
|
INFO: including mount point rootfs ('/') in backup
|
||||||
|
INFO: including mount point mp0 ('/var/lib/docker') in backup
|
||||||
|
INFO: including mount point mp1 ('/mnt/sys_drive') in backup
|
||||||
|
INFO: archive file size: 583MB
|
||||||
|
[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_07_03-18_01_21.tar.zst
|
||||||
|
```
|
||||||
|
|
||||||
|
- **mp0 + mp1 both INCLUDED** (the load-bearing vzdump guard confirmed; no exclusion lines).
|
||||||
|
- Build guest 9100 torn down (`pct destroy 9100 --purge`, volumes removed).
|
||||||
|
- Archive sha256 `b9a02ef1b6f02b9b58babc4c6aad9cf6c053ebdfba116c78c8e7830de757fd01` (612 002 238 bytes),
|
||||||
|
identical after copy to the build server.
|
||||||
|
- **Deviation caught + corrected:** the script's opt-in Gitea publish block keys on the same
|
||||||
|
`REGISTRY_*` vars the pull needs, so the bake ALSO published immediately — before Scenario C. The
|
||||||
|
premature package was DELETED (HTTP 204, anon GET 404 verified) and re-published only after
|
||||||
|
Scenario C passed (§6). Recorded as an observation (§8-O2), script not changed (no-"while-here" rule).
|
||||||
|
|
||||||
|
## 4. Scenario B — isolated hot-plug proof (PASS — the B1 STOP gate)
|
||||||
|
|
||||||
|
Scratch guest 9300 `pct restore`d from the NEW archive (the bring-up mechanism, minus the bootstrap
|
||||||
|
mount — the exact provision race window), booted 16:05:58 UTC.
|
||||||
|
|
||||||
|
**Before attach** (the R6 failure state, reproduced):
|
||||||
|
|
||||||
|
```text
|
||||||
|
felhom-controller-bootstrap.service Active: inactive (dead)
|
||||||
|
Condition: start condition unmet … ConditionPathExists=/etc/felhom-bootstrap/bootstrap.json was not met
|
||||||
|
TriggeredBy: ● felhom-controller-bootstrap.path
|
||||||
|
felhom-controller-bootstrap.path Active: active (waiting)
|
||||||
|
docker ps: (no containers) ; /etc/felhom-bootstrap: No such file or directory
|
||||||
|
```
|
||||||
|
|
||||||
|
**Attach to the RUNNING guest** — the back-half's exact op (host dir + 0600 bootstrap.json +
|
||||||
|
`chown -R 100000:100000` + `pct set 9300 -mp9 <dir>,mp=/etc/felhom-bootstrap,ro=1`) at 16:07:04.
|
||||||
|
|
||||||
|
**After (~1 s later, NO reboot — `uptime -s` unchanged at 16:05:58):**
|
||||||
|
|
||||||
|
```text
|
||||||
|
Jul 03 16:07:05 systemd[1]: Starting felhom-controller-bootstrap.service …
|
||||||
|
Jul 03 16:07:05 felhom-controller-bootstrap.sh[521]: [ctrl-bootstrap] deploying gitea.dooplex.hu/admin/felhom-controller:0.98.3 …
|
||||||
|
Jul 03 16:07:05 systemd[1]: Finished felhom-controller-bootstrap.service
|
||||||
|
felhom-controller-bootstrap.service Active: active (exited) … status=0/SUCCESS
|
||||||
|
docker ps: felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.98.3 Up (healthy)
|
||||||
|
```
|
||||||
|
|
||||||
|
The systemd-semantics assumption holds empirically: a `.path` unit DOES see a hot-plugged LXC bind
|
||||||
|
mount and DOES start a Condition-gated oneshot service; `RemainAfterExit=yes` showed no re-trigger.
|
||||||
|
(Note: the hand-crafted minimal bootstrap.json's `"id"` didn't survive shell quoting for the
|
||||||
|
`--hostname` sed parse → `hostname=<unset>` fallback, by design fail-safe; Scenario C exercises the
|
||||||
|
real agent-rendered document.) Guest 9300 + its host dir removed after.
|
||||||
|
|
||||||
|
## 5. Scenario C — full Day-0 proof, LOCAL golden (PASS — the publish gate)
|
||||||
|
|
||||||
|
Virgin-restored VM; the new archive placed as the only local vzdump of GOLDEN_VMID 9100; installer
|
||||||
|
fetched from `https://felhom.eu/scripts/felhom-host-install.sh` (served **v1.9.1**); the guide's
|
||||||
|
canonical command WITHOUT `--force-gitea-golden`:
|
||||||
|
|
||||||
|
```text
|
||||||
|
./felhom-host-install.sh --customer-id drill-1 --vmid 120 --cores 2 --memory 4096 \
|
||||||
|
--acl-storages "local local-lvm" --passphrase-file /root/.drill-pass
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Resolution order proof:** `7/8 golden archive → [SKIP] using local golden: local:backup/vzdump-lxc-9100-2026_07_03-18_01_21.tar.zst`;
|
||||||
|
final verdict `Day-0 provision SUCCESS — vmid=120 host_id=drill-1-20e7a3 … golden=local:backup/…`.
|
||||||
|
- **First boot = 0.98.3, NO D.1b performed:** in-guest
|
||||||
|
`felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.98.3 Up (healthy)`; hub `reports`
|
||||||
|
row shows `drill-1 | 0.98.3`; host row `drill-1-20e7a3 | agent 0.63.0`.
|
||||||
|
- **Self-manage proof:** `POST /api/selfupdate/check` →
|
||||||
|
`{"current_version":"0.98.3","latest_version":"0.98.3","update_available":false}` — the box is
|
||||||
|
current and the floor mechanism owns it from here (0.98.3 ≥ the v0.86.0 floor-honoring code).
|
||||||
|
- Agent `--selftest` all-OK including `pool read pool "felhom", 1 member`.
|
||||||
|
- Base infra: traefik + filebrowser healthy; cloudflared restarting on the FAKE tunnel token
|
||||||
|
(expected drill behavior, public leg demo-proven — same as the prior drill).
|
||||||
|
- **D4 smoke:** `bentopdf` deployed via the dashboard's own endpoint
|
||||||
|
(`POST /api/stacks/bentopdf/deploy` with `{"values":{}}` — endpoint-proxy method, no browser
|
||||||
|
bridge in this session) → `Up (healthy)`, and `pdf.drill.demo-felhom.eu` answers **200** through
|
||||||
|
traefik in-guest.
|
||||||
|
- In-flow B1 note: the installer's v1.9.1 belt-reboot still ran (kept by design), so the container
|
||||||
|
journal of the pre-reboot boot was not retrievable (volatile journal); the no-reboot claim rests
|
||||||
|
on the §4 isolated proof.
|
||||||
|
|
||||||
|
## 6. Publish + vouch + Scenario D — published-artifact proof
|
||||||
|
|
||||||
|
- **Publish (after Scenario C):** `PUT …/api/packages/admin/generic/felhom-golden/0.98.3/golden.tar.zst`
|
||||||
|
(exactly the installer's `step_golden` URL convention) → **HTTP 201**; round-trip GET sha256
|
||||||
|
**matches** `b9a02ef1…fd01`. Publish creds = the build server's Gitea admin credential (out-of-band);
|
||||||
|
version string = the baked controller tag per the 0.85.1 precedent.
|
||||||
|
- **Vouch (operator, password-gated UI):** hub Day-0 manifest → golden **0.98.3** / sha `b9a02ef1…fd01`
|
||||||
|
(agent 0.63.0 was already vouched). Confirmed via `GET /api/v1/artifacts/drill-1`:
|
||||||
|
`{"agent":{"version":"0.63.0",…},"golden":{"version":"0.98.3","sha256":"b9a02ef1…fd01"}}`.
|
||||||
|
- **Scenario D (PASS):** virgin-restored VM, NO local golden (dump dir + `pvesm list` both empty,
|
||||||
|
verified), same canonical command + `--force-gitea-golden`:
|
||||||
|
|
||||||
|
```text
|
||||||
|
[INFO] golden: none local — will fetch + verify from Gitea in step 7/8
|
||||||
|
[INFO] manifest: agent v0.63.0 (sha b4a89c81c33d51ea…), golden v0.98.3
|
||||||
|
[OK] verified sha256 b4a89c81c33d51ea… matches the hub manifest (agent)
|
||||||
|
[INFO] fetching golden v0.98.3 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_07_03-18_31_49.tar.zst
|
||||||
|
[OK] verified sha256 b9a02ef1b6f02b9b… matches the hub manifest (golden)
|
||||||
|
[OK] golden imported + verified: local:backup/vzdump-lxc-9100-2026_07_03-18_31_49.tar.zst
|
||||||
|
[OK] Day-0 provision SUCCESS — vmid=120 host_id=drill-1-20e7a3 customer=drill-1
|
||||||
|
```
|
||||||
|
|
||||||
|
First boot: `felhom-controller:0.98.3 Up (healthy)`; `selfupdate/check` →
|
||||||
|
`{"current_version":"0.98.3","latest_version":"0.98.3","update_available":false}`. Host-enroll
|
||||||
|
REUSED the Scenario C host id (mint-once-reuse held). This is the exact fetch+verify path a real
|
||||||
|
customer box (Peti's) takes.
|
||||||
|
|
||||||
|
## 7. Cleanup confirmation
|
||||||
|
|
||||||
|
- Hub: all `drill-1` rows deleted AFTER Scenario D (customer_configs / hosts / host_reports /
|
||||||
|
reports / guests / events / notification_log / customer_notifications — every count verified
|
||||||
|
**0**); `demo-felhom` + `peti-felhom` intact. The vouch (golden 0.98.3 in the Day-0 manifest) is
|
||||||
|
the one intended production change and stays.
|
||||||
|
- Drill VM: powered off; disk reverted to the `virgin` snapshot and kept (reusable). Local archives
|
||||||
|
and staged files on the VM gone with the revert.
|
||||||
|
- Build server: `~/drill/golden-0.98.3.tar.zst` + `bake-A.log` retained as evidence;
|
||||||
|
`.bake-cred.env` (0600) removed after the run.
|
||||||
|
|
||||||
|
## 8. Observations (recorded, not acted on)
|
||||||
|
|
||||||
|
- **O1 (SECURITY, operator follow-up):** the demo customer's `git.token` — the credential every
|
||||||
|
customer box holds for artifact fetch — is a Gitea **admin** token with **package-WRITE** rights
|
||||||
|
(the Scenario A bake proved it by successfully publishing with it). A customer-side compromise
|
||||||
|
could overwrite published artifacts. The manifest-sha trust chain means installs would reject a
|
||||||
|
tampered artifact, but the write capability should not exist on customer boxes at all → issue a
|
||||||
|
scoped read-only Gitea account/token for customer configs and rotate the current one.
|
||||||
|
- **O2:** `build-golden.sh`'s publish block auto-fires when `REGISTRY_USER/REGISTRY_TOKEN` are set
|
||||||
|
(they are also needed for the pull) — a validate-before-publish flow must either use a read-only
|
||||||
|
cred (blocked on O1) or expect the early publish and delete it. Candidate cleanup: a separate
|
||||||
|
`GOLDEN_PUBLISH=1` opt-in flag.
|
||||||
|
- **O3:** the installer's post-provision reboot (v1.9.1, R6 fix) is now a redundant belt — the path
|
||||||
|
unit deploys the controller before the reboot even happens. Removal is a candidate cleanup for a
|
||||||
|
future installer version; NOT done here (rule: keep the belt).
|
||||||
|
- **O4:** `journalctl -b -1` inside the LXC guest returns nothing (volatile journal) — pre-reboot
|
||||||
|
boot evidence must be captured before a reboot, or the journal made persistent, in future drills.
|
||||||
@@ -1,7 +1,15 @@
|
|||||||
# FOLLOW-UP — bump the golden's default controller tag + validate the full provision path
|
# FOLLOW-UP — bump the golden's default controller tag + validate the full provision path
|
||||||
|
|
||||||
**Status:** OPEN (queued 2026-06-14). Surfaced during the F9/F20 supervised session (agent v0.31.0).
|
**Status:** **RESOLVED 2026-07-03** — `build-golden.sh` v2.0.0 (felhom-agent @ `ceca355`) makes the
|
||||||
**Class:** provisioning correctness / customer-onboarding. **Risk:** SUPERVISED (golden + a real destroy→provision).
|
controller tag a MANDATORY argument (the hand-bumped default had rotted AGAIN, 0.43.0 → 0.85.1 →
|
||||||
|
stale vs 0.98.3 — a required arg cannot rot); golden **0.98.3** baked + validated clean-room on the
|
||||||
|
drill VM (first boot lands the current controller, self-update reports up-to-date, app deploys) +
|
||||||
|
published + operator-vouched. The full first-boot path was validated WITHOUT a supervised
|
||||||
|
destroy→re-provision (drill VM virgin snapshot instead — nothing touched 9201/felhom-pve).
|
||||||
|
Evidence: `../audits/DRILL-golden-098-2026-07-03.md`. Item 3 (provision re-asserting user-data
|
||||||
|
binds) was shipped separately (agent-startup re-assert + F9 part A). Original note kept below for
|
||||||
|
history.
|
||||||
|
**Class:** provisioning correctness / customer-onboarding. **Risk (as queued):** SUPERVISED (golden + a real destroy→provision).
|
||||||
|
|
||||||
## The problem
|
## The problem
|
||||||
`felhom-agent/configs/build-golden.sh:43` bakes the controller image into the golden as:
|
`felhom-agent/configs/build-golden.sh:43` bakes the controller image into the golden as:
|
||||||
|
|||||||
@@ -9,9 +9,12 @@ Verified-LIVE findings with implementable fix plans that are **not yet implement
|
|||||||
- **FIX-M19-NOTES.md** — `deriveStackName` misattribution edge (low-incidence correctness). **FIXED** in
|
- **FIX-M19-NOTES.md** — `deriveStackName` misattribution edge (low-incidence correctness). **FIXED** in
|
||||||
controller v0.62.0 @ `6bab68b` (2026-06-14). (was on the deleted branch `fix/m19-stackname-crossref`.)
|
controller v0.62.0 @ `6bab68b` (2026-06-14). (was on the deleted branch `fix/m19-stackname-crossref`.)
|
||||||
|
|
||||||
- **FOLLOWUP-golden-default-controller-tag.md** — the golden bakes a stale controller `:0.43.0`
|
- **FOLLOWUP-golden-default-controller-tag.md** — the golden bakes a stale controller (`:0.43.0` when
|
||||||
(`build-golden.sh:43`); a fresh provision would stand up an ancient controller. Bump it + validate the
|
queued; had rotted again to `:0.85.1` by resolution). **FIXED** in felhom-agent @ `ceca355`
|
||||||
full destroy→provision→first-boot path (warrants a supervised re-provision). Queued 2026-06-14.
|
(2026-07-03): `build-golden.sh` v2.0.0 makes the controller tag a MANDATORY argument (a required arg
|
||||||
|
cannot rot) and golden **0.98.3** was baked + clean-room-validated (bake → first-boot-current →
|
||||||
|
self-manage → app deploy, on the drill VM — no supervised touch of live guests needed) + published +
|
||||||
|
vouched. Evidence: `../audits/DRILL-golden-098-2026-07-03.md`.
|
||||||
|
|
||||||
Related: the live-drive fixspec (`../audits/live-drive-fixspec-2026-06-14.md`) carries the **deferred
|
Related: the live-drive fixspec (`../audits/live-drive-fixspec-2026-06-14.md`) carries the **deferred
|
||||||
supervised items** F9 (HDD provisioning/guest-attach), F20-BUG2 (durable_id scheme), F20-BUG3 (async
|
supervised items** F9 (HDD provisioning/guest-attach), F20-BUG2 (durable_id scheme), F20-BUG3 (async
|
||||||
|
|||||||
@@ -7,8 +7,10 @@
|
|||||||
>
|
>
|
||||||
> **Drill-validated: 2026-07-03** on a clean-room nested PVE 9.2.2 VM (qemu/KVM on the build
|
> **Drill-validated: 2026-07-03** on a clean-room nested PVE 9.2.2 VM (qemu/KVM on the build
|
||||||
> server) — install → verify → app deploy → uninstall → residue-diff → reinstall, all executed
|
> server) — install → verify → app deploy → uninstall → residue-diff → reinstall, all executed
|
||||||
> verbatim. Versions: script **v1.9.1**, agent **v0.62.0**, golden **v0.85.1** (controller updated
|
> verbatim. Versions: script **v1.9.1**, agent **v0.63.0**, golden **v0.98.3** (controller current
|
||||||
> to 0.98.3 via D.1b). Evidence: `documentation/audits/DRILL-day0-cleanroom-2026-07-03.md`.
|
> on first boot — no manual update step). Evidence:
|
||||||
|
> `documentation/audits/DRILL-day0-cleanroom-2026-07-03.md` (install/uninstall path) +
|
||||||
|
> `documentation/audits/DRILL-golden-098-2026-07-03.md` (current-golden rebuild).
|
||||||
>
|
>
|
||||||
> Scope: a **shared** Proxmox box (the colleague's-box model) — a box that already runs, or will run,
|
> Scope: a **shared** Proxmox box (the colleague's-box model) — a box that already runs, or will run,
|
||||||
> non-Felhom guests. A dedicated box is the same procedure minus the sizing caution.
|
> non-Felhom guests. A dedicated box is the same procedure minus the sizing caution.
|
||||||
@@ -69,16 +71,17 @@ On save the hub generates two credentials:
|
|||||||
Hub UI → **Configuration → Day-0 artifacts**. The manifest must vouch an **agent version** and a
|
Hub UI → **Configuration → Day-0 artifacts**. The manifest must vouch an **agent version** and a
|
||||||
**golden version** (with their sha256s — auto-resolved from Gitea). The install script refuses to
|
**golden version** (with their sha256s — auto-resolved from Gitea). The install script refuses to
|
||||||
install anything the manifest doesn't vouch. If the vouched versions lag the fleet, bump them here
|
install anything the manifest doesn't vouch. If the vouched versions lag the fleet, bump them here
|
||||||
first (dropdowns; the sha is fetched authoritatively).
|
first (dropdowns; the sha is fetched authoritatively). Current drilled-known-good: agent
|
||||||
|
**0.63.0** + golden **0.98.3**. Vouch a golden ≥ 0.98.3 — an older golden bakes a pre-floor
|
||||||
|
controller and re-introduces the manual update step (Part F troubleshooting).
|
||||||
|
|
||||||
### A.4 Check the global controller-version floor
|
### A.4 Check the global controller-version floor
|
||||||
|
|
||||||
Hub UI → **Configuration**. Know the current `min_controller_version` floor, but understand its
|
Hub UI → **Configuration**. Know the current `min_controller_version` floor. With golden ≥ 0.98.3
|
||||||
limit (drill-proven, OQ-2): a fresh box lands on the golden's baked-in controller (**v0.85.1**,
|
vouched (the structural fix for the old OQ-2 finding — drill-proven 2026-07-03), a fresh box lands
|
||||||
which predates the floor-honoring code), so **the floor does NOT pull a fresh install up** — the
|
on a **current, floor-honoring** controller and self-manages from first boot; the floor's job is
|
||||||
one-time update in D.1b is the required step. Raising the floor still matters for keeping the box
|
keeping it current afterwards. Recommended (1 minute, records drift-protection): raise the global
|
||||||
current afterwards. The structural fix — rebuilding + re-vouching the golden with a ≥ 0.86.0
|
floor to the current controller version whenever the golden is rebuilt.
|
||||||
controller — is a recorded operator follow-up.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -224,35 +227,29 @@ pct exec <VMID> -- curl -ksSo /dev/null -w '%{http_code}\n' \
|
|||||||
-H "Host: felhom.<CUSTOMER-DOMAIN>" https://127.0.0.1/ # want 200
|
-H "Host: felhom.<CUSTOMER-DOMAIN>" https://127.0.0.1/ # want 200
|
||||||
```
|
```
|
||||||
|
|
||||||
### D.1b One-time controller update (REQUIRED — do not skip)
|
### D.1b Controller version check (one line — the old manual update step is RETIRED)
|
||||||
|
|
||||||
The golden bakes controller **v0.85.1**, which predates the v0.86.0 floor-honoring code — **a fresh
|
The golden (≥ 0.98.3) bakes a **current, floor-honoring** controller — a fresh install is current on
|
||||||
install does NOT auto-update, regardless of the hub floor**. Bring it current once; from then on the
|
first boot and the hub floor manages it from here. Just verify:
|
||||||
floor mechanism manages it:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 1. ONLY if the installed agent is OLDER than v0.63.0 (check: felhom-agent --version): restart it
|
# controller version current on first boot; the floor manages it from here (want: update_available false)
|
||||||
# first — pre-v0.63.0 agents 401 the update's swap call until restarted (fixed in v0.63.0 by
|
|
||||||
# token reload-on-miss; which agent you got depends on what the hub Day-0 manifest vouched)
|
|
||||||
systemctl restart felhom-agent
|
|
||||||
|
|
||||||
# 2. In the dashboard: Beállítások → "Frissítés" (update to latest), or equivalently:
|
|
||||||
pct exec <VMID> -- curl -ksS -X POST -H "Host: felhom.<CUSTOMER-DOMAIN>" https://127.0.0.1/api/selfupdate/check
|
pct exec <VMID> -- curl -ksS -X POST -H "Host: felhom.<CUSTOMER-DOMAIN>" https://127.0.0.1/api/selfupdate/check
|
||||||
pct exec <VMID> -- curl -ksS -X POST -H "Host: felhom.<CUSTOMER-DOMAIN>" https://127.0.0.1/api/selfupdate/update
|
|
||||||
|
|
||||||
# 3. Wait ~1 min, confirm the new version is running (image pull happens first, then an agent swap):
|
|
||||||
pct exec <VMID> -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
If this reports the baked version as BEHIND (a newer controller shipped after the golden was baked),
|
||||||
|
the same endpoint's `/api/selfupdate/update` — or the dashboard's Beállítások → "Frissítés" button —
|
||||||
|
converges it; that is the normal self-update path, not an install-day requirement. (Installed from a
|
||||||
|
golden older than 0.86.0? See Part F troubleshooting.)
|
||||||
|
|
||||||
### D.2 In the hub UI
|
### D.2 In the hub UI
|
||||||
|
|
||||||
- **Hosts**: the new host reports (status OK, fresh last-report timestamp).
|
- **Hosts**: the new host reports (status OK, fresh last-report timestamp).
|
||||||
- The host's guest list shows `<VMID>` running.
|
- The host's guest list shows `<VMID>` running.
|
||||||
- **Capabilities**: 0 degraded. On a reduced-storage box (no `felhom-pbs`) the offsite-backup
|
- **Capabilities**: 0 degraded. On a reduced-storage box (no `felhom-pbs`) the offsite-backup
|
||||||
related capability is expected degraded — that is the documented go-live standard, not a fault.
|
related capability is expected degraded — that is the documented go-live standard, not a fault.
|
||||||
- The controller version shown for the guest matches what D.1b brought it to (the golden's baked
|
- The controller version shown for the guest is the golden's baked (current) version — it matches
|
||||||
0.85.1 does NOT self-converge — D.1b is the required manual step until the golden is rebuilt with
|
what D.1b's check reported.
|
||||||
a ≥ 0.86.0 controller).
|
|
||||||
|
|
||||||
### D.3 Customer-visible
|
### D.3 Customer-visible
|
||||||
|
|
||||||
@@ -311,8 +308,9 @@ roles/ACL/token/user, the pool (if empty), the install state file.
|
|||||||
| step 5 dies: "no git token in controller.yaml" | customer created without git credentials | Part A.2 — add `git.username`/`git.token`, regenerate config |
|
| step 5 dies: "no git token in controller.yaml" | customer created without git credentials | Part A.2 — add `git.username`/`git.token`, regenerate config |
|
||||||
| step 1: passphrase REJECTED (401) | typo / wrong customer | re-check with the hub UI's printed curl command |
|
| step 1: passphrase REJECTED (401) | typo / wrong customer | re-check with the hub UI's printed curl command |
|
||||||
| step 8 fails: "CT <vmid> already exists" | vmid collision with a hub-invisible guest | pick from `pct list` + `qm list` (Part B); the agent destroys nothing on collision — re-run with a free vmid and `--resume` |
|
| step 8 fails: "CT <vmid> already exists" | vmid collision with a hub-invisible guest | pick from `pct list` + `qm list` (Part B); the agent destroys nothing on collision — re-run with a free vmid and `--resume` |
|
||||||
| controller container missing in-guest after provision (docker ps empty) | pre-v1.9.1 script: the controller-bootstrap unit's boot-time condition lost the race with the bootstrap-mount attach | `pct reboot <VMID>` — the unit runs on the next boot (v1.9.1 does this reboot itself) |
|
| controller container missing in-guest after provision (docker ps empty) | golden older than 0.98.3 (no baked path unit) + pre-v1.9.1 script: the controller-bootstrap unit's boot-time condition lost the race with the bootstrap-mount attach | `pct reboot <VMID>` — the unit runs on the next boot (v1.9.1 reboots itself; goldens ≥ 0.98.3 bake a path unit that starts the controller on the mount hot-plug, no reboot needed) |
|
||||||
| self-update fails: "Agent swap request failed: HTTP 401" (or dashboard agent calls 401) | agent < v0.63.0: the daemon's token index missed a token minted by the provisioner after it started (FIXED in agent v0.63.0 — reload-on-miss) | `systemctl restart felhom-agent`, then retry (D.1b step 1; unnecessary from agent v0.63.0) |
|
| fresh box runs an OLD controller and never self-updates (hub floor ignored) | a golden **older than 0.86.0** was vouched in the Day-0 manifest — its baked controller predates the floor-honoring code | one-time manual update (the retired old D.1b): dashboard Beállítások → "Frissítés", or `POST /api/selfupdate/check` + `/api/selfupdate/update` via the traefik Host-header probe; if the agent is < v0.63.0, `systemctl restart felhom-agent` FIRST (pre-v0.63.0 agents 401 the swap call). Structural fix: re-vouch a golden ≥ 0.98.3 (A.3) |
|
||||||
|
| self-update fails: "Agent swap request failed: HTTP 401" (or dashboard agent calls 401) | agent < v0.63.0: the daemon's token index missed a token minted by the provisioner after it started (FIXED in agent v0.63.0 — reload-on-miss) | `systemctl restart felhom-agent`, then retry (unnecessary from agent v0.63.0) |
|
||||||
| `local-lvm free < 120 GiB` warning | small/shared box | proceed only if you sized the grows deliberately; otherwise free space first |
|
| `local-lvm free < 120 GiB` warning | small/shared box | proceed only if you sized the grows deliberately; otherwise free space first |
|
||||||
| cloudflared restarting in-guest | bad/missing tunnel token | re-check A.1/A.2 token, regenerate config, `docker restart cloudflared` in-guest |
|
| cloudflared restarting in-guest | bad/missing tunnel token | re-check A.1/A.2 token, regenerate config, `docker restart cloudflared` in-guest |
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,29 @@
|
|||||||
# Felhom scripts — Changelog
|
# Felhom scripts — Changelog
|
||||||
|
|
||||||
|
## docs — golden 0.98.3 live: D.1b RETIRED, drill ledger B1/B5 → FIXED, backlog note resolved (2026-07-03)
|
||||||
|
|
||||||
|
Companion to felhom-agent's `build-golden.sh` v2.0.0 (@ `ceca355`): the golden now bakes the CURRENT
|
||||||
|
controller (0.98.3, mandatory-tag convention — B5) and a `felhom-controller-bootstrap.path` unit
|
||||||
|
(controller deploys on the bootstrap-mount hot-plug, no reboot — B1). Clean-room validated end-to-end
|
||||||
|
BEFORE publish (bake → isolated hot-plug proof → local-golden Day-0 install → publish → vouch →
|
||||||
|
`--force-gitea-golden` install); evidence: `documentation/audits/DRILL-golden-098-2026-07-03.md`.
|
||||||
|
|
||||||
|
- `documentation/runbooks/day0-install.md`: **D.1b reduced to a one-line version check** (fresh boxes
|
||||||
|
land current + self-manage); old manual-update procedure moved to Part F troubleshooting keyed on
|
||||||
|
"golden older than 0.86.0 was vouched"; header validated-versions line → script v1.9.1 / agent
|
||||||
|
v0.63.0 / golden v0.98.3; A.3 records the drilled-known-good pair + "vouch ≥ 0.98.3"; A.4 floor
|
||||||
|
text rewritten (fresh boxes now floor-honoring; recommend raising the global floor after a golden
|
||||||
|
rebuild — operator, 1 min).
|
||||||
|
- `documentation/audits/DRILL-day0-cleanroom-2026-07-03.md` ledger: **B1, B5 → FIXED** (pointers);
|
||||||
|
R6 note: the installer's post-provision reboot is retained as a belt, removal recorded as a
|
||||||
|
candidate cleanup (not done).
|
||||||
|
- `documentation/backlog/FOLLOWUP-golden-default-controller-tag.md` + `backlog/README.md`:
|
||||||
|
**RESOLVED** per the M18/M19 convention (file kept + annotated; README entry marked FIXED).
|
||||||
|
- New evidence doc: `documentation/audits/DRILL-golden-098-2026-07-03.md` (A–D transcripts, unit
|
||||||
|
states, resolution-order + fetch/sha proofs, cleanup, observations — incl. the SECURITY
|
||||||
|
observation that the customer `git.token` has package-WRITE rights → scope-down + rotate
|
||||||
|
follow-up).
|
||||||
|
|
||||||
## docs — day0-install D.1b narrowed + drill ledger B2/B3 → FIXED agent v0.63.0 (2026-07-03)
|
## docs — day0-install D.1b narrowed + drill ledger B2/B3 → FIXED agent v0.63.0 (2026-07-03)
|
||||||
|
|
||||||
Agent v0.63.0 fixed both drill findings (B3 token reload-on-miss + B2 guesthook snippets-dir
|
Agent v0.63.0 fixed both drill findings (B3 token reload-on-miss + B2 guesthook snippets-dir
|
||||||
|
|||||||
Reference in New Issue
Block a user