R-265 CI timeout + golden 0.210.0 baked; R-221/R-259/R-258 closed, R-266 minted, G-3 unblocked
gates / gates (push) Successful in 32s
gates / gates (push) Successful in 32s
Four defects of one family, all shipped today: something the box already knows, thrown away or drawn
as its opposite. Agent v0.128.0, controller v0.210.0. NO HUB CODE, no hub bump, no ArgoCD sync.
R-265 (this repo). timeout-minutes: 5 on the gates job — every honest run in the observed session
finished in 18-34s, so this is ~9x the slowest and far under whatever reaped run 264 at 834s with no
log. The alarm mail now carries Elapsed (start stamp via $GITHUB_ENV; an absent stamp prints
"unknown (no start stamp)", never a bogus 1.7-billion-second figure) and its "names itself in the run
log" sentence is qualified so it cannot mislead when there is no log.
⚠ THE UNKNOWN IS NOT CLOSED. Whether the if: failure() alarm fires for a REAPED job is still
unverified. The timeout makes the reap unreachable in practice; it does not answer what happens in
one. Demonstrating it means deliberately hanging a run on main, which would leave the branch red for
a parallel session. Said in the workflow comment, the changelog, R-265 and the report — none of them
claiming it is answered.
GOLDEN 0.210.0 baked, published, round-trip verified, NOT VOUCHED. The currency gate went red the
moment the controller was bumped — correct — and is closed by the bake, never --no-verify. No
--no-verify anywhere this session.
⚠ THE AGENT WAS NOT PUBLISHED UNTIL THIS SESSION CHECKED, AND IT MATTERED. R-221's fix is in the
AGENT, and a fresh install takes its agent from the Day-0 manifest. The binary had been hand-deployed
to felhom-pve and never published, so agent_version 0.128.0 was not selectable and a fresh install
would have received 0.127.0 — the golden would have carried the controller fixes and NOT the one the
headline defect needed. Caught by checking each Day-0 value was FETCHABLE rather than assuming.
Published from the live-deployed bytes, sha-verified across the hop first.
Registers. R-221, R-259, R-258, R-265 CLOSED. R-266 MINTED (READY): the failed root statfs still
travels to the hub as a 0-of-0 disk; ranked LOW because it is the quiet direction — it can only miss
a true alarm, never raise a false one — and it is now a two-repo wire change governed by G-1's gate.
Highest ID moved R-265 -> R-266.
CONTEXT S-39 rules the convention this project was missing: "we do not know" is never drawn as
"fine", and the codebase has ONE way of saying it — an explicit ...Known bool companion checked in
the template. ROADMAP G-3 was explicitly blocked on that decision and is unblocked; what remains
there is a survey-and-convert of existing sites, not the gate.
Capability map row 93 CHECKED and it was NOT claiming something untrue — it is about the operator
notification path. But its narrative ("the page you open to ask whether ONE app is backed up")
invites the wrong reading, and the adjacent thing WAS false until v0.210.0, so the row now records
that the two halves disagreed and only the operator half was true.
Six red-proofs across the two code repos, each with the mutation asserted applied. The one that
matters: Part 1 Scenario A FAILED against today's tree, with the intended message.
Part 1's operator-present live validation is OWED and is the session's STOP.
repo_gates --fast: all 8 OK.
This commit is contained in:
@@ -21,9 +21,28 @@ on: [push]
|
||||
jobs:
|
||||
gates:
|
||||
runs-on: felhom-gates
|
||||
# R-265: A RUN THAT HANGS MUST FAIL ITSELF, LOUDLY AND WITH A LOG.
|
||||
#
|
||||
# Run 264 (2026-08-08) took 834 s and was reaped by the platform, leaving NO log at all — the
|
||||
# log fetch returns HTTP 500 "264.log.zst: file does not exist". Every honest run in that same
|
||||
# session finished in 18–34 s, and every real gate failure finished in under 35 s WITH a log. So
|
||||
# the alarm mail below arrived pointing at a run log that does not exist, telling the operator
|
||||
# "the failing gate names itself in the run log" when nothing could.
|
||||
#
|
||||
# 5 minutes is ~9x the slowest honest run and far under whatever reaped 264, so a hang now ends
|
||||
# as a JOB failure — which produces a log and a step record — rather than as a platform reap,
|
||||
# which produces neither.
|
||||
#
|
||||
# ⚠ WHAT THIS DOES NOT ANSWER, and must not be read as answering: whether the `if: failure()`
|
||||
# alarm step runs at all for a REAPED job is still UNKNOWN. This makes the reap unreachable in
|
||||
# practice; it does not tell us what happens in it. Recorded as still open in R-265.
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- name: Fetch the pushed commit
|
||||
run: |
|
||||
# R-265: the start stamp the alarm reports, so a mail can never again describe a run
|
||||
# without saying how long it took.
|
||||
echo "GATES_STARTED_AT=$(date +%s)" >> "$GITHUB_ENV"
|
||||
# Shallow, and pinned to the exact SHA that was pushed — not to the branch tip, which can
|
||||
# move under us if two pushes race. Probe P3 proved the two are equal when done this way.
|
||||
git init -q .
|
||||
@@ -94,7 +113,7 @@ jobs:
|
||||
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
|
||||
run: |
|
||||
python3 - <<'PY'
|
||||
import json, os, sys, urllib.request, urllib.error
|
||||
import json, os, sys, time, urllib.request, urllib.error
|
||||
|
||||
key = os.environ.get("RESEND_API_KEY", "")
|
||||
if not key:
|
||||
@@ -106,6 +125,18 @@ jobs:
|
||||
run = os.environ.get("GITHUB_RUN_NUMBER", "?")
|
||||
srv = os.environ.get("GITHUB_SERVER_URL", "https://gitea.dooplex.hu")
|
||||
|
||||
# R-265: how long the run took, so a reap is self-identifying. An honest gate failure
|
||||
# lands in well under a minute; a multi-minute figure means the job hit its own timeout
|
||||
# and the interesting question is the runner, not the gates.
|
||||
started = os.environ.get("GATES_STARTED_AT", "")
|
||||
try:
|
||||
# An ABSENT stamp is "unknown", never a number. Defaulting to 0 would print an elapsed
|
||||
# of ~1.7 billion seconds, which is a confident wrong answer — the exact failure mode
|
||||
# this whole session is about.
|
||||
elapsed = "%d s" % (int(time.time()) - int(started)) if started else "unknown (no start stamp)"
|
||||
except (TypeError, ValueError):
|
||||
elapsed = "unknown (unparseable start stamp %r)" % started
|
||||
|
||||
body = json.dumps({
|
||||
"from": "Felhom CI <monitoring@felhom.eu>",
|
||||
"to": ["admin@felhom.eu"],
|
||||
@@ -114,12 +145,18 @@ jobs:
|
||||
"The gate entry point exited non-zero.\n\n"
|
||||
"Repository : %s\n"
|
||||
"Commit : %s\n"
|
||||
"Run : %s/%s/actions/runs/%s\n\n"
|
||||
"The failing gate names itself in the run log.\n\n"
|
||||
"Run : %s/%s/actions/runs/%s\n"
|
||||
"Elapsed : %s\n\n"
|
||||
"The failing gate names itself in the run log - WHEN THERE IS ONE. A run that\n"
|
||||
"hung and was reaped by the platform leaves no log at all (R-265, run 264: 834 s,\n"
|
||||
"log fetch 500). The job now times out at 5 minutes so that case should fail as a\n"
|
||||
"job and keep its log; if Elapsed above is minutes rather than seconds, suspect\n"
|
||||
"the runner before the gates, and if the log is missing say so rather than\n"
|
||||
"guessing which gate it was.\n\n"
|
||||
"If the local pre-push hook was GREEN for this commit, then CI and the hook\n"
|
||||
"disagree - that is a finding about the gates themselves, not about CI, and it\n"
|
||||
"outranks whatever the push was for.\n"
|
||||
) % (repo, sha, srv, repo, run),
|
||||
) % (repo, sha, srv, repo, run, elapsed),
|
||||
}).encode()
|
||||
|
||||
req = urllib.request.Request(
|
||||
|
||||
+39
@@ -17,6 +17,45 @@
|
||||
|
||||
## Standing rulings
|
||||
|
||||
**S-39 — "WE DO NOT KNOW" IS NEVER DRAWN AS "FINE", AND THE CODEBASE HAS ONE WAY OF SAYING IT
|
||||
(2026-08-08, R-259 / R-258; controller v0.210.0).**
|
||||
|
||||
Two things collapse into this rule, and they looked unrelated until they were fixed on the same day:
|
||||
|
||||
- **A measurement that FAILED.** `readDiskUsage` logged a `statfs` error at DEBUG and returned,
|
||||
leaving the caller's floats at zero — and `usageColor(0)` is `"nominal"`. A disk nobody could read
|
||||
was drawn as a healthy empty one.
|
||||
- **A fact about ANOTHER SUBJECT.** The per-app backup tick was set from the box's most recent DB
|
||||
dump run, whichever app it belonged to. An answer about app Y was displayed as an answer about
|
||||
app X.
|
||||
|
||||
Both are the same error: **an absence of knowledge presented as knowledge, in the reassuring
|
||||
direction.** Neither was a wrong number; both were a confident picture over nothing.
|
||||
|
||||
**THE CONVENTION, ruled here so it can eventually be gated.** A figure that can be unknown carries an
|
||||
explicit **`…Known bool` companion** beside it, and the template checks the companion **before
|
||||
rendering anything** — no number, no percentage, no bar. This is the shape `Offbox.StatsKnown`
|
||||
already used (`backups_remote.html`), and its comment states the principle: *"a 0%-wide bar over an
|
||||
unread store is a picture of emptiness, and a picture is a claim."*
|
||||
|
||||
Pointers and separate error fields are both legitimate Go and both still exist in this codebase. The
|
||||
ruling is not that they are wrong; it is that **new** three-state figures use the companion, because
|
||||
`ROADMAP.md` G-3 (a gate for this class) is a name-based check and **a codebase with three dialects
|
||||
cannot be gated**. G-3 was explicitly blocked on this decision and is now unblocked; the remaining
|
||||
work there is a survey-and-convert of existing sites, not the gate itself.
|
||||
|
||||
**And the verdict half of the same rule:** where a status has no honest value, render **nothing** —
|
||||
`backups_apps.html` shows a check for `ok`, a cross for `error`, and no icon for any third value.
|
||||
That empty slot is where "we have no result" belongs. Deriving green from the mere **presence** of an
|
||||
artifact is the *presence is not success* rule turned into a UI badge, and this project's own
|
||||
capability-map and backup rules already forbid it in prose.
|
||||
|
||||
**A test that constructs the receiving struct by hand cannot see either defect** — the dashboard test
|
||||
therefore EXTRACTS the meter block from the shipped template rather than copying it, because a copied
|
||||
block drifts and a drifted copy passes while the page it claims to cover has changed. That is the
|
||||
fixture-is-not-the-wire mistake, hit twice already (R-262's golden, and the OOB fixture in hub
|
||||
v0.99.0).
|
||||
|
||||
**S-38 — A FACT ONE SIDE EMITS AND THE OTHER CANNOT RECEIVE IS A DEFECT, AND A CHECK NOW SAYS SO
|
||||
(2026-08-08, G-1 / R-260 / R-247).**
|
||||
|
||||
|
||||
@@ -1,169 +1,164 @@
|
||||
# REPORT — instruction-file rightsizing (core + path-scoped rules), 2026-08-06
|
||||
# REPORT — the seed that never ran twice, and three pictures that were not true (2026-08-08)
|
||||
|
||||
**Docs and gate only. No Go changed, no version bumped, no image built, nothing deployed, no
|
||||
customer machine touched.** One read-only command ran against live hardware (`ssh demo-hp "qm list"`),
|
||||
permitted by the task for exactly one purpose.
|
||||
Four defects of one family, each with a source-verified mechanism, tests and red-proofs.
|
||||
Agent **v0.128.0** · controller **v0.210.0** · `gates.yml` (workflow only). **The hub was not touched,
|
||||
not bumped and not deployed.**
|
||||
|
||||
**The headline is the contradiction count, not a token saving.** A `/context` reading measured the
|
||||
instruction files at **15k tokens against 869k free** in a 1M window. Space was never the constraint;
|
||||
five stale or conflicting facts were.
|
||||
## 1. Part 1's live sequence — NOT YET RUN
|
||||
|
||||
---
|
||||
**The code, tests and deploy are done; the live proof is the operator-present STOP** (§10). Agent
|
||||
0.128.0 is live on `felhom-pve` (`felhom-agent --version` → `0.128.0`, service `active`, normal
|
||||
smartctl/lvs/lxc-info work in the journal). Steps 3 and 4 — the preflight NOT OK, then OK after one
|
||||
tick with no daemon restart — are **owed and will be quoted verbatim when the operator says go.**
|
||||
|
||||
## 1. Baselines
|
||||
## 2. The writer of `agent.json` — ESTABLISHED
|
||||
|
||||
| Repo | `main` @ start | Clean | Note |
|
||||
`step_agent_config`, **`felhom.eu/scripts/felhom-host-install.sh:2396`**; the Python render at
|
||||
**`:2449`**; the `O_TRUNC` write at **`:2579`**. `PRESERVE_FROM` defaults empty (**`:256`**) and is set
|
||||
only by an explicit `--preserve-from` (**`:1246`**). **The render never writes an `escrow` section at
|
||||
all** — grep over the whole heredoc: zero hits. The pbsdr marker is host-side
|
||||
(`<agent-state>/pbsdr/marker.json`) and survives. **R-221's attribution was correct.**
|
||||
|
||||
A rebuild is only the case that was *measured*; the same hole opens for a hand-edited or restored
|
||||
config, which is the honest reason the fix is at the seam rather than in the installer.
|
||||
|
||||
## 3. Red-proofs — 6 of 6, each with the mutation asserted applied
|
||||
|
||||
| # | mutation | assertion it applied | outcome |
|
||||
|---|---|---|---|
|
||||
| felhom-controller | `a62bb3874b25` | yes | matched spec |
|
||||
| felhom-agent | `a2e914f683bd` | yes | matched spec |
|
||||
| felhom.eu | `d30c2a51ed2a` | yes | matched spec |
|
||||
| app-catalog-felhom.eu | `ee2c8102016a` | yes | **untouched** — 79 lines, already the target shape, cited as the model |
|
||||
| 1 | **Part 1 / Scenario A: remove the new seed call** | marker `MUTATED: the R-221 re-assert removed` present | **RED** — and **yes, it failed against today's tree**, with the intended message ✔ |
|
||||
| 2 | Part 1: remove the early return as well | marker `MUTATED: early return deleted` present | **RED** — the zero-Proxmox-calls assertion is load-bearing, not decorative ✔ |
|
||||
| 3 | Part 3 / F: revert to `status.LastDBDump.Success` | marker present | **RED** — app X's false green returns ✔ |
|
||||
| 4 | Part 3 / G: map "no result" to `ok` | marker present | **RED** — green-on-presence returns ✔ |
|
||||
| 5 | Part 2 / D: ignore `DiskKnown` in the template | marker present | **RED** — „0.0 GB / 0.0 GB (0%)" in the nominal colour returns ✔ |
|
||||
| 6 | Part 2 / E: force the flag false | marker present | **RED** — a healthy box is shown losing its numbers ✔ |
|
||||
|
||||
**One correction:** the workspace-root `CLAUDE.md` measured **16,642 B / 208 lines**, not the spec's
|
||||
15,431 / 207 — it was edited at 08:32 that morning, after the spec was written. The other five files
|
||||
matched exactly. Also structural: `/mnt/5_hdd/felhom.eu/git` **is not a git repository**, so the live
|
||||
root `CLAUDE.md` is untracked; only its copy under `felhom.eu/` is version-controlled.
|
||||
All six restored and re-verified green. **Answer to the question asked directly: the Part 1 test DID
|
||||
fail against today's tree.**
|
||||
|
||||
## 2. Contradictions: 5 before → 0 after
|
||||
## 4. The Hungarian strings as shipped
|
||||
|
||||
| # | What conflicted | Resolution |
|
||||
- `„A tárhely mérete most nem olvasható ki."` — the disk caveat line
|
||||
- `„nem ismert"` — the short label in the value slot
|
||||
- `„Erről a mentésről nincs eredményünk."` — the `title` on the no-verdict backup mark
|
||||
|
||||
## 5. The §7.3 truth table as implemented
|
||||
|
||||
| this app's own most recent dump result | restore point | verdict |
|
||||
|---|---|---|
|
||||
| 1 | agent said demo-hp hosts drill VM `300`; controller said none was provisioned | **Measured live:** `qm list` → `300 drill-r50 stopped`. **felhom-agent was right.** `nodes.md:96` already said so correctly — both `CLAUDE.md` blocks became pointers, no new text needed |
|
||||
| 2 | agent's `TEMPORARY` block (expired 2026-08-02) said felhom-pve was remote; controller said it was back on the LAN | both deleted; the audit holds the record. The gate now fails any past-dated TEMPORARY block |
|
||||
| 3 | controller pinned `agent 0.93.0`, against the root file's own no-versions rule | every version literal removed from effective text in all four files |
|
||||
| 4 | controller gave `demo-felhom` as the LAN *fallback* address as if it were the route | host tables removed from all three `CLAUDE.md` files → `nodes.md` |
|
||||
| 5 | root file said memory held `(119 files)`; it holds 157 + the index | parenthetical deleted, not corrected — derivable, and it would go stale again |
|
||||
| any of its databases failed | yes | `error` |
|
||||
| all clean | yes | `ok` |
|
||||
| none recorded | yes | **no icon**, time only, with the title above |
|
||||
| any | no | no tier-1 row, unchanged |
|
||||
|
||||
The sweep found **none beyond the five**.
|
||||
**Recency was left alone**, deliberately: an age threshold means inventing a number, and the time is
|
||||
already printed beside the icon. Recorded as an observation.
|
||||
|
||||
## 3. Before / after (effective = HTML comments stripped, i.e. what the model receives)
|
||||
## 6. The CI timeout, and what is still unknown
|
||||
|
||||
| File | before | after raw | after effective | gate |
|
||||
|---|---|---|---|---|
|
||||
| workspace-root `CLAUDE.md` | 208 ln / 16,642 B | 182 / 11,280 | **142 / 8,105** | pass |
|
||||
| versioned copy | 208 / 16,642 | 182 / 11,280 | **142 / 8,105** | pass — `cmp` identical |
|
||||
| `felhom-controller/CLAUDE.md` | 215 / 14,775 | 110 / 6,341 | **92 / 4,881** | pass |
|
||||
| `felhom-agent/CLAUDE.md` | 216 / 15,554 | 205 / 13,619 | **173 / 11,491** | pass |
|
||||
| `felhom.eu/CLAUDE.md` | 241 / 17,471 | 235 / 17,003 | **227 / 16,286** | **over — deferred** |
|
||||
**`timeout-minutes: 5`** — every honest run in the observed session finished in **18–34 s**, so 5 min
|
||||
is ~9× the slowest honest run and far under whatever reaped run 264 at 834 s. The alarm mail now
|
||||
carries **`Elapsed`** (a start stamp in step 1 via `$GITHUB_ENV`; an absent stamp prints
|
||||
`unknown (no start stamp)`, never a bogus number), and its "names itself in the run log" sentence is
|
||||
qualified so it cannot mislead when there is no log.
|
||||
|
||||
**A controller session's instruction load: 31,417 → 12,986 effective bytes (−59%).**
|
||||
**THE UNKNOWN IS NOT CLOSED.** Whether the `if: failure()` alarm fires at all for a *reaped* job is
|
||||
**still unverified**. The timeout makes the reap unreachable in practice; it does not answer what
|
||||
happens inside one. Demonstrating it would mean deliberately hanging a run on `main`, which would
|
||||
leave the branch red for a parallel session, so it was not done. Said in the workflow comment, the
|
||||
changelog, R-265 and here — four places, none of them claiming it is answered.
|
||||
|
||||
Figures are bytes, deliberately. The measured `/context` ratio shows a bytes/4 token estimate
|
||||
understates the true cost by **1.59–1.90×**, so byte counts are the honest unit here.
|
||||
## 7. Tests
|
||||
|
||||
**`felhom.eu/CLAUDE.md` is knowingly left over the ceiling.** Getting it under 200 needs the
|
||||
core+rules restructure the spec explicitly forbade for reviewability, and the gate is registered only
|
||||
in the controller and agent runners. Deferred as **R-229**, said plainly rather than quietly absorbed.
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| controller | — | **1355** total (`+7` this session: 4 verdict, 3 disk-meter) |
|
||||
| agent | — | **947** total (`+5` this session) |
|
||||
|
||||
## 4. Files created / modified
|
||||
`go build ./... && go vet ./... && go test ./...` **green in both repos** (controller 28 packages,
|
||||
agent 29), run separately from every commit. `controller_gates.py --fast` all OK; `agent_gates.py
|
||||
--fast` all OK; `repo_gates.py --fast` **all 8 OK**.
|
||||
|
||||
**Created:** `felhom-controller/.claude/rules/{gates,ui-hungarian,backup-paths,agent-coupling}.md` ·
|
||||
`felhom-agent/.claude/rules/health-checks.md` · `felhom.eu/scripts/instructions_gate.py` ·
|
||||
`felhom.eu/scripts/test_instructions_gate.py` ·
|
||||
`felhom.eu/documentation/audits/LEDGER-instruction-trim-2026-08-06.md`
|
||||
The dashboard test **extracts** the meter block from the shipped template rather than copying it — a
|
||||
copied block drifts and then passes while the page it covers has changed.
|
||||
|
||||
**Modified:** the four `CLAUDE.md` files + the versioned copy · `controller_gates.py` ·
|
||||
`agent_gates.py` · `skills/felhom-testing/SKILL.md` (gained the nine-instance table) ·
|
||||
three `CHANGELOG.md` · `CONTEXT.md` (S-35) · `OPEN-ITEMS.md` (R-229)
|
||||
## 8. Deploy
|
||||
|
||||
## 5. Mechanism verification — done before relying on it
|
||||
| | version | evidence |
|
||||
|---|---|---|
|
||||
| agent | **0.128.0** | `felhom-agent --version` on `felhom-pve`; service `active`; prior binary kept as `felhom-agent.bak-0.127.0` |
|
||||
| controller | **0.210.0** | `docker ps` on guest 9201: `felhom-controller:0.210.0 Up (healthy)` |
|
||||
|
||||
The whole design rests on three claims. Two were confirmed, one was **false**:
|
||||
**Endpoint-level validation of the controller UI was ATTEMPTED AND DID NOT SUCCEED — stated rather
|
||||
than skipped.** What was tried: login POST against the container IP `172.17.0.2:8080` with the
|
||||
mandatory `Host:` header, first with curl's cookie jar and then with the `Set-Cookie` handled
|
||||
explicitly (the known `felhom_session` jar trap). Login returned `302` and `/` returned `302` back to
|
||||
login both times, so the dashboard was never rendered. **Fallback observable, on the deployed
|
||||
artifact rather than the source** — `grep -a` inside the running container's binary:
|
||||
`SystemInfo.DiskKnown` ×1, the disk caveat line ×6, the no-verdict title ×1, `--version` →
|
||||
`0.210.0 (commit c732fe1)`. That proves the shipped bytes carry both fixes; it does **not** prove the
|
||||
rendered page, and the template-level tests are what stand for that.
|
||||
|
||||
| Claim | Verdict |
|
||||
|---|---|
|
||||
| `.claude/rules/*.md` + `paths:` loads only on a matching file read | confirmed against the installed 2.1.222 build |
|
||||
| HTML comments are stripped before injection | **confirmed empirically** — control (both markers plain → both reported) vs treatment (one commented → not reported, twice) |
|
||||
| `unconditional: true` frontmatter | **NOT a product feature.** It is this project's own marker; the gate's docstring says so, so nobody hunts for it in the docs |
|
||||
## 9. The bake, and the three Day-0 values
|
||||
|
||||
The HTML-comment test **failed its first red-proof** — an open-ended prompt returned one marker in
|
||||
both arms, so the instrument was dropping a result silently. Rewritten as forced yes/no questions
|
||||
with a both-plain control, it discriminated. Worth recording: this project's own rule ("an instrument
|
||||
that can drop results silently is not a measurement") caught a measurement made *for* this task.
|
||||
`documentation/tests/golden-0.210.0-2026-08-08/` — golden **0.210.0**, **656 787 777 B**, sha256
|
||||
`b9f701fa…4c0a00`, round-trip verified, `./etc/felhom-controller-image` read **out of the downloaded
|
||||
archive** → `felhom-controller:0.210.0`. Markers all green, token grep 0 with a control returning 1,
|
||||
bake VM destroyed, drill disk restored to `virgin`.
|
||||
|
||||
## 6. Amnesty list (the list to review)
|
||||
| field | set to | verified |
|
||||
|---|---|---|
|
||||
| `golden_version` | **0.210.0** | package `GET` **200**; hub dropdown offers it, `data-sha` matches the bake |
|
||||
| `agent_version` | **0.128.0** | package `GET` **200**; hub dropdown offers it, `data-sha` matches the deployed binary |
|
||||
| `min_agent` | **0.127.0** (unchanged) | read from the CHANGELOG header written this session |
|
||||
|
||||
Deliberately short — **one** item qualified:
|
||||
**⚠ The agent was NOT published until this session checked, and it mattered.** R-221's fix is in the
|
||||
**agent**; the binary had been hand-deployed and never published, so `agent_version 0.128.0` was not
|
||||
selectable and a fresh install would have received 0.127.0 — the golden would have carried the
|
||||
controller fixes and **not** the one the headline defect needed. Caught by checking each value was
|
||||
*fetchable* instead of assuming. Published from the **live-deployed bytes**, sha-verified across the
|
||||
hop first (`c6eba73b…` identical on both sides).
|
||||
|
||||
- Three generic code-quality bullets in `felhom.eu/CLAUDE.md` ("always double-check generated code",
|
||||
"handle edge cases without overcomplicating", "add debug capabilities"). No recorded production
|
||||
violation, recoverable worst case, and duplicated in the workspace root. **"ask first — don't
|
||||
guess" was kept** — that one is a real behavioural directive.
|
||||
**`min_agent` stays 0.127.0 deliberately:** `MinAgent` declares what the *controller* requires, and
|
||||
v0.210.0 requires nothing new from the agent. R-221 is delivered by `agent_version`, not by the floor.
|
||||
|
||||
**Nothing on the protected list was touched.** The nine-row invariant table, "presence is not
|
||||
success", the four R-96 rules, the F9 fence, secrets handling, trunk-based, the DooPlex protections,
|
||||
the destructive-target fences with their permitted targets, and the clean-tree gate all survive —
|
||||
moved or compressed, never dropped, each compressed fence keeping its target and its reason.
|
||||
**Nothing was vouched. No hub setting was touched.** The Save is the operator's.
|
||||
|
||||
## 7. Gate results
|
||||
## 10. Gate failures remaining
|
||||
|
||||
- `controller_gates.py --fast` → **all 9 gates OK**, including the new `instructions`.
|
||||
- `agent_gates.py --fast` → **both gates OK**.
|
||||
- `test_instructions_gate.py` → **20 passed, 0 failed**.
|
||||
- `go build ./... && go vet ./... && go test ./...` → **green in all three modules**
|
||||
(`felhom-controller/controller`, `felhom.eu/hub`, `felhom-agent`) — and unchanged, as required: no
|
||||
Go was touched.
|
||||
- **Registration seam asserted by parsing each runner's `GATES` list from the AST**, not by grepping
|
||||
the source — a commented-out registration would not satisfy it.
|
||||
|
||||
**Red-proof** (ceiling temporarily 100, real trimmed files):
|
||||
|
||||
```
|
||||
CLAUDE.md effective lines : 92 (ceiling 100) [felhom-controller]
|
||||
CLAUDE.md effective lines : 173 (ceiling 100) [felhom-agent]
|
||||
instructions_gate: 1 FAILURE(S)
|
||||
- .../felhom-agent/CLAUDE.md: 173 effective lines, ceiling 100. This is an ADHERENCE limit,
|
||||
not a space limit — long instruction files get followed less reliably and hide contradictions.
|
||||
```
|
||||
|
||||
It discriminated: agent failed and was named, controller still passed. Threshold restored, suite
|
||||
re-run green.
|
||||
|
||||
## 8. `MEMORY.md` — measured only, unchanged
|
||||
|
||||
146 lines / 17,688 B (17.3 KB) against the 200-line / 25 KB limits: **within both, nothing
|
||||
truncated.** The expected finding is absent; a different one is present — the index references **113**
|
||||
memory files while **157** exist on disk, and **zero** referenced files are missing. So 44 memory
|
||||
files are unindexed. Recorded, not acted on (the ruling deferred this).
|
||||
|
||||
## 9. Delegation
|
||||
|
||||
**No subagent was used.** All search, inventory and verification was done in-session. Nothing was
|
||||
delegated, so nothing needs the read-only caveat.
|
||||
|
||||
## 10. Which files actually load
|
||||
|
||||
Settled from the supplied `/context`, not assumed: at the workspace root exactly **two** memory files
|
||||
load — the root `CLAUDE.md` (6.6k tokens) and `MEMORY.md` (8.4k). Per-repo `CLAUDE.md` files are
|
||||
**not** loaded at launch; they load on demand when a file in that directory is read. This is why the
|
||||
irreversible fences were kept in the root file.
|
||||
|
||||
**Still outstanding (HUMAN):** `/context all` from **inside `felhom-controller`** after this trim, to
|
||||
give the measured after-figure and confirm which rule files a repo session actually pulls in. Claude
|
||||
Code cannot invoke a slash command on itself. The after-figures above are byte counts from disk, not
|
||||
measured tokens, and are labelled as such.
|
||||
**None.** `golden_currency_gate.py` went red the moment the controller was bumped — correct, and
|
||||
closed by the bake, not by `--no-verify`. **No `--no-verify` anywhere in this session.**
|
||||
|
||||
## 11. Register
|
||||
|
||||
`OPEN-ITEMS.md` row taken: **R-229** (READY, owner Viktor) — covering `felhom.eu/CLAUDE.md`'s
|
||||
restructure, `felhom-agent`'s remaining headroom, the auto-memory decision, and the
|
||||
spec-as-failing-test pilot.
|
||||
**Closed:** R-221, R-259, R-258, R-265 (the last with its unknown explicitly still open).
|
||||
**Minted:** **R-266** — the failed root `statfs` still travels to the hub as a 0-of-0 disk; ranked
|
||||
low because it is the quiet direction, and now a two-repo wire change governed by G-1's gate.
|
||||
**Highest ID moved R-265 → R-266.** **G-3 unblocked** in `ROADMAP.md`; **CONTEXT S-39** rules the
|
||||
convention.
|
||||
|
||||
## 12. Observations — not acted on
|
||||
**Still open, untouched:** R-246, R-255, R-256, R-257, R-261, R-262, R-263, R-264, R-240, R-243,
|
||||
R-202, R-213, R-244, R-214/R-235, C7's test-comment half, and G-8's other half.
|
||||
|
||||
1. `target-selection.md`'s known t740 off-site-tier error is still there (out of scope).
|
||||
2. **The root `CLAUDE.md` could be a symlink** to its versioned copy, removing the divergence class
|
||||
entirely — same filesystem, and Claude Code reads through symlinks (the four skills already are).
|
||||
Not done, per the spec. If adopted, the gate's copy-identity check should become a symlink-target
|
||||
check.
|
||||
3. **A blanket version-literal ban has a false-positive class the spec did not anticipate:** a bare
|
||||
`\d+\.\d+\.\d+` matches the first three octets of every IPv4. Without the dotted-quad exclusion the
|
||||
gate fails on `192.168.0.180` in the agent's own file.
|
||||
4. The spec calls the invariant table "the eight-invariant table"; **it has nine rows** and its own
|
||||
text says "Nine instances". Flagged so the protected list is not later applied to eight of nine.
|
||||
5. `demo-hp` also hosts VM `321 c11-appliance`, **running** — seen in the same `qm list`, not
|
||||
mentioned in `nodes.md`. Not investigated.
|
||||
6. Two unused Claude Code plugins (`typescript-lsp`, `context7`) and a broad user-scope allowlist
|
||||
(`Bash(python3:*)`, `Bash(curl:*)`, `Bash(scp:*)` — standing arbitrary execution and network
|
||||
egress in every project) were found by the earlier setup audit. **They live in
|
||||
`~/.claude/settings.json`, not in any repo, and are deliberately out of scope.** Recorded only.
|
||||
## 12. The capability-map row
|
||||
|
||||
`00-capability-map.md:93` — *"A failed per-app Tier-1 backup reaches the OPERATOR"*. **Checked, and
|
||||
it was NOT claiming something untrue:** it is about the operator notification path and claims nothing
|
||||
about what `/backups/apps` draws. But its narrative — *"the page you open to ask whether ONE app is
|
||||
backed up"* — invites the wrong reading, and the adjacent thing WAS false: the page's tick was green
|
||||
on presence until v0.210.0, so the two halves disagreed and only the operator half was true. The row
|
||||
now records that.
|
||||
|
||||
## 13. Observations — noticed, NOT acted on
|
||||
|
||||
1. **Other collectors in `info_linux.go` return silently on error** — `readLoadAvg`, `readMemInfo`
|
||||
and the temperature read. Only the disk one was traced to a customer-visible surface, and the
|
||||
change was deliberately not widened into a refactor of that file.
|
||||
2. **The tick's recency weakness stands.** A tick over a three-week-old restore point is still a
|
||||
tick. Adding an age threshold means inventing a number; the time is printed beside it.
|
||||
3. **The controller UI could not be driven headlessly this session** (§8). Worth one session to
|
||||
re-establish the documented headless login, because "invoke the endpoint the UI invokes" is this
|
||||
project's standard validation method and it is currently unavailable for the controller.
|
||||
4. **`HDDKnown` is wired but has no template consumer yet** — the HDD path renders through
|
||||
`StorageBars`, which has its own `Disconnected` state. Adding the flag there is the natural next
|
||||
step of the S-39 conversion and is part of G-3's survey, not this session.
|
||||
|
||||
@@ -18,64 +18,64 @@ network, and open apps from a launcher or a shared link. Backups run on their ow
|
||||
machine's drive, a second drive, and an encrypted off-site copy.
|
||||
|
||||
**The backup promise is proved, and so is getting the data back yourself.** A machine has been
|
||||
destroyed on purpose and its files came back byte for byte identical — four times now, including a
|
||||
filename with Hungarian accents. On **2026-08-07 the household's own journey passed for the first
|
||||
time**: someone with a browser and their recovery code got everything back with **no command line
|
||||
inside the machine at any point**. From logging in to seeing what is in the store took **72 seconds**.
|
||||
*(R-201 — closed.)*
|
||||
|
||||
**And the two rough edges that walk found are gone** — after a rebuild the restore used to stop dead
|
||||
twice; both refusals now say what happened, that nothing is lost, and link to the screen that fixes
|
||||
it. *(R-252, R-253 — closed 2026-08-08.)*
|
||||
destroyed on purpose and its files came back byte for byte identical — four times now. On
|
||||
**2026-08-07 the household's own journey passed for the first time**: someone with a browser and
|
||||
their recovery code got everything back with **no command line inside the machine at any point**,
|
||||
in 72 seconds. The two rough edges that walk found are also gone. *(R-201, R-252, R-253 — closed.)*
|
||||
|
||||
## What's broken
|
||||
|
||||
- **Nothing new is broken.** All three secret-in-page faults are fixed; what remains is that the
|
||||
*check* against a fourth covers 4 pages of 27, and the cheap one covering all of them is blind to
|
||||
the shape that actually shipped. *(R-255)*
|
||||
- **Nothing new is broken.** The *check* against a fourth secret-in-a-page covers 4 pages of 27, and
|
||||
the cheap one covering all of them is blind to the shape that shipped. *(R-255)*
|
||||
- **The machine's own screen keeps telling an already-paired box to pair itself** — 25 minutes after it
|
||||
was paired, on a screen that promises it refreshes itself. *(R-214, R-235)*
|
||||
- **A rebuilt machine cannot create a new recovery code at all.** *(R-221)*
|
||||
- **A backup that covered nothing still calls itself „Sikeres".** The state is honest; the word is not.
|
||||
*(R-240)*
|
||||
- **A machine waiting for its recovery code can stop backing up off-site without alarming us.** After
|
||||
a *rebuild* we ARE told; the gap is a box reaching that state with no working tier behind it. *(R-243)*
|
||||
- **The card offering to reopen set-aside backups promises more than we can deliver** — we keep the old
|
||||
sealed package, but nothing can open it. *(R-202)*
|
||||
- **Deleting a customer leaves rows behind** on every test machine ever torn down, while reporting a
|
||||
clean teardown. No secrets involved, but it accumulates with each walk. *(R-244)*
|
||||
- **A machine waiting for its recovery code can stop backing up off-site without alarming us** —
|
||||
after a *rebuild* we ARE told; the gap is reaching that state with no working tier. *(R-243)*
|
||||
- **The card offering to reopen set-aside backups promises more than we can deliver.** *(R-202)*
|
||||
- **Deleting a customer leaves rows behind** while reporting a clean teardown — no secrets, but it
|
||||
accumulates. *(R-244)*
|
||||
- **Putting restored files back where they belong is still a manual step.** *(R-213)*
|
||||
|
||||
## Fixed today — the thrown-away sentence, and a check so there is no next one
|
||||
## Fixed today — four things the machine knew and did not say
|
||||
|
||||
We could not tell whether your engineer could get into a machine. The machine says so every few
|
||||
minutes; **the hub had nowhere to put the sentence and discarded it on arrival**, so a box with the
|
||||
door open, the lock working and **no key issued** was reported as fine. Not a wrong answer — an answer
|
||||
to a question nobody was asking. Fixed, and the alert now **names the missing key** instead of saying
|
||||
"access degraded". *(R-260, R-247 — closed; hub v0.99.0, controller v0.209.0.)*
|
||||
All one family: something the box already knows, thrown away or drawn as its opposite.
|
||||
|
||||
**The check was built first and watched failing on 40 facts, before a single one was fixed** — the
|
||||
night before, an off-the-shelf tool for a neighbouring shape was rejected for failing exactly that
|
||||
test. Of the 40: three now change what we are told, sixteen are genuinely redundant, and **twenty-one
|
||||
are recorded as undecided rather than quietly waved through** *(R-264)* — the strongest being
|
||||
per-guest network health, which we already lost 1 h 15 m to once.
|
||||
- **A rebuilt machine can set up its own recovery again.** The one fact the setup needs was written
|
||||
only the first time, and a rebuild replaced the configuration while leaving the note saying
|
||||
"already done". It is now checked and re-written every minute instead of remembered once, so a
|
||||
hand-edited or restored configuration heals too. **This was the last item blocking a customer from
|
||||
something we promise them.** *(R-221 — agent 0.128.0.)*
|
||||
- **A disk we failed to read is no longer drawn as a healthy empty one.** No figures, no bar, and it
|
||||
says so: „A tárhely mérete most nem olvasható ki." *(R-259 — controller 0.210.0.)*
|
||||
- **A backup tick now answers about that app.** It went green because *some* backup file existed and
|
||||
*some other* app's database dump had succeeded most recently. Now: that app's own result, and
|
||||
**no mark at all** when we have none. *(R-258 — controller 0.210.0.)*
|
||||
- **Our own alarm no longer points at a page that may not exist.** A check run now gives up after
|
||||
five minutes rather than hanging until something else kills it, and the mail says how long it ran.
|
||||
*(R-265.)*
|
||||
|
||||
**Not fixed, and said rather than glossed:** that failed disk reading still reaches us as "0 of
|
||||
0 GB". It is the quiet direction — it can only miss a true alarm, never raise a false one. *(R-266)*
|
||||
|
||||
## What we're working on
|
||||
|
||||
- **Widening the check** so a fourth secret-in-a-page is caught by a machine. *(R-255)* · **Deciding
|
||||
the twenty-one** — each gets a reader, or stops being sent. *(R-264)*
|
||||
- **Proving the hub really keeps the old sealed key** when a machine re-seals. *(R-198)* · Still open
|
||||
from the overnight sweep, none urgent: *(R-256…R-259, R-261…R-263)*
|
||||
- **Proving the hub really keeps the old sealed key** when a machine re-seals. *(R-198)* · Still open,
|
||||
none urgent: *(R-256, R-257, R-261…R-263, R-266)*
|
||||
|
||||
## Waiting on you
|
||||
|
||||
- **One approval: move the base image on by one.** You vouched **0.208.0** during the night —
|
||||
thank you, that gap is closed. Today's release went one further, so: Hub → Configuration → Day-0
|
||||
artifacts → Golden **0.209.0** → Save. One field moves, the other two were checked and are right,
|
||||
and it is reversible. *(R-242)*
|
||||
- **Fourth time in three days, so worth a minute.** A check now catches the *baking* being forgotten
|
||||
— it caught it again today and refused the push until it was done. **Nothing catches the approval
|
||||
being forgotten.** Two ways to close that are written up, neither built. *(ROADMAP G-8)*
|
||||
- **A watching moment, five minutes.** Today's recovery fix is proved by removing one line from a
|
||||
demo machine's config — backed up first, disposable machine, no customer data near it — and
|
||||
watching the setup screen go green on its own. Nothing is destroyed. Say when.
|
||||
- **One approval, three values this time.** Hub → Configuration → Day-0 artifacts: Golden
|
||||
**0.210.0**, Agent **0.128.0**, minimum agent **0.127.0** (unchanged) → Save. Each was checked to
|
||||
be downloadable and selectable before being written here. **Agent 0.128.0 is the one that carries
|
||||
today's recovery fix**, so a new machine needs both, not just the image. It supersedes the 0.209.0
|
||||
approval you already gave, and it is reversible. *(R-242)*
|
||||
|
||||
## DooPlex infrastructure — separate from the product
|
||||
|
||||
@@ -91,3 +91,6 @@ being readable.*
|
||||
it writes PASS/FAIL to `/var/log/felhom-store-postboot-check.log`. On PASS, 34 GB comes back. *(R-209a)*
|
||||
- **Backup scripts on DooPlex are unversioned host state** *(R-231)*, and the instruction-file
|
||||
follow-ups each need a decision rather than an edit *(R-229, R-230)*.
|
||||
- **Our build-check alarm has one gap left.** A run that hangs is now cut off after five minutes and
|
||||
the mail says how long it took — but **whether the alarm fires at all when the machinery kills a
|
||||
run outright is still unverified**, and we have not claimed otherwise. *(R-265)*
|
||||
|
||||
@@ -90,7 +90,7 @@
|
||||
| An app can be **withdrawn from the catalog without orphaning the customers running it** (available / hidden / abandoned) | controller v0.158.1, catalog metadata | **PROVEN-LIVE** (2026-07-21) | TASK-F Part 1. Verified on 9201 through the real endpoints: `lifecycle: abandoned` arrived via the normal catalog sync; plant-it renders 0 times on the Alkalmazások page (control app renders 10); a direct `POST /api/stacks/plant-it/deploy` → **HTTP 409 "Ez az alkalmazás jelenleg nem telepíthető."**; the app page carries the permanent notice and offers no Telepítés button. `felhom-controller/REPORT.md` (2026-07-21) | Deployed instances keep FULL function in every state — lifecycle governs what is offered, never what runs. Orphan detection deliberately never sees the field (red-proofed): a withdrawn template stays in the catalog tree, or every deployed instance would read `Elavult` and be offered deletion. Unknown values fail OPEN; the deploy gate fails CLOSED. R-57 |
|
||||
| Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap | **PROVEN-LIVE (2026-07-25)** | **R-50 SHIPPED and deployed to the whole fleet.** The control plane now rides a host-internal, portless island bridge (`vmbr9`, `169.254.253.1/30`↔`.2/30`) with a fixed private address that no LAN/DHCP/site move can invalidate. Proven end-to-end: the spike's F1 replay (renumber the LAN → agent stays bound on the island, control plane HTTP 200; the LAN-literal contrast reproduces the original `bind: cannot assign requested address` daemon-death) + cold-reboot survival (`SPIKE-island-bridge-2026-07-25.md`), the migration runbook run verbatim (`RUNBOOK-island-migration.md`), a fresh provision auto-attaching the island `net1` (A4), and the live migration of **both demo boxes** (demo-hp + demo-felhom, 2026-07-25) — island `/storage` HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), **apps served throughout (0 container restarts)**, hub reporting 0.96.0. **Origin:** `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — the real relocation where the agent's LAN-literal bind took storage/PBS/quiesce/restore-test/DR down silently; that is now structurally impossible on a migrated box | **Fleet: DONE.** Remaining: **R-74** — bring the island to Peti's 2-node cluster (SDN vnet / bridge parity), its own supervised runbook. Related historical: R-51 (dead-primary alerting), R-52 (boot desired-state reconciliation), both shipped |
|
||||
| **The customer is warned BEFORE a filesystem fills** — per filesystem, in Hungarian, naming the drive and the free space, edge-triggered | controller **v0.191.0/.1/.2**, hub **v0.89.0** (R-167, decision D-c) | **PROVEN-LIVE (2026-08-02)** | `audits/SPIKE-r165-mp1-merge-2026-08-02.md` (context) + `felhom-controller/REPORT.md`. Exercised on guest 9201 against a REAL filesystem (`/mnt/sys_drive` filled with `fallocate`): **`disk_warning` at 90% used / 4.7 GB free** → hub `notification_log` `customer | disk_warning | sent` with the dynamic Hungarian rendered; grown to 1.7 GB free → **`disk_critical`** → `customer | sent`; file removed → `critical → ok … cleared silently, re-armed` and the persisted state emptied. **Exactly two events across three boots** — the boot in between produced none, which is the edge trigger holding | **Nothing warned before this.** The only prior signal was the healthcheck's generic `health_degraded` at 90%, for REGISTERED STORAGE PATHS ONLY — it never looked at the docker area or the system-data area, never gave a free-byte figure and never named a drive. **The two event types already existed with NO PRODUCER** (`disk_warning`/`disk_critical`: allowlisted, copy'd, in `DefaultEnabledEvents`, checkbox'd) — the **sixth** *built-but-never-wired* instance here; this ships their producer rather than a seventh near-duplicate type. **Two threshold terms, whichever trips first, and the live proof vindicated the design:** the critical crossing fired on the FREE-BYTE term (1.7 GB) at only **91%** used — a percentage-only rule would have missed it. The hub's generic `customerMessages` entries were REMOVED, because `FormatCustomerEmail` prefers the entry over the message and would discard the label and figures. **Known gap → R-177:** there is no operator-triggerable run-now path; the check is daily 03:30 + once at startup, so confirming a cleared warning on a support call needs a controller restart or a wait |
|
||||
| **A failed per-app Tier-1 backup reaches the OPERATOR — EVERY failing app, in ONE mail per run, and every failure recorded whether or not it is mailed** | controller **v0.194.0**, hub **v0.90.1** (R-158 → R-167 → **R-182**) | **PROVEN-LIVE (2026-08-03)** | `felhom-controller/REPORT.md`. Two real capture failures on guest 9201 (`mkdir …/backups: permission denied`) → both accepted and stored by the hub, `operator | recovery_unit_capture_failed | sent`, and the positive observable **`customer | recovery_unit_capture_failed | skipped | operator_only`** read from the hub's `notification_log`. One event per app, loop continuing | **Before this the failure was a `[WARN]` line and nothing else** — the manager carried three notify seams and none for the unit capture, so `/backups/apps`, the page you open to ask whether ONE app is backed up, was the one page that never said. **Deliberately NOT `backup_failed`:** that type is customer-enabled by default and carries Hungarian copy, so reusing it — which R-158's own proposal said — would email the customer about a failure they cannot act on. **D-c routes it to the operator and overrides the proposal.** Operator-only is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a `customerMessages` entry (the v0.78.0 defect); a red-proof removing the register entry shows the customer receiving it. **ROW REWRITTEN 2026-08-03 (R-182) — the 2026-08-02 claim was TRUE OF ONE APP AND FALSE OF THE REST, and it is worth saying which.** The signal existed and worked; what it did not do was scale past the first failing app. Measured: nine per-app events reached the hub in one day and **two** operator mails went out, because the cooldown key is `customerID:eventType(+tier)` and this type carries `app` but no `tier` — so the first refused app took the hour and the rest were dropped **before `LogNotification`**, leaving no row on any channel. The old row said "One event per app, loop continuing", which was true of what the CONTROLLER emitted and not of what the operator received. **Now:** the per-app event is the RECORD (hub `recordOnlyEvents`: stored + logged every time, never mailed) and `backup_run_failures` is the NOTIFICATION — one mail per run listing every failed app, its leg and its reason, with the counts and free space. A suppressed operator event of ANY type now leaves a `suppressed` row naming its key. **Proven live on demo-hp 2026-08-03** by a real 64.6 GiB fill (thin pool held 30.78 → 30.78): `notification_log` shows `recovery_unit_capture_failed | operator | recorded` ×2, `backup_run_failures | operator | sent` ×1 naming BOTH apps, and `backup_run_failures | customer | skipped | operator_only`. A second run in the same hour produced a second digest; after freeing space a run completed with 2 volume dumps and **no** digest. The suppression row proved itself on an unplanned event — `disk_critical | suppressed | key=demo-hp:disk_critical` — a collapse that yesterday would have left nothing at all. **The digest's silence is safe only because** the hub's deadline check raises `expected_backup_missed` from report freshness independently of any mail (`monitor/deadline.go:396,417`); that check is load-bearing for this row |
|
||||
| **A failed per-app Tier-1 backup reaches the OPERATOR — EVERY failing app, in ONE mail per run, and every failure recorded whether or not it is mailed** | controller **v0.194.0**, hub **v0.90.1** (R-158 → R-167 → **R-182**) | **PROVEN-LIVE (2026-08-03)** | `felhom-controller/REPORT.md`. Two real capture failures on guest 9201 (`mkdir …/backups: permission denied`) → both accepted and stored by the hub, `operator | recovery_unit_capture_failed | sent`, and the positive observable **`customer | recovery_unit_capture_failed | skipped | operator_only`** read from the hub's `notification_log`. One event per app, loop continuing | **Before this the failure was a `[WARN]` line and nothing else** — the manager carried three notify seams and none for the unit capture, so `/backups/apps`, the page you open to ask whether ONE app is backed up, was the one page that never said. **Deliberately NOT `backup_failed`:** that type is customer-enabled by default and carries Hungarian copy, so reusing it — which R-158's own proposal said — would email the customer about a failure they cannot act on. **D-c routes it to the operator and overrides the proposal.** Operator-only is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a `customerMessages` entry (the v0.78.0 defect); a red-proof removing the register entry shows the customer receiving it. **ROW REWRITTEN 2026-08-03 (R-182) — the 2026-08-02 claim was TRUE OF ONE APP AND FALSE OF THE REST, and it is worth saying which.** The signal existed and worked; what it did not do was scale past the first failing app. Measured: nine per-app events reached the hub in one day and **two** operator mails went out, because the cooldown key is `customerID:eventType(+tier)` and this type carries `app` but no `tier` — so the first refused app took the hour and the rest were dropped **before `LogNotification`**, leaving no row on any channel. The old row said "One event per app, loop continuing", which was true of what the CONTROLLER emitted and not of what the operator received. **Now:** the per-app event is the RECORD (hub `recordOnlyEvents`: stored + logged every time, never mailed) and `backup_run_failures` is the NOTIFICATION — one mail per run listing every failed app, its leg and its reason, with the counts and free space. A suppressed operator event of ANY type now leaves a `suppressed` row naming its key. **Proven live on demo-hp 2026-08-03** by a real 64.6 GiB fill (thin pool held 30.78 → 30.78): `notification_log` shows `recovery_unit_capture_failed | operator | recorded` ×2, `backup_run_failures | operator | sent` ×1 naming BOTH apps, and `backup_run_failures | customer | skipped | operator_only`. A second run in the same hour produced a second digest; after freeing space a run completed with 2 volume dumps and **no** digest. The suppression row proved itself on an unplanned event — `disk_critical | suppressed | key=demo-hp:disk_critical` — a collapse that yesterday would have left nothing at all. **The digest's silence is safe only because** the hub's deadline check raises `expected_backup_missed` from report freshness independently of any mail (`monitor/deadline.go:396,417`); that check is load-bearing for this row **⚠ CHECKED 2026-08-08 (R-258) AND THIS ROW WAS NOT CLAIMING SOMETHING UNTRUE — but the adjacent thing WAS false and the row's own narrative invites the wrong reading.** This row is about the OPERATOR notification path (`notification_log`, the per-run digest) and it claims nothing about what `/backups/apps` DRAWS. Independently of it, the page's per-app tier-1 tick was wrong until controller v0.210.0: it went green on the mere presence of a restore point and turned red only when the box's most recent DB dump — **whichever app it belonged to** — had failed. So an app whose own backup failed could show a tick while this row's operator mail correctly reported the failure; the two halves disagreed, and only the operator half was true. The tick now reads THIS app's own dump result and shows **no icon** when there is no result for it. Reading this row as evidence that the customer's page answers "is ONE app backed up" would have been wrong for five days. |
|
||||
| **A local backup is bounded by the box's FREE SPACE, not by a partition set at build time** — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime | golden `build-golden.sh` **v3.0.0**, agent **v0.120.0**, controller **v0.193.1** (R-165 / D-a / B2, completed by R-181) | **PROVEN-LIVE (2026-08-03) — BOTH halves** | `REPORT.md` (R-178 reinstalls) + `audits/SPIKE-r165-phase0-2026-08-03.md` (P1/P2/P3) + the bake transcript. **The golden bake is real evidence and is cited as such:** `build-golden.sh v3.0.0` produced `including mount point mp0 ('/var/lib/felhom')` with **no `mp1` line at all**, and its own guards printed `/var/lib/docker is a real mount`, `/mnt/sys_drive is a real mount` and `both paths are ONE filesystem`. Archive published (registry HTTP 200, sha `54e2a4c4…`). The B2 floor is unit-proven with 3 red-proofs and live on 9201 | **The row's FIRST clause is now PROVEN-LIVE; its SECOND is not, and they are separated deliberately.** **Proven (R-178, 2026-08-03):** *"a local backup is bounded by the box's FREE SPACE, not by a partition set at build time"* — both demo boxes reinstalled from this golden, by two different supply paths (demo-hp `--golden <local volid>`; demo-felhom the normal manifest route with **`verified sha256 54e2a4c431daf580… matches the hub manifest`**), each showing `mp0` at `/var/lib/felhom` with **no `mp1`**, both consumer paths real mounts on ONE filesystem (`stat -c %d` = `64519` on all three), 3/3 reboots each, and claim → deploy → backup → **restore** with a planted marker returning byte-identical. Space available to a recovery unit measured at **65 GiB / 233 GiB**, against the **19 GiB / 45 GiB** those boxes' `mp1` slices offered. **NOT proven — and measured FALSE in part:** *"a capture that would exhaust it is refused per app rather than allowed to stop the container runtime"*. The floor fired live for the first time (demo-hp 06:40:03) and does refuse per app, delete nothing, and alert — **but it is checked only in `captureAllRecoveryUnits`, while `runVolumeDumps` writes the bulk with no floor check at all**, so the leg that exhausts the volume is the unguarded one; and the refusal's claim that the previous unit is untouched was measured false (a 182,272 B dump replaced by 2,147,666,432 B under a manifest still dated 06:34:26). → **R-181, CLOSED THE SAME DAY (controller v0.193.0 + v0.193.1) and the second half is now PROVEN-LIVE TOO.** The reserve became a **per-app, per-run ADMISSION decision** taken before the app's FIRST write and covering all three legs (DB dump, volume dump, capture) — they write under one per-app root, which is what lets one verdict cover them honestly — and it gained a **size term**, so an app is no longer admitted at 96% and then allowed to write 2 GB. **Re-proven by filling demo-hp deliberately, once for EACH term, using the method that found the defect.** *Headroom @ 08:59:46* (906 MB free / 99%): both apps refused, **the whole `backups/primary` tree byte-identical — `TREE_SHA` 111d1760c18d3440f700634ab325f8b8 before and after**, opengist's tar still at its original 182,272 B; **no `Stopping <app> for safe volume dump` line at all**, which is the positive-by-absence observable that matters because that line IS present in the 08:58 baseline run; 0 volume dumps; one alert per app, HTTP 200. Space freed, re-run @ 09:01:33 → both captured normally. *Size @ 09:03:00*, reproducing the original sequence with a real 2 GiB file in opengist's volume (previous tar **2,147,666,432 B**, the exact figure the defect was measured at) and the filesystem at **91% used / 2.9 GB free — both headroom terms deliberately clear**: opengist refused `(size)` while **privatebin was ADMITTED and dumped normally**, proving the term is per-app rather than a global halt. **The refusal's wording was NOT weakened to fit** — the behaviour moved so the wording became true, and it is verified by tree fingerprint rather than by reading the log line, which is what lied. The `fallocate` instrument was re-proven on the rebuilt box before use (5 GiB step moved guest `df` while thin-pool `data_percent` held **36.83 → 36.83**), and teardown returned the pool to **29.43%**, below its own baseline. The golden **is now VOUCHED** (2026-08-03, hub `Artifact manifest set: … golden=0.192.0`), so fresh installs pick up the merged layout. Every box in the field that has not been reinstalled is still on the SPLIT layout and is unaffected: nothing assumes the merged shape at runtime, the controller's system_data_path is a path rather than a volume, and agent v0.120.0 FOLDS the retired `-sysdata-grow` into the single grow so an older `felhom-host-install.sh` still provisions the same total capacity |
|
||||
| Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | |
|
||||
| **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3. **2026-07-19:** the C6 evidence attempt ran and found a **product gap instead of evidence** — `audits/DIAG-immich-restore-2026-07-19.md`. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING **by finding, not by absence of attempt** — the rehearsal system working, not failing. **2026-07-19: the blocking product gaps are CLOSED in controller v0.148.0** (R-43 + R-44 shipped), so this row is now blocked only on the evidence run itself, not on missing capability. It flips the moment the §9 acceptance produces screenshots + the outcome flash + a snapshot ID. **2026-07-19 round 2 — PARTIAL EVIDENCE ONLY, row NOT flipped** (`audits/DIAG-immich-restore-round2-2026-07-19.md`): a deliberate run from snapshot `49e7cb46` did recover all 11 assets (`status=active`, files resolve), but the operation **reported failure** and left immich reporting schema drift, because the replay aborted against the running app (H4). Photos back ≠ clean acceptance. **2026-07-20: H4 closed in controller v0.153.0 (R-47) on BOTH paths, AND THE EVIDENCE RUN HAPPENED.** *(The "closing in v0.149" wording above was wrong — v0.149.0 was the F3 dashboard fix; R-47 shipped in v0.153.0.)* The C6 drill ran end-to-end **through the UI**: photos deleted, **trash emptied**, the full files+database restore pressed on `/backups/restore`, 40 files placed + 1 DB dump replayed rc-0, 11 assets back, no drift, timeline visually confirmed. The method note below is now DEMONSTRATED, not merely written down. Evidence: `felhom-controller/REPORT.md` 4e. **Residual: the run was performed by the OPERATOR, not by a customer** — for this row literal wording the alpha still owes one genuinely customer-driven pass, but no product gap blocks it. Method note for R-3's script: deleting in an app's own UI usually means *trash*, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash **and** verify the app's *content*, not the file count **Lane split → `07-backup-architecture.md` §3**: this row is Lane 1 (customer, unassisted). §8 rows 1–5 are the routes it would exercise |
|
||||
|
||||
@@ -118,7 +118,7 @@ and `journal-phase24.md` (Phases 2/4). Campaign document:
|
||||
| **R-215** | **`GET /recovery` rendered the recovery story on a box that never had off-site backups.** The predicate was right and the page never asked it; the POST sibling and the backups-area template both gated the same sentence correctly | **SHIPPED** (controller v0.201.0) |
|
||||
| **R-214** | **The physical console never stops asking to be paired.** Half an hour after `Day-0 provision SUCCESS`, with the host ONLINE, the console still showed the pairing banner and a stale code — on a screen whose own text promises *„Ez a képernyő magától frissül"*. Census: exactly two `/dev/console` writers in the whole day-0 path, both in the pairing loop; `felhom-host-install.sh` writes to the console not at all | **OPEN — NOT FIXED** |
|
||||
| **R-220** | **After a rebuild the customer's drives cannot be re-enrolled, and the refusal names an impossible action.** The deploy refuses (*„Válasszon a listából csatlakoztatott meghajtót"*) and the list is empty: `claim.go:84` treats a device mounted outside `/mnt/felhom-drives` as claimed, and the raw `/mnt/<name>` mount that enrolment itself creates survives the guest rebuild while the controller's registry does not. **Red-proved**: unmounting only the raw mounts flipped `attach: []` → both drives. **This is the state Campaign 10 reached by hand and recorded as its own harness error; the product's rebuild path now arrives there.** Breaches I3 | **CLOSED 2026-08-06 — shipped in agent v0.127.0 and PROVEN LIVE on a genuinely rebuilt box.** The fix is **corroborated, not a widened prefix**: a mountpoint outside `/mnt/felhom-drives` is forgiven only when the SAME device is also mounted under the managed path — a pairing only Felhom's own enrolment produces, so a disk another system is using at `/srv/data` or even `/mnt/someone-elses-disk` is still refused (own test + red-proof). Read from `/proc/mounts` deliberately: the lsblk invocation is pinned verbatim in the sudoers file, so switching to plural `MOUNTPOINTS` would have shipped a sudoers change with the binary. Fail-safe: an unreadable mount table corroborates nothing. **Measured on the Part 4 venue after a real guest purge, with both raw mounts still present on the surviving host:** `/disks/candidates` returned both drives in `attach` and `initialize` (before the fix: two empty lists), and both **re-attached through the customer endpoint** (`registered: true`). The customer-facing refusal was corrected in controller v0.203.0. |
|
||||
| **R-221** | **A rebuilt box cannot run the escrow ceremony at all.** The preflight refuses on `escrow.pbs_storage_id`, which the pbsdr bridge seeds into `agent.json` only via `finishConverged`. The convergence marker lives on the HOST and survives a guest rebuild; `agent.json` is rewritten by the installer. Unchanged descriptor → same hash → early return → the seed never runs into a config that no longer has it. **Red-proved**: moving only the marker aside seeded it instantly (`grep -c escrow`: 0 → 1). A real blocker for re-escrow, which is exactly what a rebuilt box must do | **OPEN — NOT FIXED** |
|
||||
| **R-221** | **A rebuilt box cannot run the escrow ceremony at all.** The preflight refuses on `escrow.pbs_storage_id`, which the pbsdr bridge seeds into `agent.json` only via `finishConverged`. The convergence marker lives on the HOST and survives a guest rebuild; `agent.json` is rewritten by the installer. Unchanged descriptor → same hash → early return → the seed never runs into a config that no longer has it. **Red-proved**: moving only the marker aside seeded it instantly (`grep -c escrow`: 0 → 1). A real blocker for re-escrow, which is exactly what a rebuilt box must do | **CLOSED 2026-08-08 — agent v0.128.0.** `Apply` re-asserts the seed BEFORE the idempotent early return; the return itself is kept and pinned by a zero-Proxmox-calls assertion. **The writer was established at `file:line` rather than assumed** — see the follow-through section below |
|
||||
| **R-223** | **The Day-0 manifest vouched agent 0.120.0 while the recovery feature needs 0.125.0 — and a reinstall DOWNGRADES a box that was fixed by hand.** Verbatim from the second reinstall: `agent (existing): felhom-agent 0.125.0` → `manifest: agent v0.120.0` → `installed /usr/local/bin/felhom-agent (felhom-agent 0.120.0)`. So every rebuild re-broke the recovery path — the one event that makes the feature necessary. **⚠ AND IT WAS NOT A DROPDOWN.** The first vouch attempt was REFUSED by R-120's gate (`configs.go:1162`): *"golden 0.192.0 is older than the newest controller the fleet reports (0.201.0)"*. **The artifacts form saves as a unit, so the agent could not be vouched while the golden was stale — and the golden had been stale since controller 0.193.0, meaning the Day-0 manifest had been effectively UNVOUCHABLE for days and nobody had cause to notice.** The real remedy was a golden rebake | **CLOSED 2026-08-05.** Golden **0.201.0** baked in the drill VM (658 165 766 B, sha `e730d7cab343eb35…f007654`, **round-trip verified from Gitea**), then manifest set in one save: `agent=0.125.0 golden=0.201.0 min_agent=0.125.0`. A fresh install now lands on current agent AND current controller |
|
||||
|
||||
### Phase 2 — the injected faults, 2026-08-05/06 (unattended)
|
||||
@@ -382,8 +382,8 @@ as "closed individually". **It is not — it is R-247, `READY`.** The live repo
|
||||
|---|---|---|
|
||||
| **R-256** | **C2 — „A mentéskezelő nem elérhető." names no route at all.** `web/offbox_handlers.go:47` and `:181` flash this to the customer on the off-site backup surface. It states an internal component's unavailability in the operator's vocabulary („mentéskezelő" = the backup Manager object), gives no reason the customer can act on, and names no next step — not "try again in a few minutes", not "contact support", not a page to go to. **Contrast, in the same subsystem and shipped the same week:** R-252's fix reads *„Meghajtók", „Meglévő meghajtó csatolása". Utána gyere vissza ide.* — a route. **Severity is low and stated so it is not over-ranked:** the condition is a nil backup manager, which on a healthy box does not occur; this is about the copy, not a broken path. Found by the C2 sample (19 refusals on the recovery/restore/offbox surface; **~202 of the repo's 221 refusal strings were NOT examined**) | **READY** — owner Viktor |
|
||||
| **R-257** | **C2 — „Az offsite tároló nincs elárvult állapotban." puts an English loanword and an internal state name in front of a Hungarian household customer, and names no route.** `web/offbox_handlers.go:270` (the Go error it mirrors is `backup/offbox.go:343`). „Offsite" is untranslated; „elárvult állapot" is the codebase's own `OffboxOrphaned()` predicate surfacing verbatim. A customer who pressed a button and got this cannot tell whether something failed, whether they did something wrong, or what to do instead. **This is a refusal that is CORRECT and fail-closed and still a dead end** — the same shape R-241 recorded for `--recover-offsite-install`. **Fix shape, not a decision:** say what the customer tried to do, why it does not apply right now, and where to look — or, since this is a state they cannot reach deliberately, do not offer the action at all | **READY** — owner Viktor |
|
||||
| **R-258** | **C3 — the customer's per-app backup tick is green on the PRESENCE of a restore point, and its only red condition is a GLOBAL one.** `web/handlers.go:1240-1248`: `row.Tier1LastStatus = "ok"` whenever `ListRestorePoints(app)` returns ≥1 point, and `"error"` only when `status.LastDBDump != nil && !status.LastDBDump.Success` — where `LastDBDump` is the box's **single most-recent DB dump** (`backup/backup.go:1064`, `m.lastDBDump`), **not this app's**. Three consequences, in increasing order of how wrong they look to a customer: (a) an app whose own last backup failed shows a **green tick** provided any older restore point exists and some *other* app dumped successfully afterwards; (b) an app with **no database at all** takes the `nil` branch and is green on presence alone; (c) the tick asserts nothing about **recency** — a restore point from three weeks ago is as green as one from last night, while `row.Tier1LastRun` beside it carries the real (old) time. **It is customer-visible:** `templates/backups_apps.html:173-174` renders it as a check or a cross. **This is the project's own "presence is not success" rule as a UI badge** — the artifact's existence is being read as a successful result, which is what the workspace `CLAUDE.md` section of that name forbids. **Found by a COMPLETE sweep, not a sample:** all 9 success-status assignment sites in the controller were read; this is the only new one. **NOT reproduced live** — source reading only | **READY** — owner Viktor |
|
||||
| **R-259** | **C4 — a disk read that FAILS renders as „0.0 GB / 0.0 GB (0%)" in the nominal colour, on the dashboard's most-looked-at meter.** `system/info_linux.go:259-264`: `readDiskUsage` logs at DEBUG and **returns**, leaving the caller's `TotalGB`/`UsedGB`/`AvailGB`/`Percent` at their zero values. `dashboard.html:57-63` then renders `{{fmtGB .SystemInfo.DiskUsedGB}} / {{fmtGB .SystemInfo.DiskTotalGB}} ({{printf "%.0f" .SystemInfo.DiskPercent}}%)` plus a meter whose fill is `width:0%`, and `usageColor(0)` returns **`"nominal"`** (`web/funcmap.go:189-196`) — so **a failed measurement is drawn as a healthy, empty disk.** There is **no `*Known` companion for the system disk** (grep: 0). **The fix pattern already exists in this codebase, two files away, with the reasoning written out:** R-225's `StatsKnown` guards every derived off-site figure and its own comment says *"a 0%-wide bar over an unread store is a picture of emptiness, and a picture is a claim"* (`backups_remote.html:60-62`). The dashboard does not use it. **It also travels:** `report/builder.go:94` puts the same zeroed `TotalGB`/`UsedGB` into the host report, so a failed statfs reaches the hub as a 0-of-0 disk. **Observation attached, not filed separately:** `readDiskUsage` is one of several collectors in that file that return silently on error; only this one was traced to a customer-visible surface. **NOT reproduced live** | **READY** — owner Viktor |
|
||||
| **R-258** | **C3 — the customer's per-app backup tick is green on the PRESENCE of a restore point, and its only red condition is a GLOBAL one.** `web/handlers.go:1240-1248`: `row.Tier1LastStatus = "ok"` whenever `ListRestorePoints(app)` returns ≥1 point, and `"error"` only when `status.LastDBDump != nil && !status.LastDBDump.Success` — where `LastDBDump` is the box's **single most-recent DB dump** (`backup/backup.go:1064`, `m.lastDBDump`), **not this app's**. Three consequences, in increasing order of how wrong they look to a customer: (a) an app whose own last backup failed shows a **green tick** provided any older restore point exists and some *other* app dumped successfully afterwards; (b) an app with **no database at all** takes the `nil` branch and is green on presence alone; (c) the tick asserts nothing about **recency** — a restore point from three weeks ago is as green as one from last night, while `row.Tier1LastRun` beside it carries the real (old) time. **It is customer-visible:** `templates/backups_apps.html:173-174` renders it as a check or a cross. **This is the project's own "presence is not success" rule as a UI badge** — the artifact's existence is being read as a successful result, which is what the workspace `CLAUDE.md` section of that name forbids. **Found by a COMPLETE sweep, not a sample:** all 9 success-status assignment sites in the controller were read; this is the only new one. **NOT reproduced live** — source reading only | **CLOSED 2026-08-08 — controller v0.210.0.** `appDumpVerdict` reads THIS app's own dump result; three states, no icon when nothing is known. **Recency deliberately not added** — see the observation in the follow-through section |
|
||||
| **R-259** | **C4 — a disk read that FAILS renders as „0.0 GB / 0.0 GB (0%)" in the nominal colour, on the dashboard's most-looked-at meter.** `system/info_linux.go:259-264`: `readDiskUsage` logs at DEBUG and **returns**, leaving the caller's `TotalGB`/`UsedGB`/`AvailGB`/`Percent` at their zero values. `dashboard.html:57-63` then renders `{{fmtGB .SystemInfo.DiskUsedGB}} / {{fmtGB .SystemInfo.DiskTotalGB}} ({{printf "%.0f" .SystemInfo.DiskPercent}}%)` plus a meter whose fill is `width:0%`, and `usageColor(0)` returns **`"nominal"`** (`web/funcmap.go:189-196`) — so **a failed measurement is drawn as a healthy, empty disk.** There is **no `*Known` companion for the system disk** (grep: 0). **The fix pattern already exists in this codebase, two files away, with the reasoning written out:** R-225's `StatsKnown` guards every derived off-site figure and its own comment says *"a 0%-wide bar over an unread store is a picture of emptiness, and a picture is a claim"* (`backups_remote.html:60-62`). The dashboard does not use it. **It also travels:** `report/builder.go:94` puts the same zeroed `TotalGB`/`UsedGB` into the host report, so a failed statfs reaches the hub as a 0-of-0 disk. **Observation attached, not filed separately:** `readDiskUsage` is one of several collectors in that file that return silently on error; only this one was traced to a customer-visible surface. **NOT reproduced live** | **CLOSED 2026-08-08 — controller v0.210.0.** `readDiskUsage` reports success; `SystemInfo.DiskKnown`/`HDDKnown`; the template draws no figure, no percentage and no meter fill when unknown. **The hub leg is deliberately NOT fixed and is now R-266** |
|
||||
| **R-260** | **C5 — the agent reports at least eight decision-bearing facts the hub models NOWHERE, and the sharpest one blinds the check that answers „can the operator get into this box".** Measured 2026-08-08 by a tag-reachability test over all four wire directions (465 emitted json tags; a tag whose literal string occurs nowhere in the receiving repo cannot be decoded by any struct, named or anonymous). **THE ONE THAT MATTERS: `operator_key_configured`.** The agent emits it every heartbeat (`hub/report.go:178` — *"operator authorized_key installed"*); the hub's OOB decoder (`store/host_oob.go:38-45`) mirrors **5 of the agent's 8** OOB fields and has no field for it, nor for `wg_handshake_age_s` or `healed_at`. So `oobDegraded` (`monitor/host_oob.go:60-70`) tests config-invalid and (peer-configured AND not-active-or-not-reachable) — and **a box with felhom-sshd active, reachable, valid config, peer configured and NO OPERATOR KEY INSTALLED is `ok`.** The agent knows and says so; the hub throws it away. The checker's own doc comment claims it *"answers 'can the operator get into this box right now, and if not, why' proactively"* — see R-262's class. **Break-glass is the recovery chain (G1), which is what makes this the top row of the census.** **The rest of the census, hub-side absent (0 occurrences anywhere in `felhom.eu/hub/`):** `guest_net` — **the entire per-guest network-health object**, populated every cycle at `hub/collect.go:290`, carrying `dhclient_alive`, `has_route`, `heal_succeeded`, `heals_last_hour`, `last_heal_at`, `damped`; `selfupdate_pending` + `selfupdate_pending_version` — a staged-but-unapplied agent update is invisible to the fleet view; `healed_recently` (mgmt-plane); `operator_key_configured`, `wg_handshake_age_s`, `healed_at` (OOB); `mount_parity` + `mount_inventory` (restore-test, see R-262); `cpu_temp_c`, `loadavg`, `memory_total_bytes`, `memory_used_bytes`, `uptime_seconds` (host metrics); `model_name` (SMART); `applied_at` (PBS-DR). **Controller→hub, hub-side absent:** `reporting_disabled`, `config_hash`, `last_db_dump`, `last_integrity_check`, `migrated_to`, `catalog_ref`, `storage_bindings`, `container_path`, `subpath`, `temperature_celsius`, `load_avg_{1,5,15}`, `memory_{total,used}_mb`, `uptime_seconds`. **⚠ STATED PRECISELY, because it first looked worse than it is:** the hub stores the whole report as `report_json`, so nothing is lost from the DATABASE — every hub consumer, without exception, re-unmarshals it into a typed struct, so nothing reaches a check, an alarm, a notification or a screen. And a **restore-test mount-parity mismatch is NOT hidden**: it sets `res.Err` and returns before `res.Pass = true`, so `pass:false` travels; what is lost is the *depth* of a pass, not the failure. **The class is what this row is about, not any single field** — this is `escrow_stale` (R-247) for the fourth, fifth and sixth time, and §4 of the campaign report argues it is the cheapest high-value gate available to this project. **Blind spot of the method, recorded:** 29 generically-named tags (`name`, `state`, `status`, …) were excluded by name, so a drop of a generically-named field would be missed | **CLOSED 2026-08-08** — the class is GATED (G-1, `scripts/wire_contract_gate.py`) and the sharpest instance is fixed (hub v0.99.0). The remaining unconsumed facts are **R-264, OPEN** — allowlisted with reasons, which is not the same as decided. See the follow-through section below |
|
||||
| **R-261** | **C6 — `CountSelfBindTokens` exists so that callers can assert an invariant, and no production caller asserts it.** `hub/internal/store/selfbind.go:106-111`. Its doc comment: *"it exists so callers can assert the 'after this runs, the only live link is one we just issued — or none' invariant that the auto-mint at customer-create / RESET-completion depends on."* **Census: only its own declaration in production; the two callers are `selfbind_automint_test.go:29` and `customer_delete_test.go:510`.** Tests are not callers (the campaign's rule), so the invariant the auto-mint *depends on* is checked in the test suite and never at the moment it matters. **This is the smallest of the eight rows and is filed at its true size, because the rest of the C6 sweep found INERT dead accessors rather than defects:** `OffboxOrphanedRenamedTo` and `OffboxEscrowState` have no caller but their data reaches the card another way (the template reads the settings field directly, `backups_remote.html:80`) — **R-228 is genuinely closed, and the sweep's first reading that it had regressed was wrong.** **The more consequential C6 result is a method result and is in the report, not here:** `golang.org/x/tools/cmd/deadcode` re-finds **neither** known instance, and a planted probe measured why — it reports an unreachable exported FUNCTION and not an unreachable exported METHOD on a widely-used type, and both known instances are methods | **READY** — owner Viktor |
|
||||
| **R-262** | **C7 — a comment claims a cross-repo contract is mirrored „field-for-field" and „the key-set tests guard drift"; it is two fields short, AND THE FIXTURE THE TEST READS OMITS THE SAME TWO FIELDS.** `hub/internal/api/handler.go:682-687` covers `hostBackup` **and** `hostRestoreTest`. **It is TRUE of `hostBackup`** (verified field-for-field against `agent/internal/hub/Backup`). **It is FALSE of `hostRestoreTest`:** the agent emits `mount_parity` and `mount_inventory` (`hub/report.go:432-433`, populated in production from `reconcile/restoretest.go:277-283` via `backup/runner.go:517`), and the hub has no field for either — **0 occurrences in the entire hub repo** outside the CHANGELOG. **The guard is blind in exactly the place the drift is:** `TestHostReport_GoldenContract` reads `testdata/host-report.golden.json`, the two copies of which are byte-identical as required — and **neither contains `mount_parity` or `mount_inventory` at all**, so the key sets agree on a shape that is not the shape the agent sends. A test that cannot fail on the drift it names is the R-97b lesson (*prove the consequence, not the mechanism*) landing on a contract test. **Consequence, stated precisely:** the verdict is not lost (a parity mismatch fails the test before `Pass` is set), but the hub cannot distinguish a full-fidelity restore-test pass from a boot-only one, for any agent, ever. **Fix shape, not a decision:** add the two fields and put them in the fixture — or narrow the comment to name `hostBackup` only and say plainly that `hostRestoreTest` is a subset. **Attached observation:** the same fixture carries `cpu_temp_c` and `loadavg`, which no hub struct decodes — a fixture carrying keys the receiver cannot read is the same shape one level down | **READY** — owner Viktor |
|
||||
@@ -441,13 +441,66 @@ builds the receiving struct by hand cannot see a field that never decodes, which
|
||||
|---|---|---|
|
||||
| **R-264** | **Twenty-one facts the boxes report that the hub can now decode nowhere, each allowlisted with a reason rather than silently skipped — and for these the reason is "no consumer today, and one is arguably owed".** Split out of R-260 on 2026-08-08 so that closing the CLASS (gated) and fixing its sharpest instance (`operator_key_configured`) could not be mistaken for having decided what the hub should do with the rest. **The list, grouped by what a consumer would be for.** **(a) Guest-network health — `guest_net` and its seven children** (`checked_at`, `has_route`, `dhclient_alive`, `heal_succeeded`, `heals_last_hour`, `last_heal_at`, `damped`). The R-54 watchdog reports per-guest network state and self-heal counts every cycle and the hub — the component that emails the operator — models none of it. There is a live incident in this project's own record where a killed `dhclient` took a tunnel down for 1 h 15 m (`audits/INCIDENT-guest-dhclient-killed-2026-07-20.md`); a recurring-heal signal is exactly what would have surfaced it. **This is the strongest candidate of the twenty-one.** **(b) `selfupdate_pending` + `selfupdate_pending_version`** — an agent that has flipped its binary and never committed reports pending on every heartbeat so that "the operator sees WHY the version isn't advancing", and no operator can see it. **(c) `mgmt_plane.healed_recently`** — bounded: the hub DOES alarm on the `privsep_healed_at` timestamp beside it, so the recurring-clobber signal is not lost, only this flag. **(d) `restore_tests.mount_parity` + `mount_inventory`** — R-262's subject; the verdict is not lost (a mismatch fails before `Pass` is set) but the hub cannot tell a full-fidelity pass from a boot-only one. **(e) `pbs_dr.applied_at`.** **(f) Controller-side: `config_hash`, `reporting_disabled`, `stacks`, `storage.migrated_to`, `backup.last_db_dump`, `backup.last_integrity_check`** — the last two are backup-integrity timestamps, which is the "presence is not success" neighbourhood. **For each the question is the same and is NOT answered here: is it wanted? If the hub should act on it, model it and name what consults it. If it should not, the honest end is that the emitter stops sending it** — a fact emitted forever and consumed nowhere is a future false green waiting for someone to write a check against it. **Deliberately not decided in the G-1 session**, whose scope was the gate plus the operator-access instance; unilaterally removing emitters would also break the byte-identical cross-repo host-report golden and is a coordinated two-repo change | **READY** — owner Viktor |
|
||||
|
||||
| **R-265** | **A CI run can fail with NO LOG PERSISTED, and the alarm mail then points the operator at a log that does not exist.** Observed 2026-08-08 as run **264** (`650cc8a`, a **documentation-only** commit) sat between two green runs of identical gate code. **Measured rather than assumed — the shape is unmistakable:** every other run in the session took **18–34 s and has a log (HTTP 200)**; 264 took **834 s** (07:12:40 → 07:26:34 UTC) and `GET /actions/jobs/264/logs` returns **HTTP 500 — `actions_log/…/264.log.zst: file does not exist`**. The runner pod never restarted (`act-runner`, 0 restarts, 5 d 17 h uptime), so the runner did not die; the JOB hung and was reaped. **It is NOT a gate finding, and four independent facts say so:** the diff from the green run before it is Markdown only; the same content is green two commits later (run 265, `dd55a3f`, 33 s); the gate code is byte-identical across 263/264/265; and 260–262, which WERE real gate failures, all failed in under 35 s **with** logs. **THE CAUSE OF THE HANG IS UNDETERMINED and is deliberately not guessed at.** DooPlex was doing heavy work in that window (a 139 MB `kubectl cp`, and a `go run` compiling the whole hub module for the live-validation harness), which is a plausible contention story — but the box has 40 cores and sat at load ~5, so it is **not established** and is recorded as a hypothesis, not a cause. **THE FINDING THAT MATTERS IS THE SECOND-ORDER ONE, and it is this workflow's own stated purpose turned against it.** `gates.yml` exists because "a detector nobody hears is the defect R-29 filed, rebuilt one layer up", and its alarm mail says *"The failing gate names itself in the run log."* **Here there is no run log**, so an operator following that sentence finds nothing and cannot tell an infrastructure reap from a real conviction. Worse and **unverified**: the alarm step is `if: failure()`, and whether it even ran for a reaped job is unknown — if it did not, this was a red CI that alarmed nobody, which is exactly the shape the workflow was built to prevent. **Fix shape, not a decision:** (a) make the alarm mail state the run's DURATION and whether a log exists, so a log-less reap is self-identifying; (b) give the job an explicit `timeout-minutes` well under the reap so it fails fast, loudly and with a log; (c) establish whether the alarm fires at all on a reaped job — that is one deliberate test, and until it is run, "CI alarms on failure" is an assumption | **READY** — owner Viktor |
|
||||
| **R-265** | **A CI run can fail with NO LOG PERSISTED, and the alarm mail then points the operator at a log that does not exist.** Observed 2026-08-08 as run **264** (`650cc8a`, a **documentation-only** commit) sat between two green runs of identical gate code. **Measured rather than assumed — the shape is unmistakable:** every other run in the session took **18–34 s and has a log (HTTP 200)**; 264 took **834 s** (07:12:40 → 07:26:34 UTC) and `GET /actions/jobs/264/logs` returns **HTTP 500 — `actions_log/…/264.log.zst: file does not exist`**. The runner pod never restarted (`act-runner`, 0 restarts, 5 d 17 h uptime), so the runner did not die; the JOB hung and was reaped. **It is NOT a gate finding, and four independent facts say so:** the diff from the green run before it is Markdown only; the same content is green two commits later (run 265, `dd55a3f`, 33 s); the gate code is byte-identical across 263/264/265; and 260–262, which WERE real gate failures, all failed in under 35 s **with** logs. **THE CAUSE OF THE HANG IS UNDETERMINED and is deliberately not guessed at.** DooPlex was doing heavy work in that window (a 139 MB `kubectl cp`, and a `go run` compiling the whole hub module for the live-validation harness), which is a plausible contention story — but the box has 40 cores and sat at load ~5, so it is **not established** and is recorded as a hypothesis, not a cause. **THE FINDING THAT MATTERS IS THE SECOND-ORDER ONE, and it is this workflow's own stated purpose turned against it.** `gates.yml` exists because "a detector nobody hears is the defect R-29 filed, rebuilt one layer up", and its alarm mail says *"The failing gate names itself in the run log."* **Here there is no run log**, so an operator following that sentence finds nothing and cannot tell an infrastructure reap from a real conviction. Worse and **unverified**: the alarm step is `if: failure()`, and whether it even ran for a reaped job is unknown — if it did not, this was a red CI that alarmed nobody, which is exactly the shape the workflow was built to prevent. **Fix shape, not a decision:** (a) make the alarm mail state the run's DURATION and whether a log exists, so a log-less reap is self-identifying; (b) give the job an explicit `timeout-minutes` well under the reap so it fails fast, loudly and with a log; (c) establish whether the alarm fires at all on a reaped job — that is one deliberate test, and until it is run, "CI alarms on failure" is an assumption | **CLOSED 2026-08-08 — `timeout-minutes: 5` on the gates job, and the alarm mail now states elapsed seconds and qualifies its own "names itself in the run log" sentence.** ⚠ **The unknown is NOT closed and must not be read as closed:** whether the `if: failure()` alarm fires for a REAPED job is still unverified. The timeout makes the reap unreachable in practice; it does not answer what happens inside one |
|
||||
|
||||
**Explicitly still open, untouched by this session:** R-246 (the wrong stale flag on `demo-hp` —
|
||||
clearing it is an operator act hub-side), R-255, R-256, R-257, R-258, R-259, R-261, R-262, R-263, and
|
||||
**C7's test-comment half**, which Campaign 12 recorded as *owed, not done* (60 of 2652 production
|
||||
invariant comments sampled; none of the 1440 test comments).
|
||||
|
||||
## The seed that never ran twice, and three pictures that were not true — 2026-08-08
|
||||
|
||||
Four defects of one family: something the box already knows, either thrown away or drawn as its
|
||||
opposite. Agent **v0.128.0**, controller **v0.210.0**, `gates.yml` (no hub change, no hub bump).
|
||||
|
||||
**R-221's writer, ESTABLISHED at `file:line` rather than assumed** — the prompt asked for this and it
|
||||
was owed. `step_agent_config` (`felhom.eu/scripts/felhom-host-install.sh:2396`) renders `agent.json`
|
||||
from `base = {}` unless an explicit `--preserve-from` is passed (flag `:1246`, defaulting empty at
|
||||
`:256`), and writes it with `O_TRUNC` (`:2579`). **The render never writes an `escrow` section at
|
||||
all** — grep over the whole heredoc returns zero hits. The pbsdr marker lives host-side
|
||||
(`<agent-state>/pbsdr/marker.json`) and survives. So a rebuild keeps the marker and takes the key:
|
||||
same descriptor, same hash, early return, seed never re-runs. **The attribution in R-221 was
|
||||
correct.** A rebuild is nonetheless only the case that was measured — the same hole opens for a
|
||||
hand-edited or restored config, which is the honest reason the fix is at the seam and not in the
|
||||
installer.
|
||||
|
||||
**The idempotent early return was KEPT**, and that is load-bearing: it stops a converged box
|
||||
re-running Proxmox operations every 60 s.
|
||||
`TestSeedReasserted_OnConvergedTick_WithZeroProxmoxCalls` asserts **zero** recorded runner calls on
|
||||
that tick, so a "fix" that simply deleted the return fails the test. Verified by mutation.
|
||||
|
||||
**The §7.3 truth table as implemented** (R-258):
|
||||
|
||||
| this app's own most recent dump result | restore point | verdict |
|
||||
|---|---|---|
|
||||
| any of its databases failed | yes | `error` — cross |
|
||||
| all clean | yes | `ok` — tick |
|
||||
| none recorded (no database / no run yet) | yes | **no icon**, time only, title „Erről a mentésről nincs eredményünk." |
|
||||
| any | no | no tier-1 row at all, unchanged |
|
||||
|
||||
**An existing test was asserting the defect and was corrected, not deleted.**
|
||||
`TestBuildAppBackupRows_Tier1FromRestorePoints` expected `"ok"` for a `FullBackupStatus` with **no
|
||||
`LastDBDump` at all** — a green tick derived from nothing but a file's existence, i.e. Scenario G.
|
||||
Its real subject, the `Tier1LastRun` time, is unchanged.
|
||||
|
||||
**The convention is now ruled** (§7.2, `CONTEXT.md` S-39): a `…Known bool` companion beside the
|
||||
figures. `ROADMAP.md` G-3 was blocked on that decision and is unblocked.
|
||||
|
||||
**Six red-proofs, every one demonstrated failing and restored, each with the mutation asserted
|
||||
applied.** The one that matters: Scenario A **fails against today's tree** with the intended message
|
||||
— so the test tests the defect.
|
||||
|
||||
| ID | What | State |
|
||||
|---|---|---|
|
||||
| **R-266** | **A failed root `statfs` still reaches the hub as a 0-of-0 disk, and the hub cannot tell that from an empty one.** Split out of R-259 on 2026-08-08 so that fixing the CUSTOMER-facing half could not be mistaken for fixing the wire. `report/builder.go:93-95` copies `sysInfo.DiskTotalGB` / `DiskUsedGB` / `DiskPercent` into `r.Storage[0]` (`Mount: "/"`), and those are exactly the zeros a failed `statfs` leaves behind — the controller now KNOWS the measurement failed (`SystemInfo.DiskKnown`, controller v0.210.0) and the report still does not carry it. **Deliberately not fixed here, for a reason that is now structural rather than a preference:** adding a field to that report is a change to a declared wire, which since G-1 means the receiving side must model it in the same session (`scripts/wire_contract_gate.py` refuses otherwise) — a two-repo change with a hub bump, and this session deliberately touched no hub code. **RANKED LOW, and the reason is that the consequence is bounded:** the hub bands host storage on `disk_percent`, so a failed read presents as 0% used — the *quiet* direction. It cannot raise a false "nearly full" alarm; it can only fail to raise a true one, and only while the root filesystem is unreadable, which is a state with louder symptoms of its own. **Fix shape when it is taken:** carry `disk_known` on the storage entry and have the hub's fill checker skip an unknown reading rather than band it — never treat absent as 0 | **READY** — owner Viktor |
|
||||
|
||||
**Explicitly still open, untouched by this session:** R-246, R-255, R-256, R-257, R-261, R-262,
|
||||
R-263, **R-264** (the twenty-one undecided facts — a design session of its own), R-240, R-243,
|
||||
R-202, R-213, R-244, R-214/R-235, and **C7's test-comment half**, which Campaign 12 recorded as
|
||||
*owed, not done*. **G-8's other half** (a hub-side check that notices a *vouch* has been forgotten)
|
||||
was deliberately not built: it is hub work whose payoff is a daily email, and this session already
|
||||
ends with a bake-and-vouch cycle in front of the operator.
|
||||
|
||||
## Why the TOP READY rows rank this way
|
||||
|
||||
|
||||
|
||||
@@ -139,7 +139,7 @@ by looking a fourth time.**
|
||||
|----|------|------|--------|-------|
|
||||
| ~~**G-1**~~ | ~~**Gate C5 — the cross-repo tag-reachability check.**~~ | S | **BUILT AND CLOSED 2026-08-08** — `scripts/wire_contract_gate.py`, `--fast`, registered in `repo_gates.py` | Shipped as ranked. **Built BEFORE the fixes and seen failing on 40 fields** (`documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md`) — the order was the method, because `deadcode` had been rejected for C6 the night before precisely for failing that test. 210 tags checked across 3 declared wires, 51 skipped (generic / opaque / allowlisted, each with a reason). Carries a `--selftest` that plants an unreachable tag on a real root and asserts conviction, and publishes its blind spots in both its docstring and its output. **Two instrument defects the control caught before it was trusted:** a substring false negative (`grep -F healed_at` matched `privsep_healed_at`), and treating `dr_recipe` as wholly opaque when its top-level section keys ARE decoded through an allow-list that already cost `offsite_restic` (R-122) — it is now opaque only BELOW depth 1. **Estimate held:** the size guess was right and the `--fast` judgement was right. **Not covered, and stated in the gate itself:** the hub's desired-state (served as raw stored JSON, no typed emitter) and the agent's local API (no single root). → R-260 CLOSED, R-247 CLOSED, leftover appetite R-264 |
|
||||
| **G-2** | **Gate C3 — a success verdict may not be set where an incompleteness signal is in scope.** Assert that every literal success-status assignment either has no gap/skip/missing signal available at that point, or consults it. | S | candidate | The whole population in the controller is **9 sites** — Campaign 12 read all of them, which is why this class is the one where "no others exist" is supportable. Small enough to gate by enumeration rather than by inference. **Known miss:** verdicts expressed as booleans, enum constants, or the absence of an error — and the controller does use those elsewhere. Instances: R-240 (open), R-258 (new). |
|
||||
| **G-3** | **Gate C4 — every rendered count/size/percentage needs a `*Known` companion.** | M | candidate — **needs a convention decision first** | R-225's `StatsKnown` is the house pattern and it is exemplary; a gate can only enforce it once it is *the* house style. Today three-state is also encoded with pointers and with separate error fields, both legitimate, and a name-based gate reads those as unguarded. **The decision owed is "how does this codebase say 'we could not look'", not "should we gate it".** Instance: R-259. |
|
||||
| **G-3** | **Gate C4 — every rendered count/size/percentage needs a `*Known` companion.** | M | candidate — **UNBLOCKED 2026-08-08: the convention decision it was waiting for has been made** | The decision owed was *"how does this codebase say 'we could not look'"*, and it is now ruled (`CONTEXT.md` **S-39**, shipped in controller v0.210.0 / R-259): **an explicit `…Known bool` companion beside the figures, checked in the template before anything is rendered** — the shape `Offbox.StatsKnown` already used, whose own comment carries the reasoning (*"a 0%-wide bar over an unread store is a picture of emptiness, and a picture is a claim"*). Pointers and separate error fields remain legitimate Go and both still exist here; the ruling is that **new** three-state figures use the companion, because a codebase with three dialects cannot be gated by a name-based check. **Existing call sites were deliberately NOT converted** — that conversion is the bulk of this item's M and is what remains before a gate can be turned on without a wall of false positives. **Next step is therefore a survey, not a gate:** count the rendered figures that lack a companion, decide which are genuinely three-state, convert those, then gate. Instances so far: R-225 (fixed, the pattern's origin), R-259 (fixed, the ruling). |
|
||||
| **G-4** | **Complete C1's runtime body assertion — 4 of 27 pages today.** | L | candidate — the expensive one, and honestly so | `secret_in_markup_gate.py` covers all 36 templates on the NAME-based check and is **blind to a secret under a neutral page-data key** — verified 2026-08-08 by replaying the three pre-fix templates through it: it convicts 2 of 3 and not the third. The runtime assertion catches all three; extending it means constructing each remaining page's data in a test, which is a per-page cost and is the real reason it has not been done. **Do NOT adopt the Go-side mirror Campaign 12 wrote as a gate on its own** — 27 candidates, 0 findings is bad signal-to-noise in front of every push. → R-255 |
|
||||
| **G-5** | **Gate C7, narrowly — uniqueness claims only.** A comment saying "X is the ONLY writer/place/caller of Y" is mechanically falsifiable; assert it. | S | candidate — narrow by construction | Covers ~60 of the 2652 production invariant comments. The other ~97% of the vocabulary (`never`, `always`, `must not`, `guarantees`) is not mechanical and a gate must not pretend otherwise. Instance: R-263. |
|
||||
| **G-6** | **C2 — NOT mechanically gateable.** | — | **recorded as a no** | "Names a route" is a judgement, not a predicate. The most a check could do is enforce a *convention* (e.g. every customer-visible refusal string ends in an imperative clause), which would be gamed rather than followed. Better served by the UI-copy review the `felhom-ui-design` skill already governs. Instances: R-256, R-257. |
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
# Golden 0.210.0 — baked, published, round-trip verified, **NOT VOUCHED** (2026-08-08)
|
||||
|
||||
Bumping the controller to **v0.210.0** (R-259 / R-258) made `golden_currency_gate.py` correctly red
|
||||
and it refused the `felhom.eu` push. The answer to that gate is the bake, never `--no-verify`.
|
||||
|
||||
**Vouched on arrival at this session: golden 0.209.0, agent 0.127.0** — read live from the hub, not
|
||||
from a document. The operator vouched 0.209.0 during the previous session, so that approval is
|
||||
closed and this one supersedes it.
|
||||
|
||||
## Where it ran
|
||||
|
||||
The **drill VM on DooPlex** (`/mnt/5_hdd/felhom.eu/drill/drill.qcow2`, snapshot `virgin`) — the
|
||||
accepted Tier-2 exception for bakes. Canonical §4.0 launch line, cold-booted, **restored to `virgin`
|
||||
afterwards** and the snapshot list re-read. Liveness via `ps -eo comm | grep -c qemu-system-x86`.
|
||||
|
||||
## The inputs
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| host | `drill-pve`, `pve-manager/9.2.2` |
|
||||
| template | **`debian-13-standard_13.6-1_amd64.tar.zst`** — listed with `pveam available` on the day; checksum verified on download |
|
||||
| build script | `felhom-agent/configs/build-golden.sh` v3.0.0, clean tree at `28ba859` |
|
||||
| controller baked | `gitea.dooplex.hu/admin/felhom-controller:0.210.0`, built and pushed from a clean tree at `c732fe1` |
|
||||
|
||||
## The token never crossed a shell
|
||||
|
||||
`scp` file → file into a `0600` file, read by a runner script **inside** the VM.
|
||||
`systemctl show golden-bake -p Environment -p ExecStart | grep -c -F "$(cat /root/.gitea-token)"` → **0**.
|
||||
|
||||
## The 404 pre-gate, with a control
|
||||
|
||||
```
|
||||
felhom-golden/0.209.0/golden.tar.zst → 200 ← the control: the URL SHAPE is right
|
||||
felhom-golden/0.210.0/golden.tar.zst → 404 ← the pre-gate: nothing to overwrite
|
||||
```
|
||||
|
||||
## Acceptance markers — grepped verbatim against this run's log
|
||||
|
||||
| marker | count |
|
||||
|---|---|
|
||||
| `docker OK (overlay2` | **1** |
|
||||
| `including mount point` (rootfs **and** mp0) | **2** |
|
||||
| `upload OK (HTTP 201)` | **1** |
|
||||
| `excluding` | **0** |
|
||||
| `FATAL` | **0** |
|
||||
| `mp1` | **0** |
|
||||
|
||||
`Result=success`, `ExecMainStatus=0`. Archive 626 MB.
|
||||
|
||||
## The publish, and the round trip — the published BYTES
|
||||
|
||||
```
|
||||
GOLDEN_VERSION=0.210.0
|
||||
GOLDEN_SHA256=b9f701fab813c051dd4be348a4e301f0e32518b811920406b7bdf8b3dd4c0a00
|
||||
```
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| size | **656 787 777 B** |
|
||||
| sha256 | **`b9f701fa…4c0a00`** — hashed independently on DooPlex; identical |
|
||||
| **`./etc/felhom-controller-image` read OUT of the downloaded archive** | **`gitea.dooplex.hu/admin/felhom-controller:0.210.0`** |
|
||||
|
||||
## ⚠ THE AGENT WAS NOT PUBLISHED UNTIL THIS SESSION CHECKED — and it mattered
|
||||
|
||||
R-221's fix is in the **agent**, and a fresh install takes its agent from the Day-0 manifest. The
|
||||
binary had been deployed by hand to `felhom-pve` and **never published**, so `agent_version 0.128.0`
|
||||
was not selectable and a fresh install would have received 0.127.0 — i.e. **the golden would have
|
||||
carried the controller fixes and not the one this session's headline defect needed.**
|
||||
|
||||
Caught by checking each Day-0 value was *fetchable* rather than assuming it. Published from the
|
||||
**live-deployed bytes**, sha-verified across the hop first:
|
||||
|
||||
```
|
||||
sha on felhom-pve : c6eba73bf9b9ad6980cfef57bfb3db31581abc9d643de2ff50d4254576fc1a59
|
||||
sha on DooPlex : c6eba73bf9b9ad6980cfef57bfb3db31581abc9d643de2ff50d4254576fc1a59 ← identical
|
||||
publish : upload OK (HTTP 201), round-trip GET verified (sha256 matches)
|
||||
```
|
||||
|
||||
## The vouch — the three values, each verified downloadable AND selectable
|
||||
|
||||
| field | now | **set to** | check that was run |
|
||||
|---|---|---|---|
|
||||
| `golden_version` | 0.209.0 | **0.210.0** | package `GET` → **HTTP 200**; hub dropdown offers it with `data-sha=b9f701fa…` matching the bake |
|
||||
| `agent_version` | 0.127.0 | **0.128.0** | package `GET` → **HTTP 200** (after the publish above); hub dropdown offers it with `data-sha=c6eba73b…` matching the deployed binary |
|
||||
| `min_agent` | 0.127.0 | **0.127.0** (unchanged) | read from the controller CHANGELOG header written this session: `## v0.210.0 — … — MinAgent 0.127.0` |
|
||||
|
||||
**Why `min_agent` stays 0.127.0 and is not raised to 0.128.0.** `MinAgent` declares what *this
|
||||
controller* requires, and controller v0.210.0's changes (R-259, R-258) need nothing from the agent —
|
||||
raising it would claim a coupling that does not exist. **R-221's fix is delivered by
|
||||
`agent_version 0.128.0`, not by the floor.** Setting `min_agent` above `agent_version` is the R-216
|
||||
shape, which hub v0.97.0 holds rather than serving past.
|
||||
|
||||
Hub → Configuration → Day-0 artifacts → set all three → Save. The R-120 gate sits on that save and
|
||||
refuses a golden older than the newest controller the fleet reports. **Reversible** — re-select the
|
||||
previous values and Save; no package is deleted by a bake.
|
||||
|
||||
## Teardown
|
||||
|
||||
`pct destroy 9100 --purge` · token, runner, build script and in-VM log `shred -u`'d, `/root` residue
|
||||
clean · `poweroff` · qemu confirmed gone (**0**) · `qemu-img snapshot -a virgin` restored.
|
||||
|
||||
**Token-leak grep on the log COMMITTED here, with the control that makes a `0` mean something:**
|
||||
|
||||
```
|
||||
grep -c -F "<token>" bake.log → 0
|
||||
grep -c -F "<token>" <copy with the token appended> → 1 ← the control; copy then shredded
|
||||
```
|
||||
|
||||
## What this does and does not change
|
||||
|
||||
**Does:** `golden_currency_gate.py` is green; the push it was blocking can proceed.
|
||||
|
||||
**Does NOT:** a fresh Day-0 install still lands on **0.209.0** and **agent 0.127.0** until the
|
||||
operator saves. The gate checks the BAKE, not the vouch — R-242's untouched half, unchanged.
|
||||
@@ -0,0 +1,326 @@
|
||||
[golden] build-golden.sh v3.0.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.210.0
|
||||
[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) …
|
||||
Logical volume "vm-9100-disk-0" created.
|
||||
Logical volume pve/vm-9100-disk-0 changed.
|
||||
Creating filesystem with 8388608 4k blocks and 2097152 inodes
|
||||
Filesystem UUID: 3ac5cfaf-ca37-496a-a63a-c6ee7ab6fff3
|
||||
Superblock backups stored on blocks:
|
||||
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
|
||||
4096000, 7962624
|
||||
Logical volume "vm-9100-disk-1" created.
|
||||
Logical volume pve/vm-9100-disk-1 changed.
|
||||
Creating filesystem with 6291456 4k blocks and 1572864 inodes
|
||||
Filesystem UUID: 6569b0a5-c9c1-42ce-a819-1d26622990d1
|
||||
Superblock backups stored on blocks:
|
||||
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
|
||||
extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst'
|
||||
Total bytes read: 553512960 (528MiB, 163MiB/s)
|
||||
Detected container architecture: amd64
|
||||
Creating SSH host key 'ssh_host_rsa_key' - this may take some time ...
|
||||
done: SHA256:UWJKUjrK3pjyawEPALf7P/gSRCi0H/oPhxZtAaV4W4Y root@felhom-golden
|
||||
Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ...
|
||||
done: SHA256:Q8WHUiqzfcahOTcAH1LNNGn2JH098e1xj36GrKxM1FM root@felhom-golden
|
||||
Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ...
|
||||
done: SHA256:gwx1LXjKEspMw7ijBOC8MRYjaKbJpxns1yXqPoK6JAE root@felhom-golden
|
||||
[golden] starting + installing Docker (official repo, trixie channel) …
|
||||
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
|
||||
perl: warning: Setting locale failed.
|
||||
perl: warning: Please check that your locale settings:
|
||||
LANGUAGE = (unset),
|
||||
LC_ALL = (unset),
|
||||
LC_CTYPE = (unset),
|
||||
LC_NUMERIC = (unset),
|
||||
LC_COLLATE = (unset),
|
||||
LC_TIME = (unset),
|
||||
LC_MESSAGES = (unset),
|
||||
LC_MONETARY = (unset),
|
||||
LC_ADDRESS = (unset),
|
||||
LC_IDENTIFICATION = (unset),
|
||||
LC_MEASUREMENT = (unset),
|
||||
LC_PAPER = (unset),
|
||||
LC_TELEPHONE = (unset),
|
||||
LC_NAME = (unset),
|
||||
LANG = "en_US.UTF-8"
|
||||
are supported and installed on your system.
|
||||
perl: warning: Falling back to the standard locale ("C").
|
||||
locale: Cannot set LC_CTYPE to default locale: No such file or directory
|
||||
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
|
||||
locale: Cannot set LC_ALL to default locale: No such file or directory
|
||||
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
|
||||
perl: warning: Setting locale failed.
|
||||
perl: warning: Please check that your locale settings:
|
||||
LANGUAGE = (unset),
|
||||
LC_ALL = (unset),
|
||||
LC_CTYPE = (unset),
|
||||
LC_NUMERIC = (unset),
|
||||
LC_COLLATE = (unset),
|
||||
LC_TIME = (unset),
|
||||
LC_MESSAGES = (unset),
|
||||
LC_MONETARY = (unset),
|
||||
LC_ADDRESS = (unset),
|
||||
LC_IDENTIFICATION = (unset),
|
||||
LC_MEASUREMENT = (unset),
|
||||
LC_PAPER = (unset),
|
||||
LC_TELEPHONE = (unset),
|
||||
LC_NAME = (unset),
|
||||
LANG = "en_US.UTF-8"
|
||||
are supported and installed on your system.
|
||||
perl: warning: Falling back to the standard locale ("C").
|
||||
locale: Cannot set LC_CTYPE to default locale: No such file or directory
|
||||
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
|
||||
locale: Cannot set LC_ALL to default locale: No such file or directory
|
||||
[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation …
|
||||
[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds …
|
||||
[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) …
|
||||
Unable to find image 'hello-world:latest' locally
|
||||
latest: Pulling from library/hello-world
|
||||
4f55086f7dd0: Pulling fs layer
|
||||
4f55086f7dd0: Verifying Checksum
|
||||
4f55086f7dd0: Download complete
|
||||
4f55086f7dd0: Pull complete
|
||||
Digest: sha256:7f4da0fc94bcece205a8c0b6f4d11c8196924654ffe5c4d1aa439b7f632048b2
|
||||
Status: Downloaded newer image for hello-world:latest
|
||||
docker OK (overlay2; data-root /var/lib/docker)
|
||||
/var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4
|
||||
/mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4
|
||||
both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576
|
||||
[golden] baking the in-guest controller image gitea.dooplex.hu/admin/felhom-controller:0.210.0 (no registry cred at deploy) …
|
||||
|
||||
WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
|
||||
Configure a credential helper to remove this warning. See
|
||||
https://docs.docker.com/go/credential-store/
|
||||
|
||||
0.210.0: Pulling from admin/felhom-controller
|
||||
039e6f9f9752: Pulling fs layer
|
||||
2a377a4774aa: Pulling fs layer
|
||||
0d2ce7fec68c: Pulling fs layer
|
||||
68803149072c: Pulling fs layer
|
||||
5000b1942063: Pulling fs layer
|
||||
e1e1d1e34154: Pulling fs layer
|
||||
68803149072c: Waiting
|
||||
5000b1942063: Waiting
|
||||
e1e1d1e34154: Waiting
|
||||
039e6f9f9752: Verifying Checksum
|
||||
039e6f9f9752: Download complete
|
||||
68803149072c: Verifying Checksum
|
||||
68803149072c: Download complete
|
||||
0d2ce7fec68c: Verifying Checksum
|
||||
0d2ce7fec68c: Download complete
|
||||
5000b1942063: Verifying Checksum
|
||||
5000b1942063: Download complete
|
||||
e1e1d1e34154: Verifying Checksum
|
||||
e1e1d1e34154: Download complete
|
||||
2a377a4774aa: Verifying Checksum
|
||||
2a377a4774aa: Download complete
|
||||
039e6f9f9752: Pull complete
|
||||
2a377a4774aa: Pull complete
|
||||
0d2ce7fec68c: Pull complete
|
||||
68803149072c: Pull complete
|
||||
5000b1942063: Pull complete
|
||||
e1e1d1e34154: Pull complete
|
||||
Digest: sha256:f04af2d67adca779e9eb5c8a179379dd17810848cf8dfadc450392a208850a38
|
||||
Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-controller:0.210.0
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.210.0
|
||||
[golden] asking the controller which infra images it manages …
|
||||
[golden] baking infra images (4): traefik:v3.6.7 cloudflare/cloudflared:2026.6.0 gtstef/filebrowser:1.3.3-stable gitea.dooplex.hu/admin/felhom-samba:1.1.0 …
|
||||
v3.6.7: Pulling from library/traefik
|
||||
589002ba0eae: Pulling fs layer
|
||||
ef63511ea6cc: Pulling fs layer
|
||||
0738e5cb835e: Pulling fs layer
|
||||
3e6813f70c64: Pulling fs layer
|
||||
3e6813f70c64: Waiting
|
||||
589002ba0eae: Verifying Checksum
|
||||
589002ba0eae: Download complete
|
||||
ef63511ea6cc: Verifying Checksum
|
||||
ef63511ea6cc: Download complete
|
||||
3e6813f70c64: Verifying Checksum
|
||||
3e6813f70c64: Download complete
|
||||
0738e5cb835e: Verifying Checksum
|
||||
0738e5cb835e: Download complete
|
||||
589002ba0eae: Pull complete
|
||||
ef63511ea6cc: Pull complete
|
||||
0738e5cb835e: Pull complete
|
||||
3e6813f70c64: Pull complete
|
||||
Digest: sha256:a9890c898f379c1905ee5b28342f6b408dc863f08db2dab20e46c267d1ff463a
|
||||
Status: Downloaded newer image for traefik:v3.6.7
|
||||
docker.io/library/traefik:v3.6.7
|
||||
2026.6.0: Pulling from cloudflare/cloudflared
|
||||
47de5dd0b812: Pulling fs layer
|
||||
c172f21841df: Pulling fs layer
|
||||
99515e7b4d35: Pulling fs layer
|
||||
99ba982a9142: Pulling fs layer
|
||||
d6b1b89eccac: Pulling fs layer
|
||||
2780920e5dbf: Pulling fs layer
|
||||
7c12895b777b: Pulling fs layer
|
||||
3214acf345c0: Pulling fs layer
|
||||
52630fc75a18: Pulling fs layer
|
||||
dd64bf2dd177: Pulling fs layer
|
||||
b839dfae01f6: Pulling fs layer
|
||||
ebddc55facdc: Pulling fs layer
|
||||
bdfd7f7e5bf6: Pulling fs layer
|
||||
2d4d7adf6272: Pulling fs layer
|
||||
40008157d8d2: Pulling fs layer
|
||||
bd8962e29291: Pulling fs layer
|
||||
cac2ae0193cb: Pulling fs layer
|
||||
74d1dac84ecc: Pulling fs layer
|
||||
99ba982a9142: Waiting
|
||||
d6b1b89eccac: Waiting
|
||||
2780920e5dbf: Waiting
|
||||
7c12895b777b: Waiting
|
||||
3214acf345c0: Waiting
|
||||
52630fc75a18: Waiting
|
||||
dd64bf2dd177: Waiting
|
||||
b839dfae01f6: Waiting
|
||||
ebddc55facdc: Waiting
|
||||
bdfd7f7e5bf6: Waiting
|
||||
2d4d7adf6272: Waiting
|
||||
40008157d8d2: Waiting
|
||||
bd8962e29291: Waiting
|
||||
cac2ae0193cb: Waiting
|
||||
74d1dac84ecc: Waiting
|
||||
47de5dd0b812: Download complete
|
||||
c172f21841df: Verifying Checksum
|
||||
c172f21841df: Download complete
|
||||
47de5dd0b812: Pull complete
|
||||
99515e7b4d35: Verifying Checksum
|
||||
99515e7b4d35: Download complete
|
||||
99ba982a9142: Download complete
|
||||
d6b1b89eccac: Verifying Checksum
|
||||
d6b1b89eccac: Download complete
|
||||
2780920e5dbf: Verifying Checksum
|
||||
2780920e5dbf: Download complete
|
||||
7c12895b777b: Verifying Checksum
|
||||
7c12895b777b: Download complete
|
||||
3214acf345c0: Verifying Checksum
|
||||
3214acf345c0: Download complete
|
||||
52630fc75a18: Verifying Checksum
|
||||
52630fc75a18: Download complete
|
||||
dd64bf2dd177: Download complete
|
||||
b839dfae01f6: Verifying Checksum
|
||||
b839dfae01f6: Download complete
|
||||
ebddc55facdc: Verifying Checksum
|
||||
ebddc55facdc: Download complete
|
||||
bdfd7f7e5bf6: Verifying Checksum
|
||||
bdfd7f7e5bf6: Download complete
|
||||
2d4d7adf6272: Verifying Checksum
|
||||
2d4d7adf6272: Download complete
|
||||
c172f21841df: Pull complete
|
||||
40008157d8d2: Verifying Checksum
|
||||
40008157d8d2: Download complete
|
||||
bd8962e29291: Verifying Checksum
|
||||
bd8962e29291: Download complete
|
||||
cac2ae0193cb: Verifying Checksum
|
||||
cac2ae0193cb: Download complete
|
||||
74d1dac84ecc: Verifying Checksum
|
||||
74d1dac84ecc: Download complete
|
||||
99515e7b4d35: Pull complete
|
||||
99ba982a9142: Pull complete
|
||||
d6b1b89eccac: Pull complete
|
||||
2780920e5dbf: Pull complete
|
||||
7c12895b777b: Pull complete
|
||||
3214acf345c0: Pull complete
|
||||
52630fc75a18: Pull complete
|
||||
dd64bf2dd177: Pull complete
|
||||
b839dfae01f6: Pull complete
|
||||
ebddc55facdc: Pull complete
|
||||
bdfd7f7e5bf6: Pull complete
|
||||
2d4d7adf6272: Pull complete
|
||||
40008157d8d2: Pull complete
|
||||
bd8962e29291: Pull complete
|
||||
cac2ae0193cb: Pull complete
|
||||
74d1dac84ecc: Pull complete
|
||||
Digest: sha256:ba461b8aa9c042156dbd39c38657fe7431bafa063220eab8d5330a523863da9f
|
||||
Status: Downloaded newer image for cloudflare/cloudflared:2026.6.0
|
||||
docker.io/cloudflare/cloudflared:2026.6.0
|
||||
1.3.3-stable: Pulling from gtstef/filebrowser
|
||||
6a0ac1617861: Pulling fs layer
|
||||
ef8806083e82: Pulling fs layer
|
||||
b74107c861c7: Pulling fs layer
|
||||
adc935def003: Pulling fs layer
|
||||
4f4fb700ef54: Pulling fs layer
|
||||
18695ccc900a: Pulling fs layer
|
||||
45d119d5c397: Pulling fs layer
|
||||
dac52db4fc51: Pulling fs layer
|
||||
6d598f86b2f2: Pulling fs layer
|
||||
8aa349c8396c: Pulling fs layer
|
||||
dac52db4fc51: Waiting
|
||||
6d598f86b2f2: Waiting
|
||||
8aa349c8396c: Waiting
|
||||
adc935def003: Waiting
|
||||
4f4fb700ef54: Waiting
|
||||
18695ccc900a: Waiting
|
||||
45d119d5c397: Waiting
|
||||
6a0ac1617861: Verifying Checksum
|
||||
6a0ac1617861: Download complete
|
||||
adc935def003: Verifying Checksum
|
||||
adc935def003: Download complete
|
||||
b74107c861c7: Verifying Checksum
|
||||
b74107c861c7: Download complete
|
||||
4f4fb700ef54: Verifying Checksum
|
||||
4f4fb700ef54: Download complete
|
||||
45d119d5c397: Verifying Checksum
|
||||
45d119d5c397: Download complete
|
||||
dac52db4fc51: Verifying Checksum
|
||||
dac52db4fc51: Download complete
|
||||
ef8806083e82: Verifying Checksum
|
||||
ef8806083e82: Download complete
|
||||
6a0ac1617861: Pull complete
|
||||
18695ccc900a: Verifying Checksum
|
||||
18695ccc900a: Download complete
|
||||
6d598f86b2f2: Verifying Checksum
|
||||
6d598f86b2f2: Download complete
|
||||
8aa349c8396c: Verifying Checksum
|
||||
8aa349c8396c: Download complete
|
||||
ef8806083e82: Pull complete
|
||||
b74107c861c7: Pull complete
|
||||
adc935def003: Pull complete
|
||||
4f4fb700ef54: Pull complete
|
||||
18695ccc900a: Pull complete
|
||||
45d119d5c397: Pull complete
|
||||
dac52db4fc51: Pull complete
|
||||
6d598f86b2f2: Pull complete
|
||||
8aa349c8396c: Pull complete
|
||||
Digest: sha256:eb3733681db8757412632c61a99ad656f0d94ed6781bb2ea114b4d70babab78c
|
||||
Status: Downloaded newer image for gtstef/filebrowser:1.3.3-stable
|
||||
docker.io/gtstef/filebrowser:1.3.3-stable
|
||||
1.1.0: Pulling from admin/felhom-samba
|
||||
897d797d2723: Pulling fs layer
|
||||
3051591aa250: Pulling fs layer
|
||||
ce57a3f93416: Pulling fs layer
|
||||
fb94eeec2fe1: Pulling fs layer
|
||||
fb94eeec2fe1: Waiting
|
||||
ce57a3f93416: Verifying Checksum
|
||||
ce57a3f93416: Download complete
|
||||
897d797d2723: Verifying Checksum
|
||||
897d797d2723: Download complete
|
||||
fb94eeec2fe1: Verifying Checksum
|
||||
fb94eeec2fe1: Download complete
|
||||
897d797d2723: Pull complete
|
||||
3051591aa250: Verifying Checksum
|
||||
3051591aa250: Download complete
|
||||
3051591aa250: Pull complete
|
||||
ce57a3f93416: Pull complete
|
||||
fb94eeec2fe1: Pull complete
|
||||
Digest: sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10
|
||||
Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.1.0
|
||||
gitea.dooplex.hu/admin/felhom-samba:1.1.0
|
||||
[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) …
|
||||
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'.
|
||||
[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) …
|
||||
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'.
|
||||
[golden] baking the first-boot SSH host-key regeneration unit (F3) …
|
||||
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'.
|
||||
[golden] identity-clean + minimize …
|
||||
[golden] stop + archive …
|
||||
INFO: including mount point rootfs ('/') in backup
|
||||
INFO: including mount point mp0 ('/var/lib/felhom') in backup
|
||||
INFO: archive file size: 626MB
|
||||
INFO: Finished Backup of VM 9100 (00:00:28)
|
||||
[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_08_08-16_41_56.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive)
|
||||
[golden] publishing golden (656787777 bytes, sha256 b9f701fab813c051…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.210.0/golden.tar.zst
|
||||
[golden] pre-delete existing: HTTP 404 (404/204 expected)
|
||||
[golden] upload OK (HTTP 201)
|
||||
GOLDEN_VERSION=0.210.0
|
||||
GOLDEN_SHA256=b9f701fab813c051dd4be348a4e301f0e32518b811920406b7bdf8b3dd4c0a00
|
||||
[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.210.0 / b9f701fab813c051dd4be348a4e301f0e32518b811920406b7bdf8b3dd4c0a00
|
||||
[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge)
|
||||
Reference in New Issue
Block a user