32 Commits

Author SHA1 Message Date
admin e1d349f100 hub manifest: 0.161.0 -> 0.162.0 (operator yes, 2026-10-20 morning)
gates / gates (push) Successful in 12m44s
2026-10-20 07:59:47 +02:00
admin 7d7868e5e5 Night 2026-10-19 Part 10: ep0's first weekly prune matched the dry runs (only tester-1's 10-04 copy removed; every household has a rule) -> R-1162 closed, register 161 -> 160; R-895 and R-435 due dates moved 10-20 -> 10-21 with the reason; P10 evidence
gates / gates (push) Successful in 13m14s
2026-10-20 06:00:58 +02:00
admin 189eb82eb0 Night 2026-10-19: register 161 -> 161 (closed R-921, R-953, R-986; opened R-1168..R-1170; R-1001/R-1011/R-1010/R-1070/R-1167 READY, R-893 READY with the measured design note -> D20, R-1083 gap); morning list in STATUS; 03/05/07/11 updated; R-893 evidence
gates / gates (push) Successful in 13m5s
2026-10-19 21:44:10 +02:00
admin f92fa5a8c8 nodes.md, nightly-rotation.md: scratch guest 9202 takes its address by DHCP (.114 until 2026-10-19, now .121); read it from the guest
gates / gates (push) Successful in 12m48s
2026-10-19 21:30:39 +02:00
admin 5d0dfeac5f hostinstall harness: the two R-1167 behaviour cases SKIP out loud where there is no curl (the CI runner; CI 2106/2108 red) 2026-10-19 21:30:33 +02:00
admin 15f7428871 security_scan_gate.py: govulncheck (called vulns) + gitleaks on the pushed commits in every repo's fast set and CI, warn-first, with an accepted-hits file and 8 decoys (R-986); first scan triaged: 0 real secrets, 0 called vulnerabilities; evidence for R-953 and the 0.335.0 release
gates / gates (push) Failing after 12m59s
2026-10-19 21:03:23 +02:00
admin d92a712ee4 felhom-host-install.sh 1.35.0 (not published): the hub's live-box 409 is worded in EN and HU at the preflight, host-enroll and the controller.yaml fetch, with harness cases (R-1167)
gates / gates (push) Failing after 7m16s
2026-10-19 20:35:21 +02:00
admin 5f31da04bc hub v0.162.0 (built, not deployed): no repeat mail after a restart for a known-down host or an already-reported stale restore test (R-1001); the Day-0 agent list leaves out packages without the felhom-agent file, a vouch of one is refused (R-1011); an absent tunnel reads 'none set up' (R-1070)
gates / gates (push) Successful in 7m20s
2026-10-19 20:27:01 +02:00
admin a28f162c66 Day 3 2026-10-19: passphrase locks once the box uses its own key (D19; hub 0.161.0, controller 0.334.0 floor), every ep0 household gets a prune job (tenantsync v1.3.0), demo-hp address clean; register 163 -> 161 (closed 3, opened R-1167)
gates / gates (push) Successful in 7m23s
2026-10-19 16:34:29 +02:00
admin 97f27a7528 manifests: hub 0.161.0 (R-1159, R-1165; operator yes 2026-10-19)
gates / gates (push) Successful in 7m20s
2026-10-19 16:07:37 +02:00
admin b6a911c3f9 hub v0.161.0: passphrase keys only until the box uses its own (box-config route; R-1159, D19); ep0 namespaces without a prune job alarmed, tenantsync v1.3.0 creates the job (R-1165)
gates / gates (push) Successful in 7m47s
2026-10-19 15:55:45 +02:00
admin 224fe41e06 Day 2 2026-10-19: ep0 keep-weekly 2 (D18), passphrase measured (R-1159 READY, D19), hub 0.160.0 deployed, controller 0.333.0 floor, gofmt gates; register 167 -> 163 (closed 6, opened R-1165, R-1166)
gates / gates (push) Successful in 7m15s
2026-10-19 12:20:37 +02:00
admin 73f3899266 REPORT website git-sync: CI results; the 09:5x DNS blip noted
gates / gates (push) Successful in 7m15s
2026-10-19 12:02:31 +02:00
admin e2c35007b5 Website git-sync in-cluster: R-1163 closed with evidence, R-1164 opened (P4); report
gates / gates (push) Successful in 7m9s
2026-10-19 11:51:54 +02:00
admin fd75003e17 webpage: the four git-syncs read Gitea in-cluster; sidecars retry forever (--max-failures=-1); gate 6 keeps the --repo values identical
gates / gates (push) Successful in 7m42s
On 2026-10-19 09:24 the public path to Gitea failed once and the sidecar exited (max-failures defaults to 0).
Anonymous in-cluster fetch proven first (main and installer-v1.34.0 resolve to the public hashes).
2026-10-19 11:38:44 +02:00
admin 5d3cb5c132 manifests: hub 0.160.0 (R-1156, R-1157, R-1158; operator yes 2026-10-19)
gates / gates (push) Successful in 7m36s
2026-10-19 11:25:37 +02:00
admin a17db0c3c8 hub-gofmt gate in repo_gates.py with its decoys (R-1160)
gates / gates (push) Successful in 7m14s
2026-10-19 10:40:55 +02:00
admin 503c93fada hub v0.160.0: a live re-enroll mails the operator (R-1156); further refused backups counted, alarm once a day (R-1157); the reinstall's off-site re-issue says what it still does (R-1158) 2026-10-19 10:34:41 +02:00
admin d5e3c8b0d9 hub: gofmt -w on 22 files, formatting only, no behaviour change (R-1160) 2026-10-19 10:26:36 +02:00
admin db2371c6f5 Day 2026-10-19: agent 0.160.0 (released, not vouched), controller 0.332.0 floor (one sign-in R-1100, escrow confirm R-1151), hub 0.159.0 deployed (R-1150, R-1154), R-1153 not a defect, old-restic check passed; register 168 -> 166 (closed 10, opened R-1155..R-1162)
gates / gates (push) Successful in 7m20s
2026-10-19 05:38:38 +02:00
admin 200ab6530b hub 0.159.0 to the manifest (operator yes in advance, 2026-10-19); P1 delivery evidence (agent 0.160.0 on all three hosts)
gates / gates (push) Successful in 7m18s
2026-10-19 04:10:23 +02:00
admin 5622ce2735 Day 2026-10-19 P1 evidence: agent 0.160.0 release, demo-hp delivery, backup + cap, escrow confirm/abandon, R-1093 on the format route
gates / gates (push) Successful in 7m25s
2026-10-19 03:52:34 +02:00
admin afc939bb3c R-887 closed: 381 CI jobs since the S21 change, 0 lost (the 3 re-runs' first attempts ran 47-49 s; control: the known lost attempt ran 11 m 48 s); register 168 -> 167
gates / gates (push) Successful in 7m20s
2026-10-19 03:42:11 +02:00
admin b505ce4ba0 hub v0.159.0 (R-1154, after security review): the enroll reuse path re-issues nothing, it warns the operator
The first draft (5121536f, unpushed) re-issued the claim code and the off-site login on a
reuse whose old box was stale or switched off. A security review flagged it: the caller
holds only the household's retrieval passphrase, and both re-issues are operator
session-only actions (05 section 17.5, 09 decision 252, R-1055/R-1059). Now the reuse
path classifies the old box and re-issues nothing: gone -> host_reinstall_over_old_host
(warning, mailed to the operator: delete the old host and re-run, or re-issue by hand);
live -> host_reenroll_while_live (info). The mint path after the operator's delete is
unchanged and still re-issues. The once-per-silence record and the claim engine comment
edit are reverted. 05 section 8, README and CHANGELOG updated; tests red-proved.
2026-10-19 03:29:44 +02:00
admin 5121536f10 hub v0.159.0: a refused whole-guest backup has its own alarm (R-1150); a reinstall re-issues the claim code and the off-site login when the old host is stale or off (R-1154)
R-1150: the agent's localapi:backup-cadence-cap is a separate per-host state in
HostCapabilityChecker: alone it raises whole_guest_backup_refused (warning, plain text,
agent reason in details) and whole_guest_backup_refusal_cleared (info); other critical
caps keep agent_capability_degraded/_recovered unchanged. New types operator-only and
hub-origin only (not in allowedEventTypes: the suspected controller must not post them).
agent_capability_degraded/_recovered join operatorOnlyEvents (the checker's "never the
customer" claim was configuration only).

R-1154: handleHostEnroll's reuse path re-issues (claim code + off-site login) when no
controller wait is open and either deliberate_off_at is held or both the host report and
the controller report are older than alerting.stale_threshold (wired from main); once per
silence (host_reenroll_reissued record, also written at mint); a live box gets nothing
re-issued and the skip is recorded (host_reenroll_not_reissued). 05 section 8 [DESIGN].

Tests red-proved; go build/vet/test green.
2026-10-19 03:18:34 +02:00
admin e4f022a0d9 Evening 2026-10-18: fresh install proven (R-1148), OnlyOffice walk (R-1018), one-sign-in spike (R-1100 READY), Redis volumes (R-1117), CI rows, golden 0.331.0 baked; register 168 -> 168 (closed 6, opened R-1149..R-1154)
gates / gates (push) Successful in 7m34s
2026-10-18 23:12:21 +02:00
admin b9b56c47c0 CLAUDE.md CI pull check: a rerun replaces the attempt the jobs endpoint shows; read actions/tasks when run_attempt > 1 (R-1137)
gates / gates (push) Successful in 7m7s
Measured 2026-10-18 on controller jobs 1944 and 1959: same job id,
run_attempt 2, conclusion and log are the rerun's; actions/tasks keeps one
task per attempt (1962 failure + 1964 success; 1978 failure + 1980 success).
2026-10-18 20:51:52 +02:00
admin 1fe9f84893 CI workflow: the failure mail links the public run page, by run id (R-1135)
gates / gates (push) Successful in 7m17s
The alarm step built its link from GITHUB_SERVER_URL, which on this runner
is the in-cluster Gitea address, and from the run NUMBER, which Gitea 1.26
reads as a run id first (felhom.eu /actions/runs/1116 opened run id 1116 =
run number 613). Now https://gitea.dooplex.hu + GITHUB_RUN_ID, and the link
is printed into the job log as ALARM-LINK. Same change in all four repos.
2026-10-18 20:34:59 +02:00
admin 7f7081d127 Day 2026-10-18: report, CONTEXT; old-restic check not run yet (demo-hp's clean-up window opens after the 2026-10-19 04:15 CEST run), STATUS time corrected
gates / gates (push) Successful in 7m19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-18 16:36:53 +02:00
admin e881115c43 Day 2026-10-18: register (173 -> 168; closed R-1140 R-1144 R-1145 R-1146 R-1147 R-1036; opened R-1148), STATUS, agent 0.159.2 delivery evidence
gates / gates (push) Successful in 7m20s
2026-10-18 14:41:34 +02:00
admin 37cb83c166 Estimator: Immich peak 3,290 -> 3,983 MB from the 4,000-photo import (R-1147, operator yes 2026-10-18); Immich alone on 8 GB now 'does not fit'
gates / gates (push) Successful in 7m17s
Rebuilt sizing-data.json; data cache-bust v5; SIZING.md table and the pinned test cases moved
with it. Before/after: audits/day-2026-10-18/P6-before-after.md.
2026-10-18 14:36:14 +02:00
admin 3332eb0919 manifests: publish installer 1.34.0 (tag installer-v1.34.0; vault-first step 4b, --rotate-recovery-only) — operator yes 2026-10-18
gates / gates (push) Successful in 7m19s
2026-10-18 14:28:18 +02:00
308 changed files with 8697 additions and 268 deletions
+3 -1
View File
@@ -29,5 +29,7 @@ ArgoCD's source of truth is the **manifest in git**:
`main`, and `/scripts/` from the tag `installer-v<SCRIPT_VERSION>`. Changing either `--ref` changes
what real machines download — the init container **and** the sidecar both carry it, and both must
move together. `hostinstall_gates.py` gate 6 fails if the manifest stops naming an `installer-v…`
tag or if the website stops tracking `main`. The full publish/rollback procedure is the R-110 fence
tag or if the website stops tracking `main`. All four git-sync containers (two sidecars, two init containers) read
Gitea **in-cluster** (`http://gitea.gitea-system.svc.cluster.local:3000/…`, since 2026-10-19) and must name the same
repo — gate 6 fails if they differ; the sidecars carry `--max-failures=-1` (retry, never exit). The full publish/rollback procedure is the R-110 fence
in the repo core `CLAUDE.md`.
+14 -2
View File
@@ -184,6 +184,10 @@ jobs:
# which prints at the moment a release is committed, where someone can actually act on it.
# Those are the compensating controls that make this green honest. Every other gate still
# fails this job on any push, and golden-currency still fails it on a push touching code.
env:
# R-986: the security-scan gate fetches the PINNED gitleaks here (sha256-checked); no Go, so govulncheck says
# NOT CHECKED in CI, out loud. Warn-first either way: the scan lists, it never fails this job.
FELHOM_SCAN_FETCH: "1"
run: python3 scripts/repo_gates.py --fast --scope="${PUSH_SCOPE:-code}"
- name: Manual proof - alarm key present, optional forced failure (dispatch only, R-1129)
@@ -228,8 +232,16 @@ jobs:
repo = os.environ.get("GITHUB_REPOSITORY", "?")
sha = os.environ.get("GITHUB_SHA", "?")
run = os.environ.get("GITHUB_RUN_NUMBER", "?")
srv = os.environ.get("GITHUB_SERVER_URL", "https://gitea.dooplex.hu")
# R-1135: the link is for a PERSON, so it is the public address and the run ID.
# NOT GITHUB_SERVER_URL: the runner registers at the in-cluster address, so that variable
# reads http://gitea.gitea-system.svc.cluster.local:3000, which nobody outside the cluster
# can open (forced-test mail, job 1997). NOT GITHUB_RUN_NUMBER: Gitea 1.26 reads
# /actions/runs/<n> as a run id first, so a run number that is also an older run's id
# opens THAT run (measured 2026-10-18: felhom.eu /actions/runs/1116 opened the run whose
# id is 1116 = run number 613). The link is printed too, so the job log carries it.
run = os.environ.get("GITHUB_RUN_ID", "?")
srv = "https://gitea.dooplex.hu"
print("ALARM-LINK %s/%s/actions/runs/%s" % (srv, repo, run))
# R-265: how long the run took, so a reap is self-identifying. An honest gate failure
# lands in well under a minute; a multi-minute figure means the job hit its own timeout
+37
View File
@@ -0,0 +1,37 @@
# .security-accepted — hits security_scan_gate.py (felhom.eu/scripts, R-986) lists as ACCEPTED, not WARN.
# One per line: <tool> <id> <where> # reason. govulncheck: OSV id + module dir; gitleaks: rule@<sha256[:12] of the line> + file.
# The first scan's verdicts (2026-10-19): felhom.eu documentation/audits/night-2026-10-19/P7/triage.md
gitleaks generic-api-key@0275c9c51957 documentation/audits/usenet-2026-10-12/sabnzbd-bench/13-compose-resolved.txt # accepted: a bench app's generated admin password and API key (LXC 9401, stopped; app removed from 9202)
gitleaks generic-api-key@0a189ae60ea4 documentation/tests/campaign2-evidence-2026-07-07/state/runner.log # accepted: the same dry-run deploy values in the campaign runner's log
gitleaks generic-api-key@2a27df76f661 documentation/audits/usenet-2026-10-12/sabnzbd-bench/02-telemetry-and-net.txt # accepted: a bench app's generated API key (LXC 9401, stopped; app removed from 9202 at teardown)
gitleaks generic-api-key@3a4c7b17e349 documentation/audits/usenet-2026-10-12/nzbget-bench/03-process-and-entrypoint.txt # accepted: a bench app's generated control password (LXC 9401, stopped; app removed from 9202 at teardown)
gitleaks generic-api-key@3c180a9f957c documentation/tests/campaign2-evidence-2026-07-07/state/runner.log # accepted: the same dry-run deploy values in the campaign runner's log
gitleaks generic-api-key@486819f82f58 documentation/audits/update-night-2026-09-21/apps/actualbudget/log.txt # accepted: a demo app's session token, demo-hp 2026-09-21 (Tier 0); no running demo app has it (2026-10-12 review)
gitleaks generic-api-key@486819f82f58 documentation/audits/update-night-2026-09-21/batchA.log # accepted: the same demo app session token as actualbudget/log.txt
gitleaks generic-api-key@5a64c6ffd6df documentation/audits/signup-lock-2026-09-29/B/wd.sh # accepted: a stranger-probe test password in an audit script (signup lock, Tier 0)
gitleaks generic-api-key@6e2e899b7d6e documentation/audits/offsite-lock-build-2026-10-03/partD/demo-hp-migration.txt # false: an SSH key FINGERPRINT in a log line, not a key
gitleaks generic-api-key@70380348263d hub/internal/web/bind_language_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@7134caf03160 documentation/audits/evidence-rehearsal-2026-08-09/GATE0-demo-hp-before.txt # false: a WireGuard PUBLIC key
gitleaks generic-api-key@7134caf03160 documentation/audits/evidence-rehearsal-2026-08-09/P1-demo-hp-after.txt # false: a WireGuard PUBLIC key
gitleaks generic-api-key@71ba98258279 hub/internal/api/r1140_accesslog_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@7aa746ddb210 documentation/audits/offsite-lock-build-2026-10-03/partD/demo-hp-migration.txt # false: an SSH key FINGERPRINT in a log line, not a key
gitleaks generic-api-key@7ce547845b1e documentation/audits/night-rulings-2026-09-30/D/D3-wanderer-routes-probe.txt # accepted: a throwaway app's session cookie (wanderer probe, 2026-09-30, Tier 0)
gitleaks generic-api-key@8044eef94360 documentation/tests/campaign2-evidence-2026-07-07/state/runner.log # accepted: the same dry-run deploy values in the campaign runner's log
gitleaks generic-api-key@85b662612937 hub/internal/store/r879_box_seal_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@9530a022fde8 documentation/audits/usenet-2026-10-12/sabnzbd-bench/13-compose-resolved.txt # accepted: a bench app's generated admin password and API key (LXC 9401, stopped; app removed from 9202)
gitleaks generic-api-key@9b3a9c2be48a documentation/tests/campaign2-evidence-2026-07-07/evidence/T-DEPLOY-SET/values_bookstack.json # accepted: dry-run deploy values of a 2026-07-07 campaign (generated, demo boxes since rebuilt)
gitleaks generic-api-key@a9538cfd7063 documentation/tests/campaign2-evidence-2026-07-07/evidence/T-DEPLOY-SET/values_paperless-ngx.json # accepted: dry-run deploy values of a 2026-07-07 campaign (generated, demo boxes since rebuilt)
gitleaks generic-api-key@b0414c6f59b9 documentation/audits/offsite-lock-build-2026-10-03/partD/demo-hp-migration.txt # false: an SSH key FINGERPRINT in a log line, not a key
gitleaks generic-api-key@b1db45e463b6 documentation/audits/family-gate-2026-10-02/tools/fg_items.py # false: test strings in an audit tool (websocket probe labels)
gitleaks generic-api-key@b1db45e463b6 documentation/audits/permanent-gate-2026-10-01/tools/items12.py # false: test strings in an audit tool (websocket probe labels)
gitleaks generic-api-key@bd228c851720 documentation/tests/walk5-r201-2026-08-07/journal.md # false: sha256 digests compared in a walk journal, not secrets
gitleaks generic-api-key@c1be98b7b703 documentation/tests/walk5-r201-2026-08-07/journal.md # false: sha256 digests compared in a walk journal, not secrets
gitleaks generic-api-key@c3b2b5665c9c documentation/audits/evidence-fixes-first-tester-2026-09-30/part0/cloudflare-dns.txt # false: a tunnel ID (a UUID), not the tunnel token
gitleaks generic-api-key@cf453385b707 documentation/tests/campaign2-evidence-2026-07-07/state/runner.log # accepted: the same dry-run deploy values in the campaign runner's log
gitleaks generic-api-key@e59eaeb4aee7 documentation/audits/new-apps-2026-10-01/tools/newapp_box.py # accepted: a throwaway test user's password in an audit tool for a fresh-app walk on a Tier-0 box
gitleaks generic-api-key@eada8358f62a hub/internal/web/render_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@efd4cc86d8d9 documentation/audits/power-cut-fixes-2026-10-14/E/demo-hp-journal-boot.txt # false: the id of HP's public UEFI certificate in a boot journal
gitleaks generic-api-key@f28c207293c1 documentation/tests/campaign2-evidence-2026-07-07/evidence/T-DEPLOY-SET/values_vaultwarden.json # accepted: dry-run deploy values of a 2026-07-07 campaign (generated, demo boxes since rebuilt)
gitleaks generic-api-key@f399eb9e3059 documentation/audits/night-2026-09-23/chaos/round-01-run.txt # accepted: the same demo app session token as round-01.log
gitleaks generic-api-key@f399eb9e3059 documentation/audits/night-2026-09-23/chaos/round-01.log # accepted: a demo app's session token, demo-hp 2026-09-23 (Tier 0); the 2026-10-12 review found no running demo app with it
gitleaks generic-api-key@fdabf6a76b48 documentation/audits/r890-instructions-2026-10-06/tools/wgerwalk.py # accepted: a throwaway wger walk's API token in an audit tool (Tier 0)
+6
View File
@@ -178,6 +178,12 @@ something, not only sessions that touch `documentation/` — which is why it is
rows — a run can sit several pages earlier. **Scan every page** and match on `head_sha`; with a
few hundred jobs that is a handful of requests. Guessing the page produced three consecutive
false "no CI job for this commit" readings in one session.
3. **A rerun REPLACES the attempt the `jobs` endpoint shows** (R-1137, measured 2026-10-18): the job
keeps its id, `run_attempt` becomes 2, `started_at`, `conclusion` and the log are the rerun's — the
red first attempt is gone from `jobs` and its log (controller jobs 1944, 1959: failure mails, now
green). **A matched job with `run_attempt` > 1 is not a plain green:** list `actions/tasks` (same
paging, match `head_sha`) — one task per attempt, the verdict in `status`, so it reads
„failure, then rerun success". The first attempt's log is not reachable by the API.
An unchecked green is an assumption, not an observation — and so is a green read off a field the
API never populates.
+56
View File
@@ -16,6 +16,62 @@
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
> **2026-10-19 (night) — controller 0.335.0 is the floor (MinAgent 0.158.1, 18:39 UTC); hub 0.162.0 BUILT, not deployed;
> agent `56aabc7` and installer 1.35.0 committed, not released/published.** R-921 (operator answer „add one field"): the
> agent serves its host-wide heavy-operation gate as `host_busy` on `GET /backup/status`; the controller reads it before
> any stop (held → contention, no stop). R-1070's cause: Docker 29 prints „error: no such object" in lower case, so a
> guest without cloudflared read `unknown` and failed the host OS-update rule; new state `absent` (passes when absent
> before the step too); `11` §8.2 had claimed `unknown` does not fail — corrected. R-1010: the janitor's forced snapshot
> delete is the API's `force=1`. Hub 0.162.0: R-1001 (restart seeding from the DB), R-1011 (exact-file sha). R-953 is
> gone: ArgoCD tracks none of the three Secrets now. **R-986: `scripts/security_scan_gate.py`, shared, in every repo's
> fast set and CI, WARN-FIRST**; govulncheck pin v1.1.4 → v1.8.0 (v1.1.4 cannot read Go 1.27); CI has no Go, so
> govulncheck runs only in the hook; a depth-1 CI clone scans the whole tree (the root commit), all accepted. R-893
> measured on 9202 (`audits/night-2026-10-19/P3/`) → D20. 9202 is on DHCP (.121 now).
> **2026-10-19 (late afternoon) — controller 0.334.0 is the floor (MinAgent 0.158.1, 16:31 local); hub 0.161.0 DEPLOYED;
> tenantsync v1.3.0 on ep0 (operator yes).** R-1159 (D19 „box key first, then lock"): the passphrase is used AFTER enroll
> (installer `/config` for its git login, the agent bakes it into `bootstrap.json`, the guest's first-boot pull, and —
> until 0.334.0 — every config refresh), so "keys only before a host exists" was not buildable. Now: `GET /box-config`
> with the controller key; the hub refuses passphrase keys (409, `passphrase_keys_refused`) once a ≥ 0.334.0 controller
> has reported since the host record was made — an rc (`0.334.0-rc.N`) counts as older. **A reinstall over a live box
> needs the operator's delete first.** R-1165: every new ep0 namespace gets `prune-<id>`; the hub alarms on a namespace
> with none. demo-hp's .104 clean after the router reservation. Register 163 → 161. `REPORT-day3-2026-10-19.md`.
> **2026-10-19 (afternoon) — controller 0.333.0 is the floor (MinAgent 0.158.1, 12:16 local); hub 0.160.0 DEPLOYED
> (operator yes); ep0 prune `keep-weekly 2` (operator yes, D18).** ep0 change: `prune-demo-hp`/`prune-demo-felhom` only,
> after PBS's own `--dry-run true` (API `proxmox-backup-debug api create …/prune`; it creates no task) showed nothing
> removed; households with no prune job (tester-1, Tester-2) = R-1165. R-1155: FileBrowser 1.5.6 reads its token from the
> cookie, `?auth=`, Bearer AND the Basic-auth password (`extractToken`) — the forwardAuth refusal must read all four (the
> security review caught Basic). R-1159 measured: passphrase → `/config` key and `/host-enroll` existing host key → every
> off-site key route; D19 open (CC picks A: keys only while no host exists). R-1158: the reinstall's off-site re-issue only
> rotates the hub's own registrar login; kept. gofmt gates in hub and agent runners. demo-hp's .104 is taken by another
> device (R-1166, the router). Register 167 → 163. `REPORT-day2-2026-10-19.md`.
> **2026-10-19 (day) — controller 0.332.0 is the floor (MinAgent 0.158.1, 05:24 local); agent 0.160.0 released and on the
> three test hosts, NOT vouched; hub 0.159.0 DEPLOYED (operator yes in advance).** One sign-in for the file manager (R-1100,
> D16): dashboard pass → host-only `felhom_files` cookie → traefik forwardAuth `/__felhom_gate/files` sets the per-box
> SECRET header (`<data>/filebrowser-sso-header`, never logged) naming FileBrowser user `felhom-household`; traefik makes a
> RELATIVE Location in a forwardAuth answer absolute against the auth server — always answer absolute URLs. R-1151: the
> escrow package uploads at `POST /escrow/ceremony/confirm` (two typed words), an old controller keeps the claim order.
> R-1149 (D17): local tier `keep-daily=N`. R-1154 rule (after a security review of the helper's first draft): the
> host-enroll reuse path NEVER re-issues — the passphrase must not reach the operator's re-issue powers; it mails the
> operator. R-1153 was the test file (FileBrowser previews .docx with MuPDF before OnlyOffice). R-887 closed (381 jobs, 0
> lost since S21). The old-restic check after demo-hp's first real 0.19.1 prune passed. Register 168 → 166.
> `REPORT-day-2026-10-19.md`.
> **2026-10-18 (evening) — controller 0.331.0 is the floor (MinAgent 0.158.1): OnlyOffice opens and saves from FileBrowser
> and Nextcloud, FileBrowser back at once after a restart (R-1018), a removal deletes a volume the definition dropped
> (R-1117); catalog `30eea97`: Immich/Nextcloud Redis without a volume; golden 0.331.0 baked, NOT vouched; a fresh install
> from the ISO proved installer 1.34.0 + vouched golden/agent + auto-floor (R-1148); agent `main` carries R-978, R-973,
> R-1093 unreleased. *Decided by CC unattended — operator may reverse:* the R-973 cap numbers (3 successful daily-tier runs
> per 24 h, one weekly off-site run per 6 days; `07` §6.1). Tester4 is kept as the standing fresh-install customer
> (operator). `REPORT-evening-2026-10-18.md`.**
> **2026-10-18 (day) — hub 0.158.0 DEPLOYED (access log live, agent-only vouch, R-895 first verdict 2026-10-20);
> controller 0.330.0 is the floor (R-1144); agent 0.159.2 released and on the three test hosts, NOT vouched; installer
> 1.34.0 published (vault-first step 4b, R-1145; fresh install unproven, R-1148); estimator Immich peak 3,983 MB
> (R-1147).** Independent review of the night: `audits/day-2026-10-18/P2/`. `REPORT-day-2026-10-18.md`.
> **2026-10-18 (night) — controller 0.329.0 is the floor (MinAgent 0.158.1); golden 0.329.0 baked, NOT vouched; hub 0.158.0
> built, NOT deployed.** Each app's own `appdata/<folder>` (R-1142), empty allow-list (R-981), drive retirement asks the
> agent first (R-1063), plain no-durable-id refusal (R-1069), `:latest` for releases only (R-1143), Immich 3072M/2048M
+139
View File
@@ -0,0 +1,139 @@
# REPORT — day 2026-10-18
## In plain words
- **The hub update is live.** Its own access log works and writes no secret (tested live with a fake token). You can
now vouch an agent without a new golden. The off-site snapshot check gives its first verdict on 20 October.
- **A fresh reviewer read last night's code.** One medium finding: a stale host key could have left the emergency
password unknown. Today's installer change (save in the hub first, then set) already closes it. Three low findings:
two fixed today, one is a known design limit, now written in its row.
- **„Retire anyway":** when the host says no, the drive's apps start again. Where the host's disk list already says no,
nothing is stopped at all. Controller 0.330.0 is the floor for every box.
- **Agent 0.159.2** runs on demo-hp, Tester 1 and demo-felhom (your yes). A whole-box backup ran with no pool check.
**The vouch is yours** (STATUS).
- **Installer 1.34.0 is published** (your yes): the emergency password is saved in the hub first, then set, then
checked with a login. Proven on demo-hp. A fresh install with it has not run yet (row R-1148).
- **The calculator** shows Immich's bigger figure (your yes). Immich alone on an 8 GB machine now reads „does not fit".
- **The old-restic check has not run.** demo-hp's clean-up comes only after tonight's backup (2026-10-19 about 04:16
Budapest). The STATUS item stays open.
## 1. Hub 0.158.0 deploy
ArgoCD: only `Deployment/hub` was OutOfSync; synced that resource alone → Synced/Healthy, image `felhom-hub:0.158.0`,
startup log `felhom-hub 0.158.0 starting` + `access log: /data/access-log …, kept 90 days`. Floor read back from the hub
DB with its WAL: 0.329.0 / 0.158.1 at deploy time (`audits/day-2026-10-18/P1/hub-deploy-readback.txt`). Reports from all
three boxes in the access log within two minutes (sources .101, .149, .155).
- **Access log (R-1140, closed):** today's file `access-2026-10-18.log`, mode 0600. A fresh 48-hex fake token sent in a
query, a `/bind/<token>` path, a `session=` query, a Bearer header, a Cookie and a POST body: found 0 times; control
(the same file with the token appended) 1. Secret shapes (Bearer value, a 32+ character run, `password=`, `token=`,
`session=`): 0; control 1. `/bind/<token>` is written `/bind/[redacted]` (`P1/access-log-live-test.txt`).
- **Agent-only vouch:** read-only — `/configuration` loads with CC's token (Day-0 section, golden 0.329.0, agent 0.159.1);
the save (`/configuration/artifacts`) answers 403 to the token, by design. No manifest change by CC.
- **Off-site alarm (R-895, NARROWED):** no fixture route exists in the hub for it; the unit tests are the proof so far.
Tomorrow's 07:10 run is the baseline (no verdict by design); the first verdict is 2026-10-20 07:10 — a DUE-CHECKS row.
## 2. The independent review
A fresh subagent with no context from the night reviewed controller v0.328.0→v0.329.0, hub 0.157.0→0.158.0, the agent's
orphan-check change and the installer's `--rotate-recovery-only`. Its report was written to a file and read back
(`audits/day-2026-10-18/P2/independent-review.md`, 54 lines). Verdicts (`P2/verdicts.md`):
| # | Finding | Verdict |
|---|---|---|
| F1 medium | rotate-only: unauthenticated healthz, then chpasswd, then a vault that a stale key fails → an unvaulted password | REAL in the night's code → **fixed** by R-1145 (vault first); harness `R1145-e`, fails against HEAD |
| F2 low–medium | `offsite_snapshots_unexplained` is explained by the box's own reported run time | REAL, by design (decision 68) → written into R-895; the custody alarm `_vanished` is not affected |
| F3 low | „anyway" stops apps before the agent is asked | REAL, was R-1144 → **fixed** (controller 0.330.0) |
| F4 low | host key in curl's argv during the vault PUT | REAL, pre-existing → **fixed** (header file) |
## 3. R-1144
`decommissionAnyway`: reads the agent's disk list first (a role other than user-data, or the backup target → the plain
refusal, nothing stopped); after a refusal past the stops (policy refusal or a failed agent call) starts exactly the apps
that were running; a drive the agent let go keeps its apps stopped. Household text HU + EN. Seam `decomStacks`, so no
test reaches docker. Three tests, two red-proofs (`P3/redproof-r1144.txt`). Not pressed live: no test box has a drive
that is safe to retire.
## 4. Agent 0.159.2
Released with `release-agent.sh` (code pushed first, CHANGELOG after — the `release-complete` hook refuses the other
order): sha256 `4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9`, bundle
`477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9` (read back from the registry). **demo-hp proof:** a
whole-guest backup of 9201 through the dashboard (`/api/guest-backup/trigger`, snapshot mode, completed in 7.5 min):
**0** `felhom-thin-check` sudo lines; control — the night's backup on 0.159.1 at 04:33 had **1**
(`P4/demo-hp-backup-journal.txt`). The failure path stays covered by the agent's fixture tests. **Operator: yes** →
Tester 1 and demo-felhom by signed jobs (agent_update + agent_config_update): both `signed op COMPLETED`, capability
probe 75/75, `update committed` 0.159.2, and a host report on 0.159.2 from each (`P4/`). **Not vouched** — STATUS.
## 5. The password order (R-1145, R-1146)
Step 4b (install and rotate-only): generate → PUT to the vault (200 = the hub's DB write is done) → chpasswd (one retry)
→ a PVE API login with the new value. **The states that can remain, and why each is safe or not:**
| State | When | Safe? |
|---|---|---|
| nothing changed | the vault refuses or is unreachable (a stale key included) | safe — root and the vault keep their old, matching pair |
| root = vault = new, PVE login 200 | the normal end | safe |
| vault = new, root = previous | chpasswd fails twice (or the run is killed between vault and set) | **not fully safe:** the vaulted value does not open the console; root keeps its previous password (the shadow line is fingerprinted before/after to prove it). The host key cannot read the vault back (by design), so the previous value cannot be re-vaulted; the log says the state in words, and the next run fixes it (it vaults first again). |
| root = vault = new, PVE login refused | pveproxy trouble | safe for the shell; the run dies and says the web login refused |
The old order's dangerous state (root = new, vault = old) is no longer reachable. The hub cannot tell the installer its
„last vaulted" time (no host-readable route); the installer trusts the PUT's 200, and CC checked the time moved in the
live proof. Live on demo-hp: vault `set_at` 2026-07-21 → 2026-10-18T10:43:33Z, the new value gets a PVE ticket (200),
the old one 401 (`P5/demo-hp-rotation.txt`). Harness: 52 passed / 0 failed (was 45/2); R1145-a..e red-proofed against
HEAD. Installer **1.34.0 published** on the operator's yes: tag `installer-v1.34.0`, both webpage pins, ArgoCD synced
the webpage Deployment only; the served file's sha256 = the tag's (`df722ee4…`). R-1146: SC2066 and GL8-F6 fixed, the
harness is gate `hostinstall-harness` with two decoys.
## 6. Calculator (R-1147)
Before/after (`audits/day-2026-10-18/P6-before-after.md`): Immich peak 3,290 → 3,983 MB (typical unchanged 1,557). The
one changed verdict: **Immich alone on 8 GB: fits, but tight → does not fit**. The family starter set is unchanged.
**Operator: yes** → published (cache-bust v5); live `sizing-data.json?v=5` serves 3,983. SIZING.md's table and the
pinned estimator tests moved with it. R-1036 closed (D14 = no).
## 7. The old-restic check
**Not run — the prune has not happened yet.** The hub allows demo-hp a clean-up window 6 days 12 hours after the last
one (2026-10-12 02:16:53 UTC → from 2026-10-18 14:16:53 UTC). But the box asks for a window only after its own nightly
off-site run, and that runs at 04:15 CEST. So the first real 0.19.1 prune comes **2026-10-19 at about 02:16 UTC**
(04:16 Budapest). Read at 14:33 UTC: no `[offbox]` line since the 11:09 restart, the scheduler's next run 2026-10-19
04:15 CEST, the hub's last window still id 9 (`P8/cleanup-window-not-yet.txt`, the hub DB read with its WAL). CC did not
grant a window by hand (that is the operator's). The STATUS item stays open, with the corrected time; the check script
is ready (`rx.sh` + a wrapper that lists and restores the newest, the oldest and the newest 0.14-made snapshot with both
restics and compares tree hashes).
## 8. Versions, commits, fixtures
- Hub **0.158.0** deployed (felhom.eu `4c319dde`).
- Controller **0.330.0** — `6c6bd87`; floor 0.330.0 / MinAgent 0.158.1 at 11:10:59 UTC (303 `floor_set`, read back with
the WAL); per-box floors set and cleared; box walk green on the three test boxes (0 refusals, all healthy, login and
files 200, https-only OK).
- Agent **0.159.2** — `65c1994` (code), tag `v0.159.2`, CHANGELOG `e25454c`.
- Installer **1.34.0** — felhom.eu `88aecf55` (vault-first + harness), `624bcdc3` (1.34.0, tagged), `3332eb09` (pins).
- Estimator — `37cb83c1`. Fixtures: none changed (the new texts travel in JSON).
## 9. Register — 173 before, 168 after; opened 1, closed 6
Closed: R-1140, R-1144, R-1145, R-1146, R-1147, R-1036. Opened: R-1148. Narrowed: R-895 (dated check 2026-10-20).
## 10. Observations
1. Installer 1.34.0's default path has not run in a fresh install. FILED: R-1148.
2. The review's F2 (box-reported run times explain new snapshots). NOT-A-FINDING as a new row: a known design limit,
recorded in R-895.
3. `release-agent.sh` vs the `release-complete` hook: the CHANGELOG must follow the release. NOT-A-FINDING: the documented
order (memory and skill); I tried the other order first and the hook refused it, as designed.
4. The estimator's new Immich figure is `memory.current` (with cache), not the `used` measure it normally takes.
NOT-A-FINDING: on the safe side, said in SIZING.md, and approved by the operator with the table in front of them.
5. A safety check refused a `sh -c` with an unguarded `rm "$H"`. NOT-A-FINDING: rewritten with `${H:?}` in a helper.
## Teardown — three layers
- **Machines:** demo-hp 9201 — one whole-guest backup taken (kept, like any backup); no app installed or removed today.
demo-hp host — the installer copy shredded after the rotation; root password rotated on purpose (new value in the
vault). Tester 1, demo-felhom — agent 0.159.2 and controller 0.330.0 through their normal paths; nothing else.
- **Hub:** hub 0.158.0; global floor 0.330.0 / 0.158.1; no per-box floor; signed jobs consumed. No vouch by CC.
- DooPlex: builds only; ArgoCD synced two Deployments (hub, webpage), each alone.
Instruction files: none edited. System poster: none of the three (the poster shows backup retentions; the hub's access
log is not on it). Architecture documents read: `02`, `03`, `05`, `07`, `09` §3 decision 250; `break-glass.md`.
+156
View File
@@ -0,0 +1,156 @@
# REPORT — day 2026-10-19
The operator answered the two STOPs in advance („YES to the agent and the hub release") and earlier: golden 0.331.0
vouched; D16 (one sign-in) yes; D17 „one per day, 3 days". No question was asked during the session.
## In plain words
- **Agent 0.160.0 runs on all three test hosts.** A new recovery code replaces the old one only after the household types
two of its words back; a broken-into box cannot flood the whole-box backups; the local copies are one per day for 3
days; an empty partition does not ask „data will be erased". **It is not vouched — that is yours.**
- **One sign-in for the file manager is live on every box** (controller 0.332.0, the floor). „Fájlok ↗" opens the app's
folder with no second login. A sign-out on the dashboard asks again. No app on the box can pretend to be the household.
- **Hub 0.159.0 is live.** A refused backup has its own alarm text. A reinstall over an old host record now mails you; it
does not re-issue the codes by itself (a security review stopped the first draft that did).
- **The preview error (R-1153) was a broken test file**, not the box.
- **The old backup program still reads demo-hp's off-site copies** after the first real clean-up with the new one.
## 1. Agent 0.160.0 (R-978, R-973, R-1093, R-1149, R-1151) — `audits/day-2026-10-19/P1/`
- **Content:** R-978 (an abandoned second ceremony keeps the first code), R-973 (the cadence cap: daily tier 3 per 24 h,
off-site 1 per 6 days), R-1093 (an empty partition is not data), **R-1149** (local tier `keep-daily=N`, operator D17),
**R-1151** (the package uploads only at `POST /escrow/ceremony/confirm` with two typed words; a claim without
`defer_upload` keeps the R-978 order, so an old controller still completes a ceremony). Commits `1b6e316`, `c11e3e2`,
`124164d` (CHANGELOG), `4545819` (two texts, after the release, unreleased). Released with `release-agent.sh`: sha256
`7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1`, bundle
`6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f`, round trip verified, tag `v0.160.0`.
- **Red-proofs:** keep-last for the primary → „primary tier prune = "keep-last=3"" and „holds 0 copies made before the
compromise"; the claim's defer flag ignored → „a code that was shown but never typed back replaced the package: code 1
no longer opens it (uploads=2)" (all three abandon paths); the word check skipped → „one wrong word: got 200".
- **demo-hp proof** (signed jobs 01:22 UTC, running 01:23, „update committed" 01:24; probe 75/75):
- a normal „Mentés most" of 9201: green, 6.9 GB in 443 s; yesterday's copy pruned (keep-daily=1, the same copy keep-last
would prune); the cap's history on disk holds both successes;
- escrow through the wizard's own calls: ceremony 1 — one wrong word 422, the right words 200; code 1 opens the hub's
package, code 0 is a kept earlier one (422, superseded at the confirm second 01:47:47). Ceremony 2 shown and abandoned —
code 1 still opens it (200), code 2 opens nothing (400). demo-hp's stored code is now code 1 (CC's secret store);
- R-1093 on the real route: a 64 MB RAM disk (`scsi_debug`, `/dev/sdb`), one empty GPT partition → `POST /disks/format`
→ 200 formatted, no confirmation; control: the same partition, now ext4 → 403 `needs_confirmation`. RAM disk removed.
**A loop device cannot be used:** the route and the probe accept only sd/nvme/vd/hd names (by design).
- **The answer:** the operator's „yes" (in advance). Tester 1 and demo-felhom took 0.160.0 by signed jobs (01:52 UTC →
01:54 / 01:55, probe 75/75, `keep-daily=3` armed on both); the hub's host rows read 0.160.0 for all three, and fresh
reports after the hub restart. **Old controller with the new agent:** pinned by `TestEscrowCeremony_R1151_OldControllerKeepsTheClaimOrder`; **new controller with agent 0.159.2:** pinned by `TestR1151_OldAgentClaimCompletesAtOnce` and the agentapi test — no host runs 0.159.2 any more, so not walked live.
## 2. One sign-in (R-1100) — `audits/day-2026-10-19/P2/`, `P4/05-*`, `P4/06-*`
- **Built** exactly the design note's recommendation (controller `9bdd8f8`, `601bdba`): the dashboard pass, the host-only
files cookie (derived from the session fingerprint, the R-976 shape), the per-box secret header name, the forwardAuth
answer; header user `felhom-household` with the household's file rights from userDefaults; `logoutRedirectUrl` = the
dashboard; `admin` stays a password user; the data tab's „asks for its own password" note only without the sign-in.
- **Tests, each red-proved:** replayed pass, expired pass, pass for another host (two guards: the host check and the
host-bound session key — red only with both removed), wrong X-Forwarded-Host, no session → no pass, signed out → asks
again (and after a password change), cross-site → no pass, dot-segments in `to`, the absolute redirect, the config
trusts only the secret name, the header user's rights, the route asks the controller and strips the header, the note.
- **Live, found and fixed before release:** on rc.2 the pass answer's relative redirect became
`http://felhom-controller:8080/files/` — traefik resolves a forwardAuth answer's Location against the auth server. Fixed
(absolute), red-proved, rc.3. A background security review then flagged CSRF and revocation: `/__files/open` now refuses
a cross-site request and dot-segments; the revocation residual is R-1155.
- **Live proof (rc.3, 9201, headless Chrome through the public names mapped to the box):** signed in → calibre-web's
„Fájlok ↗" → FileBrowser at `NVME 1TB/media/books`, no login form, user `felhom-household` (admin false, modify true,
login method proxy); signed out → `files.<domain>` went to the dashboard login. Screenshots `browser/01-…`, `02-…`,
`03-…`. Forged headers (`forged-headers.txt`): through traefik with no cookie — the old public name, a random name and the
REAL secret name all 401; from the bookstack container straight to `filebrowser:80` — the old name and a random name 401;
control: the real name from the same container 200 (the protection is the name's secrecy). The brief named Nextcloud;
bookstack was used (an app on 9201's network with curl).
## 3. Preview check (R-1153) — `audits/day-2026-10-19/P3/`
On 9202 (controller 0.331.0): OnlyOffice installed, the link on (FileBrowser back in 4 s). The hand-made `teszt.docx`
(912 bytes) → 500 „fitz: cannot open document"; OnlyOffice's own `sample.docx` → **200, a 256×256 JPEG, 0.3 s**. Same with
the link OFF. FileBrowser 1.5.6 previews .docx with its own MuPDF before OnlyOffice (`preview.go` `determinePreviewType`).
**Closed: not a product defect.** OnlyOffice removed through the product (both volumes gone), test files deleted.
## 4. Controller release 0.332.0 — `audits/day-2026-10-19/P4/`
Image `0.332.0` from `601bdba` (digest `sha256:245266a4…`), 03:16 UTC. Per-box floors 03:16:43 → Tester 1 and demo-felhom
on 0.332.0 at 03:16:45–55, demo-hp (from rc.3) at 03:17. Box walk on all three: 0 unhealthy, 0 allow-list refusals,
FileBrowser re-published 6 s after its re-create, public pages 200; https-only 28 OK, 0 failures, both controls right.
„Fájlok ↗" with no login through the public internet on demo-hp and Tester 1 (pass → cookie → `felhom-household`; a fresh
browser 401; sign-out → back to the dashboard). demo-felhom: CC holds no dashboard password for the lived-in box, so its
check is the files (secret 0600, config and route carry it) and the refusals (forged secret name through traefik 401, old
name from outside 401, a forged pass 403). **Global floor 0.332.0 / MinAgent 0.158.1 at 03:24:18 UTC**, read back with the
WAL; per-box floors cleared (read back empty). MinAgent 0.158.1 ≤ the vouched agent 0.159.2. i18n fixtures changed (named):
`escrow_first`, `escrow_receremony`; all others byte-identical. No golden (decision 250); the floor is above golden 0.331.0
until Sunday's bake.
## 5. Hub 0.159.0 (R-1150, R-1154) — `audits/day-2026-10-19/P5/`
- Written by a helper (`5121536f`, `b505ce4b`). **R-1154's rule, and why:** the helper's first draft re-issued the claim
code and the off-site provider password automatically when the old host was stale or switched off. A background
security review flagged it; the helper then read what the retrieval passphrase already grants (`/config` → the
controller API key; the enroll → the host key) and found the re-issues are the operator's session-only powers
(decision 252). So the passphrase must not reach them: **the reuse path re-issues nothing; when the old box looks gone it
mails the operator** („delete the old host and run the installer again, or re-issue …"), a live box gets an info record.
The operator's delete stays the gate. **R-1150:** `whole_guest_backup_refused` (operator-only, hub-origin; NOT in
`allowedEventTypes`, against the brief's wording — the suspected controller must not post or pre-empt it, R-895's
shape). Tests red-proved (helper's list in `hub/CHANGELOG.md`).
- **A push blocker on the way:** R-887's dated check was due today. Measured: 381 CI jobs since the S21 change, 0 lost (the
three re-runs' first attempts ran 47–49 s; control: the known lost attempt of 2026-10-10 ran 11 min 48 s and the attempt
count finds it). Closed.
- **Deploy:** image from a clean clone (the shared checkout held my uncommitted evidence), manifest `200ab653`, ArgoCD
synced the hub Deployment only (the only OutOfSync resource), Synced/Healthy, no ERROR, the floor read back, then fresh
reports from all three hosts (02:23–02:25 UTC) and all three controllers (02:29–02:31 UTC).
## 6. The old-restic check — `audits/evening-2026-10-18/P8/old-restic-check.txt`
The night watcher ran it: window 12 at 02:18 UTC removed 63 of demo-hp's 324 copies (→ 261); the old restic (0.14.0) and
the new (0.19.1) both list 261 and restore a 0.19.1-made copy and an August copy to identical content hashes. STATUS item
closed; `REPORT-evening-2026-10-18.md` §9 filled.
## 7. Versions, commits, fixtures, rows
- Agent **0.160.0** (released, not vouched) — `1b6e316`, `c11e3e2`, `124164d`, `4545819`. Controller **0.332.0** (floor) —
`ba4a6f0`, `9bdd8f8`, `601bdba`; rc.1–rc.3 on 9201 only. Hub **0.159.0** (deployed) — `5121536f`, `b505ce4b`, manifest
`200ab653`. felhom.eu `afc939bb` (R-887), `5622ce27`, this commit. Catalog: none.
- Fixtures: `escrow_first`, `escrow_receremony` (controller); no other page changed.
- **Register: 168 before, 166 after; closed 10** (R-887, R-978, R-973, R-1093, R-1149, R-1151, R-1100, R-1153, R-1150,
R-1154), **opened 8** (R-1155 … R-1162).
- `unproven.py --summary`: not walked 35 of 55 — unchanged. Capability map: not changed. System poster: text only — the
local whole-guest „Kept" cell („one copy per day, up to 3 days") edited in both the facts file and the drawing.
## Observations
1. FileBrowser's own 2-hour token can outlive the dashboard session if only the files cookie is lost. FILED: R-1155.
2. A reinstall within 45 min of the old box's last report reads „live" (no mail). FILED: R-1156.
3. Repeated cap refusals inside the 24 h hold raise no new alarm. FILED: R-1157.
4. The off-site leg of the clean-slate re-issue may be obsolete since hub v0.127.0. FILED: R-1158.
5. The retrieval passphrase yields the controller API key, which registers and confirms off-site keys. FILED: R-1159.
6. Hub and agent have no gofmt gate (~20 and 14 files drift). FILED: R-1160.
7. The escrow wizard calls the code „live" at the reveal; since R-1151 it is live only after the typed-back words.
FILED: R-1161.
8. R-973's off-site half (ep0 prunes by count) outlives the closed row. FILED: R-1162.
9. traefik resolves a forwardAuth answer's relative Location against the auth server. NOT-A-FINDING: found on rc.2 and
fixed before the release (red-proved); recorded as a rule in CONTEXT and memory.
10. The ceremony log line and two R-973 comments were stale. NOT-A-FINDING: fixed in agent `4545819` (the size rule).
11. My first OnlyOffice link call sent `{}` and switched the link off. NOT-A-FINDING: my call; the API wants
`{"enabled": true}`; it was already off, nothing restarted.
12. The R-1093 route refuses loop devices. NOT-A-FINDING: by design (only real disk names are allowed); a RAM disk was used.
13. CC holds no dashboard password for demo-felhom, so its „Fájlok ↗" was not walked as a household. NOT-A-FINDING: the
lived-in box's household password is not CC's (decision 254); its files and refusals were checked instead.
14. A helper's commits sat unpushed in the shared felhom.eu clone for ~40 minutes (blocked by R-887's due check).
NOT-A-FINDING: the gate did its job; pushed with my R-887 commit.
15. Two shell scripts of mine were refused by Claude Code's removal check (a cleanup `rm` it could not read behind
`bash -c`). NOT-A-FINDING: rewritten without the removal; nothing was deleted.
## Teardown — three layers
- **Machines:** 9202 — OnlyOffice removed through the product (volumes gone), test documents deleted; 9201 — on 0.332.0
(the floor); the escrow test left demo-hp on code 1 (stored); demo-hp host — the RAM disk module removed, helper scripts
deleted, no file left in `/root` (an older helper, `/root/opens.sh`, removed by name at the end). Cookie jars and
password files on DooPlex shredded or emptied. One debugging sign-in on demo-hp's dashboard (curl, 03:03 UTC) was not
signed out; its cookie file is shredded, so the session expires unused.
- **Hub:** global floor 0.332.0 / 0.158.1, no per-box floor; hub 0.159.0; no vouch.
- **DooPlex:** builds (agent, controller rc.1–3 + 0.332.0, hub 0.159.0 from a clean clone) and the hub deploy; no Docker
cleanup.
Architecture read: `01`, `02` „Logins and folders", `03` §8a, `04`, `05`, `07` §6/§6.1; the design note. Documents changed:
`02`, `03` §8a, `07` §6.1, the poster facts + drawing text, `05` §8 (helper). Instruction files: none changed.
+120
View File
@@ -0,0 +1,120 @@
# REPORT — second day brief, 2026-10-19 (ep0 one copy per week; the passphrase measured; hub 0.160.0; controller 0.333.0; gofmt gates)
## In plain words
- **ep0 keeps one whole-box copy per week, for 2 weeks** (your „yes"). Before the change, PBS's own dry run showed it
removes nothing today. The first real run is 2026-10-20 03:30 UTC; CC reads it then (row R-1162 stays open until then).
- **The household passphrase:** alone it gets the box's keys, and with them every off-site key action. Two options are
in STATUS (D19). CC picks A: the passphrase gives keys only while the customer has no box record.
- **Controller 0.333.0 is the floor for every box** (12:16 local): a dashboard sign-out now also ends the file manager's
own 2-hour pass; the recovery-code page says the code becomes live after the two words.
- **Hub 0.160.0 is live** (your „yes"): a reinstall within 45 minutes mails you too; a repeated refused backup is counted
and the alarm repeats at most once a day; the old off-site re-issue on a reinstall is kept, with true words.
- **gofmt:** the hub and the agent are formatted, and both now have the controller's gate.
- **New for you:** another device keeps taking demo-hp's network address (R-1166) — fix it in the router.
## 1. ep0 (R-1162; `07` Plane-2 row and §6.1)
- **Before** (`audits/day-2026-10-19b/P1/ep0-settings-before.txt`): `prune-demo-hp` and `prune-demo-felhom` —
`keep-last 2`, 03:30, ns-scoped, `max-depth 0`; `prune-operator-hubdb` — keep-daily 14 + keep-weekly 8, 03:45.
`tester-1` (3 copies) and `Tester-2` (1) have **no job**. ep0's clock is UTC.
- **DooPlex's copy of ep0:** `prune-ep0-copy`, `keep-weekly 8`, 07:30 (decision 71) — already by time; not changed.
- **Dry run** (PBS's own: `proxmox-backup-debug api create /admin/datastore/felhom-offsite/prune --dry-run true …`, no
task created; cross-checked with an offline model of PBS's selection, `P1/prune_sim.py`; 8 snapshots before and after):
| Box | Snapshots | keep-last 2 removes | keep-weekly 2 keeps | keep-weekly 2 removes |
|---|---|---|---|---|
| demo-hp | 2 | none | 2026-10-08, 2026-10-15 | none |
| demo-felhom | 2 | none | 2026-10-06, 2026-10-13 | none |
| tester-1 (no job) | 3 | (2026-10-04) | 2026-10-11, 2026-10-18 | (2026-10-04) |
| Tester-2 (no job) | 1 | none | 2026-10-04 | none |
- **Answer:** „Yes, the two rules" (not the tester-1 job). **After** (07:49 UTC, `P1/ep0-settings-after.txt`): both jobs
`keep-weekly 2`, schedule 03:30 unchanged, nothing run by hand. Check for 2026-10-20: the 03:30 `prunejob` tasks OK with
`keep-weekly 2` and two snapshots per namespace (the next step of R-1162).
## 2. The household passphrase (R-1159)
Measured with the hub's real router (`P2/passphrase-reach.txt`; a made-up customer, a fake off-site service). A live
measurement on Tester4 was not possible: CC holds no Tester4 passphrase, and reading it from the hub database was
refused by CC's own safety check (correctly).
- Passphrase → `GET /config/{id}` 200 with `hub.api_key` (wrong passphrase 401); `POST /host-enroll` 200 with the
EXISTING host's key.
- Controller key → `register-key`, `confirm-key` (removes every other line, the box's own key included), `move-aside`,
`window-open`, `abandon-request` (the hub deletes a set-aside copy after 7 days unless cancelled), `abandon-cancel`,
`abandon-status`: all past auth (wrong key 401 each).
- Host key → escrow GET/PUT, recovery-credential PUT, wg, desired-state, jobs, host-report: all past auth.
- The passphrase is used only at install time (the installer, the controller's setup wizard).
- **Option A (CC's pick):** keys only while the customer has no host record (`/config` leaves out the key, `/host-enroll`
refuses once a host exists). Closes the chain at its source; a reinstall needs the operator's delete first — already the
rule since R-1154. **Option B:** a second factor on the destructive off-site routes. Every reinstall waits for the
operator, and the host key stays reachable by the passphrase, so B alone does not close it. No change in this task.
## 3. Controller 0.333.0 — R-1155, R-1161, the Install chip
- **R-1155:** first measured on demo-hp (0.332.0): the household's FileBrowser token opened FileBrowser after the
dashboard sign-out (200, cookie and Bearer). Fix: the forwardAuth answer refuses a request with no valid files cookie
that carries a non-admin FileBrowser token, everywhere FileBrowser reads one (cookie, `?auth=`, Bearer, Basic
password). **The commit's security review found Basic missing** in the first version (`c809dd3`); fixed in `a6fa3c0`
before any release. Red-proofs: without the check → 200 after the sign-out; without the Basic branch → 200; with every
token refused → admin's own token 302. Live (rc.2 demo-hp, 0.333.0 Tester 1): files cookie 200; token alone 401 in all
four forms, before and after the sign-out; FileBrowser's sign-in page 200.
- **R-1161:** new key `backups_escrow.ez_a_kod_az_ellenorzes` (hu + en); the old sentence only when the claim answer says
no confirm. Fixtures re-captured: `escrow_first`, `escrow_receremony` (reason: R-1151 made the old sentence untrue);
all others byte-identical. Live on demo-hp (`P6/06-escrow-page-text.txt`): new text shown, old hidden, switch present,
negative control 0.
- **The Chrome „Install" chip: NOT-A-FINDING.** FileBrowser 1.5.6 links `/public/static/site.webmanifest` on every page;
its only PWA setting is `disablePWAInstall` (already set). Not patched (`P3/install-chip.txt`).
## 4. Hub 0.160.0
- **R-1156:** a live re-enroll is a warning mailed to the operator, same instruction; a `--resume` mails too and says it
needs nothing. **R-1157:** further refusals in one episode counted (`refusals_in_episode`), alarm at most once a day,
total on the cleared record. **R-1158:** the reinstall's off-site re-issue changes nothing a box uses (only the hub's
own registrar reads that password; no box has received it since v0.127.0); kept because it repairs a broken stored
login; comments and the `offsite_reissued` text corrected (`05` §8). Each red-proved.
- **Answer:** „Yes, deploy now". Image `0.160.0` digest `sha256:e9a360bc00d00a15014e272c4d023fef118a8b043956f3fc928dd26c00b34230`
(clean clone at `a17db0c3`; the first push hit a DNS timeout, the retry pushed it). Manifest `5d3cb5c1`; only
Deployment/hub was OutOfSync and only it was synced; Synced/Healthy; running image digest = built digest; 0 `[ERROR]`
lines; floor read back from hub.db + WAL (0.332.0 then, 0.333.0 now); host and controller reports from demo-hp, Tester 1
and demo-felhom after the restart.
## 5. gofmt (R-1160)
Hub `d5e3c8b0` (22 files) and agent `1edfd8e` (14 files), formatting only, tests green before each. Gates: `hub-gofmt`
(`scripts/hub_gofmt_gate.py`, `repo_gates.py`) and `gofmt` (felhom-agent `scripts/gofmt_gate.py`, `agent_gates.py`), fast
set, decoys on scratch trees, red-proved. No agent release.
## 6. Controller release (decision 250)
Per-box floors 0.333.0 at 09:59:38 UTC → all three on 0.333.0 within about a minute (demo-hp from rc.2 at 10:13). Box
walk (`P6/`): 0 unhealthy, 0 allow-list refusals, catalog in sync; https-only 28 OK, 0 failures; „Fájlok ↗" with no second
login on demo-hp and Tester 1, sign-out asks again. **Global floor 0.333.0 / MinAgent 0.158.1 at 10:16:22 UTC**
(303 `floor_set`, read back with the WAL); per-box floors cleared.
## 7. Versions, commits, fixtures, rows
- **Controller 0.333.0** (`c809dd3`, `a6fa3c0`, REPORT `e1a2abf`), image digest
`sha256:f150a8d8cd329f5cde9e34249e34edc59b6b5db29db1e3200900f3869a0de11b`; rc.1/rc.2 on demo-hp only. Fixtures:
`escrow_first`, `escrow_receremony`.
- **Hub 0.160.0** (`d5e3c8b0`, `503c93fa`, `a17db0c3`, manifest `5d3cb5c1`).
- **Agent:** no release (`1edfd8e`, `0cbd500`). 0.160.0 stays the vouched one.
- **Rows: 167 before, 163 after; closed 6** (R-1155, R-1156, R-1157, R-1158, R-1160, R-1161), **opened 2** (R-1165,
R-1166); R-1159 → READY (D19), R-1162 → NARROWED (one run to read).
- **System poster:** text only — `felhom-system-poster.facts.md` and the matching text in `felhom-system-poster.html`
(„last 2" → „one per week, 2 weeks").
- **Teardown:** machine — rc images only on demo-hp 9201, now on the release; no test app. Host — none. Hub — no test
customer; the per-box floors cleared. ep0 — the two prune jobs changed on the operator's yes, nothing else.
## Observations
1. Two household namespaces on ep0 have no prune job; nothing creates one for a new household. FILED: R-1165.
2. Another device takes demo-hp's address 192.168.0.104 every few minutes; demo-hp also lost its link to ep0 for 8 minutes. FILED: R-1166.
3. A config download with the passphrase is a log line only, and the host key it yields can rewrite the box's escrow and recovery credential. FILED: R-1159.
4. The first R-1155 version missed FileBrowser's Basic-auth token source (the commit's security review). NOT-A-FINDING: fixed before any release in `a6fa3c0` and red-proved (the size rule).
5. Chrome offers „Install" for the file manager. NOT-A-FINDING: FileBrowser 1.5.6 has no setting that stops its manifest; the brief forbids patching it.
6. One of CC's probe scripts on DooPlex passed the household's FileBrowser token to curl on its command line (visible to local processes for about a second; never printed or saved). NOT-A-FINDING: CC's own tooling on DooPlex, not the product; every probe now passes tokens through 0600 header files, and the token expired two hours later.
7. Two copies of CC's hub-database snapshot ran into one folder at once and one read failed („malformed"). NOT-A-FINDING: CC's helper, a local copy only; the live hub and its database were untouched, and the next read was clean.
8. The first hub image push failed on a DNS timeout at DooPlex. NOT-A-FINDING: a short DNS outage (the website session saw it too); the build's own check found the tag absent and the retry pushed it.
9. FileBrowser's `admin` password login was not walked live. NOT-A-FINDING: CC holds no FileBrowser admin password; admin's token passing is unit-pinned and the sign-in page answers 200 live.
+104
View File
@@ -0,0 +1,104 @@
# REPORT — third day brief, 2026-10-19 (the passphrase locks once the box uses its own key; every ep0 household gets a prune job; demo-hp's address)
## In plain words
- **The household passphrase no longer gives a live box's keys** (your D19). Each box now fetches its config with its own
key. The passphrase still works for a new install. A reinstall over a live box needs you to delete the old host first.
- **The brief's first shape could not be built as written.** "Keys only before a box exists" would have broken every
install and every live box's config updates: both use the passphrase after the box record exists. CC stopped and asked;
you chose "box key first, then lock".
- **ep0:** every new household gets its weekly clean-up rule automatically, and the hub alarms on a household without
one. tester-1 and Tester-2 got theirs (your yes).
- **demo-hp's address** stayed demo-hp's for a 30-minute watch after your router change.
- **Live now:** hub 0.161.0, controller 0.334.0 (the floor), tenant-sync v1.3.0 on ep0.
## 1. The passphrase (R-1159; `05` §8 [DESIGN R-1159])
- **Install-order proof** (read before building, the brief's STOP): the passphrase is used AFTER `POST /host-enroll`:
- `scripts/felhom-host-install.sh:2631` → `:480`, and `:3525`: `step_agent_install` and `step_golden` fetch the
download login from `GET /config`;
- `felhom-agent/cmd/felhom-agent/main.go:2835`: the provision bakes the passphrase into the guest's `bootstrap.json`;
- `felhom-controller/controller/internal/bootstrap/bootstrap.go:142`: the guest's first boot pulls `GET /config` with it
(that is how the box gets its own key);
- `cmd/controller/main.go:1176` (`report.ConfigRefresher`): every live box re-pulled with it on each config change;
- `:2317`: the preflight checks it with `/config` on every run.
- **Your answer:** „Box key first, then lock".
- **Built:**
- Hub 0.161.0: `GET /box-config/{id}` (the controller's own key) and `passphraseKeysClosed`. Passphrase keys are
refused (409, no key, operator event `passphrase_keys_refused`) once a ≥ 0.334.0 controller has reported since the
host record was made. `/config`, `/recovery` and the `/host-enroll` reuse path are covered; `/artifacts` has no key
and is unchanged.
- Controller 0.334.0: the refresh pulls with its own key (`report.PullConfigPreferKey`). The passphrase is used only
against an older hub or before the box has a key, and never as a retry after a refused key.
- **Tests:**
- Hub: `TestR1159_PassphraseGivesKeysOnlyUntilTheBoxUsesItsOwn` (5 cases), `TestR1159_BoxPullsItsConfigWithItsOwnKey`,
`TestR1159_BoxConfigMinControllerIsTheReleaseThatPullsWithItsKey`.
- Controller: `TestR1159_RefreshUsesTheBoxKey`, `TestR1159_MainWiresTheKeyFirstRefresh`.
- Red-proofs: lock off → all three keys served; the old pull → 0 key pulls.
- **Live on demo-hp** (`audits/day-2026-10-19c/P4/`; the probe ran on the host from the box's own `bootstrap.json`,
printing status codes only):
- on 0.333.0, the passphrase → 200;
- after its first 0.334.0 report (14:30:13 UTC), the passphrase → **409, no key**, and `passphrase_keys_refused` was
raised (the operator got the mail — CC's probe);
- its own key → 200; a wrong key → 401.
## 2. ep0 prune jobs (R-1165; `07` Plane-2 row)
- **Automatic:** `felhom-tenantsync` v1.3.0 creates `prune-<id>` (keep-weekly 2, 03:30, max-depth 0) with a household's
namespace unless a job already names it; it removes the job with the household. Its `usage` answer lists namespaces no
job covers.
- The hub alarms on that list (`pbsdr_namespace_without_prune_job`). An unread list never reads as "all covered".
- Tests: 12 script cases (`scripts/test_tenantsync_prune.py`, run on every push), the checker and the client; each
red-proved.
- **Dry run before the question** (PBS's own, `P2/ep0-dry-run-tester.txt`; 32 snapshots before and after):
- tester-1: keep 2026-10-11, 2026-10-18; remove 2026-10-04.
- Tester-2: keep 2026-10-04.
- **Answer:** „Yes, add both". **After** (13:35 UTC, `P2/ep0-jobs-after.txt`): `prune-tester-1` and `prune-Tester-2`
(keep-weekly 2, 03:30) beside the demo-hp, demo-felhom and operator jobs.
- **Script on ep0** (14:07 UTC, the same yes): the installed v1.2.0 matched the repo byte for byte; v1.3.0 is installed
(hash = the repo's `36412c6a…`), the old copy kept as `.bak-1.2.0`. Live `usage` → `unpruned: []`; `fingerprint` still
answers.
## 3. demo-hp's address (R-1166)
- A watch every 10 s for 30 minutes (10:59–11:29 UTC): **180 samples, 0 with another MAC**, SSH open throughout.
- The host's own MAC (`cat /sys/class/net/vmbr0/address`): **`7c:d3:0a:77:d9:76`** — the same as your router entry.
- Note: your DHCP pool is .120–.160, so .104 was never in it. The other device (`b8:06:0d:60:45:58`) most likely had .104
set by hand. If it comes back, look there.
## 4. Hub deploy and the controller release
- **Hub 0.161.0**
- Image digest: `sha256:d56c96a59c86ce2d006938d0980803a12949bcbf027cea2834cabd64118838b8` (clean clone at `b6a911c3`).
- Manifest `97f27a75`; only Deployment/hub was synced. Result: Synced/Healthy; the running digest is the built one;
0 `[ERROR]` lines; the ep0 read at 14:13 UTC was OK, with no alarm (every namespace has a job).
- Checks: the floor was read back; host and controller reports from demo-hp, Tester 1 and demo-felhom arrived after
the restart.
- **Controller 0.334.0**
- Image digest: `sha256:deeabe59d932cd60f8939a37bccf94f145555101fc7be64b6c5cc15f3557031a` (from `546eef3`); rc.1 ran
on demo-hp only.
- Per-box floors at 14:16:11 UTC. Tester 1 and demo-felhom moved at once; demo-hp moved at 14:30 (from rc.1).
- Box walk (`P5/`): 0 unhealthy, 0 refusals, catalog in sync; https-only 28 OK, 0 failures; „Fájlok ↗" with no second
login on demo-hp and Tester 1.
- **Global floor 0.334.0 / MinAgent 0.158.1 at 14:31:44 UTC.** The per-box floors were cleared.
## 5. Versions, commits, rows
- **Hub 0.161.0** (`b6a911c3`, manifest `97f27a75`); **controller 0.334.0** (`546eef3`); **tenantsync v1.3.0** (in
`b6a911c3`, on ep0). No agent change. No i18n fixture changed (no page changed).
- **Rows: 163 before, 161 after; closed 3** (R-1159, R-1165, R-1166), **opened 1** (R-1167).
- **System poster:** facts file only (`felhom-system-poster.facts.md`: new households get their prune job); the drawing's
text („one per week, 2 weeks") is still true.
- **Teardown:**
- Machine: rc.1 only on demo-hp 9201, now on the release; the probe scripts removed from demo-hp.
- Host: none.
- Hub: per-box floors cleared; one `passphrase_keys_refused` event on demo-hp (CC's probe).
- ep0: two jobs added and the script replaced, both on the operator's yes; nothing else.
## Observations
1. The installer prints „unexpected hub status 409" instead of the hub's sentence when a re-run meets a live box. FILED: R-1167.
2. The brief's „keys only while no host exists" would have broken every install and every live box's config refresh. NOT-A-FINDING: measured before building and put to the operator, who chose the two-step shape (D19); recorded in `05` §8.
3. A reinstall over a live box now gets two operator mails (R-1154's reinstall warning and the refusal). NOT-A-FINDING: both are true and give the same instruction; the 1-hour cooldown limits repeats.
4. An rc build (`0.334.0-rc.N`) counts as older than 0.334.0, so the lock stays open on an rc. NOT-A-FINDING: by design (an rc runs on demo-hp only, and the floor moves it to the release); proven on rc.1 (200) and the release (409).
5. demo-hp's .104 lies outside the router's DHCP pool, yet another MAC held it this morning. NOT-A-FINDING: the router reservation fixed it (30 minutes clean); the other device's source is noted in R-1166's closing text in case it returns.
+153
View File
@@ -0,0 +1,153 @@
# REPORT — evening → night 2026-10-18
## How I read the brief
- **Part 5, R-1135:** the brief says the mail should use the public Gitea URL — the register row says the same, and that
is what was done (the CI helper first read the brief's wording the other way round and followed the row).
- **R-1052** was already closed (2026-10-15); Part 1 proves R-1148 and re-proves R-1052's path with tonight's golden.
- **Part 6, Immich:** 9202's Immich lives on `/mnt/sizing-bench`, a hand-made bench folder outside `/mnt/felhom-drives/`,
so the allow-list refuses its definition (see Observations); the restore half of the measurement ran on Nextcloud.
- **The operator kept Tester4** (asked at 21:20): not deleted; its old host must go before its next install (R-1154).
## In plain words
- **The morning list:** vouch golden 0.331.0; say yes or no to the agent release with three fixes; two decisions (one
sign-in, local backup retention).
- **A brand-new box from nothing works end to end** with installer 1.34.0: emergency password saved in the hub first,
then set, then the web login took it; the approved golden and agent; the box moved itself to the floor.
- **OnlyOffice now really works** from FileBrowser and from Nextcloud. Before tonight FileBrowser could not open a
document and Nextcloud lost every save while saying „saved". Fixed in controller 0.331.0, now the floor for every box.
- **One sign-in for the file manager can be built safely**, but only with two safeguards — measured, design note written.
- **Three agent fixes are written and tested, not released.**
- **CI:** the failure mail's link opens from outside; the 5-minute CI timeout never worked (filed); a rerun hides the
first try's log, and the check now looks for it.
- **Redis for Immich and Nextcloud has no disk volume any more**, after a test; the test found that removing an app left
an old volume behind — fixed too.
- **Golden 0.331.0 is baked** (not vouched).
- **The old-restic check:** §9.
## 1. The morning list
1. **Vouch golden 0.331.0** — sha256 `3881de62b0986ccbe5e55580e3f1bb9765d65b8d898f67c6339bac105244f84f` (registry round
trip = the bake), agent 0.159.2 unchanged, **Min agent 0.158.1**.
2. **Agent release (R-978, R-973, R-1093) — may I?** If yes: release, prove on demo-hp, ask for the vouch.
3. **Decide D16** (R-1100: build one sign-in with the two safeguards?) and **D17** (R-1149: local whole-box backups
„last 3" or „one per day, 3 days"?). STATUS has both.
4. Before Tester4's next install: delete its old host `Tester4-31fa0e` (Hosts page).
## 2. Fresh install from nothing (R-1148, R-1052) — `audits/newbox-2026-10-18/`
| UTC | Step | Result |
|---|---|---|
| 17:42:21 | Tester4 created by the operator | bind mail sent |
| 17:43:15 | VM 9302 `newbox-r1148` on demo-hp, ISO 1.29.0 (sha = published) | 8 GB / 4 cores / 120 GB on `/mnt/hdd_1` root |
| 17:50:19 | Install pressed (graphical, Hungarian keyboard kept) | installed, rebooted, pairing code on the console |
| 18:02:58 | bind: mailed link + pairing code + owner phrase (operator's 0600 file, shredded) | „Sikeres összekötés" |
| 18:03:43 | step 4b: vault → chpasswd → PVE login | `[OK] … vaulted, set, and the PVE login takes it`; hub `set_at` same second |
| 18:03:44 | step 5: the hub manifest | agent **0.159.2**, golden **0.329.0** — the vouched pair, both sha-verified |
| 18:05:37 → 18:05:51 | first controller reports | 0.329.0, then **0.330.0** (`initiated_by: auto-floor` at 18:05:38) — 2 min 47 s after the bind |
| 18:17:14 | CC's PVE login from DooPlex | vaulted value **200**, installer value **401** |
| 18:18:32 → 18:20:05 | claim (setup code mail), BentoPDF | claimed; BentoPDF healthy |
| 18:24:15 | household „turn off" | hub `deliberate_off_at` set; VM stopped |
| — | teardown | name read twice, VM 9302 destroyed, disk gone; other VMs untouched; Tester4 kept (operator) |
## 3. OnlyOffice (R-1018) — `audits/onlyoffice-2026-10-18/`
On 0.330.0: FileBrowser → the editor loaded, the document never did (OnlyOffice fetched `https://files.<domain>` → 403);
Nextcloud → edit and „All changes saved", but the file kept the old text (`http://onlyoffice:80cache/…`). Switching the
link on/off left files.<domain> 502 for 4 min 8 s / 3 min 38 s. **Fixed (controller 0.331.0):** FileBrowser
`server.internalUrl`, Nextcloud's slash, the gate after a FileBrowser restart. On rc.1: both open, edit, save, read back
(„… RC1-FB-OK", „… RC1-NC-OK"); the stop revokes both (no editor; Nextcloud 404); the stop's outage one 2-s sample.
## 4. One sign-in spike (R-1100 → READY) — `audits/proxyauth-spike-2026-10-18/DESIGN-NOTE.md`
Traefik forwardAuth strips a client's header (any case, twice) and its own value wins — measured. But an app container
reaching FileBrowser directly logs in with a forged header (200), and the dashboard cookie never reaches `files.`. Verdict:
build only with a files-only ticket cookie and a per-box secret header name. Everything on 9202 removed (users, routes,
stand-in container, config).
## 5. Agent fixes (committed, NOT released) — felhom-agent `REPORT.md`
R-1093 `f7b7308` + `0963bae` (and a fail-open hole closed after a background review), R-978 `8561dc8`, R-973 `f17e197`;
tests and red-proofs for each; CI job 2062 success. R-1093 live: a loop disk on demo-hp showed the probe's condition holds.
## 6. CI (R-1135, R-1136, R-1137 closed)
Link public and by run id (forced mail 18:46:01Z, job 2067, one mail). The runner's real limit is 3 h; the job-level
5-minute timeout is not enforced (R-265 corrected; R-1152 for the step-level one). A rerun replaces the job's attempt; the
pull-check recipe in `CLAUDE.md` now reads `actions/tasks` (instruction-file edit by the CI helper: felhom.eu `CLAUDE.md`
CI recipe, item 3 added — a proven fact, stricter; `b9b56c47`).
## 7. Redis volumes (R-1117 closed) — `audits/redis-volumes-2026-10-18/`
(a) Both apps healthy on the new shape; Immich's metadata job ran (0 → 11, 0 failed); Nextcloud `cron.php` rc 0, WebDAV
200. (b) The dump skips the dropped volume (the old tar gone from the unit); a second-drive restore from a copy made WITH
the volume: 3/3 volumes, 1/1 database, healthy — it brings back that copy's compose (by design). (c) The old volume stays
attached under the tmpfs; **a removal left it behind** — fixed (controller 0.331.0, red-proofed). Templates changed
(catalog `30eea97`); Nextcloud on 9202 took it through the sync + restart (tmpfs, `save ""`).
## 8. Controller release and golden — `audits/evening-2026-10-18/P7/`, `tests/golden-0.331.0-2026-10-18/`
0.331.0 (digest `1b11d25a…`): per-box floors 20:44:00 → all three on 0.331.0 within 20 s; box walk green (0 refusals,
0 unhealthy, FileBrowser re-published in 5–6 s, public pages 200 after one 20-s blip each on two names, https-only 28 OK);
**global floor 0.331.0 / 0.158.1 at 20:48:56 UTC**, read back with the WAL; per-box floors cleared. Golden 0.331.0 baked
after it, round trip = bake sha, token checks 0 with a working control, teardown clean. No controller push after the bake.
## 9. The old-restic check (Part 8)
**Done 2026-10-19 by the watcher** (filled in by the day session, `audits/evening-2026-10-18/P8/old-restic-check.txt`):
demo-hp's clean-up window 12 ran at 02:17:53–02:18:24 UTC after the night run — „63 of 193 planned snapshot(s) removed,
324 -> 261" (the hub's row: `pruned`, 324 → 261). Then `rx.sh` (read-only): the new restic (0.19.1) and the OLD one (image
0.322.0, restic 0.14.0) both list **261**; both restore `c0b6299c` (made by 0.19.1, 02:17:48) to the same content hash
`d808e6537c7bcdb6` (7 files) and `05c3346a` (2026-08-31) to `8df241f4cf889265` (10 files). The old restic still reads and
restores demo-hp's store after its first real 0.19.1 prune. (Pre-check at 20:57 UTC: 310 and 310.)
## 10. Versions, commits, fixtures, rows
- Controller **0.331.0** — `772ef36`, `8ae9178` (+ `615940d` CI); rc.1 on 9202 only. Agent `main` `f7b7308` `8561dc8`
`0963bae` `f17e197` `79ba884` `1693da4` — unreleased. Catalog `c97eb1c`, `30eea97`. felhom.eu `1fe9f848`, `b9b56c47`,
this commit. Golden **0.331.0** baked. Hub unchanged (0.158.0).
- Fixtures: none changed — no page changed.
- **Register: 168 before, 168 after; closed 6** (R-1148, R-1018, R-1117, R-1135, R-1136, R-1137), **opened 6**
(R-1149, R-1150, R-1151, R-1152, R-1153, R-1154). Updated: R-1100 (READY), R-978, R-973, R-1093 (fixed in agent
`main`), R-265 (corrected).
## Observations
1. Installer password, golden and agent all proven in one fresh install. — closes R-1148 (not a finding).
2. FileBrowser's office preview answered 500 with the link on. FILED: R-1153.
3. A reinstall reuses an old host and skips two re-issues; now recurring with the kept Tester4. FILED: R-1154.
4. The R-973 cap leaves the local keep-last 3 floodable within a day. FILED: R-1149.
5. The cap's refusal reaches the operator with a misleading event text. FILED: R-1150.
6. R-978's typed-back confirmation is browser-only. FILED: R-1151.
7. The CI job timeout is inert. FILED: R-1152.
8. 9202's Immich sits on `/mnt/sizing-bench`, outside `/mnt/felhom-drives/`, so the allow-list refuses its definition every
15 minutes since 0.328.0 and a restore of it is refused. NOT-A-FINDING: the check works as designed; the folder is a
hand-made bench artifact (R-994), not a shape a real box has.
9. A restore brings back the backup's own compose (the old Redis volume with it). NOT-A-FINDING: by design (R-47: the
unit's compose becomes the live one); the older shape works.
10. My commit `7f7081d1` (day session) carries a „Co-Authored-By" trailer, against both briefs' rule 10. NOT-A-FINDING:
my slip; it is pushed and cannot be rewritten without a force-push (forbidden); no later commit has one.
11. I started the catalog's FULL gate run on DooPlex, whose third gate starts containers. NOT-A-FINDING: stopped at gate 2
(a network check), no container was made (checked), the fast run used instead.
12. One OnlyOffice „Connection is lost" on the first rc.1 attempt, after two blind clicks. NOT-A-FINDING: not reproduced;
the next attempt saved; no container restarted in between.
13. Two public names answered 000 once in the box walk. NOT-A-FINDING: 18 of 18 retries answered 200 in < 0.2 s.
14. The R-1100 row named the config key `proxyAuth`; it is `auth.methods.proxy`. NOT-A-FINDING: corrected in the row.
## Teardown — three layers
- **Machines:** VM 9302 destroyed (its disk gone). 9202: OnlyOffice and Nextcloud removed through the product (their
volumes, data and backup copies), the test documents deleted, the spike's route file, container, FileBrowser users and
proxy setting removed, helper files and the Immich scratch password shredded; Immich keeps the new Redis shape by hand
(= the template); 9202 on controller 0.331.0. `nextcloud_nextcloud_redis_data` (the orphan the pre-fix
removal left) removed by name afterwards; no Nextcloud volume remains. demo-hp host: the loop-disk file removed. Drill VM: CT 9100 purged, VM off,
disk `virgin`.
- **Hub:** global floor 0.331.0 / 0.158.1, no per-box floor; Tester4 and its host `Tester4-31fa0e` kept (operator); no
vouch, no deploy.
- **DooPlex:** builds and the golden bake only; the headless browser as a test client; no Docker cleanup.
Architecture read: `01`, `02`, `03` §8a, `04`, `05`, `07` §6/§8, `09` §3 decisions 162/250; `target-selection.md`,
`nightly-rotation.md`, `RUNBOOK-manual-build.md` §4. Documents changed: `03` §8a, `07` §6.1 and §8 row 10,
`target-selection.md` (Tester4). System poster: none of the three (no listed fact changed). `unproven.py --summary`:
not walked 35 of 55 — unchanged.
+164
View File
@@ -0,0 +1,164 @@
# REPORT — evening/night brief 2026-10-19 (one stop less per night, no false tunnel error, R-893 measured, hub and agent fixes ready, a scanner on every push, ep0's first weekly clean-up)
## In plain words
- **Morning list:** deploy hub 0.162.0? · release agent 0.161.0? · publish installer 1.35.0? · decide D20 (R-893).
- **One app stop less per night** (R-921): the controller (0.335.0, the floor) now asks the agent „are you busy?" before
it stops the apps. The agent's answer is written but not released, so it starts working with the agent release.
- **No false error for a box without a tunnel** (R-1070): the real cause was Docker 29 writing its „not found" message
in small letters. Fixed in the agent (not released).
- **A failed off-site restore, measured** (R-893): the box puts the newer rows over the backup's older volume; a table
only the backup had stays. The app is held stopped. Two options written; pick A (save the app's volumes first).
- **Hub fixes built, not installed:** no repeat mail after a restart; the agent list offers only real agent packages.
The ArgoCD danger to three hub secrets is already gone (ArgoCD no longer tracks them).
- **Installer message written, not published:** a re-run on a live box says „delete the old host first", EN and HU.
- **A scanner on every push, in all four repos** (warn-first). First scan: no real secret, no called weakness.
- **The scratch box cannot get its agent key without an agent change** (R-1083): the exact gap is written down.
- **ep0's first weekly clean-up ran as predicted:** only tester-1's 4 October copy went; every household has a rule.
R-1162 closed.
Decisions CC took: none beyond the brief. The operator answered one question (R-921, 19:2x): „add one field".
## Part 1 — R-921 (`07` §6.4; `03` local API) — DONE (live once the agent ships)
- No existing agent answer served the host-wide gate (read: `/backup/due`, `/backup/status`, the 409 shape). Operator:
add a read-only field to the existing route, no new route.
- Agent `56aabc7`: `host_busy` `{busy, what}` on `GET /backup/status`. Controller 0.335.0: `precheckSkips` → held →
no stop, contention for each due tier (15-minute skip, one „BLOCKED" alarm after 3 h); free / older agent / failed
read → the old order; the window's own tier in flight → attach as before.
- **The race** (free at the read, busy at the start): one short stop, resumed at once — today's behaviour, pinned.
- Tests, red-proved: agent `TestBackupStatus_R921_HostBusy` (assignment removed → `free gate: host_busy = <nil>`);
controller `TestR921_NoStopWhileTheHostIsBusy` (branch removed → 3 apps stopped, the start refused),
`…SkipsStillAlarmAfterTheCeiling`, `…OtherAnswersKeepTheOldOrder` (4 rows), `TestHostBusy_ReadsTheAgentsGate`.
- Live: demo-hp's agent answers no `host_busy` (read inside 9201) — the old order, as designed. A live busy state was
not made (it needs the new agent on the box). R-921 closed per the brief.
## Part 2 — R-1070 (`11` §8.2) — DONE in the agent (not released)
- Where: `felhom-agent/internal/osupdate/leg.go` `HostHealthVerdict` (the tunnel check) and
`internal/hub/cloudflared.go` `ClassifyTunnel`. Measured on demo-hp: `docker inspect` of a missing name prints
`error: no such object: …` (Docker 29); the match wanted `No such object` → `unknown` → the host rule failed.
- Fix: case-insensitive match → new state `absent`; the host rule passes `absent` only when the reading from before
the step (Docker answered) also had no cloudflared; `not_running` and `unknown` still fail. Hub 0.162.0 shows
`absent` as „none set up". `11` §8.2 had said `unknown` does not fail — corrected (the code always failed it).
- Tests, red-proved: `TestHostHealthVerdict_R1070_NoTunnelSetUp` (7 rows), three `TestClassifyTunnel` rows,
`TestSystemRows_R1070_AbsentTunnelIsNotAWarning`.
## Part 3 — R-893: measured and designed, no restore code changed
Helper on 9202 (fences carried in full). Evidence and the note: `documentation/audits/night-2026-10-19/P3/`.
- Rallly 4.15.3 + postgres 18; a scratch SFTP off-site on 9202's own disk; one failing SQL line in the prepared copy
of the dump, so the replay failed AFTER the volume was replaced.
- **What it left:** the newest rows (the undo dump) over the backup's volume; a table only the backup had stayed
(37 tables against 36); definition unchanged (same version); app held (`restore_mixed`), a start → 409.
- The restore page's sentence is false on both halves for this app → R-1169.
- **Options:** A — copy the app's live volumes before the replay, put them back on failure, start the app (one copy's
disk for minutes; Rallly 69 MB in ~1 s; a fit check before the stop). B — empty the DB volume before the undo load
(no disk; a bad undo copy leaves an empty database; usually still a support call). **Pick: A** (the operator's D8
direction). Not measured: the version-change branch (needs a drill-catalog commit).
- Teardown: 9202 back on 0.331.0 as found, every throwaway removed (checked again by CC); host: `/tmp` only; hub: nothing.
## Part 4 — hub 0.162.0, built, NOT deployed (`05`)
- R-1001: start-up seeding from the DB for `host_stale`/`host_down`; `restore_test_stale` edge from the DB. Tests
red-proved (each restart = a new checker over the same DB).
- R-1011: exact-file sha (`ErrNoFile`); step bundles leave the list; a vouch of one → `artifact_file_missing`. Registry
read: 6 step-bundle versions, every release holds `felhom-agent`. Tests red-proved.
- R-953: **no change needed** — read back 18:21 UTC: ArgoCD tracks none of the three Secrets (no tracking mark), so the
next deploy does nothing with them. `P4/r953-readback.txt`.
- Image `sha256:69285e640a9c5e4ce59b1fbb5cd94623fa47a71798e9d86fa0767ee6c69a6ac8` from a clean clone at `d92a712e`.
## Part 5 — agent, committed, NOT released (`03`)
R-1010 (`56aabc7`): on „Failed to find logical volume …_vzdump" one retry with the API's `force=1` (`pvesh usage`
lists `--force`; the same token privilege; no sudoers line, no root command); never a second force. Red-proved. Plus
R-921's field and R-1070. CHANGELOG „Unreleased". Not exercised live (it needs a half-removed snapshot on `pve/data`).
## Part 6 — installer 1.35.0, written, NOT published
R-1167: `die_live_box` in EN and HU at the preflight, host-enroll and the controller.yaml fetch. Harness: a static case
and two behaviour cases (the real functions against a local fake hub), red-proved on a mutated copy. **CI red twice**
(2106, 2108): the CI runner has no curl; the behaviour cases now SKIP out loud there (`5d0dfeac`).
## Part 7 — the scanner (R-986), warn-first — DONE
- Shared `scripts/security_scan_gate.py` in all four runners' fast sets (hook + CI); `.security-accepted` per repo;
8 decoys in scratch repos, red-proved; CI fetches the pinned gitleaks (both sha256 checked — seen working in CI 2108).
- govulncheck pin v1.1.4 → **v1.8.0** (v1.1.4 cannot read the Go 1.27 code). CI has no Go: govulncheck NOT CHECKED
there, out loud; the hook runs it.
- **First scan of today's `main`** (`P7/triage.md`): govulncheck 0 called in all three modules. gitleaks history:
controller 12, agent 15, catalog 1 (all as on 2026-10-12), felhom.eu 50 (+7: the usenet bench 5, a boot journal 1,
a test 1). Today's tree: controller 4, felhom.eu 42, agent 15, catalog 0 — every one triaged: **0 real**; accepted =
throwaway demo/bench app values (the bench LXC 9401 stopped, the apps gone from 9202); false = test constants, public
keys, an SSH fingerprint, a tunnel ID, hashes.
## Part 8 — R-1083: the exact gap (no agent change)
The agent mints a per-guest token only inside `--selftest=provision`, for a guest it brings up from a golden in the same
call. No product way exists for an existing guest. Row stays OPEN with the gap and two options.
## Part 9 — controller 0.335.0, the floor
rc.1 on demo-hp 18:08 UTC (0 errors) → `0.335.0` `sha256:262831f9…` → per-box floors 18:22 → all three moved by
18:37 → box walk green (0 unhealthy, 0 refusals, catalog in sync; https-only 28 OK, 0 FAIL; „Fájlok ↗" one sign-in on
demo-hp and Tester 1) → **global floor 0.335.0 / MinAgent 0.158.1 at 18:39:26 UTC**, read back; per-box floors cleared.
i18n: no page changed, no fixture changed. Evidence `P9/`.
## Part 10 — ep0's first weekly clean-up (`07` §plane-2 retention) — DONE, R-1162 closed
Read only on ep0, 03:45–03:47 UTC. Evidence `P10/01-before.txt` (19:49 UTC), `P10/02-after.txt`.
| Household | Task 03:30 UTC | Removed | Copies after |
|---|---|---|---|
| demo-hp | OK, keep-weekly 2 | nothing | 2 (10-08, 10-15) |
| demo-felhom | OK, keep-weekly 2 | nothing | 2 (10-06, 10-13) |
| Tester-2 | OK, keep-weekly 2 | nothing | 1 (10-04) |
| tester-1 | OK, keep-weekly 2 | 2026-10-04 | 2 (10-11, 10-18) |
- Exactly the dry runs' prediction. The operator job (03:45, keep-daily 14 + keep-weekly 8) removed one hub copy
(10-06) — its own rule.
- **No „household without a rule" alarm.** Positive check from a second channel: ep0's read-only `usage` op, run
directly, answers `unpruned_checked: true, unpruned: []` (tenantsync v1.3.0, sha256 same as the repo file). The hub
(0.161.0) logs its ep0 read every 15 minutes (03:31 UTC the last before my read) and no „no prune job" line since its
start (2026-10-19 14:10 UTC). The alarm is operator-channel only, so it is not in the events table (none there).
## Versions, commits
- Controller 0.335.0 (`15ea002`, REPORT `c4b3b0a`, scanner `a03c25a`). Hub 0.162.0 built (`5f31da04`). Agent
`56aabc7`, `84ef581` (not released). Installer 1.35.0 (`d92a712e`, `5d0dfeac`, not published). felhom.eu scanner
`15f74288`, docs `f92fa5a8`. Catalog `87f5588`, `57ddb20`.
- **System poster:** no listed fact changed tonight — none of the three.
- **Instruction/doc corrections:** `11-os-updates.md` §8.2 (the `unknown` tunnel sentence was false); `nodes.md` and
`nightly-rotation.md` (9202 is DHCP, .121 now).
## Rows
Register **161 before, 160 after.** **Closed 4:** R-921, R-953, R-986, R-1162. **Opened 3:** R-1168 (warn →
fail), R-1169 (the false restore sentence), R-1170 (a vanished tunnel raises no alarm). Re-stated: R-1001, R-1011,
R-1010, R-1070, R-1167 (READY, waiting on the release/deploy/publish), R-893 (READY, D20), R-1083 (the gap).
**Dates moved** (the due-checks gate fired on the new date): R-895 and R-435, 2026-10-20 → 2026-10-21. Both are reads
this brief did not name; R-895's first verdict lands at 07:10 hub time, after this session. The reason is in each row.
## Observations
1. The restore page's held-mixed sentence is false for the measured app. FILED: R-1169
2. A tunnel container that disappears on a box with a tunnel configured raises no alarm. FILED: R-1170
3. The scanner warns only. FILED: R-1168
4. `11` §8.2 said an `unknown` tunnel does not fail the host rule; the code always failed it. NOT-A-FINDING: corrected in the session (the size rule); it is R-1070's cause.
5. govulncheck v1.1.4 (the 2026-10-12 pin) cannot read Go 1.27 code. NOT-A-FINDING: the pin moved to v1.8.0 tonight, recorded in `SCANNERS.md` §5.
6. The CI runner has no curl; two new harness cases failed CI 2106 and 2108. NOT-A-FINDING: fixed the same night (`5d0dfeac`), they skip out loud there.
7. R-953's tracking marks went after 2026-10-12 with no record of who removed them. NOT-A-FINDING: the danger the row named is gone, nothing to fix.
8. 9202's DHCP address moved (.114 → .121) and the walk named the old one. NOT-A-FINDING: fixed (`box_walk.py` reads it; two docs corrected).
9. The brief said the installer already prints HU and EN; it printed English only. NOT-A-FINDING: the HU lines are new, named in the scripts CHANGELOG.
10. The agent's backup cap (R-973) can also refuse a POST after the stop. NOT-A-FINDING: it answers 409, so the apps resume at once (the BUSY path), and it fires only after a flood of presses that each stopped the apps anyway.
11. The manual off-site run dumped and restarted every app, not only the one switched on. NOT-A-FINDING: by design, the coherent app-data pair (`07`).
12. On CI's depth-1 clone, gitleaks scans the whole tree (the root commit), ~100 s for felhom.eu. NOT-A-FINDING: a stronger check than the diff, all current hits accepted; recorded in `CONTEXT.md`.
13. My 19:49 before-file listed the operator job's keep-weekly but not its keep-daily 14. NOT-A-FINDING: my read printed only some fields; the job is unchanged.
## Teardown
- Machine: 9202 back as found (0.331.0, no throwaway left — checked); 9201 on the release 0.335.0; probe scripts removed.
- Host: `/tmp` copies only, removed. Nothing on `pve/data`.
- Hub: per-box floors cleared; nothing else created.
- ep0: read only (task list, task logs, snapshot list, prune-job list, the read-only `usage` op). Nothing changed.
+87
View File
@@ -0,0 +1,87 @@
# REPORT — the website keeps syncing during an internet outage (2026-10-19)
## In plain words
1. **It worked.** The website's copy helpers now read Gitea inside the cluster, with no internet hop, and they wait
and try again instead of stopping. The new website pod runs with 0 restarts, and the site and the installer
script both answer normally.
2. **Note for later:** this ties the website to the DooPlex cluster. If it moves (for example to Hetzner), the
address must go back to the public one — register row R-1164.
## Part 1 — anonymous in-cluster fetch (before any commit)
Throwaway pod in a scratch namespace (`ccscratch-gitsync`, deleted after), no credentials:
`git ls-remote http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git main installer-v1.34.0`
| ref | in-cluster | public (anonymous) |
|---|---|---|
| `refs/heads/main` | `5d3cb5c1…` | `5d3cb5c1…` — equal |
| `refs/tags/installer-v1.34.0` | `add58eec…` (annotated tag object) | `add58eec…` — equal; peels to `624bcdc3…` |
Evidence: `audits/website-gitsync-incluster-2026-10-19/gitsync-part1-*.txt`.
## Part 2 — the change
Commit **`fd75003e`** (no Co-Authored-By, per the brief):
- `manifests/webpage.yaml`: all four git-syncs `--repo=http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git`;
the two sidecars `--max-failures=-1` (not the `--one-time` inits); one comment above the website sidecar. Refs and
`nginx-config-rev` unchanged.
- `scripts/hostinstall_gates.py` gate 6: fails unless there are four `--repo=` values and all are identical. Existing
ref checks unchanged.
Gates: `python3 scripts/repo_gates.py` → `all felhom.eu gates OK`; `python3 scripts/hostinstall_gates.py` → rc 0
(`ok: manifest: all four git-syncs read the same repo (…)`).
## Part 4 — deploy and proof
**Deploy:** hard refresh showed only `Deployment/felhom-webpage` OutOfSync; synced that resource alone at `fd75003e`
(the skill's Deployment-only sync). New pod `felhom-webpage-869ff88bf7-fswbc` ready at 09:42 UTC; the old pod
terminated. ArgoCD: Synced, operation Succeeded. (The first `rollout status` returned at once — it ran before the new
ReplicaSet existed; re-run after the sync it waited for the old replica and finished.)
**Scenario A** (`gitsync-scenarioA.txt`): restarts `0,0,0`; both sidecars started with the in-cluster `--repo` and
`--max-failures=-1`, then `updated successfully` (`main` → `fd75003e`, `installer-v1.34.0` → `624bcdc3`); both init
containers `updated successfully`; `https://felhom.eu/` **200**, `https://felhom.eu/scripts/felhom-host-install.sh`
**200**, served `SCRIPT_VERSION="1.34.0"` = manifest `--ref=installer-v1.34.0`.
**Scenarios B and C** (`gitsync-scenarioBC.txt`): scratch namespace `ccscratch-gitsync-bc` (deleted after),
`git-sync:v4.4.0 --repo=http://192.0.2.1/x.git --period=10s`:
- **B, with `--max-failures=-1`:** `error syncing repo, will retry` (`failCount:1`), restarts **0**.
- **C, without:** `too many failures, aborting`, restarts **1**.
- The first read at 95 s showed nothing for either: one fetch to an address that never answers takes **~133 s** to
time out. The re-read at 09:48 recorded both results. Instead of 90 s, a later run of this test needs about 150 s.
**Scenario D — gate red-proof:** one `--repo` changed to the public address → gate 6 `FAIL … the git-sync --repo
values differ`, rc 1; restored byte-identical → rc 0.
## Part 3 — docs changed
- `documentation/architecture/01-topology-and-trust.md` §7: a `[DESIGN]` paragraph — the website pulls `website/`
(main) and `scripts/` (installer tag) from Gitea in-cluster over plain HTTP, sidecars retry forever, tied to the
DooPlex cluster.
- `.claude/rules/manifests.md` „The two git-syncs": the fact that all four containers read Gitea in-cluster, must name
one repo (gate 6), and the sidecars carry `--max-failures=-1`.
- `scripts/CHANGELOG.md`: one entry.
## Register
Rows before **166**, after **167**. Closed **R-1163** (finding + fix, with evidence). Opened **R-1164** (P4, Hub &
operator: move the website to Hetzner; then the git-syncs must use the public address again).
## Observations
- The 95-second wait in the brief is too short for an address that never answers (~133 s TCP timeout per fetch) —
NOT-A-FINDING: recorded here as the correct wait for this test; nothing in the product depends on it.
- The skill says a whole-app sync would push git's view over de-gitted Secrets; today only the webpage Deployment was
OutOfSync — NOT-A-FINDING: the Deployment-only sync was used anyway, as the skill says.
- Around 09:50–09:55 UTC, DooPlex itself briefly could not resolve names (`Temporary failure in name resolution`;
`curl https://felhom.eu/` → `000`). By 09:56 DNS, the internet and the site (200) were back. The website pod kept
restarts `0,0,0` and synced `e2c35007` in-cluster at 09:54 with no error. NOT-A-FINDING: the same class of short
outage that triggered this task; it is not proof that the fix worked (whether the blip reached the cluster is unknown).
- CI: `fd75003e` job 2092 success, `e2c35007` job 2094 success.
## Teardown
Scratch namespaces `ccscratch-gitsync` and `ccscratch-gitsync-bc` deleted (`kubectl get ns | grep -c ccscratch` → 0).
No hub, controller, agent, customer box, demo box or ep0 touched.
+15 -9
View File
File diff suppressed because one or more lines are too long
@@ -271,6 +271,13 @@ app. Controller down → the gated app answers an error, never the app. Measured
tunnel. Since controller v0.286.0 it sits ALONE on the `felhom-tunnel` network at the fixed address `172.16.253.2`
(traefik at `.3`), so traefik can believe forwarded headers from it and from nothing else (§5). Geo-restriction WAF
is **hub-enforced** (the hub holds the CF API token; the controller only reports geo desired-state).
- **[DESIGN] The website's content comes from Gitea IN-CLUSTER** (2026-10-19). The `felhom-webpage` pod pulls
`website/` (from `main`) and `scripts/` (from the `installer-v…` tag, R-110) with four git-syncs that read
`http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git` — plain HTTP, safe only because it never
leaves the cluster (the hub's own Gitea client does the same). No internet hop: the pod can start, and keep its copy
fresh, during an internet outage. The two sidecars run with `--max-failures=-1`, so an unreachable Gitea is retried,
not a restart. **This ties the website to the DooPlex cluster:** moved elsewhere, it must read the public HTTPS
address again. `scripts/hostinstall_gates.py` gate 6 keeps the four `--repo` values identical.
---
@@ -539,8 +539,28 @@ state (both demo boxes) the panel says „Ezt a jelszót te állítottad be a F
the press. The box uses FileBrowser's own CLI (`set -u admin,<pw>`) with the server STOPPED (the CLI opens the
database with no lock timeout), verifies new=200 and admin/admin refused, stores the value encrypted and records
`generated`; a failed verify leaves `operator`. Not the `FILEBROWSER_ADMIN_PASSWORD` env: it re-applies on every
start. Spike: `audits/logins-folders-2026-10-16/P1b`. FileBrowser still asks for its own login the first time; one
sign-in for both is R-1100.
start. Spike: `audits/logins-folders-2026-10-16/P1b`. Since v0.332.0 the folder links open FileBrowser without its own
login (the next paragraph); `admin`'s password is the fallback and the operator's way in.
**[DESIGN] One sign-in for the file manager (v0.332.0, R-1100, operator decision D16 = yes 2026-10-19; spike
`audits/proxyauth-spike-2026-10-18/DESIGN-NOTE.md`).** Every dashboard link to a FileBrowser folder goes to
`/__files/open?to=<path>` on the dashboard host, which — for a signed-in, not-cross-site request — mints a pass (60 s,
one use, bound to `files.<domain>` and to the session) and sends the browser to `files.<domain>/_felhom/sso`. traefik
asks the controller before every FileBrowser request (forwardAuth `felhom-files-sso` → `/__felhom_gate/files`, the setup
gate's handshake): the pass becomes the host-only `felhom_files` cookie (a value derived from the session's fingerprint
for that host — the R-976 shape); a valid cookie → 200 with the box's **secret header** naming FileBrowser user
`felhom-household`; no cookie → 200 without it (FileBrowser's own password); a stale cookie (sign-out, password change)
→ both cookies cleared, back to the dashboard. **Why a secret header name:** FileBrowser trusts the header blindly and
every app container reaches `filebrowser:80` directly — measured: a forged fixed-name header from an app logged in. The
name (`X-Felhom-User-<32 hex>`, per box, 0600 in the data dir) is written only into FileBrowser's config and traefik's
route (`authResponseHeaders`, which strips a client's copy); never a log, a page or a report. **Why not a parent-domain
cookie:** every app subdomain is third-party code and would receive the dashboard session. **What stays:** `admin` is a
password user (R-513's reveal and reset, the R-1126 gate unchanged); the header user is a separate FileBrowser user whose
rights come from the box's userDefaults (change, add, delete, share, download — not admin, not API); FileBrowser's logout
goes to the dashboard. Residual: FileBrowser's own 2-hour token for the header user is cleared with the files cookie,
but a browser that lost only the files cookie keeps it until it expires. Measured live on all three test boxes
(`audits/day-2026-10-19/P2/`, `P4/`): opened at the folder with no login; forged headers refused through traefik and
from an app container; a sign-out asks again.
**[DESIGN] A stored FileBrowser password is checked before it is offered (v0.321.0, R-1101).** In `generated`, the
page asks FileBrowser whether the stored value still logs in; only FileBrowser's own 401/403 counts as refused (a
@@ -562,8 +582,9 @@ role, else the first declared folder, else (a media app) the first present drive
`movies`, the players both, a download client its `media_library.folder`) with NO backup sentence — the bind's class
there means „not this app's data", which is not the class-driven „temporary, not backed up". **A folder whose drive
is not bound into FileBrowser now is text, never a link** (`drivePresent` = the same gates the FileBrowser sync
applies). The button never promises one click: FileBrowser's router brings a cold visitor back to the folder after its
login (measured in the headless browser, `audits/logins-folders-2026-10-16/browser/`).
applies). Since v0.332.0 the button opens the folder signed in (one sign-in, above); on a box without it, FileBrowser's
router brings a cold visitor back to the folder after its login (measured in the headless browser,
`audits/logins-folders-2026-10-16/browser/`).
**[DESIGN] „Cím és kulcsok" is rows, not the install form.** The settings tab does not re-render the
install wizard's fields: the address is the „Cím" row, the storage is a „Tárhely: <név>" row, every
+22 -2
View File
@@ -232,7 +232,11 @@ The controller (in its LXC) reaches the agent (on the host) over the local bridg
- `POST /rollback` — roll *this* guest back to a named snapshot (post-deploy failure recovery).
- `POST /backup` — request a backup-now of *this* guest (enqueued; non-destructive).
- `GET /backup/due` — whether a policy-scheduled backup is due for *this* guest, so the controller can quiesce then call `POST /backup` (the app-consistent path, §8).
- `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI.
- `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI. **`host_busy`
(R-921, agent Unreleased `56aabc7`):** `GET /backup/status` also says whether the host-wide heavy-operation gate is
held (`{busy, what}` — another guest's backup, the restore test, an OS step, the stale-lock sweep, a power-off), the
state in which `POST /backup` answers 409; absent when the gate is not wired. The controller (≥ 0.335.0) reads it
before it stops any app. A read, never a reservation.
- **Crash-boot fact (R-856, agent v0.149.0):** `GET /host/crash-guard` — the host crash guard's last-boot record
(`present`, `last_boot_at`, `last_boot_unclean`, `tripped`) read from `/var/lib/felhom-crash-guard/state.json`; a missing
or garbled file answers 200 `present:false`, an older agent 404 — both read as a normal boot. The controller waits
@@ -399,7 +403,10 @@ per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a ba
> space on …" — so the hub raises `restore_test_failed`; it is never a pass and never dropped.
> - **Leftovers on a timer.** A failed scratch teardown and the stale-lock sweep (R-673) run every 10
> minutes, not only at agent start; the sweep holds the one-heavy-operation gate; after 3 failed
> teardown tries the operator is told.
> teardown tries the operator is told. **A half-removed `vzdump` snapshot (R-1010, agent Unreleased
> `56aabc7`):** when the sweep's delete fails ONLY with „Failed to find logical volume …_vzdump" (one disk's
> snapshot already gone), it retries once with Proxmox's own `force=1` on the same API call (no sudoers line, no
> root command); any other error is returned unchanged, and a failed forced delete is never forced again.
> - **A thin pool ≥ 90 % requests an immediate report**; the hub judges a thin pool on the worse of data
> and metadata, critical at 90 %, one alarm per pool per 6 hours (`08` §6.2).
>
@@ -544,6 +551,19 @@ nicety.
**loses Felhom's backup guarantees** — no PBS verify / monitoring on storage we can't reach. An
honest signup-time tradeoff, not a hidden one.
**When the new package reaches the hub (R-978; agent 0.160.0, 2026-10-19):** a ceremony started again no
longer replaces the hub's package when it starts. The package travels with the pending ceremony and is uploaded when its
code is claimed, before the code is handed out (an upload failure is a 502 and the code stays claimable). An abandoned
ceremony — TTL, a new start, a restart — leaves the first code opening the hub's package. **[DESIGN] Only after the
household typed it back (R-1151; agent 0.160.0 + controller 0.332.0, 2026-10-19):** a controller ≥ 0.332.0 claims with
`{"defer_upload": true}`; the agent hands out the code with two word positions it drew, keeps the package (and only those
two words, never the code) for 30 minutes, and uploads it only when `POST /escrow/ceremony/confirm` brings the two words
back — so a code opened and not written down never replaces the package. Wrong words 422 (still waiting), window gone 410,
nothing waiting 404, hub failure 502 (still waiting). A controller that has not updated claims with `{}` and gets the
R-978 order. Proven on demo-hp 2026-10-19: ceremony 1 typed back → code 1 opens the hub's package, code 0 is a kept
earlier one (422); ceremony 2 shown and abandoned → code 1 still opens it, code 2 opens nothing
(`audits/day-2026-10-19/P1/escrow-*.txt`).
#### Custody across generations — what the hub retains when a blob is superseded (hub v0.60.0; **contract corrected hub v0.93.0, R-198**)
A ceremony that seals a *different* passphrase does not destroy the blob it replaces: the hub copies
@@ -90,7 +90,11 @@ Evolves the existing staleness checker (60s **cadence**, a **configured** thresh
2026-09-17, **45 m** since operator ruling A on R-549; OK under it, down at 2× = 90 m; today: controller-report recency → `node_stale`/`down`/`recovered`):
- **Primary = host-report recency → `host_stale` / `host_down`.** The agent heartbeat is the box's
liveness signal; a silent agent = the box is gone (the critical alert).
liveness signal; a silent agent = the box is gone (the critical alert). **A hub restart does not re-mail a known
silence (R-1001, hub 0.162.0 — built 2026-10-19, not yet deployed):** a host whose newest `host_stale`/`host_down`
event is newer than its last report is seeded with that state at start, silently; `restore_test_stale` reads its edge
from the DB on the first sweep (a signal for the tier inside the read window and newer than its last proof). A failed
lookup lets the alarm through.
- **Guest up/down comes from the host report's per-guest status** — authoritative, every poll, faster
than waiting for a guest report to go stale.
- **Guest-report recency = secondary** app-level signal.
@@ -134,7 +138,11 @@ walk is green; the golden is baked weekly and vouched by the operator; a new box
itself to the floor (R-1052: not yet proven end to end). Setting the global floor today needs the operator page with the
operator's password — there is no scoped route for CC (R-1051). **An agent can be vouched mid-week; the golden stays
weekly** (hub v0.158.0): the R-120 golden guard judges only a CHANGED golden selection, so a Day-0 save that keeps the
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet.
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet. **The agent list
offers real agent packages only (R-1011, hub 0.162.0, not yet deployed):** the sha lookup asks for exactly the
`felhom-agent` file (the golden: `golden.tar.zst`) and never falls back to another file, so a step bundle
(`0.156.1-step1`, only the config bundle) is left out of the list, and a vouch of one is refused
(`artifact_file_missing`).
**The controller floor and its agent requirement (hub v0.112.0, R-472).** The managed controller floor
rides the report ACK. `store.ResolveManagedFloor` decides per box whether to serve it, from three
@@ -222,6 +230,51 @@ Today: `GET /config/{id}` with an `X-Retrieval-Password` (Hungarian passphrase)
The existing `configgen` deep-merge + Hungarian-passphrase machinery is the base; it grows the
agent-first + key-pinning + restore-mode steps.
**A reinstall over an existing host record is made loud, never acted on [DESIGN, 2026-10-19 — R-1154, brief
2026-10-19; hub v0.159.0].** `POST /host-enroll` is the installer's step 4, run on every pass (a `--resume` too), with
the customer's retrieval passphrase; an existing host record is REUSED (same id and key — the reinstalled box takes
it over). The clean-slate re-issues (a new claim code mailed to the household, the off-site provider password reset)
run only on the MINT path, i.e. after the operator deleted the old host — **the operator's delete is their gate,
and stays so.** A reinstall that kept the old record used to skip them in silence. Now the reuse path classifies the
old box and re-issues NOTHING: **gone** (no controller wait connection open now, and either `deliberate_off_at`
inside its hold or BOTH the host report and the controller report older than `alerting.stale_threshold` or absent —
the `hostStatus` threshold and comparison) → the operator-only WARNING `host_reinstall_over_old_host`, mailed to the
operator: delete the old host (Hosts page) and run the installer again, or re-issue the claim code and the off-site
login from the customer page; **live** → the operator-only `host_reenroll_while_live` — since hub v0.160.0 (R-1156) a
WARNING mailed with the same instruction (a reinstall within 45 minutes of the old box's last report reads as live; an
installer `--resume` or a repair re-run mails too, and the text says those need nothing).
**Why not re-issue automatically** (a security review of the first draft, the same day): the passphrase is a
household-held secret typed into the installer, and both re-issues are operator SESSION-ONLY actions — not even an
operator token may run them (§17.5; `09` §3 decision 252, R-1055/R-1059: an off-site re-issue „stops its backup for a
while"). Re-issuing on the reuse path would give those powers to anyone holding the passphrase whenever the box is
switched off or silent. The cost of the loud shape is two operator acts per reinstall that kept its old record. Both
event types are hub-origin only. `hub/internal/api/reenroll.go`, pinned by `hub/internal/api/r1154_reenroll_test.go`.
**What the household's retrieval passphrase can get [DESIGN, 2026-10-19 — R-1159, operator decision D19 „box key
first, then lock"; hub v0.161.0 + controller v0.334.0].** The passphrase is the household's install secret: a NEW box
needs it (the installer's preflight, download login and enroll; the guest's first boot, `bootstrap.json` → `GET
/config`, which is how the box gets its own hub key). Measured the same day, it also gave anyone who held it the box's
keys for good, and through them every off-site key action (`audits/day-2026-10-19b/P2/`). **The rule:** the passphrase
gives keys only until the box uses its own — `GET /config`, `GET /recovery` and the reuse path of `POST /host-enroll`
answer **409 with no key** (and the operator gets `passphrase_keys_refused`) once a host record exists AND a controller
report arrived since it was made AND that controller pulls its config with its own key (≥ 0.334.0, `GET /box-config`,
`boxConfigMinController`). Open for a first install, the install in progress, a reinstall after the operator deleted
the old host, and a box on an older controller (which still refreshes with the passphrase — the lock arms box by box
as the floor moves). A reinstall over a live box therefore needs the operator's delete first, which R-1154 already
made the rule. **Why this shape and not „before a host exists":** the install itself uses the passphrase AFTER enroll
(`felhom-host-install.sh` step_agent_install → `/config`; the agent bakes it into `bootstrap.json`; the guest pulls with
it), and so did every live box's config refresh — measured from the code before building, and put to the operator.
`hub/internal/api/passphrase_scope.go`; pinned by `r1159_passphrase_scope_test.go`. Proven live on demo-hp
(2026-10-19 14:30 UTC): before 0.334.0 → 200; after its first 0.334.0 report → 409, no key, the event raised; the box's
own key → 200.
**[FACT, 2026-10-19 — R-1158, hub v0.160.0] What the MINT path's off-site re-issue still does.** It resets the Storage
Box sub-account password and stores the new one — a password only the hub itself uses (`offsitekeys.TargetFor`, the key
registrar's login). No box receives it since v0.127.0 (`consume-password` answers 410; `ConsumeOneTimeSecret` has no
production caller); the fresh box registers its own key with the controller API key. So it changes nothing a box uses.
It is kept because it hands the hub a fresh, working login at each clean reinstall, which repairs a broken stored one;
the `offsite_reissued` record now says the password is the hub's and the backups do not change.
## 9. DR model
The headline: the **old heavy infra-backup push retires** — not because the hub authors everything
File diff suppressed because one or more lines are too long
+5 -1
View File
@@ -724,7 +724,11 @@ Evidence: `audits/os-host-lane-2026-10-04/` (parts A–G).
skips the host step.
- **The host health rule** (`HostHealthVerdict`, pinned in `internal/osupdate`): `felhom-agent`, `pveproxy`,
`pvedaemon`, `pvestatd` and `pve-cluster` are `active`; the customer guest runs; the guest health rule (§8.1)
passes; the tunnel is `running` — an `unknown` tunnel does not fail it, `not_running` does. Same 5-minute wait.
passes; the tunnel is `running` — `not_running` and `unknown` fail it (CORRECTED 2026-10-19: this line said
„an `unknown` tunnel does not fail it"; the code always failed it, which is what Tester4 hit, R-1070). **Since agent
Unreleased `56aabc7` (R-1070):** a guest with NO cloudflared container reads `absent` (Docker 29's lower-case
„no such object" used to fall through to `unknown`), and `absent` passes when the reading from before the step also
listed no cloudflared container (Docker answered) — a box with no tunnel set up. Same 5-minute wait.
- **Separate approved sets:** host and guest releases are separate (`os-host-…`, `os-guest-…`), each by the same rule
(24 h, 1 night of THAT layer, every ring-0 box). Ring 1 receives `host_release` beside `release`.
- **Reboot needed:** reported when PID 1 or `lxc-start` maps a replaced file, with the date of the first scanned
@@ -87,8 +87,8 @@ Title: **"Where a household's data is copied"**
| **Tier 1 — recovery unit** | each app: its settings, database dump, volumes | data drive 1, on the box | nightly at **02:30** (W) | 1 point per app | not encrypted (it never leaves the home) |
| **Tier 2 — cross-drive** | a mirror of Tier 1, plus the app's files | data drive 2, on the box (counted only on a different physical disk; in a VM made, not counted) | nightly **03:30** | mirror | not encrypted |
| **Tier 3 — off-site, per app** | Tier 1 + the app's must-keep files | **Hetzner Storage Box** (Germany), one folder per household | nightly **04:15** | 7 daily, 4 weekly, 6 monthly | the household's own restic password |
| **Whole guest — local** | the guest's system (not the data drives) | the box's own system disk | daily, in the window **04:30–08:30** | up to 3 copies (fewer on a small disk) | not encrypted |
| **Whole guest — off-site** | the same | **ep0** (Hetzner, Germany), one namespace per household | weekly, same window | last 2 | the household's own key, escrowed with a recovery code |
| **Whole guest — local** | the guest's system (not the data drives) | the box's own system disk | daily, in the window **04:30–08:30** | one copy per day, up to 3 days (fewer on a small disk) | not encrypted |
| **Whole guest — off-site** | the same | **ep0** (Hetzner, Germany), one namespace per household | weekly, same window | one per week, 2 weeks | the household's own key, escrowed with a recovery code |
Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–3 follow at fixed offsets so they cannot run out of order. A missed night is caught up 15 minutes after the box starts."
@@ -99,7 +99,7 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
- one namespace **per household** (whole-guest copies, each household's own key),
with a NEUTRAL note under it (not a red gap box): "By design: each household holds the key to its own copies; the operator cannot open them without their recovery code." — *fix (c), 2026-10-09: this is a property of the design, not a gap, and a red box said the opposite of what it means.*
- the **`operator`** namespace: two groups — **"hub database"** and **"DooPlex copy"** (code, password manager, secrets, signing keys).
Prune jobs on ep0: households keep the last 2; `operator` keeps 14 daily + 8 weekly.
Prune jobs on ep0: households keep one copy per week for 2 weeks (`keep-weekly 2`, R-1162, 2026-10-19; every new household's job is made with its namespace, R-1165); `operator` keeps 14 daily + 8 weekly.
Mark ep0 with a small shield: "**Protected** — holds the only off-premises copy of real household data."
- **Hetzner Storage Box** (grey): the shared box for the households' per-app restic copies (Tier 3), one sub-account per household, add-only keys.
- **Mail service** (grey, "Resend"): every mail the hub sends — to the operator and to households.
File diff suppressed because one or more lines are too long
@@ -0,0 +1,24 @@
loop=/dev/loop2
--- lsblk p1 (the bug input)
{"blockdevices":[{"name":"loop2p1","fstype":null,"pttype":"gpt","mountpoint":null}]}
--- blkid -p p1 (empty part)
DEVNAME=/dev/loop2p1
PART_ENTRY_SCHEME=gpt
PART_ENTRY_UUID=2b13a20b-f5a4-4906-9e30-985c170a50ad
PART_ENTRY_TYPE=0fc63daf-8483-4772-8e79-3d69d8477de4
PART_ENTRY_NUMBER=1
PART_ENTRY_OFFSET=2048
PART_ENTRY_SIZE=1048576
PART_ENTRY_DISK=7:2
rc=0
--- blkid -p p2 (ext4)
TYPE=ext4
PART_ENTRY_SCHEME=gpt
PART_ENTRY_NUMBER=2
rc=0
Re-reading the partition table failed.: Invalid argument
--- blkid -p p1 (a dos table inside)
PTTYPE=dos
PART_ENTRY_SCHEME=gpt
rc=0
ls: cannot access '/root/r1093-scratch.img': No such file or directory
@@ -0,0 +1,8 @@
tester-1-d70be4 agent_update:
uploaded signed op to the hub jobs queue
tester-1-d70be4 agent_config_update:
uploaded signed op to the hub jobs queue
demo-felhom-8363b5 agent_update:
uploaded signed op to the hub jobs queue
demo-felhom-8363b5 agent_config_update:
uploaded signed op to the hub jobs queue
@@ -0,0 +1,11 @@
tester-1: felhom-agent 0.159.2 | demo-felhom: felhom-agent 0.159.2 | 12:25:17
Oct 18 14:23:46 felhom felhom-agent[3912599]: time=2026-10-18T14:23:46.774+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=c9b02a094671155e op=agent_update
Oct 18 14:23:48 felhom felhom-agent[3912599]: time=2026-10-18T14:23:48.727+02:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=75 total=75 degraded=""
Oct 18 14:23:48 felhom felhom-agent[3912599]: time=2026-10-18T14:23:48.728+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=fa2a707792f561ed op=agent_config_update
Oct 18 14:24:52 felhom felhom-agent[1371657]: time=2026-10-18T14:24:52.767+02:00 level=WARN msg="selfupdate: update committed" version=0.159.2 wrapper=""
Oct 18 14:25:00 demo-felhom felhom-agent[2419165]: time=2026-10-18T14:25:00.552+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=9d4397117eeed3b9 op=agent_update
Oct 18 14:25:01 demo-felhom felhom-agent[2419165]: time=2026-10-18T14:25:01.930+02:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=75 total=75 degraded=""
Oct 18 14:25:01 demo-felhom felhom-agent[2419165]: time=2026-10-18T14:25:01.930+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=63f8321055a65cb8 op=agent_config_update
[exited with code 0]
Oct 18 14:26:05 demo-felhom felhom-agent[3800522]: time=2026-10-18T14:26:05.216+02:00 level=WARN msg="selfupdate: update committed" version=0.159.2 wrapper=""
@@ -0,0 +1,23 @@
Oct 18 13:12:26 demo-hp felhom-agent[3566597]: time=2026-10-18T13:12:26.461+02:00 level=INFO msg="backup: space preflight passed" vmid=9201 target=local last_archive_bytes=6916890749 need_bytes=9719855260 avail_bytes=12123615232
Oct 18 13:12:27 demo-hp felhom-agent[3566597]: time=2026-10-18T13:12:27.526+02:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=local job=backup-9201-1792321945880394998
Oct 18 13:18:22 demo-hp felhom-agent[3566597]: time=2026-10-18T13:18:22.853+02:00 level=INFO msg="janitor: stale-lock sweep deferred — a heavy operation is in flight" busy=backup:local
Oct 18 13:19:59 demo-hp felhom-agent[3566597]: time=2026-10-18T13:19:59.260+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_10_18-13_12_26.tar.zst size_bytes=6938307162 uncovered_volumes=2
Oct 18 13:19:59 demo-hp felhom-agent[3566597]: time=2026-10-18T13:19:59.260+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1792321945880394998 archive=local:backup/vzdump-lxc-9201-2026_10_18-13_12_26.tar.zst
Oct 18 13:19:59 demo-hp felhom-agent[3566597]: time=2026-10-18T13:19:59.266+02:00 level=INFO msg="local-api: no OS leg after a manual backup — it follows the night's own backup (R-899)" vmid=9201 job=backup-9201-1792321945880394998
== sudo felhom-thin-check lines since the trigger:
0
== sudo lines since the trigger (any helper):
26 COMMAND=/usr/bin/chown
36 COMMAND=/usr/bin/lsblk
98 COMMAND=/usr/bin/lxc-info
78 COMMAND=/usr/bin/mkdir
8 COMMAND=/usr/bin/wg
36 COMMAND=/usr/sbin/blkid
75 COMMAND=/usr/sbin/lvs
16 COMMAND=/usr/sbin/nft
57 COMMAND=/usr/sbin/pct
150 COMMAND=/usr/sbin/smartctl
[exited with code 0]
== control: the night's backup on agent 0.159.1 (2026-10-18 04:33 local) ran the check:
Oct 18 04:33:15 demo-hp sudo[1664888]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-thin-check pve data
@@ -0,0 +1,3 @@
== trigger 2026-10-18T11:11:36Z (local 2026-10-18 13:11:36)
200
{"data": {"started": true}, "ok": true}
@@ -0,0 +1,5 @@
felhom-agent 0.159.1
Oct 18 04:43:52 demo-hp felhom-agent[3320002]: time=2026-10-18T04:43:52.498+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
Oct 18 04:43:53 demo-hp felhom-agent[3320002]: time=2026-10-18T04:43:53.613+02:00 level=INFO msg="osupdate: DONE" run=20261018T024228Z layer=pve vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=1 not_covered=0 restart_needed=11 reboot_needed=false wrapper_seconds=16.3
Oct 18 05:23:06 demo-hp felhom-agent[3320002]: time=2026-10-18T05:23:06.521+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-10-15T20:38:47Z) is already proven"
Oct 18 11:23:06 demo-hp felhom-agent[3320002]: time=2026-10-18T11:23:06.557+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-10-15T20:38:47Z) is already proven"
@@ -0,0 +1,4 @@
wrote envelope to /dev/null
uploaded signed op to the hub jobs queue
wrote envelope to /dev/null
uploaded signed op to the hub jobs queue
@@ -0,0 +1,8 @@
felhom-agent 0.159.2 11:08:30
Oct 18 13:08:17 demo-hp felhom-agent[3320002]: time=2026-10-18T13:08:17.765+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=763e0b8971dff83c op=agent_update
Oct 18 13:08:18 demo-hp felhom-os-apply[3566414]: os-apply: BUNDLE DONE agent=0.159.2 written=0 same=30 self-check=ok signers-created=False
Oct 18 13:08:18 demo-hp felhom-agent[3320002]: time=2026-10-18T13:08:18.862+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.159.2 written=0 same=30 self-check=ok signers-created=False"
Oct 18 13:08:19 demo-hp felhom-agent[3320002]: time=2026-10-18T13:08:19.883+02:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=75 total=75 degraded=""
Oct 18 13:08:19 demo-hp felhom-agent[3320002]: time=2026-10-18T13:08:19.884+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=268dfe26d949094f op=agent_config_update
[exited with code 0]
@@ -0,0 +1,7 @@
demo-hp-bb76ea last report 2026-10-18 12:38:33 agent 0.159.2 health None
tester-1-d70be4 last report 2026-10-18 12:39:04 agent 0.159.2 health None
demo-felhom-8363b5 last report 2026-10-18 12:25:17 agent 0.159.2 health None
report keys: ['addresses', 'agent_sha256', 'agent_version', 'audit_tail', 'backups', 'capabilities', 'cloudflared', 'controller_supervisor', 'dr_recipe', 'foreign_key_archives', 'guest_disk_trim', 'guest_net', 'guests', 'host', 'host_id', 'leaf_fingerprint', 'mgmt_plane', 'oob', 'pbs_dr', 'pbs_snapshots', 'reported_at', 'restore_tests', 'storage_targets', 'system', 'wireguard']
demo-hp-bb76ea 2026-10-18 12:38:33 agent_version 0.159.2 degraded None cap_ok/total None problems None
tester-1-d70be4 2026-10-18 12:39:04 agent_version 0.159.2 degraded None cap_ok/total None problems None
demo-felhom-8363b5 2026-10-18 12:25:17 agent_version 0.159.2 degraded None cap_ok/total None problems None
@@ -0,0 +1,2 @@
felhom-agent sha256 4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9
felhom-config-bundle.json sha256 477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9
@@ -0,0 +1,102 @@
[release-agent] building 0.159.2 …
[release-agent] built ok: sha256 4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9
[release-agent] config bundle built: sha256 477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9
[release-agent] tagging v0.159.2 at 65c1994 …
[release-agent] publishing …
[publish-agent] publishing /tmp/felhom-agent-bMXfqp (16007752 bytes, sha256 4d98868e922d6c10…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/0.159.2/felhom-agent
[publish-agent] pre-delete existing artifact: HTTP 404 (404/204 expected)
[publish-agent] upload OK (HTTP 201)
[publish-agent] round-trip GET verified (sha256 matches)
[publish-agent] DONE. Record in the hub operator UI (Configs → Day-0 artifacts): agent 0.159.2 / 4d98868e922d6c10665a6c0da9323e862AGENT_VERSION=0.159.2
AGENT_SHA256=4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9
pre-push [felhom-agent]: running scripts/agent_gates.py --fast ...
agent_gates — 4 gate(s) [--fast]
--fast SKIPPED (deliberate periodic runs, never in a hook): published, gate-decoys
==============================================================================
== gate: reuse-refs (reuse_refs_check.py /mnt/5_hdd/felhom.eu/git/felhom-agent)
==============================================================================
note [felhom-agent] line 189: localapi/debuglogs_test.go (resolved by suffix → internal/localapi/debuglogs_test.go)
note [felhom-agent] line 205: hub/internal/store/dr_recipe.go (cross-repo → felhom.eu/hub/internal/store/dr_recipe.go)
OK [felhom-agent]: 117 cited paths — exact 115, suffix 1, ambiguous 0, cross-repo 1, FAILED 0 (siblings searched: app-catalog-felhom.eu, cc-secrets, felhom-controller, felhom.eu, felhom.eu-media, felhom.eu-sizing-push, felhom.eu-wt-sizing)
==============================================================================
== gate: instructions (instructions_gate.py /mnt/5_hdd/felhom.eu/git/felhom-agent)
==============================================================================
instructions_gate: /mnt/5_hdd/felhom.eu/git/felhom-agent
CLAUDE.md effective lines : 108 (ceiling 200)
version literals : 0
TEMPORARY blocks : 0
rule files : 6 (5 path-scoped)
workspace file : SYMLINK -> felhom.eu/documentation/runbooks/workspace-CLAUDE.md (resolves to the versioned copy)
memory index : 200 lines (ceiling 200), 24953 bytes (ceiling 25600)
memory index content : 28 version literal(s), 3 host address(es), 0 expired statement(s) [WARN only]
memory topic files : 181 indexed, 4 orphaned, 41 archived
register citations : 6 cited, 1109 register items known
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:31: version literal '0.14.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- **[restic 0.14.0 `--verify` is size+mtime, NOT content](restic-0140-verify-and-lock-sema
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:89: version literal '0.142.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [Offsite continuity](offsite-continuity-2026-07-17.md) — ctrl 0.142.0 orphan guard + hub
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:89: version literal '0.60.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [Offsite continuity](offsite-continuity-2026-07-17.md) — ctrl 0.142.0 orphan guard + hub
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:96: version literal '0.152.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- **[Hub operator UI: CC uses its token HUB_TOKEN, not the password](hub-operator-ui-basic
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:100: version literal '0.90.1' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [Agent 0.90.1 is wrapper-only](agent-0901-is-wrapper-only.md) — do NOT publish the binar
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:102: version literal '0.144.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [SMB „Megosztás" R-7 slice 1](smb-sharing-r7-slice1-2026-07-18.md) — ctrl 0.144.0 + felh
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md: ... and 22 more version literal(s) — full list from the tally counts above.
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:58: host address '192.168.0.192' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [hub.felhom.eu resolves to the LAN here](hub-resolves-to-lan-on-dooplex-network.md) — 19
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:77: host address '167.233.158.164' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [Offsite PBS box RAM ceiling](offsite-pbs-box-ram-ceiling-2026-07-27.md) — **root SSH =
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:185: host address '192.168.0.180' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- **CC runs ON DooPlex** (192.168.0.180) — repos `/mnt/5_hdd/felhom.eu/git/felhom-*`, buil
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory: 4 top-level topic file(s) are not referenced by MEMORY.md, so nothing will ever read them: f10-demo-hp-offsite-resolved-2026-07-23.md, launcher-v0163-2026-07-24.md, peti-return-p1-stop-2026-07-13.md, stopped-not-fault-v0164-2026-07-24.md
instructions_gate: OK (11 warning(s))
==============================================================================
== gate: release-complete (check-release-complete.py)
==============================================================================
check-release-complete — the newest CHANGELOG version is a complete release
newest CHANGELOG version: v0.159.1
ok tag v0.159.1 -> 5c3b218822, an ancestor of HEAD
ok package 0.159.1 is downloadable
check-release-complete: v0.159.1 is tagged, placed and published.
==============================================================================
== gate: observations (observations_gate.py /mnt/5_hdd/felhom.eu/git/felhom-agent)
==============================================================================
observations gate OK — REPORT.md has no observations section (nothing to check)
==============================================================================
== summary
==============================================================================
reuse-refs OK (exit 0)
instructions OK (exit 0)
release-complete OK (exit 7ea9c589094e81d27d741e625d9d8f9
[release-agent] config bundle published (HTTP 201)
[release-agent] pushing v0.159.2 …
remote: . Processing 1 references
remote: Processed 1 references in total
To https://gitea.dooplex.hu0)
observations OK (exit 0)
all agent gates OK
pre-push [felhom-agent]: gates OK - push proceeding.
RELEASED — and installable, verified by download, not by this script's own say-so.
version : 0.159.2
tag : v0.159.2
sha256 : 4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9
bundle : 477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9 (felhom-config-bundle.json — vouch it with the agent)
NOT VOUCHED. Vouching is what points machines at this version and stays you/admin/felhom-agent.git
* [new tag] v0.159.2 -> v0.159.2
[release-agent] verifying by independent download …
r deliberate act:
hub operator UI → Configs → Day-0 artifacts. Until then boxes keep installing the previous one.
@@ -0,0 +1,4 @@
== published installer read back 2026-10-18T12:32:10Z
SCRIPT_VERSION="1.34.0" # the SINGLE version source (F-1): -h and the run banners follow it.
served sha256 df722ee49f38da595e7c7017aa932b45853696c5282bc408b67a5fb8f141b67a
tag sha256 df722ee49f38da595e7c7017aa932b45853696c5282bc408b67a5fb8f141b67a
@@ -0,0 +1,28 @@
# Website calculator — Immich before/after (2026-10-18)
Immich peak: **3,290 MB → 3,983 MB** (the kernel in-use figure from the 4,000-photo import, all four containers; the heap alone peaked at 2,777 MB). Immich typical: **1,557 MB, unchanged** (rest after the import: 1,230 MB heap — the raise-only rule keeps the higher). Headroom ×1.5, guest rule and platform unchanged.
## Immich alone — peak with the platform: 3,760 → 4,453 MB
| Machine | guest memory | before | after |
|---|---|---|---|
| 8 GB | 4,096 MB | fits, but tight | does not fit **(changes)** |
| 12 GB | 7,580 MB | fits well | fits well |
| 16 GB | 11,676 MB | fits well | fits well |
| 24 GB | 19,868 MB | fits well | fits well |
| 32 GB | 28,060 MB | fits well | fits well |
| 48 GB | 44,444 MB | fits well | fits well |
| 64 GB | 60,828 MB | fits well | fits well |
## Family starter set — peak with the platform: 6,021 → 6,714 MB
| Machine | guest memory | before | after |
|---|---|---|---|
| 8 GB | 4,096 MB | does not fit | does not fit |
| 12 GB | 7,580 MB | fits, but tight | fits, but tight |
| 16 GB | 11,676 MB | fits well | fits well |
| 24 GB | 19,868 MB | fits well | fits well |
| 32 GB | 28,060 MB | fits well | fits well |
| 48 GB | 44,444 MB | fits well | fits well |
| 64 GB | 60,828 MB | fits well | fits well |
@@ -0,0 +1,27 @@
=== demo-hp (enkisfelhom.hu)
containers: 24 running; not healthy/running: traefik Up 2 days;
unhealthy: 0
2026/10/18 11:09:36 updater.go:855: [INFO] [selfupdate] Post-update startup: update successful (0.329.0 → 0.330.0)
allow-list refusals since start: 0
2026/10/18 11:09:36 sync.go:142: [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
installed apps (stacks with app.yaml): 12
public https://felhom.enkisfelhom.hu/login: 200
public https://files.enkisfelhom.hu/: 200
=== demo-felhom (demo-felhom.eu)
containers: 9 running; not healthy/running: traefik Up 2 days;
unhealthy: 0
2026/10/18 11:09:25 [INFO] [selfupdate] Post-update startup: update successful (0.329.0 → 0.330.0)
allow-list refusals since start: 0
2026/10/18 11:09:26 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
installed apps (stacks with app.yaml): 5
public https://felhom.demo-felhom.eu/login: 200
public https://files.demo-felhom.eu/: 200
=== tester-1 (enkicsifelhom.hu)
containers: 10 running; not healthy/running: traefik Up 2 days;
unhealthy: 0
2026/10/18 11:09:35 [INFO] [selfupdate] Post-update startup: update successful (0.329.0 → 0.330.0)
allow-list refusals since start: 0
2026/10/18 11:09:36 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
installed apps (stacks with app.yaml): 3
public https://felhom.enkicsifelhom.hu/login: 200
public https://files.enkicsifelhom.hu/: 200
@@ -0,0 +1,12 @@
== global floor 2026-10-18T11:10:59Z
HTTP/1.1 303 See Other
Location: /configuration?flash=floor_set
clear tester-1:
HTTP/1.1 303 See Other
Location: /customers/tester-1?flash=floor_set
clear demo-hp:
HTTP/1.1 303 See Other
Location: /customers/demo-hp?flash=floor_set
clear demo-felhom:
HTTP/1.1 303 See Other
Location: /customers/demo-felhom?flash=floor_set
@@ -0,0 +1,9 @@
set tester-1:
HTTP/1.1 303 See Other
Location: /customers/tester-1?flash=floor_set
set demo-hp:
HTTP/1.1 303 See Other
Location: /customers/demo-hp?flash=floor_set
set demo-felhom:
HTTP/1.1 303 See Other
Location: /customers/demo-felhom?flash=floor_set
@@ -0,0 +1,3 @@
('min_controller_version', '0.330.0')
('min_controller_version_declared_min_agent', '0.330.0=0.158.1')
per-box floors set: 0
@@ -0,0 +1,2 @@
OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
== every public name redirects http -> https
@@ -0,0 +1,20 @@
# P8 — old-restic check: NOT RUN (the clean-up window has not opened). Read 2026-10-18T14:33:43Z.
## demo-hp's clean-up windows (hub DB with its WAL, one tar)
id|customer|opened_at|closes_by|closed_at|box_result
9|demo-hp|2026-10-12 02:16:53|2026-10-12 02:36:53|2026-10-12 02:17:06|pruned
6|demo-hp|2026-10-05 05:25:53|2026-10-05 05:45:53|2026-10-05 05:26:04|pruned
2|demo-hp|2026-10-04 05:36:14|2026-10-04 05:56:14|2026-10-04 05:36:17|nothing
## The rule (hub/internal/offsitekeys/service.go:255)
windowCadence = 6*24*time.Hour + 12*time.Hour // "weekly", with slack for the night chain's drift
Last window 2026-10-12 02:16:53 UTC + 6 d 12 h = 2026-10-18 14:16:53 UTC (the STATUS time).
The box asks for a window only AFTER its nightly off-site run (controller backup/offbox_window.go header).
## demo-hp guest 9201 controller (docker logs since 06:00 UTC, [offbox] lines + scheduler)
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.330.0 Up 3 hours (healthy)
2026/10/18 11:09:36 scheduler.go:150: [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-10-19 04:15 CEST
2026/10/18 11:09:36 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="offbox-backup" schedule="04:15" nextRun=2026-10-19T04:15:00+02:00 totalJobs=15
2026/10/18 11:09:36 scheduler.go:85: [DEBUG] [scheduler] daily job offbox-backup: next run at 2026-10-19 04:15:00 CEST (waiting 15h5m23s)
Next run: 2026-10-19 04:15 CEST = 02:15 UTC; the window opens right after it. No window was asked for or granted by CC.
@@ -0,0 +1,23 @@
Mon Oct 19 01:24:39 UTC 2026
200
{"data": {"started": true}, "ok": true}
200
{
"data": {
"backup": {
"target_id": "local",
"vmid": 9201,
"archive": "local:backup/vzdump-lxc-9201-2026_10_19-03_25_27.tar.zst",
"mode": "snapshot",
"crash_consistent": true,
"size_bytes": 6864786948,
"success": true,
"started_at": "2026-10-19T01:25:26Z",
"duration_seconds": 443.050964897
},
"error": "",
"job_id": "backup-9201-1792373126635435331",
"phase": "done"
},
"ok": true
}
@@ -0,0 +1,13 @@
Mon Oct 19 01:52:00 UTC 2026
tester-1-d70be4 agent_update:
signed: op=agent_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=3516434bfa0c5c47295e3de0a8bd20c7 expires=2026-10-19T02:37:00Z
uploaded signed op to the hub jobs queue
tester-1-d70be4 agent_config_update:
signed: op=agent_config_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=2dfc5b3e216030c82add18ba69899b41 expires=2026-10-19T02:37:00Z
uploaded signed op to the hub jobs queue
demo-felhom-8363b5 agent_update:
signed: op=agent_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=8f3a6c280cb124c6f85a4039c5b112dc expires=2026-10-19T02:37:00Z
uploaded signed op to the hub jobs queue
demo-felhom-8363b5 agent_config_update:
signed: op=agent_config_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=dc2ead6937a38d0a2d8acd4abf57bef9 expires=2026-10-19T02:37:00Z
uploaded signed op to the hub jobs queue
@@ -0,0 +1,17 @@
2026-10-19T01:55:42Z tester-1: felhom-agent 0.160.0 | demo-felhom: felhom-agent 0.160.0
Oct 19 03:54:05 felhom felhom-agent[1371657]: time=2026-10-19T03:54:05.642+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=a254c4ec38459c7c op=agent_update
Oct 19 03:54:06 felhom felhom-agent[1371657]: time=2026-10-19T03:54:06.786+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.160.0 written=0 same=30 self-check=ok signers-created=False"
Oct 19 03:54:07 felhom felhom-agent[1371657]: time=2026-10-19T03:54:07.652+02:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=75 total=75 degraded=""
Oct 19 03:54:07 felhom felhom-agent[1371657]: time=2026-10-19T03:54:07.653+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=a66f7fc0b9fd8fb6 op=agent_config_update
Oct 19 03:54:08 felhom felhom-agent[2443463]: time=2026-10-19T03:54:08.649+02:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true
Oct 19 03:54:08 felhom felhom-agent[2443463]: time=2026-10-19T03:54:08.649+02:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
Oct 19 03:55:10 felhom felhom-agent[2443463]: time=2026-10-19T03:55:10.315+02:00 level=WARN msg="selfupdate: update committed" version=0.160.0 wrapper=""
Oct 19 03:55:15 demo-felhom felhom-os-apply[498026]: os-apply: BUNDLE DONE agent=0.160.0 written=0 same=30 self-check=ok signers-created=False
Oct 19 03:55:15 demo-felhom felhom-agent[3800522]: time=2026-10-19T03:55:15.698+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.160.0 written=0 same=30 self-check=ok signers-created=False"
Oct 19 03:55:16 demo-felhom felhom-agent[3800522]: time=2026-10-19T03:55:16.340+02:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=75 total=75 degraded=""
Oct 19 03:55:16 demo-felhom felhom-agent[3800522]: time=2026-10-19T03:55:16.340+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=7dde89f299a0684c op=agent_config_update
Oct 19 03:55:17 demo-felhom felhom-agent[498196]: time=2026-10-19T03:55:17.698+02:00 level=INFO msg="backup tier armed" target=felhom-backup cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true
Oct 19 03:55:17 demo-felhom felhom-agent[498196]: time=2026-10-19T03:55:17.698+02:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
Oct 19 03:56:19 demo-felhom felhom-agent[498196]: time=2026-10-19T03:56:19.667+02:00 level=WARN msg="selfupdate: update committed" version=0.160.0 wrapper=""
[exited with code 0]
@@ -0,0 +1,29 @@
Volid Format Type Size VMID
local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst tar.zst backup 656970239 9100
local:backup/vzdump-lxc-9201-2026_10_19-03_25_27.tar.zst tar.zst backup 6864786948 9201
local:backup/felhom-golden-0.236.0.tar.zst tar.zst backup 654115664
Oct 19 03:23:37 demo-hp felhom-agent[2622137]: time=2026-10-19T03:23:37.628+02:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=1 prune="keep-daily=1" wait_timeout=30m0s prune_pbs_allowed=false primary=true
Oct 19 03:23:37 demo-hp felhom-agent[2622137]: time=2026-10-19T03:23:37.628+02:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
Oct 19 03:24:39 demo-hp felhom-agent[2622137]: time=2026-10-19T03:24:39.730+02:00 level=WARN msg="selfupdate: update committed" version=0.160.0 wrapper=""
Oct 19 03:32:49 demo-hp felhom-agent[2622137]: time=2026-10-19T03:32:49.686+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_10_19-03_25_27.tar.zst size_bytes=6864786948 uncovered_volumes=2
Oct 19 03:32:49 demo-hp felhom-agent[2622137]: time=2026-10-19T03:32:49.686+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1792373126635435331 archive=local:backup/vzdump-lxc-9201-2026_10_19-03_25_27.tar.zst
-rw------- 1 felhom-agent felhom-agent 323 Oct 19 03:32 /var/lib/felhom-agent/backup-success-state.json
[
{
"target": "felhom-pbs",
"vmid": 9201,
"started_at": "2026-10-15T20:38:46Z",
"recent": [
"2026-10-15T20:38:46Z"
]
},
{
"target": "local",
"vmid": 9201,
"started_at": "2026-10-19T01:25:26Z",
"recent": [
"2026-10-18T11:12:25Z",
"2026-10-19T01:25:26Z"
]
}
]
@@ -0,0 +1,33 @@
Mon Oct 19 01:22:11 UTC 2026
felhom-agent 0.159.2
demo-hp-bb76ea
Volid Format Type Size VMID
local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst tar.zst backup 656970239 9100
local:backup/vzdump-lxc-9201-2026_10_18-13_12_26.tar.zst tar.zst backup 6938307162 9201
local:backup/felhom-golden-0.236.0.tar.zst tar.zst backup 654115664
"local_backup_retention": 1
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
341 night1004-tester1 running 8192 200.00 1851
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
VMID Status Lock Name
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
@@ -0,0 +1,9 @@
Mon Oct 19 01:22:47 UTC 2026
demo-hp-bb76ea agent_update:
signed: op=agent_update host=demo-hp-bb76ea guest="" key_id=felhom-op-1 nonce=71396ac328c9ea3192d027b830ec0990 expires=2026-10-19T02:07:47Z
wrote envelope to /dev/null
uploaded signed op to the hub jobs queue
demo-hp-bb76ea agent_config_update:
signed: op=agent_config_update host=demo-hp-bb76ea guest="" key_id=felhom-op-1 nonce=603f1d8ed0937492cf01a05146fc9e48 expires=2026-10-19T02:07:47Z
wrote envelope to /dev/null
uploaded signed op to the hub jobs queue
@@ -0,0 +1,5 @@
Mon Oct 19 01:47:33 UTC 2026
code 0 (the stored demo-hp code, before):
HTTP 200 opens; restic_pw_sha256 8a9e33aa
control, a wrong code:
HTTP 400 the recovery code did not open the sealed bundle — nothing was written
@@ -0,0 +1,11 @@
01:47:43Z start 200 {"data":{"job_id":"escrow-1792374463410689900","phase":"running"},"error":"","ok":true}
status 200 {"phase": "done", "claimable": true, "uploaded": false, "restic_pw_sealed": true, "detail": ""}
claim 200 code words: 10 {"confirm_required": true, "verify_positions": [3, 10]}
status after claim {"phase": "done", "claimed": true, "uploaded": false}
confirm with one wrong word 422 A megadott szavak nem egyeznek — ellenőrizd a felírt kódot.
01:47:47Z confirm with the right words 200 {"uploaded": true}
code 1 after the confirm:
HTTP 200 opens; restic_pw_sha256 8a9e33aa
code 0 after the confirm:
HTTP 422 the recovery code is correct, but it belongs to an EARLIER sealed package (superseded 2026-10-19T01:47:47Z), not the one
@@ -0,0 +1,9 @@
01:48:09Z start 200 {"data":{"job_id":"escrow-1792374489199785658","phase":"running"},"error":"","ok":true}
status 200 {"phase": "done", "claimable": true, "uploaded": false, "restic_pw_sealed": true, "detail": ""}
claim 200 code words: 10 {"confirm_required": true, "verify_positions": [5, 10]}
status after claim {"phase": "done", "claimed": true, "uploaded": false}
after the abandon — code 1 (stored):
HTTP 200 opens; restic_pw_sha256 8a9e33aa
code 2 (shown, never typed back):
HTTP 400 the recovery code did not open the sealed bundle — nothing was written
@@ -0,0 +1,23 @@
time=2026-10-19T03:47:33.706+02:00 level=INFO msg="local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)" vmid=9201
time=2026-10-19T03:47:34.786+02:00 level=INFO msg="local-api: offsite repository password RECOVERED from the sealed escrow — returning that field ONLY (the tunnel token, the PBS token and the WG key stay inside the agent and are not returned)" vmid=9201 rest
time=2026-10-19T03:47:35.044+02:00 level=INFO msg="local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)" vmid=9201
time=2026-10-19T03:47:36.053+02:00 level=WARN msg="local-api: offsite key recovery: the fetched bundle did not open with the supplied recovery code" vmid=9201 err="escrow: the recovery code did not unwrap the identity escrow (wrong recovery code, or a corrupt
time=2026-10-19T03:47:43.409+02:00 level=INFO msg="local-api: staged offsite restic repo password for escrow (field name only)" vmid=9201
time=2026-10-19T03:47:43.410+02:00 level=INFO msg="local-api: escrow ceremony started (controller-driven)" vmid=9201 job_id=escrow-1792374463410689900
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=escrow-create --upload --output=json
time=2026-10-19T03:47:45.992+02:00 level=INFO msg="local-api: escrow ceremony complete — R claimable (in-memory, one-shot); package uploads at the claim" job_id=escrow-1792374463410689900 restic_pw_sealed=true uploaded=false claim_ttl_s=600 duration_ms=2581
time=2026-10-19T03:47:47.507+02:00 level=INFO msg="local-api: escrow ceremony R claimed — the package waits for the typed-back words (R-1151)" vmid=9201 job_id=escrow-1792374463410689900 confirm_ttl_s=1800
time=2026-10-19T03:47:47.550+02:00 level=INFO msg="local-api: escrow confirm — the typed words do not match; still waiting (R-1151)" vmid=9201
time=2026-10-19T03:47:47.619+02:00 level=INFO msg="local-api: escrow confirm — words typed back; the new package is on the hub (R-1151)" vmid=9201 job_id=escrow-1792374463410689900 duration_ms=48
time=2026-10-19T03:47:48.247+02:00 level=INFO msg="local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)" vmid=9201
time=2026-10-19T03:47:49.287+02:00 level=INFO msg="local-api: offsite repository password RECOVERED from the sealed escrow — returning that field ONLY (the tunnel token, the PBS token and the WG key stay inside the agent and are not returned)" vmid=9201 rest
time=2026-10-19T03:47:49.635+02:00 level=INFO msg="local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)" vmid=9201
time=2026-10-19T03:48:09.199+02:00 level=INFO msg="local-api: staged offsite restic repo password for escrow (field name only)" vmid=9201
time=2026-10-19T03:48:09.199+02:00 level=INFO msg="local-api: escrow ceremony started (controller-driven)" vmid=9201 job_id=escrow-1792374489199785658
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=escrow-create --upload --output=json
time=2026-10-19T03:48:11.881+02:00 level=INFO msg="local-api: escrow ceremony complete — R claimable (in-memory, one-shot); package uploads at the claim" job_id=escrow-1792374489199785658 restic_pw_sealed=true uploaded=false claim_ttl_s=600 duration_ms=2681
time=2026-10-19T03:48:13.294+02:00 level=INFO msg="local-api: escrow ceremony R claimed — the package waits for the typed-back words (R-1151)" vmid=9201 job_id=escrow-1792374489199785658 confirm_ttl_s=1800
time=2026-10-19T03:48:14.059+02:00 level=INFO msg="local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)" vmid=9201
time=2026-10-19T03:48:15.141+02:00 level=INFO msg="local-api: offsite repository password RECOVERED from the sealed escrow — returning that field ONLY (the tunnel token, the PBS token and the WG key stay inside the agent and are not returned)" vmid=9201 rest
time=2026-10-19T03:48:15.389+02:00 level=INFO msg="local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)" vmid=9201
time=2026-10-19T03:48:17.435+02:00 level=WARN msg="local-api: offsite key recovery: the fetched bundle did not open with the supplied recovery code" vmid=9201 err="escrow: the recovery code did not unwrap the identity escrow (wrong recovery code, or a corrupt
@@ -0,0 +1,4 @@
Mon Oct 19 02:08:15 UTC 2026
demo-felhom-8363b5|0.160.0|2026-10-19 01:55:31
demo-hp-bb76ea|0.160.0|2026-10-19 01:53:50
tester-1-d70be4|0.160.0|2026-10-19 01:54:15
@@ -0,0 +1,36 @@
== root@192.168.0.154
Mon Oct 19 01:51:20 UTC 2026
felhom-agent 0.159.2
"host_id": "tester-1-d70be4"
"local_backup_retention": 3
Volid Format Type Size VMID
local:backup/vzdump-lxc-9100-2026_10_04-21_40_45.tar.zst tar.zst backup 648135998 9100
local:backup/vzdump-lxc-9201-2026_10_15-22_43_40.tar.zst tar.zst backup 3364084456 9201
local:backup/vzdump-lxc-9201-2026_10_17-04_32_43.tar.zst tar.zst backup 3479976346 9201
local:backup/vzdump-lxc-9201-2026_10_18-04_37_41.tar.zst tar.zst backup 1867626596 9201
== felhom-pve
Mon Oct 19 01:51:13 UTC 2026
felhom-agent 0.159.2
"host_id": "demo-felhom-8363b5"
"local_backup_retention": 3
Volid Format Type Size VMID
local:backup/vzdump-lxc-9100-2026_07_18-18_30_52.tar.zst tar.zst backup 612397450 9100
local:backup/vzdump-lxc-9100-2026_07_20-17_50_57.tar.zst tar.zst backup 648549366 9100
local:backup/vzdump-lxc-9100-2026_08_03-06_58_00.tar.zst tar.zst backup 649547835 9100
local:backup/vzdump-lxc-9100-2026_08_03-09_15_34.tar.zst tar.zst backup 649547835 9100
local:backup/vzdump-lxc-9201-2026_07_28-01_27_56.tar.zst tar.zst backup 5820882781 9201
local:backup/vzdump-lxc-9201-2026_07_28-06_44_22.tar.zst tar.zst backup 5840583923 9201
local:backup/vzdump-lxc-9201-2026_07_28-17_53_26.tar.zst tar.zst backup 5929975285 9201
{'local_backup_target': 'felhom-backup', 'local_backup_retention': 3, 'restore_test_cadence_seconds': 0, 'backup_cadence_seconds': 0, 'backup_targets': [{'target_id': 'felhom-pbs', 'cadence_seconds': 604800, 'keep_last': 0}]}
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
felhom-backup dir active 960303848 7518604 903930732 0.78%
felhom-pbs pbs active 0 0 0 0.00%
local dir active 98497780 33631712 59816520 34.14%
local-lvm lvmthin active 365760512 11082543 354677968 3.03%
time=2026-10-16T05:03:53.319+02:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-backup archive=felhom-backup:backup/vzdump-lxc-9201-2026_10_16-05_03_12.tar.zst size_bytes=1300947312 uncovered_volumes=2
time=2026-10-17T05:07:43.382+02:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-backup archive=felhom-backup:backup/vzdump-lxc-9201-2026_10_17-05_07_02.tar.zst size_bytes=1334976334 uncovered_volumes=2
time=2026-10-18T05:08:13.307+02:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-backup archive=felhom-backup:backup/vzdump-lxc-9201-2026_10_18-05_07_32.tar.zst size_bytes=1296570428 uncovered_volumes=2
Volid Format Type Size VMID
felhom-backup:backup/vzdump-lxc-9201-2026_10_16-05_03_12.tar.zst tar.zst backup 1300947312 9201
felhom-backup:backup/vzdump-lxc-9201-2026_10_17-05_07_02.tar.zst tar.zst backup 1334976334 9201
felhom-backup:backup/vzdump-lxc-9201-2026_10_18-05_07_32.tar.zst tar.zst backup 1296570428 9201
@@ -0,0 +1,32 @@
time=2026-10-19T03:49:56.342+02:00 level=INFO msg="audit: gate decision" class=storage_wipe host=demo-hp-bb76ea guest="" source=one_shot_job disposition=customer_confirmable allowed=false reason=pending_confirmation key_id="" nonce="" durab
time=2026-10-19T03:49:56.342+02:00 level=INFO msg="gate decision (storage wipe)" role=user-data disposition=customer_confirmable allowed=false reason=pending_confirmation durable_id=byid:wwn-0x3333333000000fa0-part1+fsuuid:ef8fc766-1e12-491
time=2026-10-19T03:49:56.342+02:00 level=WARN msg="local-api: user-data data-bearing format refused" vmid=9201 device=/dev/sdb1 durable_id=byid:wwn-0x3333333000000fa0-part1+fsuuid:ef8fc766-1e12-491e-b6f2-31433e1989c8 reason=pending_confirma
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
backup-success-state.json
controller-swap-9201.json
drive-intents.json
escrow-stage
felhom-sshd
format-job.json
guest-binds.json
guest-disk-trim.json
guests
journal.log
local-api.crt
local-api.key
local-tokens.log
nonces.log
os
pbsdr
restore-test-state.json
selfupdate
units
wg
@@ -0,0 +1,39 @@
Mon Oct 19 01:49:49 UTC 2026
new test disk: /dev/sdb
NAME SIZE MODEL
sdb 64M scsi_debug
--- the empty partition (the R-1093 input):
{
"blockdevices": [
{
"name": "sdb1",
"fstype": null,
"pttype": "gpt"
}
]
}
DEVNAME=/dev/sdb1
PART_ENTRY_SCHEME=gpt
PART_ENTRY_UUID=aec56a69-b9a0-46de-9c8e-911ca8558717
PART_ENTRY_TYPE=0fc63daf-8483-4772-8e79-3d69d8477de4
PART_ENTRY_NUMBER=1
PART_ENTRY_OFFSET=2048
PART_ENTRY_SIZE=126976
PART_ENTRY_DISK=8:16
blkid rc=0
lrwxrwxrwx 1 root root 9 Oct 19 03:49 scsi-33333333000000fa0 -> ../../sdb
lrwxrwxrwx 1 root root 10 Oct 19 03:49 scsi-33333333000000fa0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 9 Oct 19 03:49 wwn-0x3333333000000fa0 -> ../../sdb
--- POST /disks/format on the EMPTY partition (no confirmation sent):
HTTP 200 {"ok":true,"data":{"vmid":9201,"device":"/dev/sdb1","formatted":true,"data_bearing":false,"reason":"blank device formatted ext4","durable_id":"byid:wwn-0x3333333000000fa0-part1","fs_uuid":"ef8fc766-1e12-491e-b6f2-31433e1989c8"}}
--- the partition now:
TYPE=ext4
PART_ENTRY_SCHEME=gpt
--- CONTROL: the same call again, on the now-ext4 partition:
HTTP 403 {"ok":false,"data":{"vmid":9201,"device":"/dev/sdb1","formatted":false,"data_bearing":true,"reason":"device has a ext4 filesystem","role":"user-data","needs_confirmation":true,"durable_id":"byid:wwn-0x3333333000000fa0-part1+fsuuid:ef8fc766-1e12-491e-b6f2-31433e1989c8"},"error":"device is data-bearing — customer confirmation required"}
--- teardown
not mounted
scsi_debug removed
0
@@ -0,0 +1,102 @@
[release-agent] building 0.160.0 …
[release-agent] built ok: sha256 7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1
[release-agent] config bundle built: sha256 6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f
[release-agent] tagging v0.160.0 at c11e3e2 …
[release-agent] publishing …
[publish-agent] publishing /tmp/felhom-agent-YBD0P9 (16071264 bytes, sha256 7ebb547f2b1ab935…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/0.160.0/felhom-agent
[publish-agent] pre-delete existing artifact: HTTP 404 (404/204 expected)
[publish-agent] upload OK (HTTP 201)
[publish-agent] round-trip GET verified (sha256 matches)
[publish-agent] DONE. Record in the hub operator UI (Configs → Day-0 artifacts): agent 0.160.0 / 7ebb547f2b1ab93544ba828d3040351b6AGENT_VERSION=0.160.0
AGENT_SHA256=7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1
pre-push [felhom-agent]: running scripts/agent_gates.py --fast ...
agent_gates — 4 gate(s) [--fast]
--fast SKIPPED (deliberate periodic runs, never in a hook): published, gate-decoys
==============================================================================
== gate: reuse-refs (reuse_refs_check.py /mnt/5_hdd/felhom.eu/git/felhom-agent)
==============================================================================
note [felhom-agent] line 190: localapi/debuglogs_test.go (resolved by suffix → internal/localapi/debuglogs_test.go)
note [felhom-agent] line 206: hub/internal/store/dr_recipe.go (cross-repo → felhom.eu/hub/internal/store/dr_recipe.go)
OK [felhom-agent]: 118 cited paths — exact 116, suffix 1, ambiguous 0, cross-repo 1, FAILED 0 (siblings searched: app-catalog-felhom.eu, cc-secrets, felhom-controller, felhom.eu, felhom.eu-media, felhom.eu-sizing-push, felhom.eu-wt-sizing)
==============================================================================
== gate: instructions (instructions_gate.py /mnt/5_hdd/felhom.eu/git/felhom-agent)
==============================================================================
instructions_gate: /mnt/5_hdd/felhom.eu/git/felhom-agent
CLAUDE.md effective lines : 108 (ceiling 200)
version literals : 0
TEMPORARY blocks : 0
rule files : 6 (5 path-scoped)
workspace file : SYMLINK -> felhom.eu/documentation/runbooks/workspace-CLAUDE.md (resolves to the versioned copy)
memory index : 200 lines (ceiling 200), 25109 bytes (ceiling 25600)
memory index content : 28 version literal(s), 3 host address(es), 0 expired statement(s) [WARN only]
memory topic files : 183 indexed, 4 orphaned, 41 archived
register citations : 6 cited, 1116 register items known
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:31: version literal '0.14.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- **[restic 0.14.0 `--verify` is size+mtime, NOT content](restic-0140-verify-and-lock-sema
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:90: version literal '0.142.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [Offsite continuity](offsite-continuity-2026-07-17.md) — ctrl 0.142.0 orphan guard + hub
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:90: version literal '0.60.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [Offsite continuity](offsite-continuity-2026-07-17.md) — ctrl 0.142.0 orphan guard + hub
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:97: version literal '0.152.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- **[Hub operator UI: CC uses its token HUB_TOKEN, not the password](hub-operator-ui-basic
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:101: version literal '0.90.1' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [Agent 0.90.1 is wrapper-only](agent-0901-is-wrapper-only.md) — do NOT publish the binar
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:103: version literal '0.144.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [SMB „Megosztás" R-7 slice 1](smb-sharing-r7-slice1-2026-07-18.md) — ctrl 0.144.0 + felh
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md: ... and 22 more version literal(s) — full list from the tally counts above.
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:59: host address '192.168.0.192' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [hub.felhom.eu resolves to the LAN here](hub-resolves-to-lan-on-dooplex-network.md) — 19
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:78: host address '167.233.158.164' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [Offsite PBS box RAM ceiling](offsite-pbs-box-ram-ceiling-2026-07-27.md) — **root SSH =
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:186: host address '192.168.0.180' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- **CC runs ON DooPlex** (192.168.0.180) — repos `/mnt/5_hdd/felhom.eu/git/felhom-*`, buil
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory: 4 top-level topic file(s) are not referenced by MEMORY.md, so nothing will ever read them: f10-demo-hp-offsite-resolved-2026-07-23.md, launcher-v0163-2026-07-24.md, peti-return-p1-stop-2026-07-13.md, stopped-not-fault-v0164-2026-07-24.md
instructions_gate: OK (11 warning(s))
==============================================================================
== gate: release-complete (check-release-complete.py)
==============================================================================
check-release-complete — the newest CHANGELOG version is a complete release
newest CHANGELOG version: v0.159.2
ok tag v0.159.2 -> 65c1994d3b, an ancestor of HEAD
ok package 0.159.2 is downloadable
check-release-complete: v0.159.2 is tagged, placed and published.
==============================================================================
== gate: observations (observations_gate.py /mnt/5_hdd/felhom.eu/git/felhom-agent)
==============================================================================
observations gate OK — REPORT.md has no observations section (nothing to check)
==============================================================================
== summary
==============================================================================
reuse-refs OK (exit 0)
instructions OK (exit 0)
release-complete OK (exit 0c4126327a5b85bbcb33f8f2b3dfca1
[release-agent] config bundle published (HTTP 201)
[release-agent] pushing v0.160.0 …
remote: . Processing 1 references
remote: Processed 1 references in total
To https://gitea.dooplex.hu0)
observations OK (exit 0)
all agent gates OK
pre-push [felhom-agent]: gates OK - push proceeding.
RELEASED — and installable, verified by download, not by this script's own say-so.
version : 0.160.0
tag : v0.160.0
sha256 : 7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1
bundle : 6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f (felhom-config-bundle.json — vouch it with the agent)
NOT VOUCHED. Vouching is what points machines at this version and stays you/admin/felhom-agent.git
* [new tag] v0.160.0 -> v0.160.0
[release-agent] verifying by independent download …
r deliberate act:
hub operator UI → Configs → Day-0 artifacts. Until then boxes keep installing the previous one.
Binary file not shown.

After

Width:  |  Height:  |  Size: 99 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 79 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 36 KiB

@@ -0,0 +1,17 @@
03:05:34 goto https://felhom.enkisfelhom.hu/login -> https://felhom.enkisfelhom.hu/login
03:05:39 eval -> "focused"
03:05:43 eval -> "https://felhom.enkisfelhom.hu/launcher"
03:05:43 goto https://felhom.enkisfelhom.hu/apps/calibre-web -> https://felhom.enkisfelhom.hu/apps/calibre-web
03:05:48 eval -> "https://felhom.enkisfelhom.hu/__files/open?to=… | button: Fájlok ↗ | folder: /files/NVME 1TB/media/books"
03:05:48 shot 01-app-page-files-button
03:05:49 eval -> "pressed Fájlok"
03:05:57 eval -> "files.enkisfelhom.hu/files/NVME 1TB/media/books | login form: no | title: FileBrowser Quantum - Files - books"
03:05:57 eval -> "FileBrowser user: felhom-household | admin: false | modify: true | loginMethod: proxy"
03:05:59 shot 02-filebrowser-opened-signed-in
03:05:59 goto https://felhom.enkisfelhom.hu/launcher -> https://felhom.enkisfelhom.hu/launcher
03:06:04 eval -> "signed out on the dashboard"
03:06:08 eval -> "https://felhom.enkisfelhom.hu/login"
03:06:08 goto https://files.enkisfelhom.hu/files/ -> https://felhom.enkisfelhom.hu/login
03:06:15 eval -> "felhom.enkisfelhom.hu/login | login form: YES"
03:06:15 shot 03-after-sign-out-asks-again
LOG ('netfail', 'net::ERR_ABORTED', 'Fetch')
@@ -0,0 +1,9 @@
== through traefik (https://files.enkisfelhom.hu, no files cookie), GET /api/users?id=self
forged header named X-Felhom-User -> HTTP 401 (user None)
forged header named RANDOM -> HTTP 401 (user None)
forged header named SECRET -> HTTP 401 (user None)
== from the bookstack container (on traefik-public), straight to http://filebrowser:80/api/users?id=self — no traefik
forged header named X-Felhom-User -> HTTP 401 (user None)
forged header named RANDOM -> HTTP 401 (user None)
== CONTROL: the same container with the box's REAL secret name (read from the box, never printed)
real secret name -> HTTP 200 (user felhom-household)
@@ -0,0 +1,4 @@
Mon Oct 19 03:08:04 UTC 2026
200
{
"ok": true,
@@ -0,0 +1,5 @@
Mon Oct 19 03:08:23 UTC 2026
05:08:24 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
05:09:29 [1] deployed, controller state=running, pinned={'onlyoffice': 'onlyoffice/documentserver:9.4.0'}
deploy: True
Mon Oct 19 03:09:29 UTC 2026
@@ -0,0 +1,16 @@
Mon Oct 19 03:09:40 UTC 2026
onlyoffice health: healthy
200
{"ok": true, "data": {"enabled": false, "status": "disabled"}, "message": "Integráció kikapcsolva"}
03:09:42 files.<domain> 200
onlyoffice Up 28 seconds (healthy)
filebrowser Up 6 hours (healthy)
Mon Oct 19 03:10:35 UTC 2026
200
{"ok": true, "data": {"enabled": true, "enabled_at": "2026-10-19T03:10:36Z", "status": "active"}, "message": "Integráció engedélyezve"}
03:10:39 files.<domain> 200 (try 1)
filebrowser Up 5 seconds (healthy)
onlyoffice Up About a minute (healthy)
url: "https://office.enkisfelhom.hu"
internalUrl: "http://onlyoffice:80"
2
@@ -0,0 +1,4 @@
login 200 token_len 355
resource 200 application/vnd.openxmlformats-officedocument.wordprocessingml.document
03:12:19Z preview small HTTP 500 content-type application/json; charset=utf-8 bytes 167 in 0.0s b'{"status":500,"message":"failed to create image for PDF file: failed to open PDF from memory for file \'/srv/scratch_hdd/r1153/teszt.docx\': fitz: cannot open document"}'
03:12:19Z preview large HTTP 500 content-type application/json; charset=utf-8 bytes 167 in 0.0s b'{"status":500,"message":"failed to create image for PDF file: failed to open PDF from memory for file \'/srv/scratch_hdd/r1153/teszt.docx\': fitz: cannot open document"}'
@@ -0,0 +1,10 @@
== /r1153/sample.docx
login 200 token_len 355
resource 200 application/vnd.openxmlformats-officedocument.wordprocessingml.document
03:13:40Z preview small HTTP 200 content-type image/jpeg bytes 8714 in 0.3s
03:13:40Z preview large HTTP 200 content-type image/jpeg bytes 22734 in 0.3s
== /r1153/teszt.docx
login 200 token_len 355
resource 200 application/vnd.openxmlformats-officedocument.wordprocessingml.document
03:13:40Z preview small HTTP 500 content-type application/json; charset=utf-8 bytes 167 in 0.0s b'{"status":500,"message":"failed to create image for PDF file: failed to open PDF from memory for file \'/srv/scratch_hdd/r1153/teszt.docx\': fitz: cannot open document"}'
03:13:40Z preview large HTTP 500 content-type application/json; charset=utf-8 bytes 167 in 0.0s b'{"status":500,"message":"failed to create image for PDF file: failed to open PDF from memory for file \'/srv/scratch_hdd/r1153/teszt.docx\': fitz: cannot open document"}'
@@ -0,0 +1,9 @@
Mon Oct 19 03:13:53 UTC 2026
{"ok": true, "data": {"enabled": false, "status": "disabled"}, "message": "Integráció kikapcsolva"}
03:13:57 files.<domain> 200 (try 1)
== link OFF, /r1153/sample.docx
03:13:58Z preview small HTTP 200 content-type image/jpeg bytes 8714 in 0.2s
03:13:58Z preview large HTTP 200 content-type image/jpeg bytes 22734 in 0.2s
== link OFF, /r1153/teszt.docx
03:13:58Z preview small HTTP 500 content-type application/json; charset=utf-8 bytes 167 in 0.0s b'{"status":500,"message":"failed to create image for PDF file: failed to open PDF from memory for file \'/srv/scratch_hdd/r1153/teszt.docx\': fitz: cannot open document"}'
03:13:58Z preview large HTTP 500 content-type application/json; charset=utf-8 bytes 167 in 0.0s b'{"status":500,"message":"failed to create image for PDF file: failed to open PDF from memory for file \'/srv/scratch_hdd/r1153/teszt.docx\': fitz: cannot open document"}'
@@ -0,0 +1,8 @@
Mon Oct 19 03:14:09 UTC 2026
test files removed
05:14:54 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'onlyoffice', 'volumes_removed': ['onlyoffice_onlyoffice_data', 'onlyoffice_onlyoffice_logs'], 'hdd_paths_removed': [], 'hdd_pa
05:15:03 [X] after remove: deployed=False leftovers='/opt/docker/stacks/onlyoffice'
remove: 200
0
0
ls: cannot access '/opt/docker/stacks/onlyoffice/app.yaml': No such file or directory
Binary file not shown.

After

Width:  |  Height:  |  Size: 8.5 KiB

@@ -0,0 +1,4 @@
## per-customer floors 0.332.0 / MinAgent 0.158.1 — 03:16:43
tester-1: HTTP/1.1 303 See Other Location: /customers/tester-1?flash=floor_set
demo-hp: HTTP/1.1 303 See Other Location: /customers/demo-hp?flash=floor_set
demo-felhom: HTTP/1.1 303 See Other Location: /customers/demo-felhom?flash=floor_set
@@ -0,0 +1,14 @@
03:16:55 demo-hp=0.332.0-rc.3 tester-1=0.332.0 demo-felhom=0.332.0
03:17:28 demo-hp=0.332.0
== hp
2026/10/19 03:16:57 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="selfupdate-check" interval=6h0m0s totalJobs=23
2026/10/19 03:17:04 updater.go:104: [DEBUG] [selfupdate] SetFloor: floor "" → "0.332.0"
2026/10/19 03:17:04 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.332.0 >= floor 0.332.0 — no action
== root@192.168.0.154
2026/10/19 03:16:55 [INFO] [selfupdate] Post-update startup: update successful (0.331.0 → 0.332.0)
2026/10/19 03:16:55 [INFO] [scheduler] Registered periodic job: selfupdate-check (every 6h0m0s)
2026/10/19 03:17:26 [INFO] [selfupdate] Current version 0.332.0 is up to date
== felhom-pve
2026/10/19 03:16:45 [INFO] [selfupdate] Post-update startup: update successful (0.331.0 → 0.332.0)
2026/10/19 03:16:45 [INFO] [scheduler] Registered periodic job: selfupdate-check (every 6h0m0s)
2026/10/19 03:17:17 [INFO] [selfupdate] Current version 0.332.0 is up to date
@@ -0,0 +1,33 @@
=== demo-hp (enkisfelhom.hu) 03:17:58
containers: 24 running; not healthy/running: traefik Up 3 days;
unhealthy: 0
2026/10/19 03:16:57 updater.go:855: [INFO] [selfupdate] Post-update startup: update successful (0.332.0-rc.3 → 0.332.0)
allow-list refusals since start: 0
2026/10/19 03:16:58 sync.go:142: [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
installed apps (stacks with app.yaml): 12
public https://felhom.enkisfelhom.hu/login: 200
public https://files.enkisfelhom.hu/: 200
=== demo-felhom (demo-felhom.eu) 03:18:03
containers: 9 running; not healthy/running: traefik Up 3 days;
unhealthy: 0
2026/10/19 03:16:45 [INFO] [selfupdate] Post-update startup: update successful (0.331.0 → 0.332.0)
allow-list refusals since start: 0
2026/10/19 03:16:46 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
installed apps (stacks with app.yaml): 5
2026/10/19 03:16:45 [INFO] [infra] wrote filebrowser route → /opt/docker/stacks/traefik/dynamic/filebrowser.yml (Host files.demo-felhom.eu → filebrowser:80 by name, R-1126; one sign-in true, R-1100)
2026/10/19 03:16:46 [INFO] [web] FileBrowser mounts synced (recreated) — 1 storage path(s), config updated
2026/10/19 03:16:52 [INFO] [infra] filebrowser: default login refused for this run — joined traefik-public, files.<domain> served (R-1126)
public https://felhom.demo-felhom.eu/login: 200
public https://files.demo-felhom.eu/: 200
=== tester-1 (enkicsifelhom.hu) 03:18:06
containers: 10 running; not healthy/running: traefik Up 3 days;
unhealthy: 0
2026/10/19 03:16:55 [INFO] [selfupdate] Post-update startup: update successful (0.331.0 → 0.332.0)
allow-list refusals since start: 0
2026/10/19 03:16:56 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
installed apps (stacks with app.yaml): 3
2026/10/19 03:16:55 [INFO] [infra] wrote filebrowser route → /opt/docker/stacks/traefik/dynamic/filebrowser.yml (Host files.enkicsifelhom.hu → filebrowser:80 by name, R-1126; one sign-in true, R-1100)
2026/10/19 03:16:56 [INFO] [web] FileBrowser mounts synced (recreated) — 5 storage path(s), config updated
2026/10/19 03:17:02 [INFO] [infra] filebrowser: default login refused for this run — joined traefik-public, files.<domain> served (R-1126)
public https://felhom.enkicsifelhom.hu/login: 200
public https://files.enkicsifelhom.hu/: 200
@@ -0,0 +1,33 @@
control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -)
== tester-1 (enkicsifelhom.hu): 6 public name(s)
OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1
OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1
OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1
OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1
OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1
OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1
== demo-hp (enkisfelhom.hu): 14 public name(s)
OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1
OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1
OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1
OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1
OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1
OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1
OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1
OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1
OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1
OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1
OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1
OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1
OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1
OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1
== demo-felhom (demo-felhom.eu): 8 public name(s)
OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1
OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1
OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1
OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1
OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1
OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1
OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1
OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
== every public name redirects http -> https
@@ -0,0 +1,16 @@
=== demo-hp (enkisfelhom.hu) 03:23:50
dashboard sign-in: 302
the Fajlok button goes to: https://files.enkisfelhom.hu/_felhom/sso
the pass answer goes to: https://files.enkisfelhom.hu/files/; files cookie set: 1
FileBrowser with the files cookie: felhom-household admin: False modify: True
FileBrowser without it, a fresh browser: HTTP 401
dashboard sign-out: 302
after the sign-out, the files cookie sends the browser to: https://felhom.enkisfelhom.hu/__files/open?to=%2Ffiles%2F
=== tester-1 (enkicsifelhom.hu) 03:23:51
dashboard sign-in: 302
the Fajlok button goes to: https://files.enkicsifelhom.hu/_felhom/sso
the pass answer goes to: https://files.enkicsifelhom.hu/files/; files cookie set: 1
FileBrowser with the files cookie: felhom-household admin: False modify: True
FileBrowser without it, a fresh browser: HTTP 401
dashboard sign-out: 302
after the sign-out, the files cookie sends the browser to: https://felhom.enkicsifelhom.hu/__files/open?to=%2Ffiles%2F
@@ -0,0 +1,7 @@
=== demo-felhom (demo-felhom.eu) 03:24:06 — no dashboard password is held for this box: file and refusal checks only
-rw-------
valid shape: 1
config carries the secret: 1, route carries it: 1
forged secret name through traefik, no files cookie: HTTP 401
from outside, old public name: HTTP 401
from outside, the pass path with a forged pass: HTTP 403
@@ -0,0 +1,16 @@
## global floor 0.332.0 / 0.158.1 — 03:24:18
HTTP/1.1 303 See Other
Location: /configuration?flash=floor_set
## read back (DB with WAL) 03:24:18
min_controller_version|0.332.0
min_controller_version_declared_min_agent|0.332.0=0.158.1
## per-box floors cleared — 03:24:28
tester-1: HTTP/1.1 303 See Other Location: /customers/tester-1?flash=floor_set
demo-hp: HTTP/1.1 303 See Other Location: /customers/demo-hp?flash=floor_set
demo-felhom: HTTP/1.1 303 See Other Location: /customers/demo-felhom?flash=floor_set
min_controller_declared_min_agent
min_controller_set_at
min_controller_version
demo-felhom|''|''
demo-hp|''|''
tester-1|''|''
@@ -0,0 +1,29 @@
Mon Oct 19 02:14:15 UTC 2026
application.argoproj.io/felhom patched
deployment "hub" successfully rolled out
sync=Synced health=Progressing rev=200ab6530bf2bec9a9b341e4bc8e2bfd09f6a609
gitea.dooplex.hu/admin/felhom-hub:0.159.0
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hub-67bff8bcb6-8pgxf 0/1 Running 0 9s 10.42.0.56 dooplex <none> <none>
2026/10/19 04:14:21 [INFO] Offsite pool-box checker initialized: box=611714 fill warn=80% crit=90%, oversub warn=2.00x, unreachable after 3 consecutive failed reads, refresh 15m0s
2026/10/19 04:14:21 [INFO] osupdates: approval rule = every ring-0 box healthy for 24h0m0s and 1 night run(s)
2026/10/19 04:14:21 [INFO] osupdates: the Docker engine set is approved only by the operator, after 2 healthy ring-0 night(s)
2026/10/19 04:14:21 [INFO] osupdates: alarms = no OS leg 168h0m0s, reboot needed 336h0m0s, ring 0 stalled 168h0m0s, not covered 336h0m0s
2026/10/19 04:14:21 [INFO] Off-site key registrar enabled; daily key check at 07:10 Budapest
2026/10/19 04:14:21 [INFO] access log: /data/access-log, one file per UTC day, kept 90 days
2026/10/19 04:14:21 [INFO] Registry version checker started (every 6h)
2026/10/19 04:14:21 [INFO] offsite-key-audit: next run at 2026-10-19 07:10 CEST (in 2h55m39s)
2026/10/19 04:14:21 [INFO] WG peer-sync enabled (endpoint 167.233.158.164:22, user felhom-peersync)
2026/10/19 04:14:21 [INFO] PBS DR tenantsync enabled (endpoint 167.233.158.164:22, user felhom-peersync; WG-registration auto-provision hook armed)
2026/10/19 04:14:21 [INFO] PBS-DR box checker initialized: fill warn=80% crit=90%, unreachable after 3 consecutive failed reads, refresh 15m0s
2026/10/19 04:14:21 [INFO] agent-plane poke enabled (endpoint 167.233.158.164:22, user felhom-peersync; web + api admin seams armed)
2026/10/19 04:14:21 [INFO] PBS-DR self-heal reconciler started (interval 5m)
2026/10/19 04:14:21 [INFO] offsite credential self-heal reconciler started (interval 5m, debounce 2 reports)
2026/10/19 04:14:21 [DEBUG] Registry version check: latest = 0.331.0
Mon Oct 19 02:15:36 UTC 2026
sync=Synced health=Healthy
hub-67bff8bcb6-8pgxf 1/1 Running 0 79s
--- floor (DB with WAL):
min_controller_version|0.331.0
min_controller_version_declared_min_agent|0.331.0=0.158.1
--- errors since start:
@@ -0,0 +1,9 @@
Mon Oct 19 02:32:34 UTC 2026
hosts (agent reports) after the 02:14:21 start:
demo-felhom-8363b5|0.160.0|2026-10-19 02:25:35
demo-hp-bb76ea|0.160.0|2026-10-19 02:23:50
tester-1-d70be4|0.160.0|2026-10-19 02:24:21
controller reports:
demo-felhom|2026-10-19 02:29:12
demo-hp|2026-10-19 02:31:13
tester-1|2026-10-19 02:29:13
@@ -0,0 +1,11 @@
control (before the window): jobs with run_attempt>1: 17
('felhom-controller', 1401, 2, '2026-10-06T00:44:57Z', 'success')
('felhom-controller', 1357, 2, '2026-10-05T15:05:37Z', 'success')
('felhom-controller', 1549, 2, '2026-10-08T08:26:11Z', 'success')
('felhom-controller', 1639, 2, '2026-10-10T13:09:06Z', 'success')
('felhom-agent', 1579, 2, '2026-10-09T04:58:15Z', 'success')
('app-catalog-felhom.eu', 1368, 2, '2026-10-05T17:15:12Z', 'success')
window since 2026-10-12T12:00:00Z : jobs with run_attempt>1: 3
('felhom-controller', 1944, 2, '2026-10-16T13:50:35Z', 'success')
('felhom-controller', 1959, 2, '2026-10-16T17:07:53Z', 'success')
('felhom-controller', 1991, 2, '2026-10-17T09:59:02Z', 'success')
@@ -0,0 +1,4 @@
felhom.eu 1645 log bytes 38590
felhom.eu 1646 log bytes 38503
felhom-controller 1639 log bytes 18965
felhom.eu 1738 log bytes 46399
@@ -0,0 +1,26 @@
window since 2026-10-12T12:00:00Z: 381 jobs in 4 repos; not-success 25; with no log (lost) 0
not-success ('felhom.eu', 1738, 'c2a6cf7a', 'completed', 'failure', '2026-10-13T04:47:12Z', 1) has log
not-success ('felhom.eu', 1747, 'ed59bf3c', 'completed', 'failure', '2026-10-13T17:28:57Z', 1) has log
not-success ('felhom.eu', 1796, 'a2ca6eb4', 'completed', 'failure', '2026-10-14T16:51:48Z', 1) has log
not-success ('felhom.eu', 1889, '6749118c', 'completed', 'failure', '2026-10-15T19:21:34Z', 1) has log
not-success ('felhom.eu', 1993, 'b9c4f73b', 'completed', 'failure', '2026-10-17T10:02:08Z', 1) has log
not-success ('felhom.eu', 1997, '498b9139', 'completed', 'failure', '2026-10-17T10:42:11Z', 1) has log
not-success ('felhom-controller', 1865, 'e66d4669', 'completed', 'failure', '2026-10-15T16:37:58Z', 1) has log
not-success ('felhom-controller', 1867, 'fe0228c8', 'completed', 'failure', '2026-10-15T17:05:00Z', 1) has log
not-success ('felhom-controller', 1869, 'db52f5a4', 'completed', 'failure', '2026-10-15T17:27:25Z', 1) has log
not-success ('felhom-controller', 1870, '83392064', 'completed', 'failure', '2026-10-15T17:32:48Z', 1) has log
not-success ('felhom-controller', 1871, '9cc7ffcf', 'completed', 'failure', '2026-10-15T18:08:11Z', 1) has log
not-success ('felhom-controller', 1873, '7f3e58ff', 'completed', 'failure', '2026-10-15T18:30:00Z', 1) has log
not-success ('felhom-controller', 1875, '48d86748', 'completed', 'failure', '2026-10-15T18:33:27Z', 1) has log
not-success ('felhom-controller', 1876, '006ceab5', 'completed', 'failure', '2026-10-15T18:41:12Z', 1) has log
not-success ('felhom-controller', 1877, 'f908eee2', 'completed', 'failure', '2026-10-15T18:45:59Z', 1) has log
not-success ('felhom-controller', 1878, '02252556', 'completed', 'failure', '2026-10-15T18:46:27Z', 1) has log
not-success ('felhom-controller', 1879, '9e68e526', 'completed', 'failure', '2026-10-15T18:51:46Z', 1) has log
not-success ('felhom-controller', 1880, '8c7aa10a', 'completed', 'failure', '2026-10-15T18:52:08Z', 1) has log
not-success ('felhom-controller', 1882, 'ec438479', 'completed', 'failure', '2026-10-15T18:57:00Z', 1) has log
not-success ('felhom-controller', 1883, 'f9151f08', 'completed', 'failure', '2026-10-15T19:04:54Z', 1) has log
not-success ('felhom-controller', 1884, '4261a38b', 'completed', 'failure', '2026-10-15T19:05:47Z', 1) has log
not-success ('felhom-controller', 1887, 'c6a56712', 'completed', 'failure', '2026-10-15T19:09:40Z', 1) has log
not-success ('felhom-controller', 1990, '334df7db', 'completed', 'failure', '2026-10-17T09:53:24Z', 1) has log
not-success ('felhom-controller', 2036, 'e30091bf', 'completed', 'failure', '2026-10-17T15:30:09Z', 1) has log
not-success ('app-catalog-felhom.eu', 2067, 'c97eb1c9', 'completed', 'failure', '2026-10-18T18:45:53Z', 1) has log
@@ -0,0 +1,16 @@
task fields: ['created_at', 'display_title', 'event', 'head_branch', 'head_sha', 'id', 'name', 'run_number', 'run_started_at', 'status', 'updated_at', 'url', 'workflow_id']
{'id': 1991, 'name': 'gates', 'head_sha': 'b69585789a9fa34a7c33faf24da44926f166a9da', 'run_number': 393, 'status': 'success', 'event': 'push', 'created_at': '2026-10-16T21:50:09Z', 'run_started_at': '2026-10-16T21:50:09Z', 'updated_at': '2026-10-16T21:50:58Z'}
{'id': 1959, 'name': 'gates', 'head_sha': 'f30a80841e372abf9d7992dc9eb3eec607078e90', 'run_number': 379, 'status': 'success', 'event': 'push', 'created_at': '2026-10-16T13:19:24Z', 'run_started_at': '2026-10-16T13:19:24Z', 'updated_at': '2026-10-16T13:20:14Z'}
{'id': 1639, 'name': 'gates', 'head_sha': '1cb43a21a2dc5cd44fbcc0749b460dba1ffa8088', 'run_number': 326, 'status': 'success', 'event': 'push', 'created_at': '2026-10-10T07:55:47Z', 'run_started_at': '2026-10-10T07:55:47Z', 'updated_at': '2026-10-10T07:56:52Z'}
job 1944 sha c02c9d83 attempt 2 created 2026-10-16T13:50:35Z started 2026-10-16T14:02:29Z completed 2026-10-16T14:03:21Z
job 1959 sha 29a3ce8c attempt 2 created 2026-10-16T17:07:53Z started 2026-10-16T17:19:08Z completed 2026-10-16T17:20:00Z
job 1991 sha 3e8d0276 attempt 2 created 2026-10-17T09:59:02Z started 2026-10-17T10:15:35Z completed 2026-10-17T10:16:23Z
task 2015 for job 1991 success 2026-10-17T10:15:35Z -> 2026-10-17T10:16:24Z
task 2011 for job 1991 failure 2026-10-17T09:59:02Z -> 2026-10-17T09:59:49Z
task 1980 for job 1959 success 2026-10-16T17:19:08Z -> 2026-10-16T17:20:00Z
task 1978 for job 1959 failure 2026-10-16T17:07:54Z -> 2026-10-16T17:08:42Z
task 1964 for job 1944 success 2026-10-16T14:02:29Z -> 2026-10-16T14:03:21Z
task 1962 for job 1944 failure 2026-10-16T13:50:35Z -> 2026-10-16T13:51:24Z
CONTROL job 1639 (a known lost attempt, 2026-10-10) sha 9e5b8814
task 1655 success 2026-10-10T13:22:35Z -> 2026-10-10T13:23:51Z
task 1653 failure 2026-10-10T13:09:10Z -> 2026-10-10T13:20:58Z
@@ -0,0 +1,39 @@
== DooPlex PBS (read 2026-10-19)
[
{
"comment": "decision 71: keep the last 8 weekly copies per group (all namespaces); runs after the 05:00 pull",
"id": "prune-ep0-copy",
"keep-weekly": 8,
"schedule": "07:30",
"store": "ep0-copy"
}
]
[
{
"comment": "decision 70: nightly copy of ep0 felhom-offsite; never removes what ep0 removed",
"id": "ep0-felhom-offsite",
"remote": "ep0",
"remote-store": "felhom-offsite",
"remove-vanished": false,
"schedule": "05:00",
"store": "ep0-copy"
}
]
/mnt/5_hdd/backup/ep0-copy/ns/Tester-2/ct/9201/2026-10-04T16:31:13Z
/mnt/5_hdd/backup/ep0-copy/ns/demo-felhom/ct/9201/2026-09-22T04:12:20Z
/mnt/5_hdd/backup/ep0-copy/ns/demo-felhom/ct/9201/2026-09-29T04:16:43Z
/mnt/5_hdd/backup/ep0-copy/ns/demo-felhom/ct/9201/2026-10-06T04:21:09Z
/mnt/5_hdd/backup/ep0-copy/ns/demo-felhom/ct/9201/2026-10-13T04:22:51Z
/mnt/5_hdd/backup/ep0-copy/ns/demo-hp/ct/9201/2026-09-24T20:06:25Z
/mnt/5_hdd/backup/ep0-copy/ns/demo-hp/ct/9201/2026-10-01T20:15:29Z
/mnt/5_hdd/backup/ep0-copy/ns/demo-hp/ct/9201/2026-10-08T20:34:27Z
/mnt/5_hdd/backup/ep0-copy/ns/demo-hp/ct/9201/2026-10-15T20:38:47Z
/mnt/5_hdd/backup/ep0-copy/ns/operator/host/dooplex-gitea/2026-10-10T22:22:52Z
/mnt/5_hdd/backup/ep0-copy/ns/operator/host/dooplex-gitea/2026-10-17T22:22:32Z
/mnt/5_hdd/backup/ep0-copy/ns/operator/host/dooplex-gitea/2026-10-18T22:22:01Z
/mnt/5_hdd/backup/ep0-copy/ns/operator/host/dooplex-hub/2026-10-11T00:30:55Z
/mnt/5_hdd/backup/ep0-copy/ns/operator/host/dooplex-hub/2026-10-18T00:31:48Z
/mnt/5_hdd/backup/ep0-copy/ns/operator/host/dooplex-hub/2026-10-19T00:32:07Z
/mnt/5_hdd/backup/ep0-copy/ns/tester-1/ct/9201/2026-10-04T19:57:18Z
/mnt/5_hdd/backup/ep0-copy/ns/tester-1/ct/9201/2026-10-11T20:02:13Z
/mnt/5_hdd/backup/ep0-copy/ns/tester-1/ct/9201/2026-10-18T20:04:51Z
@@ -0,0 +1,5 @@
box | job | snapshots today | keep-last 2 keeps / removes | keep-weekly 2 keeps / removes
Tester-2 | NO JOB | 1 | 2026-10-04 (w40) / none | 2026-10-04 (w40) / none
demo-felhom | prune-demo-felhom | 2 | 2026-10-13 (w42), 2026-10-06 (w41) / none | 2026-10-13 (w42), 2026-10-06 (w41) / none
demo-hp | prune-demo-hp | 2 | 2026-10-15 (w42), 2026-10-08 (w41) / none | 2026-10-15 (w42), 2026-10-08 (w41) / none
tester-1 | NO JOB | 3 | 2026-10-18 (w42), 2026-10-11 (w41) / 2026-10-04 (w40) | 2026-10-18 (w42), 2026-10-11 (w41) / 2026-10-04 (w40)
@@ -0,0 +1,31 @@
Mon Oct 19 07:49:11 UTC 2026
[
{
"comment": "R-1162 operator D1=A 2026-10-19: one copy per ISO week, 2 weeks, server-side (box tokens are write-only)",
"id": "prune-demo-hp",
"keep-weekly": 2,
"max-depth": 0,
"ns": "demo-hp",
"schedule": "03:30",
"store": "felhom-offsite"
},
{
"comment": "R-1162 operator D1=A 2026-10-19: one copy per ISO week, 2 weeks, server-side (box tokens are write-only)",
"id": "prune-demo-felhom",
"keep-weekly": 2,
"max-depth": 0,
"ns": "demo-felhom",
"schedule": "03:30",
"store": "felhom-offsite"
},
{
"comment": "R-173 hub DB copies; operator ns only",
"id": "prune-operator-hubdb",
"keep-daily": 14,
"keep-weekly": 8,
"max-depth": 0,
"ns": "operator",
"schedule": "03:45",
"store": "felhom-offsite"
}
]
@@ -0,0 +1,40 @@
felhom-hetzner
proxmox-backup-server 4.2.8-1 running version: 4.2.5
==prune-jobs
[
{
"comment": "R-173 hub DB copies; operator ns only",
"id": "prune-operator-hubdb",
"keep-daily": 14,
"keep-weekly": 8,
"max-depth": 0,
"ns": "operator",
"schedule": "03:45",
"store": "felhom-offsite"
},
{
"comment": "R-82 retention keep-last=2, server-side (box tokens are write-only)",
"id": "prune-demo-hp",
"keep-last": 2,
"max-depth": 0,
"ns": "demo-hp",
"schedule": "03:30",
"store": "felhom-offsite"
},
{
"comment": "R-82 retention keep-last=2, server-side (box tokens are write-only)",
"id": "prune-demo-felhom",
"keep-last": 2,
"max-depth": 0,
"ns": "demo-felhom",
"schedule": "03:30",
"store": "felhom-offsite"
}
]
==datastore
{
"gc-schedule": "sun 04:30",
"name": "felhom-offsite",
"path": "/mnt/pbs-datastore",
"verify-new": true
}
@@ -0,0 +1,8 @@
ns/Tester-2/ct/9201/2026-10-04T16:31:13Z
ns/demo-felhom/ct/9201/2026-10-06T04:21:09Z
ns/demo-felhom/ct/9201/2026-10-13T04:22:51Z
ns/demo-hp/ct/9201/2026-10-08T20:34:27Z
ns/demo-hp/ct/9201/2026-10-15T20:38:47Z
ns/tester-1/ct/9201/2026-10-04T19:57:18Z
ns/tester-1/ct/9201/2026-10-11T20:02:13Z
ns/tester-1/ct/9201/2026-10-18T20:04:51Z
@@ -0,0 +1,14 @@
== ns/demo-felhom
ns/demo-felhom/ct/9201/2026-10-06T04:21:09Z
ns/demo-felhom/ct/9201/2026-10-13T04:22:51Z
== ns/demo-hp
ns/demo-hp/ct/9201/2026-10-08T20:34:27Z
ns/demo-hp/ct/9201/2026-10-15T20:38:47Z
== ns/operator
== ns/tester-1
ns/tester-1/ct/9201/2026-10-04T19:57:18Z
ns/tester-1/ct/9201/2026-10-11T20:02:13Z
ns/tester-1/ct/9201/2026-10-18T20:04:51Z
== ns/Tester-2
ns/Tester-2/ct/9201/2026-10-04T16:31:13Z
== root
@@ -0,0 +1,24 @@
== demo-hp --keep-last 2 (PBS dry-run)
2026-10-08T20:34:27Z keep
2026-10-15T20:38:47Z keep
== demo-hp --keep-weekly 2 (PBS dry-run)
2026-10-08T20:34:27Z keep
2026-10-15T20:38:47Z keep
== demo-felhom --keep-last 2 (PBS dry-run)
2026-10-06T04:21:09Z keep
2026-10-13T04:22:51Z keep
== demo-felhom --keep-weekly 2 (PBS dry-run)
2026-10-06T04:21:09Z keep
2026-10-13T04:22:51Z keep
== tester-1 --keep-last 2 (PBS dry-run)
2026-10-04T19:57:18Z REMOVE
2026-10-11T20:02:13Z keep
2026-10-18T20:04:51Z keep
== tester-1 --keep-weekly 2 (PBS dry-run)
2026-10-04T19:57:18Z REMOVE
2026-10-11T20:02:13Z keep
2026-10-18T20:04:51Z keep
== Tester-2 --keep-last 2 (PBS dry-run)
2026-10-04T16:31:13Z keep
== Tester-2 --keep-weekly 2 (PBS dry-run)
2026-10-04T16:31:13Z keep
@@ -0,0 +1,32 @@
# Offline model of PBS's prune selection (pbs-datastore prune.rs compute_prune_info / mark_selections),
# run over ep0's snapshot list. Never touches ep0. ep0's clock is UTC (timedatectl), and PBS keys
# weekly buckets by local time "%G/%V" (ISO year/week).
import sys, datetime
snaps = {}
for line in open(sys.argv[1]):
line=line.strip()
if line.startswith('ns/') and '/ct/' in line:
ns = line.split('/')[1]; t = line.rsplit('/',1)[1]
snaps.setdefault(ns, []).append(t)
def sel_last(t): return t
def sel_week(t):
d = datetime.datetime.strptime(t, '%Y-%m-%dT%H:%M:%SZ'); y,w,_ = d.isocalendar(); return f"{y}/{w:02d}"
def compute(lst, opts):
lst = sorted(lst, reverse=True); mark = {}
for keep, sel in opts:
already = {sel(t) for t in lst if mark.get(t)=='keep'}; inc=set()
for t in lst:
if t in mark: continue
s = sel(t)
if s in already: continue
if s not in inc:
if len(inc) >= keep: break
inc.add(s); mark[t]='keep'
else: mark[t]='remove'
return {t: mark.get(t,'remove') for t in lst}
jobs = {'demo-hp':'prune-demo-hp','demo-felhom':'prune-demo-felhom'}
print("box | job | snapshots today | keep-last 2 keeps / removes | keep-weekly 2 keeps / removes")
for ns in sorted(snaps):
a = compute(snaps[ns], [(2, sel_last)]); b = compute(snaps[ns], [(2, sel_week)])
f = lambda m,k: ', '.join(f"{t[:10]} (w{sel_week(t)[5:]})" for t,v in m.items() if v==k) or 'none'
print(f"{ns} | {jobs.get(ns,'NO JOB')} | {len(snaps[ns])} | {f(a,'keep')} / {f(a,'remove')} | {f(b,'keep')} / {f(b,'remove')}")
@@ -0,0 +1,24 @@
=== RUN TestR1159_Measure
r1159_measure_test.go:85: GET /config/c1 (passphrase) -> 200; hub fields: [api_key enabled url]; carries the controller key: true
r1159_measure_test.go:86: GET /config/c1 (wrong passphrase) -> 401
r1159_measure_test.go:90: POST /host-enroll (passphrase) -> 200; returns the EXISTING host's key: true
r1159_measure_test.go:105: POST /offsite/register-key/c1 controller key -> 200, wrong key -> 401
r1159_measure_test.go:105: POST /offsite/confirm-key/c1 controller key -> 200, wrong key -> 401
r1159_measure_test.go:105: POST /offsite/move-aside/c1 controller key -> 200, wrong key -> 401
r1159_measure_test.go:105: POST /offsite/window-open/c1 controller key -> 200, wrong key -> 401
r1159_measure_test.go:105: POST /offsite/abandon-request/c1 controller key -> 200, wrong key -> 401
r1159_measure_test.go:105: POST /offsite/abandon-cancel/c1 controller key -> 200, wrong key -> 401
r1159_measure_test.go:105: POST /offsite/abandon-status/c1 controller key -> 200, wrong key -> 401
r1159_measure_test.go:121: GET /hosts/h1/escrow host key -> 200, wrong key -> 401
r1159_measure_test.go:121: GET /hosts/h1/escrow/retained host key -> 200, wrong key -> 401
r1159_measure_test.go:121: PUT /hosts/h1/escrow host key -> 400, wrong key -> 401
r1159_measure_test.go:121: POST /hosts/h1/wg host key -> 400, wrong key -> 401
r1159_measure_test.go:121: POST /hosts/h1/pbs/consume-token host key -> 404, wrong key -> 401
r1159_measure_test.go:121: GET /hosts/h1/desired-state host key -> 200, wrong key -> 401
r1159_measure_test.go:121: GET /hosts/h1/jobs host key -> 200, wrong key -> 401
r1159_measure_test.go:121: PUT /hosts/h1/recovery-credential host key -> 400, wrong key -> 401
r1159_measure_test.go:121: POST /host-report host key -> 200, wrong key -> 401
r1159_measure_test.go:123: off-site service calls reached with the passphrase-derived key: RegisterKey, ConfirmKey, MoveAside, OpenWindowFor, RequestAbandon, CancelAbandon, AbandonStatusFor
--- PASS: TestR1159_Measure (0.04s)
PASS
ok gitea.dooplex.hu/admin/felhom-hub/internal/api 0.065s
@@ -0,0 +1,124 @@
package api
// R-1159 MEASUREMENT HARNESS (temporary, not committed): what a holder of the household's retrieval passphrase ALONE
// can reach through the real hub router. Field names and status codes only.
import (
"context"
"encoding/json"
"net/http"
"sort"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-hub/internal/offsitekeys"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
"gopkg.in/yaml.v3"
)
type r1159Tpl struct{}
func (r1159Tpl) Template() string { return "hub: {}\n" }
type r1159Keys struct{ calls []string }
func (f *r1159Keys) RegisterKey(_ context.Context, c, _ string) (offsitekeys.InstallResult, error) {
f.calls = append(f.calls, "RegisterKey")
return offsitekeys.InstallResult{Fingerprint: "SHA256:x"}, nil
}
func (f *r1159Keys) ConfirmKey(_ context.Context, c, _ string) (int, error) {
f.calls = append(f.calls, "ConfirmKey")
return 1, nil
}
func (f *r1159Keys) MoveAside(_ context.Context, c string) (string, error) {
f.calls = append(f.calls, "MoveAside")
return "repo.orphaned-x", nil
}
func (f *r1159Keys) OpenWindowFor(_ context.Context, c string, _ int) (offsitekeys.WindowGrant, error) {
f.calls = append(f.calls, "OpenWindowFor")
return offsitekeys.WindowGrant{}, nil
}
func (f *r1159Keys) CloseWindowFor(_ context.Context, c string, _ offsitekeys.WindowResult) error {
f.calls = append(f.calls, "CloseWindowFor")
return nil
}
func (f *r1159Keys) RequestAbandon(_ context.Context, c, _ string) (offsitekeys.AbandonStatus, error) {
f.calls = append(f.calls, "RequestAbandon")
return offsitekeys.AbandonStatus{State: "pending"}, nil
}
func (f *r1159Keys) CancelAbandon(c, _ string) (int, error) {
f.calls = append(f.calls, "CancelAbandon")
return 1, nil
}
func (f *r1159Keys) AbandonStatusFor(c string) (offsitekeys.AbandonStatus, error) {
f.calls = append(f.calls, "AbandonStatusFor")
return offsitekeys.AbandonStatus{State: "none"}, nil
}
func TestR1159_Measure(t *testing.T) {
h, st, _ := newTestHandler(t)
h.templateProvider = r1159Tpl{}
fk := &r1159Keys{}
h.SetOffsiteKeyService(fk)
if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ckey-1159", RetrievalPassword: "owner-pass"}); err != nil {
t.Fatal(err)
}
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY-1159"}); err != nil {
t.Fatal(err)
}
// Step 1: the passphrase alone.
rr := r879Get(h, "/config/c1", "owner-pass")
var y map[string]any
_ = yaml.Unmarshal(rr.Body.Bytes(), &y)
var hubFields []string
if hm, ok := y["hub"].(map[string]any); ok {
for k := range hm {
hubFields = append(hubFields, k)
}
}
sort.Strings(hubFields)
key := ""
if hm, ok := y["hub"].(map[string]any); ok {
key, _ = hm["api_key"].(string)
}
t.Logf("GET /config/c1 (passphrase) -> %d; hub fields: %v; carries the controller key: %v", rr.Code, hubFields, key == "ckey-1159")
t.Logf("GET /config/c1 (wrong passphrase) -> %d", r879Get(h, "/config/c1", "nope").Code)
er := doEnroll(h, "c1", "owner-pass")
var e enrollResp
_ = json.Unmarshal(er.Body.Bytes(), &e)
t.Logf("POST /host-enroll (passphrase) -> %d; returns the EXISTING host's key: %v", er.Code, e.APIKey == "HKEY-1159")
// Step 2: the controller key taken from the config.
routes := []struct{ path, body string }{
{"/offsite/register-key/c1", `{"public_key":"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl x"}`},
{"/offsite/confirm-key/c1", `{"fingerprint":"SHA256:x"}`},
{"/offsite/move-aside/c1", `{}`},
{"/offsite/window-open/c1", `{"count_before":10}`},
{"/offsite/abandon-request/c1", `{"path":"repo.orphaned-x"}`},
{"/offsite/abandon-cancel/c1", `{}`},
{"/offsite/abandon-status/c1", `{}`},
}
for _, r := range routes {
ok := do(h, http.MethodPost, r.path, key, r.body)
bad := do(h, http.MethodPost, r.path, "ckey-wrong", r.body)
t.Logf("POST %-30s controller key -> %d, wrong key -> %d", r.path, ok.Code, bad.Code)
}
hk := e.APIKey
for _, r := range []struct{ m, path, body string }{
{"GET", "/hosts/h1/escrow", ""},
{"GET", "/hosts/h1/escrow/retained", ""},
{"PUT", "/hosts/h1/escrow", `{}`},
{"POST", "/hosts/h1/wg", `{}`},
{"POST", "/hosts/h1/pbs/consume-token", `{}`},
{"GET", "/hosts/h1/desired-state", ""},
{"GET", "/hosts/h1/jobs", ""},
{"PUT", "/hosts/h1/recovery-credential", `{}`},
{"POST", "/host-report", validReportBody("h1")},
} {
ok := do(h, r.m, r.path, hk, r.body)
bad := do(h, r.m, r.path, "HKEY-wrong", r.body)
t.Logf("%-4s %-32s host key -> %d, wrong key -> %d", r.m, r.path, ok.Code, bad.Code)
}
t.Logf("off-site service calls reached with the passphrase-derived key: %s", strings.Join(fk.calls, ", "))
}
@@ -0,0 +1,12 @@
ARP watch for 192.168.0.104 (demo-hp vmbr0 7c:d3:0a:77:d9:76), from 09:58:07 UTC
09:58:07 mac=7c:d3:0a:77:d9:76 ssh=open
10:02:18 mac=b8:06:0d:60:45:58 ssh=shut
10:02:58 mac=7c:d3:0a:77:d9:76 ssh=open
10:03:38 mac=b8:06:0d:60:45:58 ssh=shut
10:08:59 mac=7c:d3:0a:77:d9:76 ssh=open
10:10:19 mac=b8:06:0d:60:45:58 ssh=shut
10:12:50 mac=7c:d3:0a:77:d9:76 ssh=open
10:17:10 mac=b8:06:0d:60:45:58 ssh=shut
10:17:50 mac=7c:d3:0a:77:d9:76 ssh=open
10:19:21 mac=b8:06:0d:60:45:58 ssh=shut
stopped by CC at 10:20:37 UTC (enough to show the flips)
@@ -0,0 +1,4 @@
FileBrowser 1.5.6-stable on demo-hp 9201, 2026-10-19
index.html links: <link rel="manifest" href="/public/static/site.webmanifest" crossorigin="use-credentials">
manifest: 200 application/manifest+json, display=standalone
PWA config keys in the binary: disablePWAInstall only (strings | grep -i pwa/manifest)
@@ -0,0 +1,10 @@
jwt alg: HS256 | claim names: ['Permissions', 'belongsTo', 'exp', 'iat', 'iss']
nested Permissions -> ['admin', 'api', 'create', 'delete', 'download', 'modify', 'realtime', 'share']
exp-iat/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/649cb06d-b4ca-49f2-9e2c-16c8bc36531f/scrat seconds: 7200
identity claims: {'belongsTo': 3}
signed in: cookie-token alone -> HTTP chpad/jwt_probe3.sh: line 23: T3: unbound variable
/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/649cb06d-b4ca-49f2-9e2c-16c8bc36531f/scrat401; bearer alone -> HTTP
dashboard sign-out: 302
after sign-out: coochpad/jwt_probe3.sh: line 23: T3: unbound variable
/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/649cb06d-b4ca-49f2-9e2c-16c8bc36531f/scratkie-token alone -> HTTP 401; bearer alone -> HTTP
chpad/jwt_probe3.sh: line 28: T3: unbound variable
@@ -0,0 +1,5 @@
with the files cookie (the household's normal path): HTTP 200
token alone, still signed in: cookie 401 · bearer 401 · basic 401 · ?auth= 401
dashboard sign-out: 302
token alone, after sign-out: cookie 401 · bearer 401 · basic 401 · ?auth= 401
FileBrowser's own sign-in page, no token: HTTP 200
@@ -0,0 +1,27 @@
=== demo-hp (enkisfelhom.hu) 10:15:18
containers: 24 running; not healthy/running: traefik Up 3 days;
unhealthy: 0
2026/10/19 10:13:21 updater.go:855: [INFO] [selfupdate] Post-update startup: update successful (0.333.0-rc.2 → 0.333.0)
allow-list refusals since start: 0
2026/10/19 10:13:21 sync.go:142: [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
installed apps (stacks with app.yaml): 12
public https://felhom.enkisfelhom.hu/: 302
public https://files.enkisfelhom.hu/: 200
=== demo-felhom (demo-felhom.eu) 10:15:20
containers: 9 running; not healthy/running: traefik Up 3 days;
unhealthy: 0
2026/10/19 09:59:39 [INFO] [selfupdate] Post-update startup: update successful (0.332.0 → 0.333.0)
allow-list refusals since start: 0
2026/10/19 09:59:40 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
installed apps (stacks with app.yaml): 5
public https://felhom.demo-felhom.eu/: 302
public https://files.demo-felhom.eu/: 200
=== tester-1 (enkicsifelhom.hu) 10:15:22
containers: 10 running; not healthy/running: traefik Up 3 days;
unhealthy: 0
2026/10/19 09:59:48 [INFO] [selfupdate] Post-update startup: update successful (0.332.0 → 0.333.0)
allow-list refusals since start: 0
2026/10/19 09:59:49 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
installed apps (stacks with app.yaml): 3
public https://felhom.enkicsifelhom.hu/: 302
public https://files.enkicsifelhom.hu/: 200
@@ -0,0 +1,33 @@
control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -)
== tester-1 (enkicsifelhom.hu): 6 public name(s)
OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1
OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1
OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1
OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1
OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1
OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1
== demo-hp (enkisfelhom.hu): 14 public name(s)
OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1
OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1
OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1
OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1
OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1
OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1
OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1
OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1
OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1
OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1
OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1
OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1
OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1
OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1
== demo-felhom (demo-felhom.eu): 8 public name(s)
OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1
OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1
OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1
OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1
OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1
OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1
OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1
OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
== every public name redirects http -> https
@@ -0,0 +1,26 @@
== demo-hp
sec: cannot run /tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/649cb06d-b4ca-49f2-9e2c-16c8bc36531f/scratchpad/sso_check.sh: Permission denied
== tester-1
sec: cannot run /tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/649cb06d-b4ca-49f2-9e2c-16c8bc36531f/scratchpad/sso_check.sh: Permission denied
== tester-1 R-1155
with the files cookie (the household's normal path): HTTP 200
token alone, still signed in: cookie 401 · bearer 401 · basic 401 · ?auth= 401
dashboard sign-out: 302
token alone, after sign-out: cookie 401 · bearer 401 · basic 401 · ?auth= 401
FileBrowser's own sign-in page, no token: HTTP 200
== demo-hp
dashboard sign-in: 302
the Fajlok button goes to: https://files.enkisfelhom.hu/_felhom/sso
the pass answer goes to: https://files.enkisfelhom.hu/files/; files cookie set: 1
FileBrowser with the files cookie: felhom-household admin: False modify: True
FileBrowser without it, a fresh browser: HTTP 401
dashboard sign-out: 302
after the sign-out, the files cookie sends the browser to: https://felhom.enkisfelhom.hu/__files/open?to=%2Ffiles%2F
== tester-1
dashboard sign-in: 302
the Fajlok button goes to: https://files.enkicsifelhom.hu/_felhom/sso
the pass answer goes to: https://files.enkicsifelhom.hu/files/; files cookie set: 1
FileBrowser with the files cookie: felhom-household admin: False modify: True
FileBrowser without it, a fresh browser: HTTP 401
dashboard sign-out: 302
after the sign-out, the files cookie sends the browser to: https://felhom.enkicsifelhom.hu/__files/open?to=%2Ffiles%2F
@@ -0,0 +1,5 @@
page bytes: 53890
new text, shown by default (id=live-after-check, ASCII fragment 'lent a k'): 1
old text, hidden until the claim says no confirm (id=live-now display:none): 1
the switch in the page script: 1
negative control (a sentence that is not on the page): 0
@@ -0,0 +1,13 @@
## global floor 0.333.0 / MinAgent 0.158.1 — 10:16:22 UTC
HTTP/1.1 303 See Other
Location: /configuration?flash=floor_set
## clear per-box floors
tester-1: HTTP/1.1 303 See Other Location: /customers/tester-1?flash=floor_set
demo-hp: HTTP/1.1 303 See Other Location: /customers/demo-hp?flash=floor_set
demo-felhom: HTTP/1.1 303 See Other Location: /customers/demo-felhom?flash=floor_set
## read back (hub.db + WAL)
min_controller_version|0.333.0
min_controller_version_declared_min_agent|0.333.0=0.158.1
demo-felhom|
demo-hp|
tester-1|

Some files were not shown because too many files have changed in this diff Show More