Compare commits
33 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| b0ae2c574a | |||
| e1d349f100 | |||
| 7d7868e5e5 | |||
| 189eb82eb0 | |||
| f92fa5a8c8 | |||
| 5d0dfeac5f | |||
| 15f7428871 | |||
| d92a712ee4 | |||
| 5f31da04bc | |||
| a28f162c66 | |||
| 97f27a7528 | |||
| b6a911c3f9 | |||
| 224fe41e06 | |||
| 73f3899266 | |||
| e2c35007b5 | |||
| fd75003e17 | |||
| 5d3cb5c132 | |||
| a17db0c3c8 | |||
| 503c93fada | |||
| d5e3c8b0d9 | |||
| db2371c6f5 | |||
| 200ab6530b | |||
| 5622ce2735 | |||
| afc939bb3c | |||
| b505ce4ba0 | |||
| 5121536f10 | |||
| e4f022a0d9 | |||
| b9b56c47c0 | |||
| 1fe9f84893 | |||
| 7f7081d127 | |||
| e881115c43 | |||
| 37cb83c166 | |||
| 3332eb0919 |
@@ -29,5 +29,7 @@ ArgoCD's source of truth is the **manifest in git**:
|
||||
`main`, and `/scripts/` from the tag `installer-v<SCRIPT_VERSION>`. Changing either `--ref` changes
|
||||
what real machines download — the init container **and** the sidecar both carry it, and both must
|
||||
move together. `hostinstall_gates.py` gate 6 fails if the manifest stops naming an `installer-v…`
|
||||
tag or if the website stops tracking `main`. The full publish/rollback procedure is the R-110 fence
|
||||
tag or if the website stops tracking `main`. All four git-sync containers (two sidecars, two init containers) read
|
||||
Gitea **in-cluster** (`http://gitea.gitea-system.svc.cluster.local:3000/…`, since 2026-10-19) and must name the same
|
||||
repo — gate 6 fails if they differ; the sidecars carry `--max-failures=-1` (retry, never exit). The full publish/rollback procedure is the R-110 fence
|
||||
in the repo core `CLAUDE.md`.
|
||||
|
||||
@@ -184,6 +184,10 @@ jobs:
|
||||
# which prints at the moment a release is committed, where someone can actually act on it.
|
||||
# Those are the compensating controls that make this green honest. Every other gate still
|
||||
# fails this job on any push, and golden-currency still fails it on a push touching code.
|
||||
env:
|
||||
# R-986: the security-scan gate fetches the PINNED gitleaks here (sha256-checked); no Go, so govulncheck says
|
||||
# NOT CHECKED in CI, out loud. Warn-first either way: the scan lists, it never fails this job.
|
||||
FELHOM_SCAN_FETCH: "1"
|
||||
run: python3 scripts/repo_gates.py --fast --scope="${PUSH_SCOPE:-code}"
|
||||
|
||||
- name: Manual proof - alarm key present, optional forced failure (dispatch only, R-1129)
|
||||
@@ -228,8 +232,16 @@ jobs:
|
||||
|
||||
repo = os.environ.get("GITHUB_REPOSITORY", "?")
|
||||
sha = os.environ.get("GITHUB_SHA", "?")
|
||||
run = os.environ.get("GITHUB_RUN_NUMBER", "?")
|
||||
srv = os.environ.get("GITHUB_SERVER_URL", "https://gitea.dooplex.hu")
|
||||
# R-1135: the link is for a PERSON, so it is the public address and the run ID.
|
||||
# NOT GITHUB_SERVER_URL: the runner registers at the in-cluster address, so that variable
|
||||
# reads http://gitea.gitea-system.svc.cluster.local:3000, which nobody outside the cluster
|
||||
# can open (forced-test mail, job 1997). NOT GITHUB_RUN_NUMBER: Gitea 1.26 reads
|
||||
# /actions/runs/<n> as a run id first, so a run number that is also an older run's id
|
||||
# opens THAT run (measured 2026-10-18: felhom.eu /actions/runs/1116 opened the run whose
|
||||
# id is 1116 = run number 613). The link is printed too, so the job log carries it.
|
||||
run = os.environ.get("GITHUB_RUN_ID", "?")
|
||||
srv = "https://gitea.dooplex.hu"
|
||||
print("ALARM-LINK %s/%s/actions/runs/%s" % (srv, repo, run))
|
||||
|
||||
# R-265: how long the run took, so a reap is self-identifying. An honest gate failure
|
||||
# lands in well under a minute; a multi-minute figure means the job hit its own timeout
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
# .security-accepted — hits security_scan_gate.py (felhom.eu/scripts, R-986) lists as ACCEPTED, not WARN.
|
||||
# One per line: <tool> <id> <where> # reason. govulncheck: OSV id + module dir; gitleaks: rule@<sha256[:12] of the line> + file.
|
||||
# The first scan's verdicts (2026-10-19): felhom.eu documentation/audits/night-2026-10-19/P7/triage.md
|
||||
gitleaks generic-api-key@0275c9c51957 documentation/audits/usenet-2026-10-12/sabnzbd-bench/13-compose-resolved.txt # accepted: a bench app's generated admin password and API key (LXC 9401, stopped; app removed from 9202)
|
||||
gitleaks generic-api-key@0a189ae60ea4 documentation/tests/campaign2-evidence-2026-07-07/state/runner.log # accepted: the same dry-run deploy values in the campaign runner's log
|
||||
gitleaks generic-api-key@2a27df76f661 documentation/audits/usenet-2026-10-12/sabnzbd-bench/02-telemetry-and-net.txt # accepted: a bench app's generated API key (LXC 9401, stopped; app removed from 9202 at teardown)
|
||||
gitleaks generic-api-key@3a4c7b17e349 documentation/audits/usenet-2026-10-12/nzbget-bench/03-process-and-entrypoint.txt # accepted: a bench app's generated control password (LXC 9401, stopped; app removed from 9202 at teardown)
|
||||
gitleaks generic-api-key@3c180a9f957c documentation/tests/campaign2-evidence-2026-07-07/state/runner.log # accepted: the same dry-run deploy values in the campaign runner's log
|
||||
gitleaks generic-api-key@486819f82f58 documentation/audits/update-night-2026-09-21/apps/actualbudget/log.txt # accepted: a demo app's session token, demo-hp 2026-09-21 (Tier 0); no running demo app has it (2026-10-12 review)
|
||||
gitleaks generic-api-key@486819f82f58 documentation/audits/update-night-2026-09-21/batchA.log # accepted: the same demo app session token as actualbudget/log.txt
|
||||
gitleaks generic-api-key@5a64c6ffd6df documentation/audits/signup-lock-2026-09-29/B/wd.sh # accepted: a stranger-probe test password in an audit script (signup lock, Tier 0)
|
||||
gitleaks generic-api-key@6e2e899b7d6e documentation/audits/offsite-lock-build-2026-10-03/partD/demo-hp-migration.txt # false: an SSH key FINGERPRINT in a log line, not a key
|
||||
gitleaks generic-api-key@70380348263d hub/internal/web/bind_language_test.go # false: a test constant or a decoy value, no live secret
|
||||
gitleaks generic-api-key@7134caf03160 documentation/audits/evidence-rehearsal-2026-08-09/GATE0-demo-hp-before.txt # false: a WireGuard PUBLIC key
|
||||
gitleaks generic-api-key@7134caf03160 documentation/audits/evidence-rehearsal-2026-08-09/P1-demo-hp-after.txt # false: a WireGuard PUBLIC key
|
||||
gitleaks generic-api-key@71ba98258279 hub/internal/api/r1140_accesslog_test.go # false: a test constant or a decoy value, no live secret
|
||||
gitleaks generic-api-key@7aa746ddb210 documentation/audits/offsite-lock-build-2026-10-03/partD/demo-hp-migration.txt # false: an SSH key FINGERPRINT in a log line, not a key
|
||||
gitleaks generic-api-key@7ce547845b1e documentation/audits/night-rulings-2026-09-30/D/D3-wanderer-routes-probe.txt # accepted: a throwaway app's session cookie (wanderer probe, 2026-09-30, Tier 0)
|
||||
gitleaks generic-api-key@8044eef94360 documentation/tests/campaign2-evidence-2026-07-07/state/runner.log # accepted: the same dry-run deploy values in the campaign runner's log
|
||||
gitleaks generic-api-key@85b662612937 hub/internal/store/r879_box_seal_test.go # false: a test constant or a decoy value, no live secret
|
||||
gitleaks generic-api-key@9530a022fde8 documentation/audits/usenet-2026-10-12/sabnzbd-bench/13-compose-resolved.txt # accepted: a bench app's generated admin password and API key (LXC 9401, stopped; app removed from 9202)
|
||||
gitleaks generic-api-key@9b3a9c2be48a documentation/tests/campaign2-evidence-2026-07-07/evidence/T-DEPLOY-SET/values_bookstack.json # accepted: dry-run deploy values of a 2026-07-07 campaign (generated, demo boxes since rebuilt)
|
||||
gitleaks generic-api-key@a9538cfd7063 documentation/tests/campaign2-evidence-2026-07-07/evidence/T-DEPLOY-SET/values_paperless-ngx.json # accepted: dry-run deploy values of a 2026-07-07 campaign (generated, demo boxes since rebuilt)
|
||||
gitleaks generic-api-key@b0414c6f59b9 documentation/audits/offsite-lock-build-2026-10-03/partD/demo-hp-migration.txt # false: an SSH key FINGERPRINT in a log line, not a key
|
||||
gitleaks generic-api-key@b1db45e463b6 documentation/audits/family-gate-2026-10-02/tools/fg_items.py # false: test strings in an audit tool (websocket probe labels)
|
||||
gitleaks generic-api-key@b1db45e463b6 documentation/audits/permanent-gate-2026-10-01/tools/items12.py # false: test strings in an audit tool (websocket probe labels)
|
||||
gitleaks generic-api-key@bd228c851720 documentation/tests/walk5-r201-2026-08-07/journal.md # false: sha256 digests compared in a walk journal, not secrets
|
||||
gitleaks generic-api-key@c1be98b7b703 documentation/tests/walk5-r201-2026-08-07/journal.md # false: sha256 digests compared in a walk journal, not secrets
|
||||
gitleaks generic-api-key@c3b2b5665c9c documentation/audits/evidence-fixes-first-tester-2026-09-30/part0/cloudflare-dns.txt # false: a tunnel ID (a UUID), not the tunnel token
|
||||
gitleaks generic-api-key@cf453385b707 documentation/tests/campaign2-evidence-2026-07-07/state/runner.log # accepted: the same dry-run deploy values in the campaign runner's log
|
||||
gitleaks generic-api-key@e59eaeb4aee7 documentation/audits/new-apps-2026-10-01/tools/newapp_box.py # accepted: a throwaway test user's password in an audit tool for a fresh-app walk on a Tier-0 box
|
||||
gitleaks generic-api-key@eada8358f62a hub/internal/web/render_test.go # false: a test constant or a decoy value, no live secret
|
||||
gitleaks generic-api-key@efd4cc86d8d9 documentation/audits/power-cut-fixes-2026-10-14/E/demo-hp-journal-boot.txt # false: the id of HP's public UEFI certificate in a boot journal
|
||||
gitleaks generic-api-key@f28c207293c1 documentation/tests/campaign2-evidence-2026-07-07/evidence/T-DEPLOY-SET/values_vaultwarden.json # accepted: dry-run deploy values of a 2026-07-07 campaign (generated, demo boxes since rebuilt)
|
||||
gitleaks generic-api-key@f399eb9e3059 documentation/audits/night-2026-09-23/chaos/round-01-run.txt # accepted: the same demo app session token as round-01.log
|
||||
gitleaks generic-api-key@f399eb9e3059 documentation/audits/night-2026-09-23/chaos/round-01.log # accepted: a demo app's session token, demo-hp 2026-09-23 (Tier 0); the 2026-10-12 review found no running demo app with it
|
||||
gitleaks generic-api-key@fdabf6a76b48 documentation/audits/r890-instructions-2026-10-06/tools/wgerwalk.py # accepted: a throwaway wger walk's API token in an audit tool (Tier 0)
|
||||
@@ -178,6 +178,12 @@ something, not only sessions that touch `documentation/` — which is why it is
|
||||
rows — a run can sit several pages earlier. **Scan every page** and match on `head_sha`; with a
|
||||
few hundred jobs that is a handful of requests. Guessing the page produced three consecutive
|
||||
false "no CI job for this commit" readings in one session.
|
||||
3. **A rerun REPLACES the attempt the `jobs` endpoint shows** (R-1137, measured 2026-10-18): the job
|
||||
keeps its id, `run_attempt` becomes 2, `started_at`, `conclusion` and the log are the rerun's — the
|
||||
red first attempt is gone from `jobs` and its log (controller jobs 1944, 1959: failure mails, now
|
||||
green). **A matched job with `run_attempt` > 1 is not a plain green:** list `actions/tasks` (same
|
||||
paging, match `head_sha`) — one task per attempt, the verdict in `status`, so it reads
|
||||
„failure, then rerun success". The first attempt's log is not reachable by the API.
|
||||
|
||||
An unchecked green is an assumption, not an observation — and so is a green read off a field the
|
||||
API never populates.
|
||||
|
||||
+56
@@ -16,6 +16,62 @@
|
||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||
|
||||
|
||||
> **2026-10-19 (night) — controller 0.335.0 is the floor (MinAgent 0.158.1, 18:39 UTC); hub 0.162.0 BUILT, not deployed;
|
||||
> agent `56aabc7` and installer 1.35.0 committed, not released/published.** R-921 (operator answer „add one field"): the
|
||||
> agent serves its host-wide heavy-operation gate as `host_busy` on `GET /backup/status`; the controller reads it before
|
||||
> any stop (held → contention, no stop). R-1070's cause: Docker 29 prints „error: no such object" in lower case, so a
|
||||
> guest without cloudflared read `unknown` and failed the host OS-update rule; new state `absent` (passes when absent
|
||||
> before the step too); `11` §8.2 had claimed `unknown` does not fail — corrected. R-1010: the janitor's forced snapshot
|
||||
> delete is the API's `force=1`. Hub 0.162.0: R-1001 (restart seeding from the DB), R-1011 (exact-file sha). R-953 is
|
||||
> gone: ArgoCD tracks none of the three Secrets now. **R-986: `scripts/security_scan_gate.py`, shared, in every repo's
|
||||
> fast set and CI, WARN-FIRST**; govulncheck pin v1.1.4 → v1.8.0 (v1.1.4 cannot read Go 1.27); CI has no Go, so
|
||||
> govulncheck runs only in the hook; a depth-1 CI clone scans the whole tree (the root commit), all accepted. R-893
|
||||
> measured on 9202 (`audits/night-2026-10-19/P3/`) → D20. 9202 is on DHCP (.121 now).
|
||||
|
||||
> **2026-10-19 (late afternoon) — controller 0.334.0 is the floor (MinAgent 0.158.1, 16:31 local); hub 0.161.0 DEPLOYED;
|
||||
> tenantsync v1.3.0 on ep0 (operator yes).** R-1159 (D19 „box key first, then lock"): the passphrase is used AFTER enroll
|
||||
> (installer `/config` for its git login, the agent bakes it into `bootstrap.json`, the guest's first-boot pull, and —
|
||||
> until 0.334.0 — every config refresh), so "keys only before a host exists" was not buildable. Now: `GET /box-config`
|
||||
> with the controller key; the hub refuses passphrase keys (409, `passphrase_keys_refused`) once a ≥ 0.334.0 controller
|
||||
> has reported since the host record was made — an rc (`0.334.0-rc.N`) counts as older. **A reinstall over a live box
|
||||
> needs the operator's delete first.** R-1165: every new ep0 namespace gets `prune-<id>`; the hub alarms on a namespace
|
||||
> with none. demo-hp's .104 clean after the router reservation. Register 163 → 161. `REPORT-day3-2026-10-19.md`.
|
||||
|
||||
> **2026-10-19 (afternoon) — controller 0.333.0 is the floor (MinAgent 0.158.1, 12:16 local); hub 0.160.0 DEPLOYED
|
||||
> (operator yes); ep0 prune `keep-weekly 2` (operator yes, D18).** ep0 change: `prune-demo-hp`/`prune-demo-felhom` only,
|
||||
> after PBS's own `--dry-run true` (API `proxmox-backup-debug api create …/prune`; it creates no task) showed nothing
|
||||
> removed; households with no prune job (tester-1, Tester-2) = R-1165. R-1155: FileBrowser 1.5.6 reads its token from the
|
||||
> cookie, `?auth=`, Bearer AND the Basic-auth password (`extractToken`) — the forwardAuth refusal must read all four (the
|
||||
> security review caught Basic). R-1159 measured: passphrase → `/config` key and `/host-enroll` existing host key → every
|
||||
> off-site key route; D19 open (CC picks A: keys only while no host exists). R-1158: the reinstall's off-site re-issue only
|
||||
> rotates the hub's own registrar login; kept. gofmt gates in hub and agent runners. demo-hp's .104 is taken by another
|
||||
> device (R-1166, the router). Register 167 → 163. `REPORT-day2-2026-10-19.md`.
|
||||
|
||||
> **2026-10-19 (day) — controller 0.332.0 is the floor (MinAgent 0.158.1, 05:24 local); agent 0.160.0 released and on the
|
||||
> three test hosts, NOT vouched; hub 0.159.0 DEPLOYED (operator yes in advance).** One sign-in for the file manager (R-1100,
|
||||
> D16): dashboard pass → host-only `felhom_files` cookie → traefik forwardAuth `/__felhom_gate/files` sets the per-box
|
||||
> SECRET header (`<data>/filebrowser-sso-header`, never logged) naming FileBrowser user `felhom-household`; traefik makes a
|
||||
> RELATIVE Location in a forwardAuth answer absolute against the auth server — always answer absolute URLs. R-1151: the
|
||||
> escrow package uploads at `POST /escrow/ceremony/confirm` (two typed words), an old controller keeps the claim order.
|
||||
> R-1149 (D17): local tier `keep-daily=N`. R-1154 rule (after a security review of the helper's first draft): the
|
||||
> host-enroll reuse path NEVER re-issues — the passphrase must not reach the operator's re-issue powers; it mails the
|
||||
> operator. R-1153 was the test file (FileBrowser previews .docx with MuPDF before OnlyOffice). R-887 closed (381 jobs, 0
|
||||
> lost since S21). The old-restic check after demo-hp's first real 0.19.1 prune passed. Register 168 → 166.
|
||||
> `REPORT-day-2026-10-19.md`.
|
||||
|
||||
> **2026-10-18 (evening) — controller 0.331.0 is the floor (MinAgent 0.158.1): OnlyOffice opens and saves from FileBrowser
|
||||
> and Nextcloud, FileBrowser back at once after a restart (R-1018), a removal deletes a volume the definition dropped
|
||||
> (R-1117); catalog `30eea97`: Immich/Nextcloud Redis without a volume; golden 0.331.0 baked, NOT vouched; a fresh install
|
||||
> from the ISO proved installer 1.34.0 + vouched golden/agent + auto-floor (R-1148); agent `main` carries R-978, R-973,
|
||||
> R-1093 unreleased. *Decided by CC unattended — operator may reverse:* the R-973 cap numbers (3 successful daily-tier runs
|
||||
> per 24 h, one weekly off-site run per 6 days; `07` §6.1). Tester4 is kept as the standing fresh-install customer
|
||||
> (operator). `REPORT-evening-2026-10-18.md`.**
|
||||
|
||||
> **2026-10-18 (day) — hub 0.158.0 DEPLOYED (access log live, agent-only vouch, R-895 first verdict 2026-10-20);
|
||||
> controller 0.330.0 is the floor (R-1144); agent 0.159.2 released and on the three test hosts, NOT vouched; installer
|
||||
> 1.34.0 published (vault-first step 4b, R-1145; fresh install unproven, R-1148); estimator Immich peak 3,983 MB
|
||||
> (R-1147).** Independent review of the night: `audits/day-2026-10-18/P2/`. `REPORT-day-2026-10-18.md`.
|
||||
|
||||
> **2026-10-18 (night) — controller 0.329.0 is the floor (MinAgent 0.158.1); golden 0.329.0 baked, NOT vouched; hub 0.158.0
|
||||
> built, NOT deployed.** Each app's own `appdata/<folder>` (R-1142), empty allow-list (R-981), drive retirement asks the
|
||||
> agent first (R-1063), plain no-durable-id refusal (R-1069), `:latest` for releases only (R-1143), Immich 3072M/2048M
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
# REPORT — day 2026-10-18
|
||||
|
||||
## In plain words
|
||||
|
||||
- **The hub update is live.** Its own access log works and writes no secret (tested live with a fake token). You can
|
||||
now vouch an agent without a new golden. The off-site snapshot check gives its first verdict on 20 October.
|
||||
- **A fresh reviewer read last night's code.** One medium finding: a stale host key could have left the emergency
|
||||
password unknown. Today's installer change (save in the hub first, then set) already closes it. Three low findings:
|
||||
two fixed today, one is a known design limit, now written in its row.
|
||||
- **„Retire anyway":** when the host says no, the drive's apps start again. Where the host's disk list already says no,
|
||||
nothing is stopped at all. Controller 0.330.0 is the floor for every box.
|
||||
- **Agent 0.159.2** runs on demo-hp, Tester 1 and demo-felhom (your yes). A whole-box backup ran with no pool check.
|
||||
**The vouch is yours** (STATUS).
|
||||
- **Installer 1.34.0 is published** (your yes): the emergency password is saved in the hub first, then set, then
|
||||
checked with a login. Proven on demo-hp. A fresh install with it has not run yet (row R-1148).
|
||||
- **The calculator** shows Immich's bigger figure (your yes). Immich alone on an 8 GB machine now reads „does not fit".
|
||||
- **The old-restic check has not run.** demo-hp's clean-up comes only after tonight's backup (2026-10-19 about 04:16
|
||||
Budapest). The STATUS item stays open.
|
||||
|
||||
## 1. Hub 0.158.0 deploy
|
||||
|
||||
ArgoCD: only `Deployment/hub` was OutOfSync; synced that resource alone → Synced/Healthy, image `felhom-hub:0.158.0`,
|
||||
startup log `felhom-hub 0.158.0 starting` + `access log: /data/access-log …, kept 90 days`. Floor read back from the hub
|
||||
DB with its WAL: 0.329.0 / 0.158.1 at deploy time (`audits/day-2026-10-18/P1/hub-deploy-readback.txt`). Reports from all
|
||||
three boxes in the access log within two minutes (sources .101, .149, .155).
|
||||
- **Access log (R-1140, closed):** today's file `access-2026-10-18.log`, mode 0600. A fresh 48-hex fake token sent in a
|
||||
query, a `/bind/<token>` path, a `session=` query, a Bearer header, a Cookie and a POST body: found 0 times; control
|
||||
(the same file with the token appended) 1. Secret shapes (Bearer value, a 32+ character run, `password=`, `token=`,
|
||||
`session=`): 0; control 1. `/bind/<token>` is written `/bind/[redacted]` (`P1/access-log-live-test.txt`).
|
||||
- **Agent-only vouch:** read-only — `/configuration` loads with CC's token (Day-0 section, golden 0.329.0, agent 0.159.1);
|
||||
the save (`/configuration/artifacts`) answers 403 to the token, by design. No manifest change by CC.
|
||||
- **Off-site alarm (R-895, NARROWED):** no fixture route exists in the hub for it; the unit tests are the proof so far.
|
||||
Tomorrow's 07:10 run is the baseline (no verdict by design); the first verdict is 2026-10-20 07:10 — a DUE-CHECKS row.
|
||||
|
||||
## 2. The independent review
|
||||
|
||||
A fresh subagent with no context from the night reviewed controller v0.328.0→v0.329.0, hub 0.157.0→0.158.0, the agent's
|
||||
orphan-check change and the installer's `--rotate-recovery-only`. Its report was written to a file and read back
|
||||
(`audits/day-2026-10-18/P2/independent-review.md`, 54 lines). Verdicts (`P2/verdicts.md`):
|
||||
|
||||
| # | Finding | Verdict |
|
||||
|---|---|---|
|
||||
| F1 medium | rotate-only: unauthenticated healthz, then chpasswd, then a vault that a stale key fails → an unvaulted password | REAL in the night's code → **fixed** by R-1145 (vault first); harness `R1145-e`, fails against HEAD |
|
||||
| F2 low–medium | `offsite_snapshots_unexplained` is explained by the box's own reported run time | REAL, by design (decision 68) → written into R-895; the custody alarm `_vanished` is not affected |
|
||||
| F3 low | „anyway" stops apps before the agent is asked | REAL, was R-1144 → **fixed** (controller 0.330.0) |
|
||||
| F4 low | host key in curl's argv during the vault PUT | REAL, pre-existing → **fixed** (header file) |
|
||||
|
||||
## 3. R-1144
|
||||
|
||||
`decommissionAnyway`: reads the agent's disk list first (a role other than user-data, or the backup target → the plain
|
||||
refusal, nothing stopped); after a refusal past the stops (policy refusal or a failed agent call) starts exactly the apps
|
||||
that were running; a drive the agent let go keeps its apps stopped. Household text HU + EN. Seam `decomStacks`, so no
|
||||
test reaches docker. Three tests, two red-proofs (`P3/redproof-r1144.txt`). Not pressed live: no test box has a drive
|
||||
that is safe to retire.
|
||||
|
||||
## 4. Agent 0.159.2
|
||||
|
||||
Released with `release-agent.sh` (code pushed first, CHANGELOG after — the `release-complete` hook refuses the other
|
||||
order): sha256 `4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9`, bundle
|
||||
`477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9` (read back from the registry). **demo-hp proof:** a
|
||||
whole-guest backup of 9201 through the dashboard (`/api/guest-backup/trigger`, snapshot mode, completed in 7.5 min):
|
||||
**0** `felhom-thin-check` sudo lines; control — the night's backup on 0.159.1 at 04:33 had **1**
|
||||
(`P4/demo-hp-backup-journal.txt`). The failure path stays covered by the agent's fixture tests. **Operator: yes** →
|
||||
Tester 1 and demo-felhom by signed jobs (agent_update + agent_config_update): both `signed op COMPLETED`, capability
|
||||
probe 75/75, `update committed` 0.159.2, and a host report on 0.159.2 from each (`P4/`). **Not vouched** — STATUS.
|
||||
|
||||
## 5. The password order (R-1145, R-1146)
|
||||
|
||||
Step 4b (install and rotate-only): generate → PUT to the vault (200 = the hub's DB write is done) → chpasswd (one retry)
|
||||
→ a PVE API login with the new value. **The states that can remain, and why each is safe or not:**
|
||||
|
||||
| State | When | Safe? |
|
||||
|---|---|---|
|
||||
| nothing changed | the vault refuses or is unreachable (a stale key included) | safe — root and the vault keep their old, matching pair |
|
||||
| root = vault = new, PVE login 200 | the normal end | safe |
|
||||
| vault = new, root = previous | chpasswd fails twice (or the run is killed between vault and set) | **not fully safe:** the vaulted value does not open the console; root keeps its previous password (the shadow line is fingerprinted before/after to prove it). The host key cannot read the vault back (by design), so the previous value cannot be re-vaulted; the log says the state in words, and the next run fixes it (it vaults first again). |
|
||||
| root = vault = new, PVE login refused | pveproxy trouble | safe for the shell; the run dies and says the web login refused |
|
||||
|
||||
The old order's dangerous state (root = new, vault = old) is no longer reachable. The hub cannot tell the installer its
|
||||
„last vaulted" time (no host-readable route); the installer trusts the PUT's 200, and CC checked the time moved in the
|
||||
live proof. Live on demo-hp: vault `set_at` 2026-07-21 → 2026-10-18T10:43:33Z, the new value gets a PVE ticket (200),
|
||||
the old one 401 (`P5/demo-hp-rotation.txt`). Harness: 52 passed / 0 failed (was 45/2); R1145-a..e red-proofed against
|
||||
HEAD. Installer **1.34.0 published** on the operator's yes: tag `installer-v1.34.0`, both webpage pins, ArgoCD synced
|
||||
the webpage Deployment only; the served file's sha256 = the tag's (`df722ee4…`). R-1146: SC2066 and GL8-F6 fixed, the
|
||||
harness is gate `hostinstall-harness` with two decoys.
|
||||
|
||||
## 6. Calculator (R-1147)
|
||||
|
||||
Before/after (`audits/day-2026-10-18/P6-before-after.md`): Immich peak 3,290 → 3,983 MB (typical unchanged 1,557). The
|
||||
one changed verdict: **Immich alone on 8 GB: fits, but tight → does not fit**. The family starter set is unchanged.
|
||||
**Operator: yes** → published (cache-bust v5); live `sizing-data.json?v=5` serves 3,983. SIZING.md's table and the
|
||||
pinned estimator tests moved with it. R-1036 closed (D14 = no).
|
||||
|
||||
## 7. The old-restic check
|
||||
|
||||
**Not run — the prune has not happened yet.** The hub allows demo-hp a clean-up window 6 days 12 hours after the last
|
||||
one (2026-10-12 02:16:53 UTC → from 2026-10-18 14:16:53 UTC). But the box asks for a window only after its own nightly
|
||||
off-site run, and that runs at 04:15 CEST. So the first real 0.19.1 prune comes **2026-10-19 at about 02:16 UTC**
|
||||
(04:16 Budapest). Read at 14:33 UTC: no `[offbox]` line since the 11:09 restart, the scheduler's next run 2026-10-19
|
||||
04:15 CEST, the hub's last window still id 9 (`P8/cleanup-window-not-yet.txt`, the hub DB read with its WAL). CC did not
|
||||
grant a window by hand (that is the operator's). The STATUS item stays open, with the corrected time; the check script
|
||||
is ready (`rx.sh` + a wrapper that lists and restores the newest, the oldest and the newest 0.14-made snapshot with both
|
||||
restics and compares tree hashes).
|
||||
|
||||
## 8. Versions, commits, fixtures
|
||||
|
||||
- Hub **0.158.0** deployed (felhom.eu `4c319dde`).
|
||||
- Controller **0.330.0** — `6c6bd87`; floor 0.330.0 / MinAgent 0.158.1 at 11:10:59 UTC (303 `floor_set`, read back with
|
||||
the WAL); per-box floors set and cleared; box walk green on the three test boxes (0 refusals, all healthy, login and
|
||||
files 200, https-only OK).
|
||||
- Agent **0.159.2** — `65c1994` (code), tag `v0.159.2`, CHANGELOG `e25454c`.
|
||||
- Installer **1.34.0** — felhom.eu `88aecf55` (vault-first + harness), `624bcdc3` (1.34.0, tagged), `3332eb09` (pins).
|
||||
- Estimator — `37cb83c1`. Fixtures: none changed (the new texts travel in JSON).
|
||||
|
||||
## 9. Register — 173 before, 168 after; opened 1, closed 6
|
||||
|
||||
Closed: R-1140, R-1144, R-1145, R-1146, R-1147, R-1036. Opened: R-1148. Narrowed: R-895 (dated check 2026-10-20).
|
||||
|
||||
## 10. Observations
|
||||
|
||||
1. Installer 1.34.0's default path has not run in a fresh install. FILED: R-1148.
|
||||
2. The review's F2 (box-reported run times explain new snapshots). NOT-A-FINDING as a new row: a known design limit,
|
||||
recorded in R-895.
|
||||
3. `release-agent.sh` vs the `release-complete` hook: the CHANGELOG must follow the release. NOT-A-FINDING: the documented
|
||||
order (memory and skill); I tried the other order first and the hook refused it, as designed.
|
||||
4. The estimator's new Immich figure is `memory.current` (with cache), not the `used` measure it normally takes.
|
||||
NOT-A-FINDING: on the safe side, said in SIZING.md, and approved by the operator with the table in front of them.
|
||||
5. A safety check refused a `sh -c` with an unguarded `rm "$H"`. NOT-A-FINDING: rewritten with `${H:?}` in a helper.
|
||||
|
||||
## Teardown — three layers
|
||||
|
||||
- **Machines:** demo-hp 9201 — one whole-guest backup taken (kept, like any backup); no app installed or removed today.
|
||||
demo-hp host — the installer copy shredded after the rotation; root password rotated on purpose (new value in the
|
||||
vault). Tester 1, demo-felhom — agent 0.159.2 and controller 0.330.0 through their normal paths; nothing else.
|
||||
- **Hub:** hub 0.158.0; global floor 0.330.0 / 0.158.1; no per-box floor; signed jobs consumed. No vouch by CC.
|
||||
- DooPlex: builds only; ArgoCD synced two Deployments (hub, webpage), each alone.
|
||||
|
||||
Instruction files: none edited. System poster: none of the three (the poster shows backup retentions; the hub's access
|
||||
log is not on it). Architecture documents read: `02`, `03`, `05`, `07`, `09` §3 decision 250; `break-glass.md`.
|
||||
@@ -0,0 +1,156 @@
|
||||
# REPORT — day 2026-10-19
|
||||
|
||||
The operator answered the two STOPs in advance („YES to the agent and the hub release") and earlier: golden 0.331.0
|
||||
vouched; D16 (one sign-in) yes; D17 „one per day, 3 days". No question was asked during the session.
|
||||
|
||||
## In plain words
|
||||
|
||||
- **Agent 0.160.0 runs on all three test hosts.** A new recovery code replaces the old one only after the household types
|
||||
two of its words back; a broken-into box cannot flood the whole-box backups; the local copies are one per day for 3
|
||||
days; an empty partition does not ask „data will be erased". **It is not vouched — that is yours.**
|
||||
- **One sign-in for the file manager is live on every box** (controller 0.332.0, the floor). „Fájlok ↗" opens the app's
|
||||
folder with no second login. A sign-out on the dashboard asks again. No app on the box can pretend to be the household.
|
||||
- **Hub 0.159.0 is live.** A refused backup has its own alarm text. A reinstall over an old host record now mails you; it
|
||||
does not re-issue the codes by itself (a security review stopped the first draft that did).
|
||||
- **The preview error (R-1153) was a broken test file**, not the box.
|
||||
- **The old backup program still reads demo-hp's off-site copies** after the first real clean-up with the new one.
|
||||
|
||||
## 1. Agent 0.160.0 (R-978, R-973, R-1093, R-1149, R-1151) — `audits/day-2026-10-19/P1/`
|
||||
|
||||
- **Content:** R-978 (an abandoned second ceremony keeps the first code), R-973 (the cadence cap: daily tier 3 per 24 h,
|
||||
off-site 1 per 6 days), R-1093 (an empty partition is not data), **R-1149** (local tier `keep-daily=N`, operator D17),
|
||||
**R-1151** (the package uploads only at `POST /escrow/ceremony/confirm` with two typed words; a claim without
|
||||
`defer_upload` keeps the R-978 order, so an old controller still completes a ceremony). Commits `1b6e316`, `c11e3e2`,
|
||||
`124164d` (CHANGELOG), `4545819` (two texts, after the release, unreleased). Released with `release-agent.sh`: sha256
|
||||
`7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1`, bundle
|
||||
`6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f`, round trip verified, tag `v0.160.0`.
|
||||
- **Red-proofs:** keep-last for the primary → „primary tier prune = "keep-last=3"" and „holds 0 copies made before the
|
||||
compromise"; the claim's defer flag ignored → „a code that was shown but never typed back replaced the package: code 1
|
||||
no longer opens it (uploads=2)" (all three abandon paths); the word check skipped → „one wrong word: got 200".
|
||||
- **demo-hp proof** (signed jobs 01:22 UTC, running 01:23, „update committed" 01:24; probe 75/75):
|
||||
- a normal „Mentés most" of 9201: green, 6.9 GB in 443 s; yesterday's copy pruned (keep-daily=1, the same copy keep-last
|
||||
would prune); the cap's history on disk holds both successes;
|
||||
- escrow through the wizard's own calls: ceremony 1 — one wrong word 422, the right words 200; code 1 opens the hub's
|
||||
package, code 0 is a kept earlier one (422, superseded at the confirm second 01:47:47). Ceremony 2 shown and abandoned —
|
||||
code 1 still opens it (200), code 2 opens nothing (400). demo-hp's stored code is now code 1 (CC's secret store);
|
||||
- R-1093 on the real route: a 64 MB RAM disk (`scsi_debug`, `/dev/sdb`), one empty GPT partition → `POST /disks/format`
|
||||
→ 200 formatted, no confirmation; control: the same partition, now ext4 → 403 `needs_confirmation`. RAM disk removed.
|
||||
**A loop device cannot be used:** the route and the probe accept only sd/nvme/vd/hd names (by design).
|
||||
- **The answer:** the operator's „yes" (in advance). Tester 1 and demo-felhom took 0.160.0 by signed jobs (01:52 UTC →
|
||||
01:54 / 01:55, probe 75/75, `keep-daily=3` armed on both); the hub's host rows read 0.160.0 for all three, and fresh
|
||||
reports after the hub restart. **Old controller with the new agent:** pinned by `TestEscrowCeremony_R1151_OldControllerKeepsTheClaimOrder`; **new controller with agent 0.159.2:** pinned by `TestR1151_OldAgentClaimCompletesAtOnce` and the agentapi test — no host runs 0.159.2 any more, so not walked live.
|
||||
|
||||
## 2. One sign-in (R-1100) — `audits/day-2026-10-19/P2/`, `P4/05-*`, `P4/06-*`
|
||||
|
||||
- **Built** exactly the design note's recommendation (controller `9bdd8f8`, `601bdba`): the dashboard pass, the host-only
|
||||
files cookie (derived from the session fingerprint, the R-976 shape), the per-box secret header name, the forwardAuth
|
||||
answer; header user `felhom-household` with the household's file rights from userDefaults; `logoutRedirectUrl` = the
|
||||
dashboard; `admin` stays a password user; the data tab's „asks for its own password" note only without the sign-in.
|
||||
- **Tests, each red-proved:** replayed pass, expired pass, pass for another host (two guards: the host check and the
|
||||
host-bound session key — red only with both removed), wrong X-Forwarded-Host, no session → no pass, signed out → asks
|
||||
again (and after a password change), cross-site → no pass, dot-segments in `to`, the absolute redirect, the config
|
||||
trusts only the secret name, the header user's rights, the route asks the controller and strips the header, the note.
|
||||
- **Live, found and fixed before release:** on rc.2 the pass answer's relative redirect became
|
||||
`http://felhom-controller:8080/files/` — traefik resolves a forwardAuth answer's Location against the auth server. Fixed
|
||||
(absolute), red-proved, rc.3. A background security review then flagged CSRF and revocation: `/__files/open` now refuses
|
||||
a cross-site request and dot-segments; the revocation residual is R-1155.
|
||||
- **Live proof (rc.3, 9201, headless Chrome through the public names mapped to the box):** signed in → calibre-web's
|
||||
„Fájlok ↗" → FileBrowser at `NVME 1TB/media/books`, no login form, user `felhom-household` (admin false, modify true,
|
||||
login method proxy); signed out → `files.<domain>` went to the dashboard login. Screenshots `browser/01-…`, `02-…`,
|
||||
`03-…`. Forged headers (`forged-headers.txt`): through traefik with no cookie — the old public name, a random name and the
|
||||
REAL secret name all 401; from the bookstack container straight to `filebrowser:80` — the old name and a random name 401;
|
||||
control: the real name from the same container 200 (the protection is the name's secrecy). The brief named Nextcloud;
|
||||
bookstack was used (an app on 9201's network with curl).
|
||||
|
||||
## 3. Preview check (R-1153) — `audits/day-2026-10-19/P3/`
|
||||
|
||||
On 9202 (controller 0.331.0): OnlyOffice installed, the link on (FileBrowser back in 4 s). The hand-made `teszt.docx`
|
||||
(912 bytes) → 500 „fitz: cannot open document"; OnlyOffice's own `sample.docx` → **200, a 256×256 JPEG, 0.3 s**. Same with
|
||||
the link OFF. FileBrowser 1.5.6 previews .docx with its own MuPDF before OnlyOffice (`preview.go` `determinePreviewType`).
|
||||
**Closed: not a product defect.** OnlyOffice removed through the product (both volumes gone), test files deleted.
|
||||
|
||||
## 4. Controller release 0.332.0 — `audits/day-2026-10-19/P4/`
|
||||
|
||||
Image `0.332.0` from `601bdba` (digest `sha256:245266a4…`), 03:16 UTC. Per-box floors 03:16:43 → Tester 1 and demo-felhom
|
||||
on 0.332.0 at 03:16:45–55, demo-hp (from rc.3) at 03:17. Box walk on all three: 0 unhealthy, 0 allow-list refusals,
|
||||
FileBrowser re-published 6 s after its re-create, public pages 200; https-only 28 OK, 0 failures, both controls right.
|
||||
„Fájlok ↗" with no login through the public internet on demo-hp and Tester 1 (pass → cookie → `felhom-household`; a fresh
|
||||
browser 401; sign-out → back to the dashboard). demo-felhom: CC holds no dashboard password for the lived-in box, so its
|
||||
check is the files (secret 0600, config and route carry it) and the refusals (forged secret name through traefik 401, old
|
||||
name from outside 401, a forged pass 403). **Global floor 0.332.0 / MinAgent 0.158.1 at 03:24:18 UTC**, read back with the
|
||||
WAL; per-box floors cleared (read back empty). MinAgent 0.158.1 ≤ the vouched agent 0.159.2. i18n fixtures changed (named):
|
||||
`escrow_first`, `escrow_receremony`; all others byte-identical. No golden (decision 250); the floor is above golden 0.331.0
|
||||
until Sunday's bake.
|
||||
|
||||
## 5. Hub 0.159.0 (R-1150, R-1154) — `audits/day-2026-10-19/P5/`
|
||||
|
||||
- Written by a helper (`5121536f`, `b505ce4b`). **R-1154's rule, and why:** the helper's first draft re-issued the claim
|
||||
code and the off-site provider password automatically when the old host was stale or switched off. A background
|
||||
security review flagged it; the helper then read what the retrieval passphrase already grants (`/config` → the
|
||||
controller API key; the enroll → the host key) and found the re-issues are the operator's session-only powers
|
||||
(decision 252). So the passphrase must not reach them: **the reuse path re-issues nothing; when the old box looks gone it
|
||||
mails the operator** („delete the old host and run the installer again, or re-issue …"), a live box gets an info record.
|
||||
The operator's delete stays the gate. **R-1150:** `whole_guest_backup_refused` (operator-only, hub-origin; NOT in
|
||||
`allowedEventTypes`, against the brief's wording — the suspected controller must not post or pre-empt it, R-895's
|
||||
shape). Tests red-proved (helper's list in `hub/CHANGELOG.md`).
|
||||
- **A push blocker on the way:** R-887's dated check was due today. Measured: 381 CI jobs since the S21 change, 0 lost (the
|
||||
three re-runs' first attempts ran 47–49 s; control: the known lost attempt of 2026-10-10 ran 11 min 48 s and the attempt
|
||||
count finds it). Closed.
|
||||
- **Deploy:** image from a clean clone (the shared checkout held my uncommitted evidence), manifest `200ab653`, ArgoCD
|
||||
synced the hub Deployment only (the only OutOfSync resource), Synced/Healthy, no ERROR, the floor read back, then fresh
|
||||
reports from all three hosts (02:23–02:25 UTC) and all three controllers (02:29–02:31 UTC).
|
||||
|
||||
## 6. The old-restic check — `audits/evening-2026-10-18/P8/old-restic-check.txt`
|
||||
|
||||
The night watcher ran it: window 12 at 02:18 UTC removed 63 of demo-hp's 324 copies (→ 261); the old restic (0.14.0) and
|
||||
the new (0.19.1) both list 261 and restore a 0.19.1-made copy and an August copy to identical content hashes. STATUS item
|
||||
closed; `REPORT-evening-2026-10-18.md` §9 filled.
|
||||
|
||||
## 7. Versions, commits, fixtures, rows
|
||||
|
||||
- Agent **0.160.0** (released, not vouched) — `1b6e316`, `c11e3e2`, `124164d`, `4545819`. Controller **0.332.0** (floor) —
|
||||
`ba4a6f0`, `9bdd8f8`, `601bdba`; rc.1–rc.3 on 9201 only. Hub **0.159.0** (deployed) — `5121536f`, `b505ce4b`, manifest
|
||||
`200ab653`. felhom.eu `afc939bb` (R-887), `5622ce27`, this commit. Catalog: none.
|
||||
- Fixtures: `escrow_first`, `escrow_receremony` (controller); no other page changed.
|
||||
- **Register: 168 before, 166 after; closed 10** (R-887, R-978, R-973, R-1093, R-1149, R-1151, R-1100, R-1153, R-1150,
|
||||
R-1154), **opened 8** (R-1155 … R-1162).
|
||||
- `unproven.py --summary`: not walked 35 of 55 — unchanged. Capability map: not changed. System poster: text only — the
|
||||
local whole-guest „Kept" cell („one copy per day, up to 3 days") edited in both the facts file and the drawing.
|
||||
|
||||
## Observations
|
||||
|
||||
1. FileBrowser's own 2-hour token can outlive the dashboard session if only the files cookie is lost. FILED: R-1155.
|
||||
2. A reinstall within 45 min of the old box's last report reads „live" (no mail). FILED: R-1156.
|
||||
3. Repeated cap refusals inside the 24 h hold raise no new alarm. FILED: R-1157.
|
||||
4. The off-site leg of the clean-slate re-issue may be obsolete since hub v0.127.0. FILED: R-1158.
|
||||
5. The retrieval passphrase yields the controller API key, which registers and confirms off-site keys. FILED: R-1159.
|
||||
6. Hub and agent have no gofmt gate (~20 and 14 files drift). FILED: R-1160.
|
||||
7. The escrow wizard calls the code „live" at the reveal; since R-1151 it is live only after the typed-back words.
|
||||
FILED: R-1161.
|
||||
8. R-973's off-site half (ep0 prunes by count) outlives the closed row. FILED: R-1162.
|
||||
9. traefik resolves a forwardAuth answer's relative Location against the auth server. NOT-A-FINDING: found on rc.2 and
|
||||
fixed before the release (red-proved); recorded as a rule in CONTEXT and memory.
|
||||
10. The ceremony log line and two R-973 comments were stale. NOT-A-FINDING: fixed in agent `4545819` (the size rule).
|
||||
11. My first OnlyOffice link call sent `{}` and switched the link off. NOT-A-FINDING: my call; the API wants
|
||||
`{"enabled": true}`; it was already off, nothing restarted.
|
||||
12. The R-1093 route refuses loop devices. NOT-A-FINDING: by design (only real disk names are allowed); a RAM disk was used.
|
||||
13. CC holds no dashboard password for demo-felhom, so its „Fájlok ↗" was not walked as a household. NOT-A-FINDING: the
|
||||
lived-in box's household password is not CC's (decision 254); its files and refusals were checked instead.
|
||||
14. A helper's commits sat unpushed in the shared felhom.eu clone for ~40 minutes (blocked by R-887's due check).
|
||||
NOT-A-FINDING: the gate did its job; pushed with my R-887 commit.
|
||||
15. Two shell scripts of mine were refused by Claude Code's removal check (a cleanup `rm` it could not read behind
|
||||
`bash -c`). NOT-A-FINDING: rewritten without the removal; nothing was deleted.
|
||||
|
||||
## Teardown — three layers
|
||||
|
||||
- **Machines:** 9202 — OnlyOffice removed through the product (volumes gone), test documents deleted; 9201 — on 0.332.0
|
||||
(the floor); the escrow test left demo-hp on code 1 (stored); demo-hp host — the RAM disk module removed, helper scripts
|
||||
deleted, no file left in `/root` (an older helper, `/root/opens.sh`, removed by name at the end). Cookie jars and
|
||||
password files on DooPlex shredded or emptied. One debugging sign-in on demo-hp's dashboard (curl, 03:03 UTC) was not
|
||||
signed out; its cookie file is shredded, so the session expires unused.
|
||||
- **Hub:** global floor 0.332.0 / 0.158.1, no per-box floor; hub 0.159.0; no vouch.
|
||||
- **DooPlex:** builds (agent, controller rc.1–3 + 0.332.0, hub 0.159.0 from a clean clone) and the hub deploy; no Docker
|
||||
cleanup.
|
||||
|
||||
Architecture read: `01`, `02` „Logins and folders", `03` §8a, `04`, `05`, `07` §6/§6.1; the design note. Documents changed:
|
||||
`02`, `03` §8a, `07` §6.1, the poster facts + drawing text, `05` §8 (helper). Instruction files: none changed.
|
||||
@@ -0,0 +1,120 @@
|
||||
# REPORT — second day brief, 2026-10-19 (ep0 one copy per week; the passphrase measured; hub 0.160.0; controller 0.333.0; gofmt gates)
|
||||
|
||||
## In plain words
|
||||
|
||||
- **ep0 keeps one whole-box copy per week, for 2 weeks** (your „yes"). Before the change, PBS's own dry run showed it
|
||||
removes nothing today. The first real run is 2026-10-20 03:30 UTC; CC reads it then (row R-1162 stays open until then).
|
||||
- **The household passphrase:** alone it gets the box's keys, and with them every off-site key action. Two options are
|
||||
in STATUS (D19). CC picks A: the passphrase gives keys only while the customer has no box record.
|
||||
- **Controller 0.333.0 is the floor for every box** (12:16 local): a dashboard sign-out now also ends the file manager's
|
||||
own 2-hour pass; the recovery-code page says the code becomes live after the two words.
|
||||
- **Hub 0.160.0 is live** (your „yes"): a reinstall within 45 minutes mails you too; a repeated refused backup is counted
|
||||
and the alarm repeats at most once a day; the old off-site re-issue on a reinstall is kept, with true words.
|
||||
- **gofmt:** the hub and the agent are formatted, and both now have the controller's gate.
|
||||
- **New for you:** another device keeps taking demo-hp's network address (R-1166) — fix it in the router.
|
||||
|
||||
## 1. ep0 (R-1162; `07` Plane-2 row and §6.1)
|
||||
|
||||
- **Before** (`audits/day-2026-10-19b/P1/ep0-settings-before.txt`): `prune-demo-hp` and `prune-demo-felhom` —
|
||||
`keep-last 2`, 03:30, ns-scoped, `max-depth 0`; `prune-operator-hubdb` — keep-daily 14 + keep-weekly 8, 03:45.
|
||||
`tester-1` (3 copies) and `Tester-2` (1) have **no job**. ep0's clock is UTC.
|
||||
- **DooPlex's copy of ep0:** `prune-ep0-copy`, `keep-weekly 8`, 07:30 (decision 71) — already by time; not changed.
|
||||
- **Dry run** (PBS's own: `proxmox-backup-debug api create /admin/datastore/felhom-offsite/prune --dry-run true …`, no
|
||||
task created; cross-checked with an offline model of PBS's selection, `P1/prune_sim.py`; 8 snapshots before and after):
|
||||
|
||||
| Box | Snapshots | keep-last 2 removes | keep-weekly 2 keeps | keep-weekly 2 removes |
|
||||
|---|---|---|---|---|
|
||||
| demo-hp | 2 | none | 2026-10-08, 2026-10-15 | none |
|
||||
| demo-felhom | 2 | none | 2026-10-06, 2026-10-13 | none |
|
||||
| tester-1 (no job) | 3 | (2026-10-04) | 2026-10-11, 2026-10-18 | (2026-10-04) |
|
||||
| Tester-2 (no job) | 1 | none | 2026-10-04 | none |
|
||||
|
||||
- **Answer:** „Yes, the two rules" (not the tester-1 job). **After** (07:49 UTC, `P1/ep0-settings-after.txt`): both jobs
|
||||
`keep-weekly 2`, schedule 03:30 unchanged, nothing run by hand. Check for 2026-10-20: the 03:30 `prunejob` tasks OK with
|
||||
`keep-weekly 2` and two snapshots per namespace (the next step of R-1162).
|
||||
|
||||
## 2. The household passphrase (R-1159)
|
||||
|
||||
Measured with the hub's real router (`P2/passphrase-reach.txt`; a made-up customer, a fake off-site service). A live
|
||||
measurement on Tester4 was not possible: CC holds no Tester4 passphrase, and reading it from the hub database was
|
||||
refused by CC's own safety check (correctly).
|
||||
|
||||
- Passphrase → `GET /config/{id}` 200 with `hub.api_key` (wrong passphrase 401); `POST /host-enroll` 200 with the
|
||||
EXISTING host's key.
|
||||
- Controller key → `register-key`, `confirm-key` (removes every other line, the box's own key included), `move-aside`,
|
||||
`window-open`, `abandon-request` (the hub deletes a set-aside copy after 7 days unless cancelled), `abandon-cancel`,
|
||||
`abandon-status`: all past auth (wrong key 401 each).
|
||||
- Host key → escrow GET/PUT, recovery-credential PUT, wg, desired-state, jobs, host-report: all past auth.
|
||||
- The passphrase is used only at install time (the installer, the controller's setup wizard).
|
||||
- **Option A (CC's pick):** keys only while the customer has no host record (`/config` leaves out the key, `/host-enroll`
|
||||
refuses once a host exists). Closes the chain at its source; a reinstall needs the operator's delete first — already the
|
||||
rule since R-1154. **Option B:** a second factor on the destructive off-site routes. Every reinstall waits for the
|
||||
operator, and the host key stays reachable by the passphrase, so B alone does not close it. No change in this task.
|
||||
|
||||
## 3. Controller 0.333.0 — R-1155, R-1161, the Install chip
|
||||
|
||||
- **R-1155:** first measured on demo-hp (0.332.0): the household's FileBrowser token opened FileBrowser after the
|
||||
dashboard sign-out (200, cookie and Bearer). Fix: the forwardAuth answer refuses a request with no valid files cookie
|
||||
that carries a non-admin FileBrowser token, everywhere FileBrowser reads one (cookie, `?auth=`, Bearer, Basic
|
||||
password). **The commit's security review found Basic missing** in the first version (`c809dd3`); fixed in `a6fa3c0`
|
||||
before any release. Red-proofs: without the check → 200 after the sign-out; without the Basic branch → 200; with every
|
||||
token refused → admin's own token 302. Live (rc.2 demo-hp, 0.333.0 Tester 1): files cookie 200; token alone 401 in all
|
||||
four forms, before and after the sign-out; FileBrowser's sign-in page 200.
|
||||
- **R-1161:** new key `backups_escrow.ez_a_kod_az_ellenorzes` (hu + en); the old sentence only when the claim answer says
|
||||
no confirm. Fixtures re-captured: `escrow_first`, `escrow_receremony` (reason: R-1151 made the old sentence untrue);
|
||||
all others byte-identical. Live on demo-hp (`P6/06-escrow-page-text.txt`): new text shown, old hidden, switch present,
|
||||
negative control 0.
|
||||
- **The Chrome „Install" chip: NOT-A-FINDING.** FileBrowser 1.5.6 links `/public/static/site.webmanifest` on every page;
|
||||
its only PWA setting is `disablePWAInstall` (already set). Not patched (`P3/install-chip.txt`).
|
||||
|
||||
## 4. Hub 0.160.0
|
||||
|
||||
- **R-1156:** a live re-enroll is a warning mailed to the operator, same instruction; a `--resume` mails too and says it
|
||||
needs nothing. **R-1157:** further refusals in one episode counted (`refusals_in_episode`), alarm at most once a day,
|
||||
total on the cleared record. **R-1158:** the reinstall's off-site re-issue changes nothing a box uses (only the hub's
|
||||
own registrar reads that password; no box has received it since v0.127.0); kept because it repairs a broken stored
|
||||
login; comments and the `offsite_reissued` text corrected (`05` §8). Each red-proved.
|
||||
- **Answer:** „Yes, deploy now". Image `0.160.0` digest `sha256:e9a360bc00d00a15014e272c4d023fef118a8b043956f3fc928dd26c00b34230`
|
||||
(clean clone at `a17db0c3`; the first push hit a DNS timeout, the retry pushed it). Manifest `5d3cb5c1`; only
|
||||
Deployment/hub was OutOfSync and only it was synced; Synced/Healthy; running image digest = built digest; 0 `[ERROR]`
|
||||
lines; floor read back from hub.db + WAL (0.332.0 then, 0.333.0 now); host and controller reports from demo-hp, Tester 1
|
||||
and demo-felhom after the restart.
|
||||
|
||||
## 5. gofmt (R-1160)
|
||||
|
||||
Hub `d5e3c8b0` (22 files) and agent `1edfd8e` (14 files), formatting only, tests green before each. Gates: `hub-gofmt`
|
||||
(`scripts/hub_gofmt_gate.py`, `repo_gates.py`) and `gofmt` (felhom-agent `scripts/gofmt_gate.py`, `agent_gates.py`), fast
|
||||
set, decoys on scratch trees, red-proved. No agent release.
|
||||
|
||||
## 6. Controller release (decision 250)
|
||||
|
||||
Per-box floors 0.333.0 at 09:59:38 UTC → all three on 0.333.0 within about a minute (demo-hp from rc.2 at 10:13). Box
|
||||
walk (`P6/`): 0 unhealthy, 0 allow-list refusals, catalog in sync; https-only 28 OK, 0 failures; „Fájlok ↗" with no second
|
||||
login on demo-hp and Tester 1, sign-out asks again. **Global floor 0.333.0 / MinAgent 0.158.1 at 10:16:22 UTC**
|
||||
(303 `floor_set`, read back with the WAL); per-box floors cleared.
|
||||
|
||||
## 7. Versions, commits, fixtures, rows
|
||||
|
||||
- **Controller 0.333.0** (`c809dd3`, `a6fa3c0`, REPORT `e1a2abf`), image digest
|
||||
`sha256:f150a8d8cd329f5cde9e34249e34edc59b6b5db29db1e3200900f3869a0de11b`; rc.1/rc.2 on demo-hp only. Fixtures:
|
||||
`escrow_first`, `escrow_receremony`.
|
||||
- **Hub 0.160.0** (`d5e3c8b0`, `503c93fa`, `a17db0c3`, manifest `5d3cb5c1`).
|
||||
- **Agent:** no release (`1edfd8e`, `0cbd500`). 0.160.0 stays the vouched one.
|
||||
- **Rows: 167 before, 163 after; closed 6** (R-1155, R-1156, R-1157, R-1158, R-1160, R-1161), **opened 2** (R-1165,
|
||||
R-1166); R-1159 → READY (D19), R-1162 → NARROWED (one run to read).
|
||||
- **System poster:** text only — `felhom-system-poster.facts.md` and the matching text in `felhom-system-poster.html`
|
||||
(„last 2" → „one per week, 2 weeks").
|
||||
- **Teardown:** machine — rc images only on demo-hp 9201, now on the release; no test app. Host — none. Hub — no test
|
||||
customer; the per-box floors cleared. ep0 — the two prune jobs changed on the operator's yes, nothing else.
|
||||
|
||||
## Observations
|
||||
|
||||
1. Two household namespaces on ep0 have no prune job; nothing creates one for a new household. FILED: R-1165.
|
||||
2. Another device takes demo-hp's address 192.168.0.104 every few minutes; demo-hp also lost its link to ep0 for 8 minutes. FILED: R-1166.
|
||||
3. A config download with the passphrase is a log line only, and the host key it yields can rewrite the box's escrow and recovery credential. FILED: R-1159.
|
||||
4. The first R-1155 version missed FileBrowser's Basic-auth token source (the commit's security review). NOT-A-FINDING: fixed before any release in `a6fa3c0` and red-proved (the size rule).
|
||||
5. Chrome offers „Install" for the file manager. NOT-A-FINDING: FileBrowser 1.5.6 has no setting that stops its manifest; the brief forbids patching it.
|
||||
6. One of CC's probe scripts on DooPlex passed the household's FileBrowser token to curl on its command line (visible to local processes for about a second; never printed or saved). NOT-A-FINDING: CC's own tooling on DooPlex, not the product; every probe now passes tokens through 0600 header files, and the token expired two hours later.
|
||||
7. Two copies of CC's hub-database snapshot ran into one folder at once and one read failed („malformed"). NOT-A-FINDING: CC's helper, a local copy only; the live hub and its database were untouched, and the next read was clean.
|
||||
8. The first hub image push failed on a DNS timeout at DooPlex. NOT-A-FINDING: a short DNS outage (the website session saw it too); the build's own check found the tag absent and the retry pushed it.
|
||||
9. FileBrowser's `admin` password login was not walked live. NOT-A-FINDING: CC holds no FileBrowser admin password; admin's token passing is unit-pinned and the sign-in page answers 200 live.
|
||||
@@ -0,0 +1,104 @@
|
||||
# REPORT — third day brief, 2026-10-19 (the passphrase locks once the box uses its own key; every ep0 household gets a prune job; demo-hp's address)
|
||||
|
||||
## In plain words
|
||||
|
||||
- **The household passphrase no longer gives a live box's keys** (your D19). Each box now fetches its config with its own
|
||||
key. The passphrase still works for a new install. A reinstall over a live box needs you to delete the old host first.
|
||||
- **The brief's first shape could not be built as written.** "Keys only before a box exists" would have broken every
|
||||
install and every live box's config updates: both use the passphrase after the box record exists. CC stopped and asked;
|
||||
you chose "box key first, then lock".
|
||||
- **ep0:** every new household gets its weekly clean-up rule automatically, and the hub alarms on a household without
|
||||
one. tester-1 and Tester-2 got theirs (your yes).
|
||||
- **demo-hp's address** stayed demo-hp's for a 30-minute watch after your router change.
|
||||
- **Live now:** hub 0.161.0, controller 0.334.0 (the floor), tenant-sync v1.3.0 on ep0.
|
||||
|
||||
## 1. The passphrase (R-1159; `05` §8 [DESIGN R-1159])
|
||||
|
||||
- **Install-order proof** (read before building, the brief's STOP): the passphrase is used AFTER `POST /host-enroll`:
|
||||
- `scripts/felhom-host-install.sh:2631` → `:480`, and `:3525`: `step_agent_install` and `step_golden` fetch the
|
||||
download login from `GET /config`;
|
||||
- `felhom-agent/cmd/felhom-agent/main.go:2835`: the provision bakes the passphrase into the guest's `bootstrap.json`;
|
||||
- `felhom-controller/controller/internal/bootstrap/bootstrap.go:142`: the guest's first boot pulls `GET /config` with it
|
||||
(that is how the box gets its own key);
|
||||
- `cmd/controller/main.go:1176` (`report.ConfigRefresher`): every live box re-pulled with it on each config change;
|
||||
- `:2317`: the preflight checks it with `/config` on every run.
|
||||
- **Your answer:** „Box key first, then lock".
|
||||
- **Built:**
|
||||
- Hub 0.161.0: `GET /box-config/{id}` (the controller's own key) and `passphraseKeysClosed`. Passphrase keys are
|
||||
refused (409, no key, operator event `passphrase_keys_refused`) once a ≥ 0.334.0 controller has reported since the
|
||||
host record was made. `/config`, `/recovery` and the `/host-enroll` reuse path are covered; `/artifacts` has no key
|
||||
and is unchanged.
|
||||
- Controller 0.334.0: the refresh pulls with its own key (`report.PullConfigPreferKey`). The passphrase is used only
|
||||
against an older hub or before the box has a key, and never as a retry after a refused key.
|
||||
- **Tests:**
|
||||
- Hub: `TestR1159_PassphraseGivesKeysOnlyUntilTheBoxUsesItsOwn` (5 cases), `TestR1159_BoxPullsItsConfigWithItsOwnKey`,
|
||||
`TestR1159_BoxConfigMinControllerIsTheReleaseThatPullsWithItsKey`.
|
||||
- Controller: `TestR1159_RefreshUsesTheBoxKey`, `TestR1159_MainWiresTheKeyFirstRefresh`.
|
||||
- Red-proofs: lock off → all three keys served; the old pull → 0 key pulls.
|
||||
- **Live on demo-hp** (`audits/day-2026-10-19c/P4/`; the probe ran on the host from the box's own `bootstrap.json`,
|
||||
printing status codes only):
|
||||
- on 0.333.0, the passphrase → 200;
|
||||
- after its first 0.334.0 report (14:30:13 UTC), the passphrase → **409, no key**, and `passphrase_keys_refused` was
|
||||
raised (the operator got the mail — CC's probe);
|
||||
- its own key → 200; a wrong key → 401.
|
||||
|
||||
## 2. ep0 prune jobs (R-1165; `07` Plane-2 row)
|
||||
|
||||
- **Automatic:** `felhom-tenantsync` v1.3.0 creates `prune-<id>` (keep-weekly 2, 03:30, max-depth 0) with a household's
|
||||
namespace unless a job already names it; it removes the job with the household. Its `usage` answer lists namespaces no
|
||||
job covers.
|
||||
- The hub alarms on that list (`pbsdr_namespace_without_prune_job`). An unread list never reads as "all covered".
|
||||
- Tests: 12 script cases (`scripts/test_tenantsync_prune.py`, run on every push), the checker and the client; each
|
||||
red-proved.
|
||||
- **Dry run before the question** (PBS's own, `P2/ep0-dry-run-tester.txt`; 32 snapshots before and after):
|
||||
- tester-1: keep 2026-10-11, 2026-10-18; remove 2026-10-04.
|
||||
- Tester-2: keep 2026-10-04.
|
||||
- **Answer:** „Yes, add both". **After** (13:35 UTC, `P2/ep0-jobs-after.txt`): `prune-tester-1` and `prune-Tester-2`
|
||||
(keep-weekly 2, 03:30) beside the demo-hp, demo-felhom and operator jobs.
|
||||
- **Script on ep0** (14:07 UTC, the same yes): the installed v1.2.0 matched the repo byte for byte; v1.3.0 is installed
|
||||
(hash = the repo's `36412c6a…`), the old copy kept as `.bak-1.2.0`. Live `usage` → `unpruned: []`; `fingerprint` still
|
||||
answers.
|
||||
|
||||
## 3. demo-hp's address (R-1166)
|
||||
|
||||
- A watch every 10 s for 30 minutes (10:59–11:29 UTC): **180 samples, 0 with another MAC**, SSH open throughout.
|
||||
- The host's own MAC (`cat /sys/class/net/vmbr0/address`): **`7c:d3:0a:77:d9:76`** — the same as your router entry.
|
||||
- Note: your DHCP pool is .120–.160, so .104 was never in it. The other device (`b8:06:0d:60:45:58`) most likely had .104
|
||||
set by hand. If it comes back, look there.
|
||||
|
||||
## 4. Hub deploy and the controller release
|
||||
|
||||
- **Hub 0.161.0**
|
||||
- Image digest: `sha256:d56c96a59c86ce2d006938d0980803a12949bcbf027cea2834cabd64118838b8` (clean clone at `b6a911c3`).
|
||||
- Manifest `97f27a75`; only Deployment/hub was synced. Result: Synced/Healthy; the running digest is the built one;
|
||||
0 `[ERROR]` lines; the ep0 read at 14:13 UTC was OK, with no alarm (every namespace has a job).
|
||||
- Checks: the floor was read back; host and controller reports from demo-hp, Tester 1 and demo-felhom arrived after
|
||||
the restart.
|
||||
- **Controller 0.334.0**
|
||||
- Image digest: `sha256:deeabe59d932cd60f8939a37bccf94f145555101fc7be64b6c5cc15f3557031a` (from `546eef3`); rc.1 ran
|
||||
on demo-hp only.
|
||||
- Per-box floors at 14:16:11 UTC. Tester 1 and demo-felhom moved at once; demo-hp moved at 14:30 (from rc.1).
|
||||
- Box walk (`P5/`): 0 unhealthy, 0 refusals, catalog in sync; https-only 28 OK, 0 failures; „Fájlok ↗" with no second
|
||||
login on demo-hp and Tester 1.
|
||||
- **Global floor 0.334.0 / MinAgent 0.158.1 at 14:31:44 UTC.** The per-box floors were cleared.
|
||||
|
||||
## 5. Versions, commits, rows
|
||||
|
||||
- **Hub 0.161.0** (`b6a911c3`, manifest `97f27a75`); **controller 0.334.0** (`546eef3`); **tenantsync v1.3.0** (in
|
||||
`b6a911c3`, on ep0). No agent change. No i18n fixture changed (no page changed).
|
||||
- **Rows: 163 before, 161 after; closed 3** (R-1159, R-1165, R-1166), **opened 1** (R-1167).
|
||||
- **System poster:** facts file only (`felhom-system-poster.facts.md`: new households get their prune job); the drawing's
|
||||
text („one per week, 2 weeks") is still true.
|
||||
- **Teardown:**
|
||||
- Machine: rc.1 only on demo-hp 9201, now on the release; the probe scripts removed from demo-hp.
|
||||
- Host: none.
|
||||
- Hub: per-box floors cleared; one `passphrase_keys_refused` event on demo-hp (CC's probe).
|
||||
- ep0: two jobs added and the script replaced, both on the operator's yes; nothing else.
|
||||
|
||||
## Observations
|
||||
|
||||
1. The installer prints „unexpected hub status 409" instead of the hub's sentence when a re-run meets a live box. FILED: R-1167.
|
||||
2. The brief's „keys only while no host exists" would have broken every install and every live box's config refresh. NOT-A-FINDING: measured before building and put to the operator, who chose the two-step shape (D19); recorded in `05` §8.
|
||||
3. A reinstall over a live box now gets two operator mails (R-1154's reinstall warning and the refusal). NOT-A-FINDING: both are true and give the same instruction; the 1-hour cooldown limits repeats.
|
||||
4. An rc build (`0.334.0-rc.N`) counts as older than 0.334.0, so the lock stays open on an rc. NOT-A-FINDING: by design (an rc runs on demo-hp only, and the floor moves it to the release); proven on rc.1 (200) and the release (409).
|
||||
5. demo-hp's .104 lies outside the router's DHCP pool, yet another MAC held it this morning. NOT-A-FINDING: the router reservation fixed it (30 minutes clean); the other device's source is noted in R-1166's closing text in case it returns.
|
||||
@@ -0,0 +1,153 @@
|
||||
# REPORT — evening → night 2026-10-18
|
||||
|
||||
## How I read the brief
|
||||
|
||||
- **Part 5, R-1135:** the brief says the mail should use the public Gitea URL — the register row says the same, and that
|
||||
is what was done (the CI helper first read the brief's wording the other way round and followed the row).
|
||||
- **R-1052** was already closed (2026-10-15); Part 1 proves R-1148 and re-proves R-1052's path with tonight's golden.
|
||||
- **Part 6, Immich:** 9202's Immich lives on `/mnt/sizing-bench`, a hand-made bench folder outside `/mnt/felhom-drives/`,
|
||||
so the allow-list refuses its definition (see Observations); the restore half of the measurement ran on Nextcloud.
|
||||
- **The operator kept Tester4** (asked at 21:20): not deleted; its old host must go before its next install (R-1154).
|
||||
|
||||
## In plain words
|
||||
|
||||
- **The morning list:** vouch golden 0.331.0; say yes or no to the agent release with three fixes; two decisions (one
|
||||
sign-in, local backup retention).
|
||||
- **A brand-new box from nothing works end to end** with installer 1.34.0: emergency password saved in the hub first,
|
||||
then set, then the web login took it; the approved golden and agent; the box moved itself to the floor.
|
||||
- **OnlyOffice now really works** from FileBrowser and from Nextcloud. Before tonight FileBrowser could not open a
|
||||
document and Nextcloud lost every save while saying „saved". Fixed in controller 0.331.0, now the floor for every box.
|
||||
- **One sign-in for the file manager can be built safely**, but only with two safeguards — measured, design note written.
|
||||
- **Three agent fixes are written and tested, not released.**
|
||||
- **CI:** the failure mail's link opens from outside; the 5-minute CI timeout never worked (filed); a rerun hides the
|
||||
first try's log, and the check now looks for it.
|
||||
- **Redis for Immich and Nextcloud has no disk volume any more**, after a test; the test found that removing an app left
|
||||
an old volume behind — fixed too.
|
||||
- **Golden 0.331.0 is baked** (not vouched).
|
||||
- **The old-restic check:** §9.
|
||||
|
||||
## 1. The morning list
|
||||
|
||||
1. **Vouch golden 0.331.0** — sha256 `3881de62b0986ccbe5e55580e3f1bb9765d65b8d898f67c6339bac105244f84f` (registry round
|
||||
trip = the bake), agent 0.159.2 unchanged, **Min agent 0.158.1**.
|
||||
2. **Agent release (R-978, R-973, R-1093) — may I?** If yes: release, prove on demo-hp, ask for the vouch.
|
||||
3. **Decide D16** (R-1100: build one sign-in with the two safeguards?) and **D17** (R-1149: local whole-box backups
|
||||
„last 3" or „one per day, 3 days"?). STATUS has both.
|
||||
4. Before Tester4's next install: delete its old host `Tester4-31fa0e` (Hosts page).
|
||||
|
||||
## 2. Fresh install from nothing (R-1148, R-1052) — `audits/newbox-2026-10-18/`
|
||||
|
||||
| UTC | Step | Result |
|
||||
|---|---|---|
|
||||
| 17:42:21 | Tester4 created by the operator | bind mail sent |
|
||||
| 17:43:15 | VM 9302 `newbox-r1148` on demo-hp, ISO 1.29.0 (sha = published) | 8 GB / 4 cores / 120 GB on `/mnt/hdd_1` root |
|
||||
| 17:50:19 | Install pressed (graphical, Hungarian keyboard kept) | installed, rebooted, pairing code on the console |
|
||||
| 18:02:58 | bind: mailed link + pairing code + owner phrase (operator's 0600 file, shredded) | „Sikeres összekötés" |
|
||||
| 18:03:43 | step 4b: vault → chpasswd → PVE login | `[OK] … vaulted, set, and the PVE login takes it`; hub `set_at` same second |
|
||||
| 18:03:44 | step 5: the hub manifest | agent **0.159.2**, golden **0.329.0** — the vouched pair, both sha-verified |
|
||||
| 18:05:37 → 18:05:51 | first controller reports | 0.329.0, then **0.330.0** (`initiated_by: auto-floor` at 18:05:38) — 2 min 47 s after the bind |
|
||||
| 18:17:14 | CC's PVE login from DooPlex | vaulted value **200**, installer value **401** |
|
||||
| 18:18:32 → 18:20:05 | claim (setup code mail), BentoPDF | claimed; BentoPDF healthy |
|
||||
| 18:24:15 | household „turn off" | hub `deliberate_off_at` set; VM stopped |
|
||||
| — | teardown | name read twice, VM 9302 destroyed, disk gone; other VMs untouched; Tester4 kept (operator) |
|
||||
|
||||
## 3. OnlyOffice (R-1018) — `audits/onlyoffice-2026-10-18/`
|
||||
|
||||
On 0.330.0: FileBrowser → the editor loaded, the document never did (OnlyOffice fetched `https://files.<domain>` → 403);
|
||||
Nextcloud → edit and „All changes saved", but the file kept the old text (`http://onlyoffice:80cache/…`). Switching the
|
||||
link on/off left files.<domain> 502 for 4 min 8 s / 3 min 38 s. **Fixed (controller 0.331.0):** FileBrowser
|
||||
`server.internalUrl`, Nextcloud's slash, the gate after a FileBrowser restart. On rc.1: both open, edit, save, read back
|
||||
(„… RC1-FB-OK", „… RC1-NC-OK"); the stop revokes both (no editor; Nextcloud 404); the stop's outage one 2-s sample.
|
||||
|
||||
## 4. One sign-in spike (R-1100 → READY) — `audits/proxyauth-spike-2026-10-18/DESIGN-NOTE.md`
|
||||
|
||||
Traefik forwardAuth strips a client's header (any case, twice) and its own value wins — measured. But an app container
|
||||
reaching FileBrowser directly logs in with a forged header (200), and the dashboard cookie never reaches `files.`. Verdict:
|
||||
build only with a files-only ticket cookie and a per-box secret header name. Everything on 9202 removed (users, routes,
|
||||
stand-in container, config).
|
||||
|
||||
## 5. Agent fixes (committed, NOT released) — felhom-agent `REPORT.md`
|
||||
|
||||
R-1093 `f7b7308` + `0963bae` (and a fail-open hole closed after a background review), R-978 `8561dc8`, R-973 `f17e197`;
|
||||
tests and red-proofs for each; CI job 2062 success. R-1093 live: a loop disk on demo-hp showed the probe's condition holds.
|
||||
|
||||
## 6. CI (R-1135, R-1136, R-1137 closed)
|
||||
|
||||
Link public and by run id (forced mail 18:46:01Z, job 2067, one mail). The runner's real limit is 3 h; the job-level
|
||||
5-minute timeout is not enforced (R-265 corrected; R-1152 for the step-level one). A rerun replaces the job's attempt; the
|
||||
pull-check recipe in `CLAUDE.md` now reads `actions/tasks` (instruction-file edit by the CI helper: felhom.eu `CLAUDE.md`
|
||||
CI recipe, item 3 added — a proven fact, stricter; `b9b56c47`).
|
||||
|
||||
## 7. Redis volumes (R-1117 closed) — `audits/redis-volumes-2026-10-18/`
|
||||
|
||||
(a) Both apps healthy on the new shape; Immich's metadata job ran (0 → 11, 0 failed); Nextcloud `cron.php` rc 0, WebDAV
|
||||
200. (b) The dump skips the dropped volume (the old tar gone from the unit); a second-drive restore from a copy made WITH
|
||||
the volume: 3/3 volumes, 1/1 database, healthy — it brings back that copy's compose (by design). (c) The old volume stays
|
||||
attached under the tmpfs; **a removal left it behind** — fixed (controller 0.331.0, red-proofed). Templates changed
|
||||
(catalog `30eea97`); Nextcloud on 9202 took it through the sync + restart (tmpfs, `save ""`).
|
||||
|
||||
## 8. Controller release and golden — `audits/evening-2026-10-18/P7/`, `tests/golden-0.331.0-2026-10-18/`
|
||||
|
||||
0.331.0 (digest `1b11d25a…`): per-box floors 20:44:00 → all three on 0.331.0 within 20 s; box walk green (0 refusals,
|
||||
0 unhealthy, FileBrowser re-published in 5–6 s, public pages 200 after one 20-s blip each on two names, https-only 28 OK);
|
||||
**global floor 0.331.0 / 0.158.1 at 20:48:56 UTC**, read back with the WAL; per-box floors cleared. Golden 0.331.0 baked
|
||||
after it, round trip = bake sha, token checks 0 with a working control, teardown clean. No controller push after the bake.
|
||||
|
||||
## 9. The old-restic check (Part 8)
|
||||
|
||||
**Done 2026-10-19 by the watcher** (filled in by the day session, `audits/evening-2026-10-18/P8/old-restic-check.txt`):
|
||||
demo-hp's clean-up window 12 ran at 02:17:53–02:18:24 UTC after the night run — „63 of 193 planned snapshot(s) removed,
|
||||
324 -> 261" (the hub's row: `pruned`, 324 → 261). Then `rx.sh` (read-only): the new restic (0.19.1) and the OLD one (image
|
||||
0.322.0, restic 0.14.0) both list **261**; both restore `c0b6299c` (made by 0.19.1, 02:17:48) to the same content hash
|
||||
`d808e6537c7bcdb6` (7 files) and `05c3346a` (2026-08-31) to `8df241f4cf889265` (10 files). The old restic still reads and
|
||||
restores demo-hp's store after its first real 0.19.1 prune. (Pre-check at 20:57 UTC: 310 and 310.)
|
||||
|
||||
## 10. Versions, commits, fixtures, rows
|
||||
|
||||
- Controller **0.331.0** — `772ef36`, `8ae9178` (+ `615940d` CI); rc.1 on 9202 only. Agent `main` `f7b7308` `8561dc8`
|
||||
`0963bae` `f17e197` `79ba884` `1693da4` — unreleased. Catalog `c97eb1c`, `30eea97`. felhom.eu `1fe9f848`, `b9b56c47`,
|
||||
this commit. Golden **0.331.0** baked. Hub unchanged (0.158.0).
|
||||
- Fixtures: none changed — no page changed.
|
||||
- **Register: 168 before, 168 after; closed 6** (R-1148, R-1018, R-1117, R-1135, R-1136, R-1137), **opened 6**
|
||||
(R-1149, R-1150, R-1151, R-1152, R-1153, R-1154). Updated: R-1100 (READY), R-978, R-973, R-1093 (fixed in agent
|
||||
`main`), R-265 (corrected).
|
||||
|
||||
## Observations
|
||||
|
||||
1. Installer password, golden and agent all proven in one fresh install. — closes R-1148 (not a finding).
|
||||
2. FileBrowser's office preview answered 500 with the link on. FILED: R-1153.
|
||||
3. A reinstall reuses an old host and skips two re-issues; now recurring with the kept Tester4. FILED: R-1154.
|
||||
4. The R-973 cap leaves the local keep-last 3 floodable within a day. FILED: R-1149.
|
||||
5. The cap's refusal reaches the operator with a misleading event text. FILED: R-1150.
|
||||
6. R-978's typed-back confirmation is browser-only. FILED: R-1151.
|
||||
7. The CI job timeout is inert. FILED: R-1152.
|
||||
8. 9202's Immich sits on `/mnt/sizing-bench`, outside `/mnt/felhom-drives/`, so the allow-list refuses its definition every
|
||||
15 minutes since 0.328.0 and a restore of it is refused. NOT-A-FINDING: the check works as designed; the folder is a
|
||||
hand-made bench artifact (R-994), not a shape a real box has.
|
||||
9. A restore brings back the backup's own compose (the old Redis volume with it). NOT-A-FINDING: by design (R-47: the
|
||||
unit's compose becomes the live one); the older shape works.
|
||||
10. My commit `7f7081d1` (day session) carries a „Co-Authored-By" trailer, against both briefs' rule 10. NOT-A-FINDING:
|
||||
my slip; it is pushed and cannot be rewritten without a force-push (forbidden); no later commit has one.
|
||||
11. I started the catalog's FULL gate run on DooPlex, whose third gate starts containers. NOT-A-FINDING: stopped at gate 2
|
||||
(a network check), no container was made (checked), the fast run used instead.
|
||||
12. One OnlyOffice „Connection is lost" on the first rc.1 attempt, after two blind clicks. NOT-A-FINDING: not reproduced;
|
||||
the next attempt saved; no container restarted in between.
|
||||
13. Two public names answered 000 once in the box walk. NOT-A-FINDING: 18 of 18 retries answered 200 in < 0.2 s.
|
||||
14. The R-1100 row named the config key `proxyAuth`; it is `auth.methods.proxy`. NOT-A-FINDING: corrected in the row.
|
||||
|
||||
## Teardown — three layers
|
||||
|
||||
- **Machines:** VM 9302 destroyed (its disk gone). 9202: OnlyOffice and Nextcloud removed through the product (their
|
||||
volumes, data and backup copies), the test documents deleted, the spike's route file, container, FileBrowser users and
|
||||
proxy setting removed, helper files and the Immich scratch password shredded; Immich keeps the new Redis shape by hand
|
||||
(= the template); 9202 on controller 0.331.0. `nextcloud_nextcloud_redis_data` (the orphan the pre-fix
|
||||
removal left) removed by name afterwards; no Nextcloud volume remains. demo-hp host: the loop-disk file removed. Drill VM: CT 9100 purged, VM off,
|
||||
disk `virgin`.
|
||||
- **Hub:** global floor 0.331.0 / 0.158.1, no per-box floor; Tester4 and its host `Tester4-31fa0e` kept (operator); no
|
||||
vouch, no deploy.
|
||||
- **DooPlex:** builds and the golden bake only; the headless browser as a test client; no Docker cleanup.
|
||||
|
||||
Architecture read: `01`, `02`, `03` §8a, `04`, `05`, `07` §6/§8, `09` §3 decisions 162/250; `target-selection.md`,
|
||||
`nightly-rotation.md`, `RUNBOOK-manual-build.md` §4. Documents changed: `03` §8a, `07` §6.1 and §8 row 10,
|
||||
`target-selection.md` (Tester4). System poster: none of the three (no listed fact changed). `unproven.py --summary`:
|
||||
not walked 35 of 55 — unchanged.
|
||||
@@ -0,0 +1,164 @@
|
||||
# REPORT — evening/night brief 2026-10-19 (one stop less per night, no false tunnel error, R-893 measured, hub and agent fixes ready, a scanner on every push, ep0's first weekly clean-up)
|
||||
|
||||
## In plain words
|
||||
|
||||
- **Morning list:** deploy hub 0.162.0? · release agent 0.161.0? · publish installer 1.35.0? · decide D20 (R-893).
|
||||
- **One app stop less per night** (R-921): the controller (0.335.0, the floor) now asks the agent „are you busy?" before
|
||||
it stops the apps. The agent's answer is written but not released, so it starts working with the agent release.
|
||||
- **No false error for a box without a tunnel** (R-1070): the real cause was Docker 29 writing its „not found" message
|
||||
in small letters. Fixed in the agent (not released).
|
||||
- **A failed off-site restore, measured** (R-893): the box puts the newer rows over the backup's older volume; a table
|
||||
only the backup had stays. The app is held stopped. Two options written; pick A (save the app's volumes first).
|
||||
- **Hub fixes built, not installed:** no repeat mail after a restart; the agent list offers only real agent packages.
|
||||
The ArgoCD danger to three hub secrets is already gone (ArgoCD no longer tracks them).
|
||||
- **Installer message written, not published:** a re-run on a live box says „delete the old host first", EN and HU.
|
||||
- **A scanner on every push, in all four repos** (warn-first). First scan: no real secret, no called weakness.
|
||||
- **The scratch box cannot get its agent key without an agent change** (R-1083): the exact gap is written down.
|
||||
- **ep0's first weekly clean-up ran as predicted:** only tester-1's 4 October copy went; every household has a rule.
|
||||
R-1162 closed.
|
||||
|
||||
Decisions CC took: none beyond the brief. The operator answered one question (R-921, 19:2x): „add one field".
|
||||
|
||||
## Part 1 — R-921 (`07` §6.4; `03` local API) — DONE (live once the agent ships)
|
||||
|
||||
- No existing agent answer served the host-wide gate (read: `/backup/due`, `/backup/status`, the 409 shape). Operator:
|
||||
add a read-only field to the existing route, no new route.
|
||||
- Agent `56aabc7`: `host_busy` `{busy, what}` on `GET /backup/status`. Controller 0.335.0: `precheckSkips` → held →
|
||||
no stop, contention for each due tier (15-minute skip, one „BLOCKED" alarm after 3 h); free / older agent / failed
|
||||
read → the old order; the window's own tier in flight → attach as before.
|
||||
- **The race** (free at the read, busy at the start): one short stop, resumed at once — today's behaviour, pinned.
|
||||
- Tests, red-proved: agent `TestBackupStatus_R921_HostBusy` (assignment removed → `free gate: host_busy = <nil>`);
|
||||
controller `TestR921_NoStopWhileTheHostIsBusy` (branch removed → 3 apps stopped, the start refused),
|
||||
`…SkipsStillAlarmAfterTheCeiling`, `…OtherAnswersKeepTheOldOrder` (4 rows), `TestHostBusy_ReadsTheAgentsGate`.
|
||||
- Live: demo-hp's agent answers no `host_busy` (read inside 9201) — the old order, as designed. A live busy state was
|
||||
not made (it needs the new agent on the box). R-921 closed per the brief.
|
||||
|
||||
## Part 2 — R-1070 (`11` §8.2) — DONE in the agent (not released)
|
||||
|
||||
- Where: `felhom-agent/internal/osupdate/leg.go` `HostHealthVerdict` (the tunnel check) and
|
||||
`internal/hub/cloudflared.go` `ClassifyTunnel`. Measured on demo-hp: `docker inspect` of a missing name prints
|
||||
`error: no such object: …` (Docker 29); the match wanted `No such object` → `unknown` → the host rule failed.
|
||||
- Fix: case-insensitive match → new state `absent`; the host rule passes `absent` only when the reading from before
|
||||
the step (Docker answered) also had no cloudflared; `not_running` and `unknown` still fail. Hub 0.162.0 shows
|
||||
`absent` as „none set up". `11` §8.2 had said `unknown` does not fail — corrected (the code always failed it).
|
||||
- Tests, red-proved: `TestHostHealthVerdict_R1070_NoTunnelSetUp` (7 rows), three `TestClassifyTunnel` rows,
|
||||
`TestSystemRows_R1070_AbsentTunnelIsNotAWarning`.
|
||||
|
||||
## Part 3 — R-893: measured and designed, no restore code changed
|
||||
|
||||
Helper on 9202 (fences carried in full). Evidence and the note: `documentation/audits/night-2026-10-19/P3/`.
|
||||
|
||||
- Rallly 4.15.3 + postgres 18; a scratch SFTP off-site on 9202's own disk; one failing SQL line in the prepared copy
|
||||
of the dump, so the replay failed AFTER the volume was replaced.
|
||||
- **What it left:** the newest rows (the undo dump) over the backup's volume; a table only the backup had stayed
|
||||
(37 tables against 36); definition unchanged (same version); app held (`restore_mixed`), a start → 409.
|
||||
- The restore page's sentence is false on both halves for this app → R-1169.
|
||||
- **Options:** A — copy the app's live volumes before the replay, put them back on failure, start the app (one copy's
|
||||
disk for minutes; Rallly 69 MB in ~1 s; a fit check before the stop). B — empty the DB volume before the undo load
|
||||
(no disk; a bad undo copy leaves an empty database; usually still a support call). **Pick: A** (the operator's D8
|
||||
direction). Not measured: the version-change branch (needs a drill-catalog commit).
|
||||
- Teardown: 9202 back on 0.331.0 as found, every throwaway removed (checked again by CC); host: `/tmp` only; hub: nothing.
|
||||
|
||||
## Part 4 — hub 0.162.0, built, NOT deployed (`05`)
|
||||
|
||||
- R-1001: start-up seeding from the DB for `host_stale`/`host_down`; `restore_test_stale` edge from the DB. Tests
|
||||
red-proved (each restart = a new checker over the same DB).
|
||||
- R-1011: exact-file sha (`ErrNoFile`); step bundles leave the list; a vouch of one → `artifact_file_missing`. Registry
|
||||
read: 6 step-bundle versions, every release holds `felhom-agent`. Tests red-proved.
|
||||
- R-953: **no change needed** — read back 18:21 UTC: ArgoCD tracks none of the three Secrets (no tracking mark), so the
|
||||
next deploy does nothing with them. `P4/r953-readback.txt`.
|
||||
- Image `sha256:69285e640a9c5e4ce59b1fbb5cd94623fa47a71798e9d86fa0767ee6c69a6ac8` from a clean clone at `d92a712e`.
|
||||
|
||||
## Part 5 — agent, committed, NOT released (`03`)
|
||||
|
||||
R-1010 (`56aabc7`): on „Failed to find logical volume …_vzdump" one retry with the API's `force=1` (`pvesh usage`
|
||||
lists `--force`; the same token privilege; no sudoers line, no root command); never a second force. Red-proved. Plus
|
||||
R-921's field and R-1070. CHANGELOG „Unreleased". Not exercised live (it needs a half-removed snapshot on `pve/data`).
|
||||
|
||||
## Part 6 — installer 1.35.0, written, NOT published
|
||||
|
||||
R-1167: `die_live_box` in EN and HU at the preflight, host-enroll and the controller.yaml fetch. Harness: a static case
|
||||
and two behaviour cases (the real functions against a local fake hub), red-proved on a mutated copy. **CI red twice**
|
||||
(2106, 2108): the CI runner has no curl; the behaviour cases now SKIP out loud there (`5d0dfeac`).
|
||||
|
||||
## Part 7 — the scanner (R-986), warn-first — DONE
|
||||
|
||||
- Shared `scripts/security_scan_gate.py` in all four runners' fast sets (hook + CI); `.security-accepted` per repo;
|
||||
8 decoys in scratch repos, red-proved; CI fetches the pinned gitleaks (both sha256 checked — seen working in CI 2108).
|
||||
- govulncheck pin v1.1.4 → **v1.8.0** (v1.1.4 cannot read the Go 1.27 code). CI has no Go: govulncheck NOT CHECKED
|
||||
there, out loud; the hook runs it.
|
||||
- **First scan of today's `main`** (`P7/triage.md`): govulncheck 0 called in all three modules. gitleaks history:
|
||||
controller 12, agent 15, catalog 1 (all as on 2026-10-12), felhom.eu 50 (+7: the usenet bench 5, a boot journal 1,
|
||||
a test 1). Today's tree: controller 4, felhom.eu 42, agent 15, catalog 0 — every one triaged: **0 real**; accepted =
|
||||
throwaway demo/bench app values (the bench LXC 9401 stopped, the apps gone from 9202); false = test constants, public
|
||||
keys, an SSH fingerprint, a tunnel ID, hashes.
|
||||
|
||||
## Part 8 — R-1083: the exact gap (no agent change)
|
||||
|
||||
The agent mints a per-guest token only inside `--selftest=provision`, for a guest it brings up from a golden in the same
|
||||
call. No product way exists for an existing guest. Row stays OPEN with the gap and two options.
|
||||
|
||||
## Part 9 — controller 0.335.0, the floor
|
||||
|
||||
rc.1 on demo-hp 18:08 UTC (0 errors) → `0.335.0` `sha256:262831f9…` → per-box floors 18:22 → all three moved by
|
||||
18:37 → box walk green (0 unhealthy, 0 refusals, catalog in sync; https-only 28 OK, 0 FAIL; „Fájlok ↗" one sign-in on
|
||||
demo-hp and Tester 1) → **global floor 0.335.0 / MinAgent 0.158.1 at 18:39:26 UTC**, read back; per-box floors cleared.
|
||||
i18n: no page changed, no fixture changed. Evidence `P9/`.
|
||||
|
||||
## Part 10 — ep0's first weekly clean-up (`07` §plane-2 retention) — DONE, R-1162 closed
|
||||
|
||||
Read only on ep0, 03:45–03:47 UTC. Evidence `P10/01-before.txt` (19:49 UTC), `P10/02-after.txt`.
|
||||
|
||||
| Household | Task 03:30 UTC | Removed | Copies after |
|
||||
|---|---|---|---|
|
||||
| demo-hp | OK, keep-weekly 2 | nothing | 2 (10-08, 10-15) |
|
||||
| demo-felhom | OK, keep-weekly 2 | nothing | 2 (10-06, 10-13) |
|
||||
| Tester-2 | OK, keep-weekly 2 | nothing | 1 (10-04) |
|
||||
| tester-1 | OK, keep-weekly 2 | 2026-10-04 | 2 (10-11, 10-18) |
|
||||
|
||||
- Exactly the dry runs' prediction. The operator job (03:45, keep-daily 14 + keep-weekly 8) removed one hub copy
|
||||
(10-06) — its own rule.
|
||||
- **No „household without a rule" alarm.** Positive check from a second channel: ep0's read-only `usage` op, run
|
||||
directly, answers `unpruned_checked: true, unpruned: []` (tenantsync v1.3.0, sha256 same as the repo file). The hub
|
||||
(0.161.0) logs its ep0 read every 15 minutes (03:31 UTC the last before my read) and no „no prune job" line since its
|
||||
start (2026-10-19 14:10 UTC). The alarm is operator-channel only, so it is not in the events table (none there).
|
||||
|
||||
## Versions, commits
|
||||
|
||||
- Controller 0.335.0 (`15ea002`, REPORT `c4b3b0a`, scanner `a03c25a`). Hub 0.162.0 built (`5f31da04`). Agent
|
||||
`56aabc7`, `84ef581` (not released). Installer 1.35.0 (`d92a712e`, `5d0dfeac`, not published). felhom.eu scanner
|
||||
`15f74288`, docs `f92fa5a8`. Catalog `87f5588`, `57ddb20`.
|
||||
- **System poster:** no listed fact changed tonight — none of the three.
|
||||
- **Instruction/doc corrections:** `11-os-updates.md` §8.2 (the `unknown` tunnel sentence was false); `nodes.md` and
|
||||
`nightly-rotation.md` (9202 is DHCP, .121 now).
|
||||
|
||||
## Rows
|
||||
|
||||
Register **161 before, 160 after.** **Closed 4:** R-921, R-953, R-986, R-1162. **Opened 3:** R-1168 (warn →
|
||||
fail), R-1169 (the false restore sentence), R-1170 (a vanished tunnel raises no alarm). Re-stated: R-1001, R-1011,
|
||||
R-1010, R-1070, R-1167 (READY, waiting on the release/deploy/publish), R-893 (READY, D20), R-1083 (the gap).
|
||||
**Dates moved** (the due-checks gate fired on the new date): R-895 and R-435, 2026-10-20 → 2026-10-21. Both are reads
|
||||
this brief did not name; R-895's first verdict lands at 07:10 hub time, after this session. The reason is in each row.
|
||||
|
||||
## Observations
|
||||
|
||||
1. The restore page's held-mixed sentence is false for the measured app. FILED: R-1169
|
||||
2. A tunnel container that disappears on a box with a tunnel configured raises no alarm. FILED: R-1170
|
||||
3. The scanner warns only. FILED: R-1168
|
||||
4. `11` §8.2 said an `unknown` tunnel does not fail the host rule; the code always failed it. NOT-A-FINDING: corrected in the session (the size rule); it is R-1070's cause.
|
||||
5. govulncheck v1.1.4 (the 2026-10-12 pin) cannot read Go 1.27 code. NOT-A-FINDING: the pin moved to v1.8.0 tonight, recorded in `SCANNERS.md` §5.
|
||||
6. The CI runner has no curl; two new harness cases failed CI 2106 and 2108. NOT-A-FINDING: fixed the same night (`5d0dfeac`), they skip out loud there.
|
||||
7. R-953's tracking marks went after 2026-10-12 with no record of who removed them. NOT-A-FINDING: the danger the row named is gone, nothing to fix.
|
||||
8. 9202's DHCP address moved (.114 → .121) and the walk named the old one. NOT-A-FINDING: fixed (`box_walk.py` reads it; two docs corrected).
|
||||
9. The brief said the installer already prints HU and EN; it printed English only. NOT-A-FINDING: the HU lines are new, named in the scripts CHANGELOG.
|
||||
10. The agent's backup cap (R-973) can also refuse a POST after the stop. NOT-A-FINDING: it answers 409, so the apps resume at once (the BUSY path), and it fires only after a flood of presses that each stopped the apps anyway.
|
||||
11. The manual off-site run dumped and restarted every app, not only the one switched on. NOT-A-FINDING: by design, the coherent app-data pair (`07`).
|
||||
12. On CI's depth-1 clone, gitleaks scans the whole tree (the root commit), ~100 s for felhom.eu. NOT-A-FINDING: a stronger check than the diff, all current hits accepted; recorded in `CONTEXT.md`.
|
||||
13. My 19:49 before-file listed the operator job's keep-weekly but not its keep-daily 14. NOT-A-FINDING: my read printed only some fields; the job is unchanged.
|
||||
|
||||
## Teardown
|
||||
|
||||
- Machine: 9202 back as found (0.331.0, no throwaway left — checked); 9201 on the release 0.335.0; probe scripts removed.
|
||||
- Host: `/tmp` copies only, removed. Nothing on `pve/data`.
|
||||
- Hub: per-box floors cleared; nothing else created.
|
||||
- ep0: read only (task list, task logs, snapshot list, prune-job list, the read-only `usage` op). Nothing changed.
|
||||
@@ -0,0 +1,87 @@
|
||||
# REPORT — the website keeps syncing during an internet outage (2026-10-19)
|
||||
|
||||
## In plain words
|
||||
|
||||
1. **It worked.** The website's copy helpers now read Gitea inside the cluster, with no internet hop, and they wait
|
||||
and try again instead of stopping. The new website pod runs with 0 restarts, and the site and the installer
|
||||
script both answer normally.
|
||||
2. **Note for later:** this ties the website to the DooPlex cluster. If it moves (for example to Hetzner), the
|
||||
address must go back to the public one — register row R-1164.
|
||||
|
||||
## Part 1 — anonymous in-cluster fetch (before any commit)
|
||||
|
||||
Throwaway pod in a scratch namespace (`ccscratch-gitsync`, deleted after), no credentials:
|
||||
`git ls-remote http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git main installer-v1.34.0`
|
||||
|
||||
| ref | in-cluster | public (anonymous) |
|
||||
|---|---|---|
|
||||
| `refs/heads/main` | `5d3cb5c1…` | `5d3cb5c1…` — equal |
|
||||
| `refs/tags/installer-v1.34.0` | `add58eec…` (annotated tag object) | `add58eec…` — equal; peels to `624bcdc3…` |
|
||||
|
||||
Evidence: `audits/website-gitsync-incluster-2026-10-19/gitsync-part1-*.txt`.
|
||||
|
||||
## Part 2 — the change
|
||||
|
||||
Commit **`fd75003e`** (no Co-Authored-By, per the brief):
|
||||
- `manifests/webpage.yaml`: all four git-syncs `--repo=http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git`;
|
||||
the two sidecars `--max-failures=-1` (not the `--one-time` inits); one comment above the website sidecar. Refs and
|
||||
`nginx-config-rev` unchanged.
|
||||
- `scripts/hostinstall_gates.py` gate 6: fails unless there are four `--repo=` values and all are identical. Existing
|
||||
ref checks unchanged.
|
||||
|
||||
Gates: `python3 scripts/repo_gates.py` → `all felhom.eu gates OK`; `python3 scripts/hostinstall_gates.py` → rc 0
|
||||
(`ok: manifest: all four git-syncs read the same repo (…)`).
|
||||
|
||||
## Part 4 — deploy and proof
|
||||
|
||||
**Deploy:** hard refresh showed only `Deployment/felhom-webpage` OutOfSync; synced that resource alone at `fd75003e`
|
||||
(the skill's Deployment-only sync). New pod `felhom-webpage-869ff88bf7-fswbc` ready at 09:42 UTC; the old pod
|
||||
terminated. ArgoCD: Synced, operation Succeeded. (The first `rollout status` returned at once — it ran before the new
|
||||
ReplicaSet existed; re-run after the sync it waited for the old replica and finished.)
|
||||
|
||||
**Scenario A** (`gitsync-scenarioA.txt`): restarts `0,0,0`; both sidecars started with the in-cluster `--repo` and
|
||||
`--max-failures=-1`, then `updated successfully` (`main` → `fd75003e`, `installer-v1.34.0` → `624bcdc3`); both init
|
||||
containers `updated successfully`; `https://felhom.eu/` **200**, `https://felhom.eu/scripts/felhom-host-install.sh`
|
||||
**200**, served `SCRIPT_VERSION="1.34.0"` = manifest `--ref=installer-v1.34.0`.
|
||||
|
||||
**Scenarios B and C** (`gitsync-scenarioBC.txt`): scratch namespace `ccscratch-gitsync-bc` (deleted after),
|
||||
`git-sync:v4.4.0 --repo=http://192.0.2.1/x.git --period=10s`:
|
||||
- **B, with `--max-failures=-1`:** `error syncing repo, will retry` (`failCount:1`), restarts **0**.
|
||||
- **C, without:** `too many failures, aborting`, restarts **1**.
|
||||
- The first read at 95 s showed nothing for either: one fetch to an address that never answers takes **~133 s** to
|
||||
time out. The re-read at 09:48 recorded both results. Instead of 90 s, a later run of this test needs about 150 s.
|
||||
|
||||
**Scenario D — gate red-proof:** one `--repo` changed to the public address → gate 6 `FAIL … the git-sync --repo
|
||||
values differ`, rc 1; restored byte-identical → rc 0.
|
||||
|
||||
## Part 3 — docs changed
|
||||
|
||||
- `documentation/architecture/01-topology-and-trust.md` §7: a `[DESIGN]` paragraph — the website pulls `website/`
|
||||
(main) and `scripts/` (installer tag) from Gitea in-cluster over plain HTTP, sidecars retry forever, tied to the
|
||||
DooPlex cluster.
|
||||
- `.claude/rules/manifests.md` „The two git-syncs": the fact that all four containers read Gitea in-cluster, must name
|
||||
one repo (gate 6), and the sidecars carry `--max-failures=-1`.
|
||||
- `scripts/CHANGELOG.md`: one entry.
|
||||
|
||||
## Register
|
||||
|
||||
Rows before **166**, after **167**. Closed **R-1163** (finding + fix, with evidence). Opened **R-1164** (P4, Hub &
|
||||
operator: move the website to Hetzner; then the git-syncs must use the public address again).
|
||||
|
||||
## Observations
|
||||
|
||||
- The 95-second wait in the brief is too short for an address that never answers (~133 s TCP timeout per fetch) —
|
||||
NOT-A-FINDING: recorded here as the correct wait for this test; nothing in the product depends on it.
|
||||
- The skill says a whole-app sync would push git's view over de-gitted Secrets; today only the webpage Deployment was
|
||||
OutOfSync — NOT-A-FINDING: the Deployment-only sync was used anyway, as the skill says.
|
||||
|
||||
- Around 09:50–09:55 UTC, DooPlex itself briefly could not resolve names (`Temporary failure in name resolution`;
|
||||
`curl https://felhom.eu/` → `000`). By 09:56 DNS, the internet and the site (200) were back. The website pod kept
|
||||
restarts `0,0,0` and synced `e2c35007` in-cluster at 09:54 with no error. NOT-A-FINDING: the same class of short
|
||||
outage that triggered this task; it is not proof that the fix worked (whether the blip reached the cluster is unknown).
|
||||
- CI: `fd75003e` job 2092 success, `e2c35007` job 2094 success.
|
||||
|
||||
## Teardown
|
||||
|
||||
Scratch namespaces `ccscratch-gitsync` and `ccscratch-gitsync-bc` deleted (`kubectl get ns | grep -c ccscratch` → 0).
|
||||
No hub, controller, agent, customer box, demo box or ep0 touched.
|
||||
@@ -271,6 +271,13 @@ app. Controller down → the gated app answers an error, never the app. Measured
|
||||
tunnel. Since controller v0.286.0 it sits ALONE on the `felhom-tunnel` network at the fixed address `172.16.253.2`
|
||||
(traefik at `.3`), so traefik can believe forwarded headers from it and from nothing else (§5). Geo-restriction WAF
|
||||
is **hub-enforced** (the hub holds the CF API token; the controller only reports geo desired-state).
|
||||
- **[DESIGN] The website's content comes from Gitea IN-CLUSTER** (2026-10-19). The `felhom-webpage` pod pulls
|
||||
`website/` (from `main`) and `scripts/` (from the `installer-v…` tag, R-110) with four git-syncs that read
|
||||
`http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git` — plain HTTP, safe only because it never
|
||||
leaves the cluster (the hub's own Gitea client does the same). No internet hop: the pod can start, and keep its copy
|
||||
fresh, during an internet outage. The two sidecars run with `--max-failures=-1`, so an unreachable Gitea is retried,
|
||||
not a restart. **This ties the website to the DooPlex cluster:** moved elsewhere, it must read the public HTTPS
|
||||
address again. `scripts/hostinstall_gates.py` gate 6 keeps the four `--repo` values identical.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -539,8 +539,28 @@ state (both demo boxes) the panel says „Ezt a jelszót te állítottad be a F
|
||||
the press. The box uses FileBrowser's own CLI (`set -u admin,<pw>`) with the server STOPPED (the CLI opens the
|
||||
database with no lock timeout), verifies new=200 and admin/admin refused, stores the value encrypted and records
|
||||
`generated`; a failed verify leaves `operator`. Not the `FILEBROWSER_ADMIN_PASSWORD` env: it re-applies on every
|
||||
start. Spike: `audits/logins-folders-2026-10-16/P1b`. FileBrowser still asks for its own login the first time; one
|
||||
sign-in for both is R-1100.
|
||||
start. Spike: `audits/logins-folders-2026-10-16/P1b`. Since v0.332.0 the folder links open FileBrowser without its own
|
||||
login (the next paragraph); `admin`'s password is the fallback and the operator's way in.
|
||||
|
||||
**[DESIGN] One sign-in for the file manager (v0.332.0, R-1100, operator decision D16 = yes 2026-10-19; spike
|
||||
`audits/proxyauth-spike-2026-10-18/DESIGN-NOTE.md`).** Every dashboard link to a FileBrowser folder goes to
|
||||
`/__files/open?to=<path>` on the dashboard host, which — for a signed-in, not-cross-site request — mints a pass (60 s,
|
||||
one use, bound to `files.<domain>` and to the session) and sends the browser to `files.<domain>/_felhom/sso`. traefik
|
||||
asks the controller before every FileBrowser request (forwardAuth `felhom-files-sso` → `/__felhom_gate/files`, the setup
|
||||
gate's handshake): the pass becomes the host-only `felhom_files` cookie (a value derived from the session's fingerprint
|
||||
for that host — the R-976 shape); a valid cookie → 200 with the box's **secret header** naming FileBrowser user
|
||||
`felhom-household`; no cookie → 200 without it (FileBrowser's own password); a stale cookie (sign-out, password change)
|
||||
→ both cookies cleared, back to the dashboard. **Why a secret header name:** FileBrowser trusts the header blindly and
|
||||
every app container reaches `filebrowser:80` directly — measured: a forged fixed-name header from an app logged in. The
|
||||
name (`X-Felhom-User-<32 hex>`, per box, 0600 in the data dir) is written only into FileBrowser's config and traefik's
|
||||
route (`authResponseHeaders`, which strips a client's copy); never a log, a page or a report. **Why not a parent-domain
|
||||
cookie:** every app subdomain is third-party code and would receive the dashboard session. **What stays:** `admin` is a
|
||||
password user (R-513's reveal and reset, the R-1126 gate unchanged); the header user is a separate FileBrowser user whose
|
||||
rights come from the box's userDefaults (change, add, delete, share, download — not admin, not API); FileBrowser's logout
|
||||
goes to the dashboard. Residual: FileBrowser's own 2-hour token for the header user is cleared with the files cookie,
|
||||
but a browser that lost only the files cookie keeps it until it expires. Measured live on all three test boxes
|
||||
(`audits/day-2026-10-19/P2/`, `P4/`): opened at the folder with no login; forged headers refused through traefik and
|
||||
from an app container; a sign-out asks again.
|
||||
|
||||
**[DESIGN] A stored FileBrowser password is checked before it is offered (v0.321.0, R-1101).** In `generated`, the
|
||||
page asks FileBrowser whether the stored value still logs in; only FileBrowser's own 401/403 counts as refused (a
|
||||
@@ -562,8 +582,9 @@ role, else the first declared folder, else (a media app) the first present drive
|
||||
`movies`, the players both, a download client its `media_library.folder`) with NO backup sentence — the bind's class
|
||||
there means „not this app's data", which is not the class-driven „temporary, not backed up". **A folder whose drive
|
||||
is not bound into FileBrowser now is text, never a link** (`drivePresent` = the same gates the FileBrowser sync
|
||||
applies). The button never promises one click: FileBrowser's router brings a cold visitor back to the folder after its
|
||||
login (measured in the headless browser, `audits/logins-folders-2026-10-16/browser/`).
|
||||
applies). Since v0.332.0 the button opens the folder signed in (one sign-in, above); on a box without it, FileBrowser's
|
||||
router brings a cold visitor back to the folder after its login (measured in the headless browser,
|
||||
`audits/logins-folders-2026-10-16/browser/`).
|
||||
|
||||
**[DESIGN] „Cím és kulcsok" is rows, not the install form.** The settings tab does not re-render the
|
||||
install wizard's fields: the address is the „Cím" row, the storage is a „Tárhely: <név>" row, every
|
||||
|
||||
@@ -232,7 +232,11 @@ The controller (in its LXC) reaches the agent (on the host) over the local bridg
|
||||
- `POST /rollback` — roll *this* guest back to a named snapshot (post-deploy failure recovery).
|
||||
- `POST /backup` — request a backup-now of *this* guest (enqueued; non-destructive).
|
||||
- `GET /backup/due` — whether a policy-scheduled backup is due for *this* guest, so the controller can quiesce then call `POST /backup` (the app-consistent path, §8).
|
||||
- `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI.
|
||||
- `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI. **`host_busy`
|
||||
(R-921, agent Unreleased `56aabc7`):** `GET /backup/status` also says whether the host-wide heavy-operation gate is
|
||||
held (`{busy, what}` — another guest's backup, the restore test, an OS step, the stale-lock sweep, a power-off), the
|
||||
state in which `POST /backup` answers 409; absent when the gate is not wired. The controller (≥ 0.335.0) reads it
|
||||
before it stops any app. A read, never a reservation.
|
||||
- **Crash-boot fact (R-856, agent v0.149.0):** `GET /host/crash-guard` — the host crash guard's last-boot record
|
||||
(`present`, `last_boot_at`, `last_boot_unclean`, `tripped`) read from `/var/lib/felhom-crash-guard/state.json`; a missing
|
||||
or garbled file answers 200 `present:false`, an older agent 404 — both read as a normal boot. The controller waits
|
||||
@@ -399,7 +403,10 @@ per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a ba
|
||||
> space on …" — so the hub raises `restore_test_failed`; it is never a pass and never dropped.
|
||||
> - **Leftovers on a timer.** A failed scratch teardown and the stale-lock sweep (R-673) run every 10
|
||||
> minutes, not only at agent start; the sweep holds the one-heavy-operation gate; after 3 failed
|
||||
> teardown tries the operator is told.
|
||||
> teardown tries the operator is told. **A half-removed `vzdump` snapshot (R-1010, agent Unreleased
|
||||
> `56aabc7`):** when the sweep's delete fails ONLY with „Failed to find logical volume …_vzdump" (one disk's
|
||||
> snapshot already gone), it retries once with Proxmox's own `force=1` on the same API call (no sudoers line, no
|
||||
> root command); any other error is returned unchanged, and a failed forced delete is never forced again.
|
||||
> - **A thin pool ≥ 90 % requests an immediate report**; the hub judges a thin pool on the worse of data
|
||||
> and metadata, critical at 90 %, one alarm per pool per 6 hours (`08` §6.2).
|
||||
>
|
||||
@@ -544,6 +551,19 @@ nicety.
|
||||
**loses Felhom's backup guarantees** — no PBS verify / monitoring on storage we can't reach. An
|
||||
honest signup-time tradeoff, not a hidden one.
|
||||
|
||||
**When the new package reaches the hub (R-978; agent 0.160.0, 2026-10-19):** a ceremony started again no
|
||||
longer replaces the hub's package when it starts. The package travels with the pending ceremony and is uploaded when its
|
||||
code is claimed, before the code is handed out (an upload failure is a 502 and the code stays claimable). An abandoned
|
||||
ceremony — TTL, a new start, a restart — leaves the first code opening the hub's package. **[DESIGN] Only after the
|
||||
household typed it back (R-1151; agent 0.160.0 + controller 0.332.0, 2026-10-19):** a controller ≥ 0.332.0 claims with
|
||||
`{"defer_upload": true}`; the agent hands out the code with two word positions it drew, keeps the package (and only those
|
||||
two words, never the code) for 30 minutes, and uploads it only when `POST /escrow/ceremony/confirm` brings the two words
|
||||
back — so a code opened and not written down never replaces the package. Wrong words 422 (still waiting), window gone 410,
|
||||
nothing waiting 404, hub failure 502 (still waiting). A controller that has not updated claims with `{}` and gets the
|
||||
R-978 order. Proven on demo-hp 2026-10-19: ceremony 1 typed back → code 1 opens the hub's package, code 0 is a kept
|
||||
earlier one (422); ceremony 2 shown and abandoned → code 1 still opens it, code 2 opens nothing
|
||||
(`audits/day-2026-10-19/P1/escrow-*.txt`).
|
||||
|
||||
#### Custody across generations — what the hub retains when a blob is superseded (hub v0.60.0; **contract corrected hub v0.93.0, R-198**)
|
||||
|
||||
A ceremony that seals a *different* passphrase does not destroy the blob it replaces: the hub copies
|
||||
|
||||
@@ -90,7 +90,11 @@ Evolves the existing staleness checker (60s **cadence**, a **configured** thresh
|
||||
2026-09-17, **45 m** since operator ruling A on R-549; OK under it, down at 2× = 90 m; today: controller-report recency → `node_stale`/`down`/`recovered`):
|
||||
|
||||
- **Primary = host-report recency → `host_stale` / `host_down`.** The agent heartbeat is the box's
|
||||
liveness signal; a silent agent = the box is gone (the critical alert).
|
||||
liveness signal; a silent agent = the box is gone (the critical alert). **A hub restart does not re-mail a known
|
||||
silence (R-1001, hub 0.162.0 — built 2026-10-19, not yet deployed):** a host whose newest `host_stale`/`host_down`
|
||||
event is newer than its last report is seeded with that state at start, silently; `restore_test_stale` reads its edge
|
||||
from the DB on the first sweep (a signal for the tier inside the read window and newer than its last proof). A failed
|
||||
lookup lets the alarm through.
|
||||
- **Guest up/down comes from the host report's per-guest status** — authoritative, every poll, faster
|
||||
than waiting for a guest report to go stale.
|
||||
- **Guest-report recency = secondary** app-level signal.
|
||||
@@ -134,7 +138,11 @@ walk is green; the golden is baked weekly and vouched by the operator; a new box
|
||||
itself to the floor (R-1052: not yet proven end to end). Setting the global floor today needs the operator page with the
|
||||
operator's password — there is no scoped route for CC (R-1051). **An agent can be vouched mid-week; the golden stays
|
||||
weekly** (hub v0.158.0): the R-120 golden guard judges only a CHANGED golden selection, so a Day-0 save that keeps the
|
||||
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet.
|
||||
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet. **The agent list
|
||||
offers real agent packages only (R-1011, hub 0.162.0, not yet deployed):** the sha lookup asks for exactly the
|
||||
`felhom-agent` file (the golden: `golden.tar.zst`) and never falls back to another file, so a step bundle
|
||||
(`0.156.1-step1`, only the config bundle) is left out of the list, and a vouch of one is refused
|
||||
(`artifact_file_missing`).
|
||||
|
||||
**The controller floor and its agent requirement (hub v0.112.0, R-472).** The managed controller floor
|
||||
rides the report ACK. `store.ResolveManagedFloor` decides per box whether to serve it, from three
|
||||
@@ -222,6 +230,51 @@ Today: `GET /config/{id}` with an `X-Retrieval-Password` (Hungarian passphrase)
|
||||
The existing `configgen` deep-merge + Hungarian-passphrase machinery is the base; it grows the
|
||||
agent-first + key-pinning + restore-mode steps.
|
||||
|
||||
**A reinstall over an existing host record is made loud, never acted on [DESIGN, 2026-10-19 — R-1154, brief
|
||||
2026-10-19; hub v0.159.0].** `POST /host-enroll` is the installer's step 4, run on every pass (a `--resume` too), with
|
||||
the customer's retrieval passphrase; an existing host record is REUSED (same id and key — the reinstalled box takes
|
||||
it over). The clean-slate re-issues (a new claim code mailed to the household, the off-site provider password reset)
|
||||
run only on the MINT path, i.e. after the operator deleted the old host — **the operator's delete is their gate,
|
||||
and stays so.** A reinstall that kept the old record used to skip them in silence. Now the reuse path classifies the
|
||||
old box and re-issues NOTHING: **gone** (no controller wait connection open now, and either `deliberate_off_at`
|
||||
inside its hold or BOTH the host report and the controller report older than `alerting.stale_threshold` or absent —
|
||||
the `hostStatus` threshold and comparison) → the operator-only WARNING `host_reinstall_over_old_host`, mailed to the
|
||||
operator: delete the old host (Hosts page) and run the installer again, or re-issue the claim code and the off-site
|
||||
login from the customer page; **live** → the operator-only `host_reenroll_while_live` — since hub v0.160.0 (R-1156) a
|
||||
WARNING mailed with the same instruction (a reinstall within 45 minutes of the old box's last report reads as live; an
|
||||
installer `--resume` or a repair re-run mails too, and the text says those need nothing).
|
||||
**Why not re-issue automatically** (a security review of the first draft, the same day): the passphrase is a
|
||||
household-held secret typed into the installer, and both re-issues are operator SESSION-ONLY actions — not even an
|
||||
operator token may run them (§17.5; `09` §3 decision 252, R-1055/R-1059: an off-site re-issue „stops its backup for a
|
||||
while"). Re-issuing on the reuse path would give those powers to anyone holding the passphrase whenever the box is
|
||||
switched off or silent. The cost of the loud shape is two operator acts per reinstall that kept its old record. Both
|
||||
event types are hub-origin only. `hub/internal/api/reenroll.go`, pinned by `hub/internal/api/r1154_reenroll_test.go`.
|
||||
|
||||
**What the household's retrieval passphrase can get [DESIGN, 2026-10-19 — R-1159, operator decision D19 „box key
|
||||
first, then lock"; hub v0.161.0 + controller v0.334.0].** The passphrase is the household's install secret: a NEW box
|
||||
needs it (the installer's preflight, download login and enroll; the guest's first boot, `bootstrap.json` → `GET
|
||||
/config`, which is how the box gets its own hub key). Measured the same day, it also gave anyone who held it the box's
|
||||
keys for good, and through them every off-site key action (`audits/day-2026-10-19b/P2/`). **The rule:** the passphrase
|
||||
gives keys only until the box uses its own — `GET /config`, `GET /recovery` and the reuse path of `POST /host-enroll`
|
||||
answer **409 with no key** (and the operator gets `passphrase_keys_refused`) once a host record exists AND a controller
|
||||
report arrived since it was made AND that controller pulls its config with its own key (≥ 0.334.0, `GET /box-config`,
|
||||
`boxConfigMinController`). Open for a first install, the install in progress, a reinstall after the operator deleted
|
||||
the old host, and a box on an older controller (which still refreshes with the passphrase — the lock arms box by box
|
||||
as the floor moves). A reinstall over a live box therefore needs the operator's delete first, which R-1154 already
|
||||
made the rule. **Why this shape and not „before a host exists":** the install itself uses the passphrase AFTER enroll
|
||||
(`felhom-host-install.sh` step_agent_install → `/config`; the agent bakes it into `bootstrap.json`; the guest pulls with
|
||||
it), and so did every live box's config refresh — measured from the code before building, and put to the operator.
|
||||
`hub/internal/api/passphrase_scope.go`; pinned by `r1159_passphrase_scope_test.go`. Proven live on demo-hp
|
||||
(2026-10-19 14:30 UTC): before 0.334.0 → 200; after its first 0.334.0 report → 409, no key, the event raised; the box's
|
||||
own key → 200.
|
||||
|
||||
**[FACT, 2026-10-19 — R-1158, hub v0.160.0] What the MINT path's off-site re-issue still does.** It resets the Storage
|
||||
Box sub-account password and stores the new one — a password only the hub itself uses (`offsitekeys.TargetFor`, the key
|
||||
registrar's login). No box receives it since v0.127.0 (`consume-password` answers 410; `ConsumeOneTimeSecret` has no
|
||||
production caller); the fresh box registers its own key with the controller API key. So it changes nothing a box uses.
|
||||
It is kept because it hands the hub a fresh, working login at each clean reinstall, which repairs a broken stored one;
|
||||
the `offsite_reissued` record now says the password is the hub's and the backups do not change.
|
||||
|
||||
## 9. DR model
|
||||
|
||||
The headline: the **old heavy infra-backup push retires** — not because the hub authors everything
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -724,7 +724,11 @@ Evidence: `audits/os-host-lane-2026-10-04/` (parts A–G).
|
||||
skips the host step.
|
||||
- **The host health rule** (`HostHealthVerdict`, pinned in `internal/osupdate`): `felhom-agent`, `pveproxy`,
|
||||
`pvedaemon`, `pvestatd` and `pve-cluster` are `active`; the customer guest runs; the guest health rule (§8.1)
|
||||
passes; the tunnel is `running` — an `unknown` tunnel does not fail it, `not_running` does. Same 5-minute wait.
|
||||
passes; the tunnel is `running` — `not_running` and `unknown` fail it (CORRECTED 2026-10-19: this line said
|
||||
„an `unknown` tunnel does not fail it"; the code always failed it, which is what Tester4 hit, R-1070). **Since agent
|
||||
Unreleased `56aabc7` (R-1070):** a guest with NO cloudflared container reads `absent` (Docker 29's lower-case
|
||||
„no such object" used to fall through to `unknown`), and `absent` passes when the reading from before the step also
|
||||
listed no cloudflared container (Docker answered) — a box with no tunnel set up. Same 5-minute wait.
|
||||
- **Separate approved sets:** host and guest releases are separate (`os-host-…`, `os-guest-…`), each by the same rule
|
||||
(24 h, 1 night of THAT layer, every ring-0 box). Ring 1 receives `host_release` beside `release`.
|
||||
- **Reboot needed:** reported when PID 1 or `lxc-start` maps a replaced file, with the date of the first scanned
|
||||
|
||||
@@ -87,8 +87,8 @@ Title: **"Where a household's data is copied"**
|
||||
| **Tier 1 — recovery unit** | each app: its settings, database dump, volumes | data drive 1, on the box | nightly at **02:30** (W) | 1 point per app | not encrypted (it never leaves the home) |
|
||||
| **Tier 2 — cross-drive** | a mirror of Tier 1, plus the app's files | data drive 2, on the box (counted only on a different physical disk; in a VM made, not counted) | nightly **03:30** | mirror | not encrypted |
|
||||
| **Tier 3 — off-site, per app** | Tier 1 + the app's must-keep files | **Hetzner Storage Box** (Germany), one folder per household | nightly **04:15** | 7 daily, 4 weekly, 6 monthly | the household's own restic password |
|
||||
| **Whole guest — local** | the guest's system (not the data drives) | the box's own system disk | daily, in the window **04:30–08:30** | up to 3 copies (fewer on a small disk) | not encrypted |
|
||||
| **Whole guest — off-site** | the same | **ep0** (Hetzner, Germany), one namespace per household | weekly, same window | last 2 | the household's own key, escrowed with a recovery code |
|
||||
| **Whole guest — local** | the guest's system (not the data drives) | the box's own system disk | daily, in the window **04:30–08:30** | one copy per day, up to 3 days (fewer on a small disk) | not encrypted |
|
||||
| **Whole guest — off-site** | the same | **ep0** (Hetzner, Germany), one namespace per household | weekly, same window | one per week, 2 weeks | the household's own key, escrowed with a recovery code |
|
||||
|
||||
Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–3 follow at fixed offsets so they cannot run out of order. A missed night is caught up 15 minutes after the box starts."
|
||||
|
||||
@@ -99,7 +99,7 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
|
||||
- one namespace **per household** (whole-guest copies, each household's own key),
|
||||
with a NEUTRAL note under it (not a red gap box): "By design: each household holds the key to its own copies; the operator cannot open them without their recovery code." — *fix (c), 2026-10-09: this is a property of the design, not a gap, and a red box said the opposite of what it means.*
|
||||
- the **`operator`** namespace: two groups — **"hub database"** and **"DooPlex copy"** (code, password manager, secrets, signing keys).
|
||||
Prune jobs on ep0: households keep the last 2; `operator` keeps 14 daily + 8 weekly.
|
||||
Prune jobs on ep0: households keep one copy per week for 2 weeks (`keep-weekly 2`, R-1162, 2026-10-19; every new household's job is made with its namespace, R-1165); `operator` keeps 14 daily + 8 weekly.
|
||||
Mark ep0 with a small shield: "**Protected** — holds the only off-premises copy of real household data."
|
||||
- **Hetzner Storage Box** (grey): the shared box for the households' per-app restic copies (Tier 3), one sub-account per household, add-only keys.
|
||||
- **Mail service** (grey, "Resend"): every mail the hub sends — to the operator and to households.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,24 @@
|
||||
loop=/dev/loop2
|
||||
--- lsblk p1 (the bug input)
|
||||
{"blockdevices":[{"name":"loop2p1","fstype":null,"pttype":"gpt","mountpoint":null}]}
|
||||
--- blkid -p p1 (empty part)
|
||||
DEVNAME=/dev/loop2p1
|
||||
PART_ENTRY_SCHEME=gpt
|
||||
PART_ENTRY_UUID=2b13a20b-f5a4-4906-9e30-985c170a50ad
|
||||
PART_ENTRY_TYPE=0fc63daf-8483-4772-8e79-3d69d8477de4
|
||||
PART_ENTRY_NUMBER=1
|
||||
PART_ENTRY_OFFSET=2048
|
||||
PART_ENTRY_SIZE=1048576
|
||||
PART_ENTRY_DISK=7:2
|
||||
rc=0
|
||||
--- blkid -p p2 (ext4)
|
||||
TYPE=ext4
|
||||
PART_ENTRY_SCHEME=gpt
|
||||
PART_ENTRY_NUMBER=2
|
||||
rc=0
|
||||
Re-reading the partition table failed.: Invalid argument
|
||||
--- blkid -p p1 (a dos table inside)
|
||||
PTTYPE=dos
|
||||
PART_ENTRY_SCHEME=gpt
|
||||
rc=0
|
||||
ls: cannot access '/root/r1093-scratch.img': No such file or directory
|
||||
@@ -0,0 +1,8 @@
|
||||
tester-1-d70be4 agent_update:
|
||||
uploaded signed op to the hub jobs queue
|
||||
tester-1-d70be4 agent_config_update:
|
||||
uploaded signed op to the hub jobs queue
|
||||
demo-felhom-8363b5 agent_update:
|
||||
uploaded signed op to the hub jobs queue
|
||||
demo-felhom-8363b5 agent_config_update:
|
||||
uploaded signed op to the hub jobs queue
|
||||
@@ -0,0 +1,11 @@
|
||||
tester-1: felhom-agent 0.159.2 | demo-felhom: felhom-agent 0.159.2 | 12:25:17
|
||||
Oct 18 14:23:46 felhom felhom-agent[3912599]: time=2026-10-18T14:23:46.774+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=c9b02a094671155e op=agent_update
|
||||
Oct 18 14:23:48 felhom felhom-agent[3912599]: time=2026-10-18T14:23:48.727+02:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=75 total=75 degraded=""
|
||||
Oct 18 14:23:48 felhom felhom-agent[3912599]: time=2026-10-18T14:23:48.728+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=fa2a707792f561ed op=agent_config_update
|
||||
Oct 18 14:24:52 felhom felhom-agent[1371657]: time=2026-10-18T14:24:52.767+02:00 level=WARN msg="selfupdate: update committed" version=0.159.2 wrapper=""
|
||||
Oct 18 14:25:00 demo-felhom felhom-agent[2419165]: time=2026-10-18T14:25:00.552+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=9d4397117eeed3b9 op=agent_update
|
||||
Oct 18 14:25:01 demo-felhom felhom-agent[2419165]: time=2026-10-18T14:25:01.930+02:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=75 total=75 degraded=""
|
||||
Oct 18 14:25:01 demo-felhom felhom-agent[2419165]: time=2026-10-18T14:25:01.930+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=63f8321055a65cb8 op=agent_config_update
|
||||
|
||||
[exited with code 0]
|
||||
Oct 18 14:26:05 demo-felhom felhom-agent[3800522]: time=2026-10-18T14:26:05.216+02:00 level=WARN msg="selfupdate: update committed" version=0.159.2 wrapper=""
|
||||
@@ -0,0 +1,23 @@
|
||||
Oct 18 13:12:26 demo-hp felhom-agent[3566597]: time=2026-10-18T13:12:26.461+02:00 level=INFO msg="backup: space preflight passed" vmid=9201 target=local last_archive_bytes=6916890749 need_bytes=9719855260 avail_bytes=12123615232
|
||||
Oct 18 13:12:27 demo-hp felhom-agent[3566597]: time=2026-10-18T13:12:27.526+02:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=local job=backup-9201-1792321945880394998
|
||||
Oct 18 13:18:22 demo-hp felhom-agent[3566597]: time=2026-10-18T13:18:22.853+02:00 level=INFO msg="janitor: stale-lock sweep deferred — a heavy operation is in flight" busy=backup:local
|
||||
Oct 18 13:19:59 demo-hp felhom-agent[3566597]: time=2026-10-18T13:19:59.260+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_10_18-13_12_26.tar.zst size_bytes=6938307162 uncovered_volumes=2
|
||||
Oct 18 13:19:59 demo-hp felhom-agent[3566597]: time=2026-10-18T13:19:59.260+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1792321945880394998 archive=local:backup/vzdump-lxc-9201-2026_10_18-13_12_26.tar.zst
|
||||
Oct 18 13:19:59 demo-hp felhom-agent[3566597]: time=2026-10-18T13:19:59.266+02:00 level=INFO msg="local-api: no OS leg after a manual backup — it follows the night's own backup (R-899)" vmid=9201 job=backup-9201-1792321945880394998
|
||||
== sudo felhom-thin-check lines since the trigger:
|
||||
0
|
||||
== sudo lines since the trigger (any helper):
|
||||
26 COMMAND=/usr/bin/chown
|
||||
36 COMMAND=/usr/bin/lsblk
|
||||
98 COMMAND=/usr/bin/lxc-info
|
||||
78 COMMAND=/usr/bin/mkdir
|
||||
8 COMMAND=/usr/bin/wg
|
||||
36 COMMAND=/usr/sbin/blkid
|
||||
75 COMMAND=/usr/sbin/lvs
|
||||
16 COMMAND=/usr/sbin/nft
|
||||
57 COMMAND=/usr/sbin/pct
|
||||
150 COMMAND=/usr/sbin/smartctl
|
||||
|
||||
[exited with code 0]
|
||||
== control: the night's backup on agent 0.159.1 (2026-10-18 04:33 local) ran the check:
|
||||
Oct 18 04:33:15 demo-hp sudo[1664888]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-thin-check pve data
|
||||
@@ -0,0 +1,3 @@
|
||||
== trigger 2026-10-18T11:11:36Z (local 2026-10-18 13:11:36)
|
||||
200
|
||||
{"data": {"started": true}, "ok": true}
|
||||
@@ -0,0 +1,5 @@
|
||||
felhom-agent 0.159.1
|
||||
Oct 18 04:43:52 demo-hp felhom-agent[3320002]: time=2026-10-18T04:43:52.498+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
|
||||
Oct 18 04:43:53 demo-hp felhom-agent[3320002]: time=2026-10-18T04:43:53.613+02:00 level=INFO msg="osupdate: DONE" run=20261018T024228Z layer=pve vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=1 not_covered=0 restart_needed=11 reboot_needed=false wrapper_seconds=16.3
|
||||
Oct 18 05:23:06 demo-hp felhom-agent[3320002]: time=2026-10-18T05:23:06.521+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-10-15T20:38:47Z) is already proven"
|
||||
Oct 18 11:23:06 demo-hp felhom-agent[3320002]: time=2026-10-18T11:23:06.557+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-10-15T20:38:47Z) is already proven"
|
||||
@@ -0,0 +1,4 @@
|
||||
wrote envelope to /dev/null
|
||||
uploaded signed op to the hub jobs queue
|
||||
wrote envelope to /dev/null
|
||||
uploaded signed op to the hub jobs queue
|
||||
@@ -0,0 +1,8 @@
|
||||
felhom-agent 0.159.2 11:08:30
|
||||
Oct 18 13:08:17 demo-hp felhom-agent[3320002]: time=2026-10-18T13:08:17.765+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=763e0b8971dff83c op=agent_update
|
||||
Oct 18 13:08:18 demo-hp felhom-os-apply[3566414]: os-apply: BUNDLE DONE agent=0.159.2 written=0 same=30 self-check=ok signers-created=False
|
||||
Oct 18 13:08:18 demo-hp felhom-agent[3320002]: time=2026-10-18T13:08:18.862+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.159.2 written=0 same=30 self-check=ok signers-created=False"
|
||||
Oct 18 13:08:19 demo-hp felhom-agent[3320002]: time=2026-10-18T13:08:19.883+02:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=75 total=75 degraded=""
|
||||
Oct 18 13:08:19 demo-hp felhom-agent[3320002]: time=2026-10-18T13:08:19.884+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=268dfe26d949094f op=agent_config_update
|
||||
|
||||
[exited with code 0]
|
||||
@@ -0,0 +1,7 @@
|
||||
demo-hp-bb76ea last report 2026-10-18 12:38:33 agent 0.159.2 health None
|
||||
tester-1-d70be4 last report 2026-10-18 12:39:04 agent 0.159.2 health None
|
||||
demo-felhom-8363b5 last report 2026-10-18 12:25:17 agent 0.159.2 health None
|
||||
report keys: ['addresses', 'agent_sha256', 'agent_version', 'audit_tail', 'backups', 'capabilities', 'cloudflared', 'controller_supervisor', 'dr_recipe', 'foreign_key_archives', 'guest_disk_trim', 'guest_net', 'guests', 'host', 'host_id', 'leaf_fingerprint', 'mgmt_plane', 'oob', 'pbs_dr', 'pbs_snapshots', 'reported_at', 'restore_tests', 'storage_targets', 'system', 'wireguard']
|
||||
demo-hp-bb76ea 2026-10-18 12:38:33 agent_version 0.159.2 degraded None cap_ok/total None problems None
|
||||
tester-1-d70be4 2026-10-18 12:39:04 agent_version 0.159.2 degraded None cap_ok/total None problems None
|
||||
demo-felhom-8363b5 2026-10-18 12:25:17 agent_version 0.159.2 degraded None cap_ok/total None problems None
|
||||
@@ -0,0 +1,2 @@
|
||||
felhom-agent sha256 4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9
|
||||
felhom-config-bundle.json sha256 477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9
|
||||
@@ -0,0 +1,102 @@
|
||||
[release-agent] building 0.159.2 …
|
||||
[release-agent] built ok: sha256 4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9
|
||||
[release-agent] config bundle built: sha256 477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9
|
||||
[release-agent] tagging v0.159.2 at 65c1994 …
|
||||
[release-agent] publishing …
|
||||
[publish-agent] publishing /tmp/felhom-agent-bMXfqp (16007752 bytes, sha256 4d98868e922d6c10…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/0.159.2/felhom-agent
|
||||
[publish-agent] pre-delete existing artifact: HTTP 404 (404/204 expected)
|
||||
[publish-agent] upload OK (HTTP 201)
|
||||
[publish-agent] round-trip GET verified (sha256 matches)
|
||||
[publish-agent] DONE. Record in the hub operator UI (Configs → Day-0 artifacts): agent 0.159.2 / 4d98868e922d6c10665a6c0da9323e862AGENT_VERSION=0.159.2
|
||||
AGENT_SHA256=4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9
|
||||
pre-push [felhom-agent]: running scripts/agent_gates.py --fast ...
|
||||
agent_gates — 4 gate(s) [--fast]
|
||||
--fast SKIPPED (deliberate periodic runs, never in a hook): published, gate-decoys
|
||||
|
||||
==============================================================================
|
||||
== gate: reuse-refs (reuse_refs_check.py /mnt/5_hdd/felhom.eu/git/felhom-agent)
|
||||
==============================================================================
|
||||
note [felhom-agent] line 189: localapi/debuglogs_test.go (resolved by suffix → internal/localapi/debuglogs_test.go)
|
||||
note [felhom-agent] line 205: hub/internal/store/dr_recipe.go (cross-repo → felhom.eu/hub/internal/store/dr_recipe.go)
|
||||
OK [felhom-agent]: 117 cited paths — exact 115, suffix 1, ambiguous 0, cross-repo 1, FAILED 0 (siblings searched: app-catalog-felhom.eu, cc-secrets, felhom-controller, felhom.eu, felhom.eu-media, felhom.eu-sizing-push, felhom.eu-wt-sizing)
|
||||
|
||||
==============================================================================
|
||||
== gate: instructions (instructions_gate.py /mnt/5_hdd/felhom.eu/git/felhom-agent)
|
||||
==============================================================================
|
||||
instructions_gate: /mnt/5_hdd/felhom.eu/git/felhom-agent
|
||||
CLAUDE.md effective lines : 108 (ceiling 200)
|
||||
version literals : 0
|
||||
TEMPORARY blocks : 0
|
||||
rule files : 6 (5 path-scoped)
|
||||
workspace file : SYMLINK -> felhom.eu/documentation/runbooks/workspace-CLAUDE.md (resolves to the versioned copy)
|
||||
memory index : 200 lines (ceiling 200), 24953 bytes (ceiling 25600)
|
||||
memory index content : 28 version literal(s), 3 host address(es), 0 expired statement(s) [WARN only]
|
||||
memory topic files : 181 indexed, 4 orphaned, 41 archived
|
||||
register citations : 6 cited, 1109 register items known
|
||||
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:31: version literal '0.14.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||
- **[restic 0.14.0 `--verify` is size+mtime, NOT content](restic-0140-verify-and-lock-sema
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:89: version literal '0.142.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||
- [Offsite continuity](offsite-continuity-2026-07-17.md) — ctrl 0.142.0 orphan guard + hub
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:89: version literal '0.60.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||
- [Offsite continuity](offsite-continuity-2026-07-17.md) — ctrl 0.142.0 orphan guard + hub
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:96: version literal '0.152.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||
- **[Hub operator UI: CC uses its token HUB_TOKEN, not the password](hub-operator-ui-basic
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:100: version literal '0.90.1' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||
- [Agent 0.90.1 is wrapper-only](agent-0901-is-wrapper-only.md) — do NOT publish the binar
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:102: version literal '0.144.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||
- [SMB „Megosztás" R-7 slice 1](smb-sharing-r7-slice1-2026-07-18.md) — ctrl 0.144.0 + felh
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md: ... and 22 more version literal(s) — full list from the tally counts above.
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:58: host address '192.168.0.192' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||
- [hub.felhom.eu resolves to the LAN here](hub-resolves-to-lan-on-dooplex-network.md) — 19
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:77: host address '167.233.158.164' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||
- [Offsite PBS box RAM ceiling](offsite-pbs-box-ram-ceiling-2026-07-27.md) — **root SSH =
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:185: host address '192.168.0.180' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||
- **CC runs ON DooPlex** (192.168.0.180) — repos `/mnt/5_hdd/felhom.eu/git/felhom-*`, buil
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory: 4 top-level topic file(s) are not referenced by MEMORY.md, so nothing will ever read them: f10-demo-hp-offsite-resolved-2026-07-23.md, launcher-v0163-2026-07-24.md, peti-return-p1-stop-2026-07-13.md, stopped-not-fault-v0164-2026-07-24.md
|
||||
|
||||
instructions_gate: OK (11 warning(s))
|
||||
|
||||
==============================================================================
|
||||
== gate: release-complete (check-release-complete.py)
|
||||
==============================================================================
|
||||
check-release-complete — the newest CHANGELOG version is a complete release
|
||||
newest CHANGELOG version: v0.159.1
|
||||
ok tag v0.159.1 -> 5c3b218822, an ancestor of HEAD
|
||||
ok package 0.159.1 is downloadable
|
||||
|
||||
check-release-complete: v0.159.1 is tagged, placed and published.
|
||||
|
||||
==============================================================================
|
||||
== gate: observations (observations_gate.py /mnt/5_hdd/felhom.eu/git/felhom-agent)
|
||||
==============================================================================
|
||||
observations gate OK — REPORT.md has no observations section (nothing to check)
|
||||
|
||||
==============================================================================
|
||||
== summary
|
||||
==============================================================================
|
||||
reuse-refs OK (exit 0)
|
||||
instructions OK (exit 0)
|
||||
release-complete OK (exit 7ea9c589094e81d27d741e625d9d8f9
|
||||
[release-agent] config bundle published (HTTP 201)
|
||||
[release-agent] pushing v0.159.2 …
|
||||
remote: . Processing 1 references
|
||||
remote: Processed 1 references in total
|
||||
To https://gitea.dooplex.hu0)
|
||||
observations OK (exit 0)
|
||||
|
||||
all agent gates OK
|
||||
pre-push [felhom-agent]: gates OK - push proceeding.
|
||||
|
||||
RELEASED — and installable, verified by download, not by this script's own say-so.
|
||||
|
||||
version : 0.159.2
|
||||
tag : v0.159.2
|
||||
sha256 : 4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9
|
||||
bundle : 477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9 (felhom-config-bundle.json — vouch it with the agent)
|
||||
|
||||
NOT VOUCHED. Vouching is what points machines at this version and stays you/admin/felhom-agent.git
|
||||
* [new tag] v0.159.2 -> v0.159.2
|
||||
[release-agent] verifying by independent download …
|
||||
r deliberate act:
|
||||
hub operator UI → Configs → Day-0 artifacts. Until then boxes keep installing the previous one.
|
||||
@@ -0,0 +1,4 @@
|
||||
== published installer read back 2026-10-18T12:32:10Z
|
||||
SCRIPT_VERSION="1.34.0" # the SINGLE version source (F-1): -h and the run banners follow it.
|
||||
served sha256 df722ee49f38da595e7c7017aa932b45853696c5282bc408b67a5fb8f141b67a
|
||||
tag sha256 df722ee49f38da595e7c7017aa932b45853696c5282bc408b67a5fb8f141b67a
|
||||
@@ -0,0 +1,28 @@
|
||||
# Website calculator — Immich before/after (2026-10-18)
|
||||
|
||||
Immich peak: **3,290 MB → 3,983 MB** (the kernel in-use figure from the 4,000-photo import, all four containers; the heap alone peaked at 2,777 MB). Immich typical: **1,557 MB, unchanged** (rest after the import: 1,230 MB heap — the raise-only rule keeps the higher). Headroom ×1.5, guest rule and platform unchanged.
|
||||
|
||||
## Immich alone — peak with the platform: 3,760 → 4,453 MB
|
||||
|
||||
| Machine | guest memory | before | after |
|
||||
|---|---|---|---|
|
||||
| 8 GB | 4,096 MB | fits, but tight | does not fit **(changes)** |
|
||||
| 12 GB | 7,580 MB | fits well | fits well |
|
||||
| 16 GB | 11,676 MB | fits well | fits well |
|
||||
| 24 GB | 19,868 MB | fits well | fits well |
|
||||
| 32 GB | 28,060 MB | fits well | fits well |
|
||||
| 48 GB | 44,444 MB | fits well | fits well |
|
||||
| 64 GB | 60,828 MB | fits well | fits well |
|
||||
|
||||
## Family starter set — peak with the platform: 6,021 → 6,714 MB
|
||||
|
||||
| Machine | guest memory | before | after |
|
||||
|---|---|---|---|
|
||||
| 8 GB | 4,096 MB | does not fit | does not fit |
|
||||
| 12 GB | 7,580 MB | fits, but tight | fits, but tight |
|
||||
| 16 GB | 11,676 MB | fits well | fits well |
|
||||
| 24 GB | 19,868 MB | fits well | fits well |
|
||||
| 32 GB | 28,060 MB | fits well | fits well |
|
||||
| 48 GB | 44,444 MB | fits well | fits well |
|
||||
| 64 GB | 60,828 MB | fits well | fits well |
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
=== demo-hp (enkisfelhom.hu)
|
||||
containers: 24 running; not healthy/running: traefik Up 2 days;
|
||||
unhealthy: 0
|
||||
2026/10/18 11:09:36 updater.go:855: [INFO] [selfupdate] Post-update startup: update successful (0.329.0 → 0.330.0)
|
||||
allow-list refusals since start: 0
|
||||
2026/10/18 11:09:36 sync.go:142: [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
|
||||
installed apps (stacks with app.yaml): 12
|
||||
public https://felhom.enkisfelhom.hu/login: 200
|
||||
public https://files.enkisfelhom.hu/: 200
|
||||
=== demo-felhom (demo-felhom.eu)
|
||||
containers: 9 running; not healthy/running: traefik Up 2 days;
|
||||
unhealthy: 0
|
||||
2026/10/18 11:09:25 [INFO] [selfupdate] Post-update startup: update successful (0.329.0 → 0.330.0)
|
||||
allow-list refusals since start: 0
|
||||
2026/10/18 11:09:26 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
|
||||
installed apps (stacks with app.yaml): 5
|
||||
public https://felhom.demo-felhom.eu/login: 200
|
||||
public https://files.demo-felhom.eu/: 200
|
||||
=== tester-1 (enkicsifelhom.hu)
|
||||
containers: 10 running; not healthy/running: traefik Up 2 days;
|
||||
unhealthy: 0
|
||||
2026/10/18 11:09:35 [INFO] [selfupdate] Post-update startup: update successful (0.329.0 → 0.330.0)
|
||||
allow-list refusals since start: 0
|
||||
2026/10/18 11:09:36 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
|
||||
installed apps (stacks with app.yaml): 3
|
||||
public https://felhom.enkicsifelhom.hu/login: 200
|
||||
public https://files.enkicsifelhom.hu/: 200
|
||||
@@ -0,0 +1,12 @@
|
||||
== global floor 2026-10-18T11:10:59Z
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /configuration?flash=floor_set
|
||||
clear tester-1:
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /customers/tester-1?flash=floor_set
|
||||
clear demo-hp:
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /customers/demo-hp?flash=floor_set
|
||||
clear demo-felhom:
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /customers/demo-felhom?flash=floor_set
|
||||
@@ -0,0 +1,9 @@
|
||||
set tester-1:
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /customers/tester-1?flash=floor_set
|
||||
set demo-hp:
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /customers/demo-hp?flash=floor_set
|
||||
set demo-felhom:
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /customers/demo-felhom?flash=floor_set
|
||||
@@ -0,0 +1,3 @@
|
||||
('min_controller_version', '0.330.0')
|
||||
('min_controller_version_declared_min_agent', '0.330.0=0.158.1')
|
||||
per-box floors set: 0
|
||||
@@ -0,0 +1,2 @@
|
||||
OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
|
||||
== every public name redirects http -> https
|
||||
@@ -0,0 +1,20 @@
|
||||
# P8 — old-restic check: NOT RUN (the clean-up window has not opened). Read 2026-10-18T14:33:43Z.
|
||||
|
||||
## demo-hp's clean-up windows (hub DB with its WAL, one tar)
|
||||
id|customer|opened_at|closes_by|closed_at|box_result
|
||||
9|demo-hp|2026-10-12 02:16:53|2026-10-12 02:36:53|2026-10-12 02:17:06|pruned
|
||||
6|demo-hp|2026-10-05 05:25:53|2026-10-05 05:45:53|2026-10-05 05:26:04|pruned
|
||||
2|demo-hp|2026-10-04 05:36:14|2026-10-04 05:56:14|2026-10-04 05:36:17|nothing
|
||||
|
||||
## The rule (hub/internal/offsitekeys/service.go:255)
|
||||
windowCadence = 6*24*time.Hour + 12*time.Hour // "weekly", with slack for the night chain's drift
|
||||
Last window 2026-10-12 02:16:53 UTC + 6 d 12 h = 2026-10-18 14:16:53 UTC (the STATUS time).
|
||||
The box asks for a window only AFTER its nightly off-site run (controller backup/offbox_window.go header).
|
||||
|
||||
## demo-hp guest 9201 controller (docker logs since 06:00 UTC, [offbox] lines + scheduler)
|
||||
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.330.0 Up 3 hours (healthy)
|
||||
2026/10/18 11:09:36 scheduler.go:150: [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-10-19 04:15 CEST
|
||||
2026/10/18 11:09:36 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="offbox-backup" schedule="04:15" nextRun=2026-10-19T04:15:00+02:00 totalJobs=15
|
||||
2026/10/18 11:09:36 scheduler.go:85: [DEBUG] [scheduler] daily job offbox-backup: next run at 2026-10-19 04:15:00 CEST (waiting 15h5m23s)
|
||||
|
||||
Next run: 2026-10-19 04:15 CEST = 02:15 UTC; the window opens right after it. No window was asked for or granted by CC.
|
||||
@@ -0,0 +1,23 @@
|
||||
Mon Oct 19 01:24:39 UTC 2026
|
||||
200
|
||||
{"data": {"started": true}, "ok": true}
|
||||
200
|
||||
{
|
||||
"data": {
|
||||
"backup": {
|
||||
"target_id": "local",
|
||||
"vmid": 9201,
|
||||
"archive": "local:backup/vzdump-lxc-9201-2026_10_19-03_25_27.tar.zst",
|
||||
"mode": "snapshot",
|
||||
"crash_consistent": true,
|
||||
"size_bytes": 6864786948,
|
||||
"success": true,
|
||||
"started_at": "2026-10-19T01:25:26Z",
|
||||
"duration_seconds": 443.050964897
|
||||
},
|
||||
"error": "",
|
||||
"job_id": "backup-9201-1792373126635435331",
|
||||
"phase": "done"
|
||||
},
|
||||
"ok": true
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
Mon Oct 19 01:52:00 UTC 2026
|
||||
tester-1-d70be4 agent_update:
|
||||
signed: op=agent_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=3516434bfa0c5c47295e3de0a8bd20c7 expires=2026-10-19T02:37:00Z
|
||||
uploaded signed op to the hub jobs queue
|
||||
tester-1-d70be4 agent_config_update:
|
||||
signed: op=agent_config_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=2dfc5b3e216030c82add18ba69899b41 expires=2026-10-19T02:37:00Z
|
||||
uploaded signed op to the hub jobs queue
|
||||
demo-felhom-8363b5 agent_update:
|
||||
signed: op=agent_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=8f3a6c280cb124c6f85a4039c5b112dc expires=2026-10-19T02:37:00Z
|
||||
uploaded signed op to the hub jobs queue
|
||||
demo-felhom-8363b5 agent_config_update:
|
||||
signed: op=agent_config_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=dc2ead6937a38d0a2d8acd4abf57bef9 expires=2026-10-19T02:37:00Z
|
||||
uploaded signed op to the hub jobs queue
|
||||
@@ -0,0 +1,17 @@
|
||||
2026-10-19T01:55:42Z tester-1: felhom-agent 0.160.0 | demo-felhom: felhom-agent 0.160.0
|
||||
Oct 19 03:54:05 felhom felhom-agent[1371657]: time=2026-10-19T03:54:05.642+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=a254c4ec38459c7c op=agent_update
|
||||
Oct 19 03:54:06 felhom felhom-agent[1371657]: time=2026-10-19T03:54:06.786+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.160.0 written=0 same=30 self-check=ok signers-created=False"
|
||||
Oct 19 03:54:07 felhom felhom-agent[1371657]: time=2026-10-19T03:54:07.652+02:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=75 total=75 degraded=""
|
||||
Oct 19 03:54:07 felhom felhom-agent[1371657]: time=2026-10-19T03:54:07.653+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=a66f7fc0b9fd8fb6 op=agent_config_update
|
||||
Oct 19 03:54:08 felhom felhom-agent[2443463]: time=2026-10-19T03:54:08.649+02:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true
|
||||
Oct 19 03:54:08 felhom felhom-agent[2443463]: time=2026-10-19T03:54:08.649+02:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
|
||||
Oct 19 03:55:10 felhom felhom-agent[2443463]: time=2026-10-19T03:55:10.315+02:00 level=WARN msg="selfupdate: update committed" version=0.160.0 wrapper=""
|
||||
Oct 19 03:55:15 demo-felhom felhom-os-apply[498026]: os-apply: BUNDLE DONE agent=0.160.0 written=0 same=30 self-check=ok signers-created=False
|
||||
Oct 19 03:55:15 demo-felhom felhom-agent[3800522]: time=2026-10-19T03:55:15.698+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.160.0 written=0 same=30 self-check=ok signers-created=False"
|
||||
Oct 19 03:55:16 demo-felhom felhom-agent[3800522]: time=2026-10-19T03:55:16.340+02:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=75 total=75 degraded=""
|
||||
Oct 19 03:55:16 demo-felhom felhom-agent[3800522]: time=2026-10-19T03:55:16.340+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=7dde89f299a0684c op=agent_config_update
|
||||
Oct 19 03:55:17 demo-felhom felhom-agent[498196]: time=2026-10-19T03:55:17.698+02:00 level=INFO msg="backup tier armed" target=felhom-backup cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true
|
||||
Oct 19 03:55:17 demo-felhom felhom-agent[498196]: time=2026-10-19T03:55:17.698+02:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
|
||||
Oct 19 03:56:19 demo-felhom felhom-agent[498196]: time=2026-10-19T03:56:19.667+02:00 level=WARN msg="selfupdate: update committed" version=0.160.0 wrapper=""
|
||||
|
||||
[exited with code 0]
|
||||
@@ -0,0 +1,29 @@
|
||||
Volid Format Type Size VMID
|
||||
local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst tar.zst backup 656970239 9100
|
||||
local:backup/vzdump-lxc-9201-2026_10_19-03_25_27.tar.zst tar.zst backup 6864786948 9201
|
||||
local:backup/felhom-golden-0.236.0.tar.zst tar.zst backup 654115664
|
||||
Oct 19 03:23:37 demo-hp felhom-agent[2622137]: time=2026-10-19T03:23:37.628+02:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=1 prune="keep-daily=1" wait_timeout=30m0s prune_pbs_allowed=false primary=true
|
||||
Oct 19 03:23:37 demo-hp felhom-agent[2622137]: time=2026-10-19T03:23:37.628+02:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
|
||||
Oct 19 03:24:39 demo-hp felhom-agent[2622137]: time=2026-10-19T03:24:39.730+02:00 level=WARN msg="selfupdate: update committed" version=0.160.0 wrapper=""
|
||||
Oct 19 03:32:49 demo-hp felhom-agent[2622137]: time=2026-10-19T03:32:49.686+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_10_19-03_25_27.tar.zst size_bytes=6864786948 uncovered_volumes=2
|
||||
Oct 19 03:32:49 demo-hp felhom-agent[2622137]: time=2026-10-19T03:32:49.686+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1792373126635435331 archive=local:backup/vzdump-lxc-9201-2026_10_19-03_25_27.tar.zst
|
||||
-rw------- 1 felhom-agent felhom-agent 323 Oct 19 03:32 /var/lib/felhom-agent/backup-success-state.json
|
||||
[
|
||||
{
|
||||
"target": "felhom-pbs",
|
||||
"vmid": 9201,
|
||||
"started_at": "2026-10-15T20:38:46Z",
|
||||
"recent": [
|
||||
"2026-10-15T20:38:46Z"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target": "local",
|
||||
"vmid": 9201,
|
||||
"started_at": "2026-10-19T01:25:26Z",
|
||||
"recent": [
|
||||
"2026-10-18T11:12:25Z",
|
||||
"2026-10-19T01:25:26Z"
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,33 @@
|
||||
Mon Oct 19 01:22:11 UTC 2026
|
||||
felhom-agent 0.159.2
|
||||
demo-hp-bb76ea
|
||||
Volid Format Type Size VMID
|
||||
local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst tar.zst backup 656970239 9100
|
||||
local:backup/vzdump-lxc-9201-2026_10_18-13_12_26.tar.zst tar.zst backup 6938307162 9201
|
||||
local:backup/felhom-golden-0.236.0.tar.zst tar.zst backup 654115664
|
||||
"local_backup_retention": 1
|
||||
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
|
||||
341 night1004-tester1 running 8192 200.00 1851
|
||||
perl: warning: Setting locale failed.
|
||||
perl: warning: Please check that your locale settings:
|
||||
LANGUAGE = (unset),
|
||||
LC_ALL = (unset),
|
||||
LC_CTYPE = "UTF-8",
|
||||
LC_NUMERIC = (unset),
|
||||
LC_COLLATE = (unset),
|
||||
LC_TIME = (unset),
|
||||
LC_MESSAGES = (unset),
|
||||
LC_MONETARY = (unset),
|
||||
LC_ADDRESS = (unset),
|
||||
LC_IDENTIFICATION = (unset),
|
||||
LC_MEASUREMENT = (unset),
|
||||
LC_PAPER = (unset),
|
||||
LC_TELEPHONE = (unset),
|
||||
LC_NAME = (unset),
|
||||
LANG = "en_US.UTF-8"
|
||||
are supported and installed on your system.
|
||||
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
|
||||
VMID Status Lock Name
|
||||
9201 running demo-hp
|
||||
9202 running demo-hp-scratch
|
||||
9401 stopped upgrade-harness
|
||||
@@ -0,0 +1,9 @@
|
||||
Mon Oct 19 01:22:47 UTC 2026
|
||||
demo-hp-bb76ea agent_update:
|
||||
signed: op=agent_update host=demo-hp-bb76ea guest="" key_id=felhom-op-1 nonce=71396ac328c9ea3192d027b830ec0990 expires=2026-10-19T02:07:47Z
|
||||
wrote envelope to /dev/null
|
||||
uploaded signed op to the hub jobs queue
|
||||
demo-hp-bb76ea agent_config_update:
|
||||
signed: op=agent_config_update host=demo-hp-bb76ea guest="" key_id=felhom-op-1 nonce=603f1d8ed0937492cf01a05146fc9e48 expires=2026-10-19T02:07:47Z
|
||||
wrote envelope to /dev/null
|
||||
uploaded signed op to the hub jobs queue
|
||||
@@ -0,0 +1,5 @@
|
||||
Mon Oct 19 01:47:33 UTC 2026
|
||||
code 0 (the stored demo-hp code, before):
|
||||
HTTP 200 opens; restic_pw_sha256 8a9e33aa
|
||||
control, a wrong code:
|
||||
HTTP 400 the recovery code did not open the sealed bundle — nothing was written
|
||||
@@ -0,0 +1,11 @@
|
||||
01:47:43Z start 200 {"data":{"job_id":"escrow-1792374463410689900","phase":"running"},"error":"","ok":true}
|
||||
|
||||
status 200 {"phase": "done", "claimable": true, "uploaded": false, "restic_pw_sealed": true, "detail": ""}
|
||||
claim 200 code words: 10 {"confirm_required": true, "verify_positions": [3, 10]}
|
||||
status after claim {"phase": "done", "claimed": true, "uploaded": false}
|
||||
confirm with one wrong word 422 A megadott szavak nem egyeznek — ellenőrizd a felírt kódot.
|
||||
01:47:47Z confirm with the right words 200 {"uploaded": true}
|
||||
code 1 after the confirm:
|
||||
HTTP 200 opens; restic_pw_sha256 8a9e33aa
|
||||
code 0 after the confirm:
|
||||
HTTP 422 the recovery code is correct, but it belongs to an EARLIER sealed package (superseded 2026-10-19T01:47:47Z), not the one
|
||||
@@ -0,0 +1,9 @@
|
||||
01:48:09Z start 200 {"data":{"job_id":"escrow-1792374489199785658","phase":"running"},"error":"","ok":true}
|
||||
|
||||
status 200 {"phase": "done", "claimable": true, "uploaded": false, "restic_pw_sealed": true, "detail": ""}
|
||||
claim 200 code words: 10 {"confirm_required": true, "verify_positions": [5, 10]}
|
||||
status after claim {"phase": "done", "claimed": true, "uploaded": false}
|
||||
after the abandon — code 1 (stored):
|
||||
HTTP 200 opens; restic_pw_sha256 8a9e33aa
|
||||
code 2 (shown, never typed back):
|
||||
HTTP 400 the recovery code did not open the sealed bundle — nothing was written
|
||||
@@ -0,0 +1,23 @@
|
||||
time=2026-10-19T03:47:33.706+02:00 level=INFO msg="local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)" vmid=9201
|
||||
time=2026-10-19T03:47:34.786+02:00 level=INFO msg="local-api: offsite repository password RECOVERED from the sealed escrow — returning that field ONLY (the tunnel token, the PBS token and the WG key stay inside the agent and are not returned)" vmid=9201 rest
|
||||
time=2026-10-19T03:47:35.044+02:00 level=INFO msg="local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)" vmid=9201
|
||||
time=2026-10-19T03:47:36.053+02:00 level=WARN msg="local-api: offsite key recovery: the fetched bundle did not open with the supplied recovery code" vmid=9201 err="escrow: the recovery code did not unwrap the identity escrow (wrong recovery code, or a corrupt
|
||||
time=2026-10-19T03:47:43.409+02:00 level=INFO msg="local-api: staged offsite restic repo password for escrow (field name only)" vmid=9201
|
||||
time=2026-10-19T03:47:43.410+02:00 level=INFO msg="local-api: escrow ceremony started (controller-driven)" vmid=9201 job_id=escrow-1792374463410689900
|
||||
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=escrow-create --upload --output=json
|
||||
time=2026-10-19T03:47:45.992+02:00 level=INFO msg="local-api: escrow ceremony complete — R claimable (in-memory, one-shot); package uploads at the claim" job_id=escrow-1792374463410689900 restic_pw_sealed=true uploaded=false claim_ttl_s=600 duration_ms=2581
|
||||
time=2026-10-19T03:47:47.507+02:00 level=INFO msg="local-api: escrow ceremony R claimed — the package waits for the typed-back words (R-1151)" vmid=9201 job_id=escrow-1792374463410689900 confirm_ttl_s=1800
|
||||
time=2026-10-19T03:47:47.550+02:00 level=INFO msg="local-api: escrow confirm — the typed words do not match; still waiting (R-1151)" vmid=9201
|
||||
time=2026-10-19T03:47:47.619+02:00 level=INFO msg="local-api: escrow confirm — words typed back; the new package is on the hub (R-1151)" vmid=9201 job_id=escrow-1792374463410689900 duration_ms=48
|
||||
time=2026-10-19T03:47:48.247+02:00 level=INFO msg="local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)" vmid=9201
|
||||
time=2026-10-19T03:47:49.287+02:00 level=INFO msg="local-api: offsite repository password RECOVERED from the sealed escrow — returning that field ONLY (the tunnel token, the PBS token and the WG key stay inside the agent and are not returned)" vmid=9201 rest
|
||||
time=2026-10-19T03:47:49.635+02:00 level=INFO msg="local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)" vmid=9201
|
||||
time=2026-10-19T03:48:09.199+02:00 level=INFO msg="local-api: staged offsite restic repo password for escrow (field name only)" vmid=9201
|
||||
time=2026-10-19T03:48:09.199+02:00 level=INFO msg="local-api: escrow ceremony started (controller-driven)" vmid=9201 job_id=escrow-1792374489199785658
|
||||
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=escrow-create --upload --output=json
|
||||
time=2026-10-19T03:48:11.881+02:00 level=INFO msg="local-api: escrow ceremony complete — R claimable (in-memory, one-shot); package uploads at the claim" job_id=escrow-1792374489199785658 restic_pw_sealed=true uploaded=false claim_ttl_s=600 duration_ms=2681
|
||||
time=2026-10-19T03:48:13.294+02:00 level=INFO msg="local-api: escrow ceremony R claimed — the package waits for the typed-back words (R-1151)" vmid=9201 job_id=escrow-1792374489199785658 confirm_ttl_s=1800
|
||||
time=2026-10-19T03:48:14.059+02:00 level=INFO msg="local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)" vmid=9201
|
||||
time=2026-10-19T03:48:15.141+02:00 level=INFO msg="local-api: offsite repository password RECOVERED from the sealed escrow — returning that field ONLY (the tunnel token, the PBS token and the WG key stay inside the agent and are not returned)" vmid=9201 rest
|
||||
time=2026-10-19T03:48:15.389+02:00 level=INFO msg="local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)" vmid=9201
|
||||
time=2026-10-19T03:48:17.435+02:00 level=WARN msg="local-api: offsite key recovery: the fetched bundle did not open with the supplied recovery code" vmid=9201 err="escrow: the recovery code did not unwrap the identity escrow (wrong recovery code, or a corrupt
|
||||
@@ -0,0 +1,4 @@
|
||||
Mon Oct 19 02:08:15 UTC 2026
|
||||
demo-felhom-8363b5|0.160.0|2026-10-19 01:55:31
|
||||
demo-hp-bb76ea|0.160.0|2026-10-19 01:53:50
|
||||
tester-1-d70be4|0.160.0|2026-10-19 01:54:15
|
||||
@@ -0,0 +1,36 @@
|
||||
== root@192.168.0.154
|
||||
Mon Oct 19 01:51:20 UTC 2026
|
||||
felhom-agent 0.159.2
|
||||
"host_id": "tester-1-d70be4"
|
||||
"local_backup_retention": 3
|
||||
Volid Format Type Size VMID
|
||||
local:backup/vzdump-lxc-9100-2026_10_04-21_40_45.tar.zst tar.zst backup 648135998 9100
|
||||
local:backup/vzdump-lxc-9201-2026_10_15-22_43_40.tar.zst tar.zst backup 3364084456 9201
|
||||
local:backup/vzdump-lxc-9201-2026_10_17-04_32_43.tar.zst tar.zst backup 3479976346 9201
|
||||
local:backup/vzdump-lxc-9201-2026_10_18-04_37_41.tar.zst tar.zst backup 1867626596 9201
|
||||
== felhom-pve
|
||||
Mon Oct 19 01:51:13 UTC 2026
|
||||
felhom-agent 0.159.2
|
||||
"host_id": "demo-felhom-8363b5"
|
||||
"local_backup_retention": 3
|
||||
Volid Format Type Size VMID
|
||||
local:backup/vzdump-lxc-9100-2026_07_18-18_30_52.tar.zst tar.zst backup 612397450 9100
|
||||
local:backup/vzdump-lxc-9100-2026_07_20-17_50_57.tar.zst tar.zst backup 648549366 9100
|
||||
local:backup/vzdump-lxc-9100-2026_08_03-06_58_00.tar.zst tar.zst backup 649547835 9100
|
||||
local:backup/vzdump-lxc-9100-2026_08_03-09_15_34.tar.zst tar.zst backup 649547835 9100
|
||||
local:backup/vzdump-lxc-9201-2026_07_28-01_27_56.tar.zst tar.zst backup 5820882781 9201
|
||||
local:backup/vzdump-lxc-9201-2026_07_28-06_44_22.tar.zst tar.zst backup 5840583923 9201
|
||||
local:backup/vzdump-lxc-9201-2026_07_28-17_53_26.tar.zst tar.zst backup 5929975285 9201
|
||||
{'local_backup_target': 'felhom-backup', 'local_backup_retention': 3, 'restore_test_cadence_seconds': 0, 'backup_cadence_seconds': 0, 'backup_targets': [{'target_id': 'felhom-pbs', 'cadence_seconds': 604800, 'keep_last': 0}]}
|
||||
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
|
||||
felhom-backup dir active 960303848 7518604 903930732 0.78%
|
||||
felhom-pbs pbs active 0 0 0 0.00%
|
||||
local dir active 98497780 33631712 59816520 34.14%
|
||||
local-lvm lvmthin active 365760512 11082543 354677968 3.03%
|
||||
time=2026-10-16T05:03:53.319+02:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-backup archive=felhom-backup:backup/vzdump-lxc-9201-2026_10_16-05_03_12.tar.zst size_bytes=1300947312 uncovered_volumes=2
|
||||
time=2026-10-17T05:07:43.382+02:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-backup archive=felhom-backup:backup/vzdump-lxc-9201-2026_10_17-05_07_02.tar.zst size_bytes=1334976334 uncovered_volumes=2
|
||||
time=2026-10-18T05:08:13.307+02:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-backup archive=felhom-backup:backup/vzdump-lxc-9201-2026_10_18-05_07_32.tar.zst size_bytes=1296570428 uncovered_volumes=2
|
||||
Volid Format Type Size VMID
|
||||
felhom-backup:backup/vzdump-lxc-9201-2026_10_16-05_03_12.tar.zst tar.zst backup 1300947312 9201
|
||||
felhom-backup:backup/vzdump-lxc-9201-2026_10_17-05_07_02.tar.zst tar.zst backup 1334976334 9201
|
||||
felhom-backup:backup/vzdump-lxc-9201-2026_10_18-05_07_32.tar.zst tar.zst backup 1296570428 9201
|
||||
@@ -0,0 +1,32 @@
|
||||
time=2026-10-19T03:49:56.342+02:00 level=INFO msg="audit: gate decision" class=storage_wipe host=demo-hp-bb76ea guest="" source=one_shot_job disposition=customer_confirmable allowed=false reason=pending_confirmation key_id="" nonce="" durab
|
||||
time=2026-10-19T03:49:56.342+02:00 level=INFO msg="gate decision (storage wipe)" role=user-data disposition=customer_confirmable allowed=false reason=pending_confirmation durable_id=byid:wwn-0x3333333000000fa0-part1+fsuuid:ef8fc766-1e12-491
|
||||
time=2026-10-19T03:49:56.342+02:00 level=WARN msg="local-api: user-data data-bearing format refused" vmid=9201 device=/dev/sdb1 durable_id=byid:wwn-0x3333333000000fa0-part1+fsuuid:ef8fc766-1e12-491e-b6f2-31433e1989c8 reason=pending_confirma
|
||||
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||
backup-success-state.json
|
||||
controller-swap-9201.json
|
||||
drive-intents.json
|
||||
escrow-stage
|
||||
felhom-sshd
|
||||
format-job.json
|
||||
guest-binds.json
|
||||
guest-disk-trim.json
|
||||
guests
|
||||
journal.log
|
||||
local-api.crt
|
||||
local-api.key
|
||||
local-tokens.log
|
||||
nonces.log
|
||||
os
|
||||
pbsdr
|
||||
restore-test-state.json
|
||||
selfupdate
|
||||
units
|
||||
wg
|
||||
@@ -0,0 +1,39 @@
|
||||
Mon Oct 19 01:49:49 UTC 2026
|
||||
new test disk: /dev/sdb
|
||||
NAME SIZE MODEL
|
||||
sdb 64M scsi_debug
|
||||
--- the empty partition (the R-1093 input):
|
||||
{
|
||||
"blockdevices": [
|
||||
{
|
||||
"name": "sdb1",
|
||||
"fstype": null,
|
||||
"pttype": "gpt"
|
||||
}
|
||||
]
|
||||
}
|
||||
DEVNAME=/dev/sdb1
|
||||
PART_ENTRY_SCHEME=gpt
|
||||
PART_ENTRY_UUID=aec56a69-b9a0-46de-9c8e-911ca8558717
|
||||
PART_ENTRY_TYPE=0fc63daf-8483-4772-8e79-3d69d8477de4
|
||||
PART_ENTRY_NUMBER=1
|
||||
PART_ENTRY_OFFSET=2048
|
||||
PART_ENTRY_SIZE=126976
|
||||
PART_ENTRY_DISK=8:16
|
||||
blkid rc=0
|
||||
lrwxrwxrwx 1 root root 9 Oct 19 03:49 scsi-33333333000000fa0 -> ../../sdb
|
||||
lrwxrwxrwx 1 root root 10 Oct 19 03:49 scsi-33333333000000fa0-part1 -> ../../sdb1
|
||||
lrwxrwxrwx 1 root root 9 Oct 19 03:49 wwn-0x3333333000000fa0 -> ../../sdb
|
||||
--- POST /disks/format on the EMPTY partition (no confirmation sent):
|
||||
HTTP 200 {"ok":true,"data":{"vmid":9201,"device":"/dev/sdb1","formatted":true,"data_bearing":false,"reason":"blank device formatted ext4","durable_id":"byid:wwn-0x3333333000000fa0-part1","fs_uuid":"ef8fc766-1e12-491e-b6f2-31433e1989c8"}}
|
||||
|
||||
--- the partition now:
|
||||
TYPE=ext4
|
||||
PART_ENTRY_SCHEME=gpt
|
||||
--- CONTROL: the same call again, on the now-ext4 partition:
|
||||
HTTP 403 {"ok":false,"data":{"vmid":9201,"device":"/dev/sdb1","formatted":false,"data_bearing":true,"reason":"device has a ext4 filesystem","role":"user-data","needs_confirmation":true,"durable_id":"byid:wwn-0x3333333000000fa0-part1+fsuuid:ef8fc766-1e12-491e-b6f2-31433e1989c8"},"error":"device is data-bearing — customer confirmation required"}
|
||||
|
||||
--- teardown
|
||||
not mounted
|
||||
scsi_debug removed
|
||||
0
|
||||
@@ -0,0 +1,102 @@
|
||||
[release-agent] building 0.160.0 …
|
||||
[release-agent] built ok: sha256 7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1
|
||||
[release-agent] config bundle built: sha256 6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f
|
||||
[release-agent] tagging v0.160.0 at c11e3e2 …
|
||||
[release-agent] publishing …
|
||||
[publish-agent] publishing /tmp/felhom-agent-YBD0P9 (16071264 bytes, sha256 7ebb547f2b1ab935…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/0.160.0/felhom-agent
|
||||
[publish-agent] pre-delete existing artifact: HTTP 404 (404/204 expected)
|
||||
[publish-agent] upload OK (HTTP 201)
|
||||
[publish-agent] round-trip GET verified (sha256 matches)
|
||||
[publish-agent] DONE. Record in the hub operator UI (Configs → Day-0 artifacts): agent 0.160.0 / 7ebb547f2b1ab93544ba828d3040351b6AGENT_VERSION=0.160.0
|
||||
AGENT_SHA256=7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1
|
||||
pre-push [felhom-agent]: running scripts/agent_gates.py --fast ...
|
||||
agent_gates — 4 gate(s) [--fast]
|
||||
--fast SKIPPED (deliberate periodic runs, never in a hook): published, gate-decoys
|
||||
|
||||
==============================================================================
|
||||
== gate: reuse-refs (reuse_refs_check.py /mnt/5_hdd/felhom.eu/git/felhom-agent)
|
||||
==============================================================================
|
||||
note [felhom-agent] line 190: localapi/debuglogs_test.go (resolved by suffix → internal/localapi/debuglogs_test.go)
|
||||
note [felhom-agent] line 206: hub/internal/store/dr_recipe.go (cross-repo → felhom.eu/hub/internal/store/dr_recipe.go)
|
||||
OK [felhom-agent]: 118 cited paths — exact 116, suffix 1, ambiguous 0, cross-repo 1, FAILED 0 (siblings searched: app-catalog-felhom.eu, cc-secrets, felhom-controller, felhom.eu, felhom.eu-media, felhom.eu-sizing-push, felhom.eu-wt-sizing)
|
||||
|
||||
==============================================================================
|
||||
== gate: instructions (instructions_gate.py /mnt/5_hdd/felhom.eu/git/felhom-agent)
|
||||
==============================================================================
|
||||
instructions_gate: /mnt/5_hdd/felhom.eu/git/felhom-agent
|
||||
CLAUDE.md effective lines : 108 (ceiling 200)
|
||||
version literals : 0
|
||||
TEMPORARY blocks : 0
|
||||
rule files : 6 (5 path-scoped)
|
||||
workspace file : SYMLINK -> felhom.eu/documentation/runbooks/workspace-CLAUDE.md (resolves to the versioned copy)
|
||||
memory index : 200 lines (ceiling 200), 25109 bytes (ceiling 25600)
|
||||
memory index content : 28 version literal(s), 3 host address(es), 0 expired statement(s) [WARN only]
|
||||
memory topic files : 183 indexed, 4 orphaned, 41 archived
|
||||
register citations : 6 cited, 1116 register items known
|
||||
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:31: version literal '0.14.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||
- **[restic 0.14.0 `--verify` is size+mtime, NOT content](restic-0140-verify-and-lock-sema
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:90: version literal '0.142.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||
- [Offsite continuity](offsite-continuity-2026-07-17.md) — ctrl 0.142.0 orphan guard + hub
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:90: version literal '0.60.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||
- [Offsite continuity](offsite-continuity-2026-07-17.md) — ctrl 0.142.0 orphan guard + hub
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:97: version literal '0.152.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||
- **[Hub operator UI: CC uses its token HUB_TOKEN, not the password](hub-operator-ui-basic
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:101: version literal '0.90.1' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||
- [Agent 0.90.1 is wrapper-only](agent-0901-is-wrapper-only.md) — do NOT publish the binar
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:103: version literal '0.144.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||
- [SMB „Megosztás" R-7 slice 1](smb-sharing-r7-slice1-2026-07-18.md) — ctrl 0.144.0 + felh
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md: ... and 22 more version literal(s) — full list from the tally counts above.
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:59: host address '192.168.0.192' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||
- [hub.felhom.eu resolves to the LAN here](hub-resolves-to-lan-on-dooplex-network.md) — 19
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:78: host address '167.233.158.164' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||
- [Offsite PBS box RAM ceiling](offsite-pbs-box-ram-ceiling-2026-07-27.md) — **root SSH =
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:186: host address '192.168.0.180' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||
- **CC runs ON DooPlex** (192.168.0.180) — repos `/mnt/5_hdd/felhom.eu/git/felhom-*`, buil
|
||||
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory: 4 top-level topic file(s) are not referenced by MEMORY.md, so nothing will ever read them: f10-demo-hp-offsite-resolved-2026-07-23.md, launcher-v0163-2026-07-24.md, peti-return-p1-stop-2026-07-13.md, stopped-not-fault-v0164-2026-07-24.md
|
||||
|
||||
instructions_gate: OK (11 warning(s))
|
||||
|
||||
==============================================================================
|
||||
== gate: release-complete (check-release-complete.py)
|
||||
==============================================================================
|
||||
check-release-complete — the newest CHANGELOG version is a complete release
|
||||
newest CHANGELOG version: v0.159.2
|
||||
ok tag v0.159.2 -> 65c1994d3b, an ancestor of HEAD
|
||||
ok package 0.159.2 is downloadable
|
||||
|
||||
check-release-complete: v0.159.2 is tagged, placed and published.
|
||||
|
||||
==============================================================================
|
||||
== gate: observations (observations_gate.py /mnt/5_hdd/felhom.eu/git/felhom-agent)
|
||||
==============================================================================
|
||||
observations gate OK — REPORT.md has no observations section (nothing to check)
|
||||
|
||||
==============================================================================
|
||||
== summary
|
||||
==============================================================================
|
||||
reuse-refs OK (exit 0)
|
||||
instructions OK (exit 0)
|
||||
release-complete OK (exit 0c4126327a5b85bbcb33f8f2b3dfca1
|
||||
[release-agent] config bundle published (HTTP 201)
|
||||
[release-agent] pushing v0.160.0 …
|
||||
remote: . Processing 1 references
|
||||
remote: Processed 1 references in total
|
||||
To https://gitea.dooplex.hu0)
|
||||
observations OK (exit 0)
|
||||
|
||||
all agent gates OK
|
||||
pre-push [felhom-agent]: gates OK - push proceeding.
|
||||
|
||||
RELEASED — and installable, verified by download, not by this script's own say-so.
|
||||
|
||||
version : 0.160.0
|
||||
tag : v0.160.0
|
||||
sha256 : 7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1
|
||||
bundle : 6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f (felhom-config-bundle.json — vouch it with the agent)
|
||||
|
||||
NOT VOUCHED. Vouching is what points machines at this version and stays you/admin/felhom-agent.git
|
||||
* [new tag] v0.160.0 -> v0.160.0
|
||||
[release-agent] verifying by independent download …
|
||||
r deliberate act:
|
||||
hub operator UI → Configs → Day-0 artifacts. Until then boxes keep installing the previous one.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 99 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 79 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 36 KiB |
@@ -0,0 +1,17 @@
|
||||
03:05:34 goto https://felhom.enkisfelhom.hu/login -> https://felhom.enkisfelhom.hu/login
|
||||
03:05:39 eval -> "focused"
|
||||
03:05:43 eval -> "https://felhom.enkisfelhom.hu/launcher"
|
||||
03:05:43 goto https://felhom.enkisfelhom.hu/apps/calibre-web -> https://felhom.enkisfelhom.hu/apps/calibre-web
|
||||
03:05:48 eval -> "https://felhom.enkisfelhom.hu/__files/open?to=… | button: Fájlok ↗ | folder: /files/NVME 1TB/media/books"
|
||||
03:05:48 shot 01-app-page-files-button
|
||||
03:05:49 eval -> "pressed Fájlok"
|
||||
03:05:57 eval -> "files.enkisfelhom.hu/files/NVME 1TB/media/books | login form: no | title: FileBrowser Quantum - Files - books"
|
||||
03:05:57 eval -> "FileBrowser user: felhom-household | admin: false | modify: true | loginMethod: proxy"
|
||||
03:05:59 shot 02-filebrowser-opened-signed-in
|
||||
03:05:59 goto https://felhom.enkisfelhom.hu/launcher -> https://felhom.enkisfelhom.hu/launcher
|
||||
03:06:04 eval -> "signed out on the dashboard"
|
||||
03:06:08 eval -> "https://felhom.enkisfelhom.hu/login"
|
||||
03:06:08 goto https://files.enkisfelhom.hu/files/ -> https://felhom.enkisfelhom.hu/login
|
||||
03:06:15 eval -> "felhom.enkisfelhom.hu/login | login form: YES"
|
||||
03:06:15 shot 03-after-sign-out-asks-again
|
||||
LOG ('netfail', 'net::ERR_ABORTED', 'Fetch')
|
||||
@@ -0,0 +1,9 @@
|
||||
== through traefik (https://files.enkisfelhom.hu, no files cookie), GET /api/users?id=self
|
||||
forged header named X-Felhom-User -> HTTP 401 (user None)
|
||||
forged header named RANDOM -> HTTP 401 (user None)
|
||||
forged header named SECRET -> HTTP 401 (user None)
|
||||
== from the bookstack container (on traefik-public), straight to http://filebrowser:80/api/users?id=self — no traefik
|
||||
forged header named X-Felhom-User -> HTTP 401 (user None)
|
||||
forged header named RANDOM -> HTTP 401 (user None)
|
||||
== CONTROL: the same container with the box's REAL secret name (read from the box, never printed)
|
||||
real secret name -> HTTP 200 (user felhom-household)
|
||||
@@ -0,0 +1,4 @@
|
||||
Mon Oct 19 03:08:04 UTC 2026
|
||||
200
|
||||
{
|
||||
"ok": true,
|
||||
@@ -0,0 +1,5 @@
|
||||
Mon Oct 19 03:08:23 UTC 2026
|
||||
05:08:24 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||
05:09:29 [1] deployed, controller state=running, pinned={'onlyoffice': 'onlyoffice/documentserver:9.4.0'}
|
||||
deploy: True
|
||||
Mon Oct 19 03:09:29 UTC 2026
|
||||
@@ -0,0 +1,16 @@
|
||||
Mon Oct 19 03:09:40 UTC 2026
|
||||
onlyoffice health: healthy
|
||||
200
|
||||
{"ok": true, "data": {"enabled": false, "status": "disabled"}, "message": "Integráció kikapcsolva"}
|
||||
03:09:42 files.<domain> 200
|
||||
onlyoffice Up 28 seconds (healthy)
|
||||
filebrowser Up 6 hours (healthy)
|
||||
Mon Oct 19 03:10:35 UTC 2026
|
||||
200
|
||||
{"ok": true, "data": {"enabled": true, "enabled_at": "2026-10-19T03:10:36Z", "status": "active"}, "message": "Integráció engedélyezve"}
|
||||
03:10:39 files.<domain> 200 (try 1)
|
||||
filebrowser Up 5 seconds (healthy)
|
||||
onlyoffice Up About a minute (healthy)
|
||||
url: "https://office.enkisfelhom.hu"
|
||||
internalUrl: "http://onlyoffice:80"
|
||||
2
|
||||
@@ -0,0 +1,4 @@
|
||||
login 200 token_len 355
|
||||
resource 200 application/vnd.openxmlformats-officedocument.wordprocessingml.document
|
||||
03:12:19Z preview small HTTP 500 content-type application/json; charset=utf-8 bytes 167 in 0.0s b'{"status":500,"message":"failed to create image for PDF file: failed to open PDF from memory for file \'/srv/scratch_hdd/r1153/teszt.docx\': fitz: cannot open document"}'
|
||||
03:12:19Z preview large HTTP 500 content-type application/json; charset=utf-8 bytes 167 in 0.0s b'{"status":500,"message":"failed to create image for PDF file: failed to open PDF from memory for file \'/srv/scratch_hdd/r1153/teszt.docx\': fitz: cannot open document"}'
|
||||
@@ -0,0 +1,10 @@
|
||||
== /r1153/sample.docx
|
||||
login 200 token_len 355
|
||||
resource 200 application/vnd.openxmlformats-officedocument.wordprocessingml.document
|
||||
03:13:40Z preview small HTTP 200 content-type image/jpeg bytes 8714 in 0.3s
|
||||
03:13:40Z preview large HTTP 200 content-type image/jpeg bytes 22734 in 0.3s
|
||||
== /r1153/teszt.docx
|
||||
login 200 token_len 355
|
||||
resource 200 application/vnd.openxmlformats-officedocument.wordprocessingml.document
|
||||
03:13:40Z preview small HTTP 500 content-type application/json; charset=utf-8 bytes 167 in 0.0s b'{"status":500,"message":"failed to create image for PDF file: failed to open PDF from memory for file \'/srv/scratch_hdd/r1153/teszt.docx\': fitz: cannot open document"}'
|
||||
03:13:40Z preview large HTTP 500 content-type application/json; charset=utf-8 bytes 167 in 0.0s b'{"status":500,"message":"failed to create image for PDF file: failed to open PDF from memory for file \'/srv/scratch_hdd/r1153/teszt.docx\': fitz: cannot open document"}'
|
||||
@@ -0,0 +1,9 @@
|
||||
Mon Oct 19 03:13:53 UTC 2026
|
||||
{"ok": true, "data": {"enabled": false, "status": "disabled"}, "message": "Integráció kikapcsolva"}
|
||||
03:13:57 files.<domain> 200 (try 1)
|
||||
== link OFF, /r1153/sample.docx
|
||||
03:13:58Z preview small HTTP 200 content-type image/jpeg bytes 8714 in 0.2s
|
||||
03:13:58Z preview large HTTP 200 content-type image/jpeg bytes 22734 in 0.2s
|
||||
== link OFF, /r1153/teszt.docx
|
||||
03:13:58Z preview small HTTP 500 content-type application/json; charset=utf-8 bytes 167 in 0.0s b'{"status":500,"message":"failed to create image for PDF file: failed to open PDF from memory for file \'/srv/scratch_hdd/r1153/teszt.docx\': fitz: cannot open document"}'
|
||||
03:13:58Z preview large HTTP 500 content-type application/json; charset=utf-8 bytes 167 in 0.0s b'{"status":500,"message":"failed to create image for PDF file: failed to open PDF from memory for file \'/srv/scratch_hdd/r1153/teszt.docx\': fitz: cannot open document"}'
|
||||
@@ -0,0 +1,8 @@
|
||||
Mon Oct 19 03:14:09 UTC 2026
|
||||
test files removed
|
||||
05:14:54 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'onlyoffice', 'volumes_removed': ['onlyoffice_onlyoffice_data', 'onlyoffice_onlyoffice_logs'], 'hdd_paths_removed': [], 'hdd_pa
|
||||
05:15:03 [X] after remove: deployed=False leftovers='/opt/docker/stacks/onlyoffice'
|
||||
remove: 200
|
||||
0
|
||||
0
|
||||
ls: cannot access '/opt/docker/stacks/onlyoffice/app.yaml': No such file or directory
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 8.5 KiB |
@@ -0,0 +1,4 @@
|
||||
## per-customer floors 0.332.0 / MinAgent 0.158.1 — 03:16:43
|
||||
tester-1: HTTP/1.1 303 See Other Location: /customers/tester-1?flash=floor_set
|
||||
demo-hp: HTTP/1.1 303 See Other Location: /customers/demo-hp?flash=floor_set
|
||||
demo-felhom: HTTP/1.1 303 See Other Location: /customers/demo-felhom?flash=floor_set
|
||||
@@ -0,0 +1,14 @@
|
||||
03:16:55 demo-hp=0.332.0-rc.3 tester-1=0.332.0 demo-felhom=0.332.0
|
||||
03:17:28 demo-hp=0.332.0
|
||||
== hp
|
||||
2026/10/19 03:16:57 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="selfupdate-check" interval=6h0m0s totalJobs=23
|
||||
2026/10/19 03:17:04 updater.go:104: [DEBUG] [selfupdate] SetFloor: floor "" → "0.332.0"
|
||||
2026/10/19 03:17:04 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.332.0 >= floor 0.332.0 — no action
|
||||
== root@192.168.0.154
|
||||
2026/10/19 03:16:55 [INFO] [selfupdate] Post-update startup: update successful (0.331.0 → 0.332.0)
|
||||
2026/10/19 03:16:55 [INFO] [scheduler] Registered periodic job: selfupdate-check (every 6h0m0s)
|
||||
2026/10/19 03:17:26 [INFO] [selfupdate] Current version 0.332.0 is up to date
|
||||
== felhom-pve
|
||||
2026/10/19 03:16:45 [INFO] [selfupdate] Post-update startup: update successful (0.331.0 → 0.332.0)
|
||||
2026/10/19 03:16:45 [INFO] [scheduler] Registered periodic job: selfupdate-check (every 6h0m0s)
|
||||
2026/10/19 03:17:17 [INFO] [selfupdate] Current version 0.332.0 is up to date
|
||||
@@ -0,0 +1,33 @@
|
||||
=== demo-hp (enkisfelhom.hu) 03:17:58
|
||||
containers: 24 running; not healthy/running: traefik Up 3 days;
|
||||
unhealthy: 0
|
||||
2026/10/19 03:16:57 updater.go:855: [INFO] [selfupdate] Post-update startup: update successful (0.332.0-rc.3 → 0.332.0)
|
||||
allow-list refusals since start: 0
|
||||
2026/10/19 03:16:58 sync.go:142: [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
|
||||
installed apps (stacks with app.yaml): 12
|
||||
public https://felhom.enkisfelhom.hu/login: 200
|
||||
public https://files.enkisfelhom.hu/: 200
|
||||
=== demo-felhom (demo-felhom.eu) 03:18:03
|
||||
containers: 9 running; not healthy/running: traefik Up 3 days;
|
||||
unhealthy: 0
|
||||
2026/10/19 03:16:45 [INFO] [selfupdate] Post-update startup: update successful (0.331.0 → 0.332.0)
|
||||
allow-list refusals since start: 0
|
||||
2026/10/19 03:16:46 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
|
||||
installed apps (stacks with app.yaml): 5
|
||||
2026/10/19 03:16:45 [INFO] [infra] wrote filebrowser route → /opt/docker/stacks/traefik/dynamic/filebrowser.yml (Host files.demo-felhom.eu → filebrowser:80 by name, R-1126; one sign-in true, R-1100)
|
||||
2026/10/19 03:16:46 [INFO] [web] FileBrowser mounts synced (recreated) — 1 storage path(s), config updated
|
||||
2026/10/19 03:16:52 [INFO] [infra] filebrowser: default login refused for this run — joined traefik-public, files.<domain> served (R-1126)
|
||||
public https://felhom.demo-felhom.eu/login: 200
|
||||
public https://files.demo-felhom.eu/: 200
|
||||
=== tester-1 (enkicsifelhom.hu) 03:18:06
|
||||
containers: 10 running; not healthy/running: traefik Up 3 days;
|
||||
unhealthy: 0
|
||||
2026/10/19 03:16:55 [INFO] [selfupdate] Post-update startup: update successful (0.331.0 → 0.332.0)
|
||||
allow-list refusals since start: 0
|
||||
2026/10/19 03:16:56 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
|
||||
installed apps (stacks with app.yaml): 3
|
||||
2026/10/19 03:16:55 [INFO] [infra] wrote filebrowser route → /opt/docker/stacks/traefik/dynamic/filebrowser.yml (Host files.enkicsifelhom.hu → filebrowser:80 by name, R-1126; one sign-in true, R-1100)
|
||||
2026/10/19 03:16:56 [INFO] [web] FileBrowser mounts synced (recreated) — 5 storage path(s), config updated
|
||||
2026/10/19 03:17:02 [INFO] [infra] filebrowser: default login refused for this run — joined traefik-public, files.<domain> served (R-1126)
|
||||
public https://felhom.enkicsifelhom.hu/login: 200
|
||||
public https://files.enkicsifelhom.hu/: 200
|
||||
@@ -0,0 +1,33 @@
|
||||
control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -)
|
||||
== tester-1 (enkicsifelhom.hu): 6 public name(s)
|
||||
OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1
|
||||
== demo-hp (enkisfelhom.hu): 14 public name(s)
|
||||
OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1
|
||||
== demo-felhom (demo-felhom.eu): 8 public name(s)
|
||||
OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
|
||||
== every public name redirects http -> https
|
||||
@@ -0,0 +1,16 @@
|
||||
=== demo-hp (enkisfelhom.hu) 03:23:50
|
||||
dashboard sign-in: 302
|
||||
the Fajlok button goes to: https://files.enkisfelhom.hu/_felhom/sso
|
||||
the pass answer goes to: https://files.enkisfelhom.hu/files/; files cookie set: 1
|
||||
FileBrowser with the files cookie: felhom-household admin: False modify: True
|
||||
FileBrowser without it, a fresh browser: HTTP 401
|
||||
dashboard sign-out: 302
|
||||
after the sign-out, the files cookie sends the browser to: https://felhom.enkisfelhom.hu/__files/open?to=%2Ffiles%2F
|
||||
=== tester-1 (enkicsifelhom.hu) 03:23:51
|
||||
dashboard sign-in: 302
|
||||
the Fajlok button goes to: https://files.enkicsifelhom.hu/_felhom/sso
|
||||
the pass answer goes to: https://files.enkicsifelhom.hu/files/; files cookie set: 1
|
||||
FileBrowser with the files cookie: felhom-household admin: False modify: True
|
||||
FileBrowser without it, a fresh browser: HTTP 401
|
||||
dashboard sign-out: 302
|
||||
after the sign-out, the files cookie sends the browser to: https://felhom.enkicsifelhom.hu/__files/open?to=%2Ffiles%2F
|
||||
@@ -0,0 +1,7 @@
|
||||
=== demo-felhom (demo-felhom.eu) 03:24:06 — no dashboard password is held for this box: file and refusal checks only
|
||||
-rw-------
|
||||
valid shape: 1
|
||||
config carries the secret: 1, route carries it: 1
|
||||
forged secret name through traefik, no files cookie: HTTP 401
|
||||
from outside, old public name: HTTP 401
|
||||
from outside, the pass path with a forged pass: HTTP 403
|
||||
@@ -0,0 +1,16 @@
|
||||
## global floor 0.332.0 / 0.158.1 — 03:24:18
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /configuration?flash=floor_set
|
||||
## read back (DB with WAL) 03:24:18
|
||||
min_controller_version|0.332.0
|
||||
min_controller_version_declared_min_agent|0.332.0=0.158.1
|
||||
## per-box floors cleared — 03:24:28
|
||||
tester-1: HTTP/1.1 303 See Other Location: /customers/tester-1?flash=floor_set
|
||||
demo-hp: HTTP/1.1 303 See Other Location: /customers/demo-hp?flash=floor_set
|
||||
demo-felhom: HTTP/1.1 303 See Other Location: /customers/demo-felhom?flash=floor_set
|
||||
min_controller_declared_min_agent
|
||||
min_controller_set_at
|
||||
min_controller_version
|
||||
demo-felhom|''|''
|
||||
demo-hp|''|''
|
||||
tester-1|''|''
|
||||
@@ -0,0 +1,29 @@
|
||||
Mon Oct 19 02:14:15 UTC 2026
|
||||
application.argoproj.io/felhom patched
|
||||
deployment "hub" successfully rolled out
|
||||
sync=Synced health=Progressing rev=200ab6530bf2bec9a9b341e4bc8e2bfd09f6a609
|
||||
gitea.dooplex.hu/admin/felhom-hub:0.159.0
|
||||
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
|
||||
hub-67bff8bcb6-8pgxf 0/1 Running 0 9s 10.42.0.56 dooplex <none> <none>
|
||||
2026/10/19 04:14:21 [INFO] Offsite pool-box checker initialized: box=611714 fill warn=80% crit=90%, oversub warn=2.00x, unreachable after 3 consecutive failed reads, refresh 15m0s
|
||||
2026/10/19 04:14:21 [INFO] osupdates: approval rule = every ring-0 box healthy for 24h0m0s and 1 night run(s)
|
||||
2026/10/19 04:14:21 [INFO] osupdates: the Docker engine set is approved only by the operator, after 2 healthy ring-0 night(s)
|
||||
2026/10/19 04:14:21 [INFO] osupdates: alarms = no OS leg 168h0m0s, reboot needed 336h0m0s, ring 0 stalled 168h0m0s, not covered 336h0m0s
|
||||
2026/10/19 04:14:21 [INFO] Off-site key registrar enabled; daily key check at 07:10 Budapest
|
||||
2026/10/19 04:14:21 [INFO] access log: /data/access-log, one file per UTC day, kept 90 days
|
||||
2026/10/19 04:14:21 [INFO] Registry version checker started (every 6h)
|
||||
2026/10/19 04:14:21 [INFO] offsite-key-audit: next run at 2026-10-19 07:10 CEST (in 2h55m39s)
|
||||
2026/10/19 04:14:21 [INFO] WG peer-sync enabled (endpoint 167.233.158.164:22, user felhom-peersync)
|
||||
2026/10/19 04:14:21 [INFO] PBS DR tenantsync enabled (endpoint 167.233.158.164:22, user felhom-peersync; WG-registration auto-provision hook armed)
|
||||
2026/10/19 04:14:21 [INFO] PBS-DR box checker initialized: fill warn=80% crit=90%, unreachable after 3 consecutive failed reads, refresh 15m0s
|
||||
2026/10/19 04:14:21 [INFO] agent-plane poke enabled (endpoint 167.233.158.164:22, user felhom-peersync; web + api admin seams armed)
|
||||
2026/10/19 04:14:21 [INFO] PBS-DR self-heal reconciler started (interval 5m)
|
||||
2026/10/19 04:14:21 [INFO] offsite credential self-heal reconciler started (interval 5m, debounce 2 reports)
|
||||
2026/10/19 04:14:21 [DEBUG] Registry version check: latest = 0.331.0
|
||||
Mon Oct 19 02:15:36 UTC 2026
|
||||
sync=Synced health=Healthy
|
||||
hub-67bff8bcb6-8pgxf 1/1 Running 0 79s
|
||||
--- floor (DB with WAL):
|
||||
min_controller_version|0.331.0
|
||||
min_controller_version_declared_min_agent|0.331.0=0.158.1
|
||||
--- errors since start:
|
||||
@@ -0,0 +1,9 @@
|
||||
Mon Oct 19 02:32:34 UTC 2026
|
||||
hosts (agent reports) after the 02:14:21 start:
|
||||
demo-felhom-8363b5|0.160.0|2026-10-19 02:25:35
|
||||
demo-hp-bb76ea|0.160.0|2026-10-19 02:23:50
|
||||
tester-1-d70be4|0.160.0|2026-10-19 02:24:21
|
||||
controller reports:
|
||||
demo-felhom|2026-10-19 02:29:12
|
||||
demo-hp|2026-10-19 02:31:13
|
||||
tester-1|2026-10-19 02:29:13
|
||||
@@ -0,0 +1,11 @@
|
||||
control (before the window): jobs with run_attempt>1: 17
|
||||
('felhom-controller', 1401, 2, '2026-10-06T00:44:57Z', 'success')
|
||||
('felhom-controller', 1357, 2, '2026-10-05T15:05:37Z', 'success')
|
||||
('felhom-controller', 1549, 2, '2026-10-08T08:26:11Z', 'success')
|
||||
('felhom-controller', 1639, 2, '2026-10-10T13:09:06Z', 'success')
|
||||
('felhom-agent', 1579, 2, '2026-10-09T04:58:15Z', 'success')
|
||||
('app-catalog-felhom.eu', 1368, 2, '2026-10-05T17:15:12Z', 'success')
|
||||
window since 2026-10-12T12:00:00Z : jobs with run_attempt>1: 3
|
||||
('felhom-controller', 1944, 2, '2026-10-16T13:50:35Z', 'success')
|
||||
('felhom-controller', 1959, 2, '2026-10-16T17:07:53Z', 'success')
|
||||
('felhom-controller', 1991, 2, '2026-10-17T09:59:02Z', 'success')
|
||||
@@ -0,0 +1,4 @@
|
||||
felhom.eu 1645 log bytes 38590
|
||||
felhom.eu 1646 log bytes 38503
|
||||
felhom-controller 1639 log bytes 18965
|
||||
felhom.eu 1738 log bytes 46399
|
||||
@@ -0,0 +1,26 @@
|
||||
window since 2026-10-12T12:00:00Z: 381 jobs in 4 repos; not-success 25; with no log (lost) 0
|
||||
not-success ('felhom.eu', 1738, 'c2a6cf7a', 'completed', 'failure', '2026-10-13T04:47:12Z', 1) has log
|
||||
not-success ('felhom.eu', 1747, 'ed59bf3c', 'completed', 'failure', '2026-10-13T17:28:57Z', 1) has log
|
||||
not-success ('felhom.eu', 1796, 'a2ca6eb4', 'completed', 'failure', '2026-10-14T16:51:48Z', 1) has log
|
||||
not-success ('felhom.eu', 1889, '6749118c', 'completed', 'failure', '2026-10-15T19:21:34Z', 1) has log
|
||||
not-success ('felhom.eu', 1993, 'b9c4f73b', 'completed', 'failure', '2026-10-17T10:02:08Z', 1) has log
|
||||
not-success ('felhom.eu', 1997, '498b9139', 'completed', 'failure', '2026-10-17T10:42:11Z', 1) has log
|
||||
not-success ('felhom-controller', 1865, 'e66d4669', 'completed', 'failure', '2026-10-15T16:37:58Z', 1) has log
|
||||
not-success ('felhom-controller', 1867, 'fe0228c8', 'completed', 'failure', '2026-10-15T17:05:00Z', 1) has log
|
||||
not-success ('felhom-controller', 1869, 'db52f5a4', 'completed', 'failure', '2026-10-15T17:27:25Z', 1) has log
|
||||
not-success ('felhom-controller', 1870, '83392064', 'completed', 'failure', '2026-10-15T17:32:48Z', 1) has log
|
||||
not-success ('felhom-controller', 1871, '9cc7ffcf', 'completed', 'failure', '2026-10-15T18:08:11Z', 1) has log
|
||||
not-success ('felhom-controller', 1873, '7f3e58ff', 'completed', 'failure', '2026-10-15T18:30:00Z', 1) has log
|
||||
not-success ('felhom-controller', 1875, '48d86748', 'completed', 'failure', '2026-10-15T18:33:27Z', 1) has log
|
||||
not-success ('felhom-controller', 1876, '006ceab5', 'completed', 'failure', '2026-10-15T18:41:12Z', 1) has log
|
||||
not-success ('felhom-controller', 1877, 'f908eee2', 'completed', 'failure', '2026-10-15T18:45:59Z', 1) has log
|
||||
not-success ('felhom-controller', 1878, '02252556', 'completed', 'failure', '2026-10-15T18:46:27Z', 1) has log
|
||||
not-success ('felhom-controller', 1879, '9e68e526', 'completed', 'failure', '2026-10-15T18:51:46Z', 1) has log
|
||||
not-success ('felhom-controller', 1880, '8c7aa10a', 'completed', 'failure', '2026-10-15T18:52:08Z', 1) has log
|
||||
not-success ('felhom-controller', 1882, 'ec438479', 'completed', 'failure', '2026-10-15T18:57:00Z', 1) has log
|
||||
not-success ('felhom-controller', 1883, 'f9151f08', 'completed', 'failure', '2026-10-15T19:04:54Z', 1) has log
|
||||
not-success ('felhom-controller', 1884, '4261a38b', 'completed', 'failure', '2026-10-15T19:05:47Z', 1) has log
|
||||
not-success ('felhom-controller', 1887, 'c6a56712', 'completed', 'failure', '2026-10-15T19:09:40Z', 1) has log
|
||||
not-success ('felhom-controller', 1990, '334df7db', 'completed', 'failure', '2026-10-17T09:53:24Z', 1) has log
|
||||
not-success ('felhom-controller', 2036, 'e30091bf', 'completed', 'failure', '2026-10-17T15:30:09Z', 1) has log
|
||||
not-success ('app-catalog-felhom.eu', 2067, 'c97eb1c9', 'completed', 'failure', '2026-10-18T18:45:53Z', 1) has log
|
||||
@@ -0,0 +1,16 @@
|
||||
task fields: ['created_at', 'display_title', 'event', 'head_branch', 'head_sha', 'id', 'name', 'run_number', 'run_started_at', 'status', 'updated_at', 'url', 'workflow_id']
|
||||
{'id': 1991, 'name': 'gates', 'head_sha': 'b69585789a9fa34a7c33faf24da44926f166a9da', 'run_number': 393, 'status': 'success', 'event': 'push', 'created_at': '2026-10-16T21:50:09Z', 'run_started_at': '2026-10-16T21:50:09Z', 'updated_at': '2026-10-16T21:50:58Z'}
|
||||
{'id': 1959, 'name': 'gates', 'head_sha': 'f30a80841e372abf9d7992dc9eb3eec607078e90', 'run_number': 379, 'status': 'success', 'event': 'push', 'created_at': '2026-10-16T13:19:24Z', 'run_started_at': '2026-10-16T13:19:24Z', 'updated_at': '2026-10-16T13:20:14Z'}
|
||||
{'id': 1639, 'name': 'gates', 'head_sha': '1cb43a21a2dc5cd44fbcc0749b460dba1ffa8088', 'run_number': 326, 'status': 'success', 'event': 'push', 'created_at': '2026-10-10T07:55:47Z', 'run_started_at': '2026-10-10T07:55:47Z', 'updated_at': '2026-10-10T07:56:52Z'}
|
||||
job 1944 sha c02c9d83 attempt 2 created 2026-10-16T13:50:35Z started 2026-10-16T14:02:29Z completed 2026-10-16T14:03:21Z
|
||||
job 1959 sha 29a3ce8c attempt 2 created 2026-10-16T17:07:53Z started 2026-10-16T17:19:08Z completed 2026-10-16T17:20:00Z
|
||||
job 1991 sha 3e8d0276 attempt 2 created 2026-10-17T09:59:02Z started 2026-10-17T10:15:35Z completed 2026-10-17T10:16:23Z
|
||||
task 2015 for job 1991 success 2026-10-17T10:15:35Z -> 2026-10-17T10:16:24Z
|
||||
task 2011 for job 1991 failure 2026-10-17T09:59:02Z -> 2026-10-17T09:59:49Z
|
||||
task 1980 for job 1959 success 2026-10-16T17:19:08Z -> 2026-10-16T17:20:00Z
|
||||
task 1978 for job 1959 failure 2026-10-16T17:07:54Z -> 2026-10-16T17:08:42Z
|
||||
task 1964 for job 1944 success 2026-10-16T14:02:29Z -> 2026-10-16T14:03:21Z
|
||||
task 1962 for job 1944 failure 2026-10-16T13:50:35Z -> 2026-10-16T13:51:24Z
|
||||
CONTROL job 1639 (a known lost attempt, 2026-10-10) sha 9e5b8814
|
||||
task 1655 success 2026-10-10T13:22:35Z -> 2026-10-10T13:23:51Z
|
||||
task 1653 failure 2026-10-10T13:09:10Z -> 2026-10-10T13:20:58Z
|
||||
@@ -0,0 +1,39 @@
|
||||
== DooPlex PBS (read 2026-10-19)
|
||||
[
|
||||
{
|
||||
"comment": "decision 71: keep the last 8 weekly copies per group (all namespaces); runs after the 05:00 pull",
|
||||
"id": "prune-ep0-copy",
|
||||
"keep-weekly": 8,
|
||||
"schedule": "07:30",
|
||||
"store": "ep0-copy"
|
||||
}
|
||||
]
|
||||
[
|
||||
{
|
||||
"comment": "decision 70: nightly copy of ep0 felhom-offsite; never removes what ep0 removed",
|
||||
"id": "ep0-felhom-offsite",
|
||||
"remote": "ep0",
|
||||
"remote-store": "felhom-offsite",
|
||||
"remove-vanished": false,
|
||||
"schedule": "05:00",
|
||||
"store": "ep0-copy"
|
||||
}
|
||||
]
|
||||
/mnt/5_hdd/backup/ep0-copy/ns/Tester-2/ct/9201/2026-10-04T16:31:13Z
|
||||
/mnt/5_hdd/backup/ep0-copy/ns/demo-felhom/ct/9201/2026-09-22T04:12:20Z
|
||||
/mnt/5_hdd/backup/ep0-copy/ns/demo-felhom/ct/9201/2026-09-29T04:16:43Z
|
||||
/mnt/5_hdd/backup/ep0-copy/ns/demo-felhom/ct/9201/2026-10-06T04:21:09Z
|
||||
/mnt/5_hdd/backup/ep0-copy/ns/demo-felhom/ct/9201/2026-10-13T04:22:51Z
|
||||
/mnt/5_hdd/backup/ep0-copy/ns/demo-hp/ct/9201/2026-09-24T20:06:25Z
|
||||
/mnt/5_hdd/backup/ep0-copy/ns/demo-hp/ct/9201/2026-10-01T20:15:29Z
|
||||
/mnt/5_hdd/backup/ep0-copy/ns/demo-hp/ct/9201/2026-10-08T20:34:27Z
|
||||
/mnt/5_hdd/backup/ep0-copy/ns/demo-hp/ct/9201/2026-10-15T20:38:47Z
|
||||
/mnt/5_hdd/backup/ep0-copy/ns/operator/host/dooplex-gitea/2026-10-10T22:22:52Z
|
||||
/mnt/5_hdd/backup/ep0-copy/ns/operator/host/dooplex-gitea/2026-10-17T22:22:32Z
|
||||
/mnt/5_hdd/backup/ep0-copy/ns/operator/host/dooplex-gitea/2026-10-18T22:22:01Z
|
||||
/mnt/5_hdd/backup/ep0-copy/ns/operator/host/dooplex-hub/2026-10-11T00:30:55Z
|
||||
/mnt/5_hdd/backup/ep0-copy/ns/operator/host/dooplex-hub/2026-10-18T00:31:48Z
|
||||
/mnt/5_hdd/backup/ep0-copy/ns/operator/host/dooplex-hub/2026-10-19T00:32:07Z
|
||||
/mnt/5_hdd/backup/ep0-copy/ns/tester-1/ct/9201/2026-10-04T19:57:18Z
|
||||
/mnt/5_hdd/backup/ep0-copy/ns/tester-1/ct/9201/2026-10-11T20:02:13Z
|
||||
/mnt/5_hdd/backup/ep0-copy/ns/tester-1/ct/9201/2026-10-18T20:04:51Z
|
||||
@@ -0,0 +1,5 @@
|
||||
box | job | snapshots today | keep-last 2 keeps / removes | keep-weekly 2 keeps / removes
|
||||
Tester-2 | NO JOB | 1 | 2026-10-04 (w40) / none | 2026-10-04 (w40) / none
|
||||
demo-felhom | prune-demo-felhom | 2 | 2026-10-13 (w42), 2026-10-06 (w41) / none | 2026-10-13 (w42), 2026-10-06 (w41) / none
|
||||
demo-hp | prune-demo-hp | 2 | 2026-10-15 (w42), 2026-10-08 (w41) / none | 2026-10-15 (w42), 2026-10-08 (w41) / none
|
||||
tester-1 | NO JOB | 3 | 2026-10-18 (w42), 2026-10-11 (w41) / 2026-10-04 (w40) | 2026-10-18 (w42), 2026-10-11 (w41) / 2026-10-04 (w40)
|
||||
@@ -0,0 +1,31 @@
|
||||
Mon Oct 19 07:49:11 UTC 2026
|
||||
[
|
||||
{
|
||||
"comment": "R-1162 operator D1=A 2026-10-19: one copy per ISO week, 2 weeks, server-side (box tokens are write-only)",
|
||||
"id": "prune-demo-hp",
|
||||
"keep-weekly": 2,
|
||||
"max-depth": 0,
|
||||
"ns": "demo-hp",
|
||||
"schedule": "03:30",
|
||||
"store": "felhom-offsite"
|
||||
},
|
||||
{
|
||||
"comment": "R-1162 operator D1=A 2026-10-19: one copy per ISO week, 2 weeks, server-side (box tokens are write-only)",
|
||||
"id": "prune-demo-felhom",
|
||||
"keep-weekly": 2,
|
||||
"max-depth": 0,
|
||||
"ns": "demo-felhom",
|
||||
"schedule": "03:30",
|
||||
"store": "felhom-offsite"
|
||||
},
|
||||
{
|
||||
"comment": "R-173 hub DB copies; operator ns only",
|
||||
"id": "prune-operator-hubdb",
|
||||
"keep-daily": 14,
|
||||
"keep-weekly": 8,
|
||||
"max-depth": 0,
|
||||
"ns": "operator",
|
||||
"schedule": "03:45",
|
||||
"store": "felhom-offsite"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,40 @@
|
||||
felhom-hetzner
|
||||
proxmox-backup-server 4.2.8-1 running version: 4.2.5
|
||||
==prune-jobs
|
||||
[
|
||||
{
|
||||
"comment": "R-173 hub DB copies; operator ns only",
|
||||
"id": "prune-operator-hubdb",
|
||||
"keep-daily": 14,
|
||||
"keep-weekly": 8,
|
||||
"max-depth": 0,
|
||||
"ns": "operator",
|
||||
"schedule": "03:45",
|
||||
"store": "felhom-offsite"
|
||||
},
|
||||
{
|
||||
"comment": "R-82 retention keep-last=2, server-side (box tokens are write-only)",
|
||||
"id": "prune-demo-hp",
|
||||
"keep-last": 2,
|
||||
"max-depth": 0,
|
||||
"ns": "demo-hp",
|
||||
"schedule": "03:30",
|
||||
"store": "felhom-offsite"
|
||||
},
|
||||
{
|
||||
"comment": "R-82 retention keep-last=2, server-side (box tokens are write-only)",
|
||||
"id": "prune-demo-felhom",
|
||||
"keep-last": 2,
|
||||
"max-depth": 0,
|
||||
"ns": "demo-felhom",
|
||||
"schedule": "03:30",
|
||||
"store": "felhom-offsite"
|
||||
}
|
||||
]
|
||||
==datastore
|
||||
{
|
||||
"gc-schedule": "sun 04:30",
|
||||
"name": "felhom-offsite",
|
||||
"path": "/mnt/pbs-datastore",
|
||||
"verify-new": true
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
ns/Tester-2/ct/9201/2026-10-04T16:31:13Z
|
||||
ns/demo-felhom/ct/9201/2026-10-06T04:21:09Z
|
||||
ns/demo-felhom/ct/9201/2026-10-13T04:22:51Z
|
||||
ns/demo-hp/ct/9201/2026-10-08T20:34:27Z
|
||||
ns/demo-hp/ct/9201/2026-10-15T20:38:47Z
|
||||
ns/tester-1/ct/9201/2026-10-04T19:57:18Z
|
||||
ns/tester-1/ct/9201/2026-10-11T20:02:13Z
|
||||
ns/tester-1/ct/9201/2026-10-18T20:04:51Z
|
||||
@@ -0,0 +1,14 @@
|
||||
== ns/demo-felhom
|
||||
ns/demo-felhom/ct/9201/2026-10-06T04:21:09Z
|
||||
ns/demo-felhom/ct/9201/2026-10-13T04:22:51Z
|
||||
== ns/demo-hp
|
||||
ns/demo-hp/ct/9201/2026-10-08T20:34:27Z
|
||||
ns/demo-hp/ct/9201/2026-10-15T20:38:47Z
|
||||
== ns/operator
|
||||
== ns/tester-1
|
||||
ns/tester-1/ct/9201/2026-10-04T19:57:18Z
|
||||
ns/tester-1/ct/9201/2026-10-11T20:02:13Z
|
||||
ns/tester-1/ct/9201/2026-10-18T20:04:51Z
|
||||
== ns/Tester-2
|
||||
ns/Tester-2/ct/9201/2026-10-04T16:31:13Z
|
||||
== root
|
||||
@@ -0,0 +1,24 @@
|
||||
== demo-hp --keep-last 2 (PBS dry-run)
|
||||
2026-10-08T20:34:27Z keep
|
||||
2026-10-15T20:38:47Z keep
|
||||
== demo-hp --keep-weekly 2 (PBS dry-run)
|
||||
2026-10-08T20:34:27Z keep
|
||||
2026-10-15T20:38:47Z keep
|
||||
== demo-felhom --keep-last 2 (PBS dry-run)
|
||||
2026-10-06T04:21:09Z keep
|
||||
2026-10-13T04:22:51Z keep
|
||||
== demo-felhom --keep-weekly 2 (PBS dry-run)
|
||||
2026-10-06T04:21:09Z keep
|
||||
2026-10-13T04:22:51Z keep
|
||||
== tester-1 --keep-last 2 (PBS dry-run)
|
||||
2026-10-04T19:57:18Z REMOVE
|
||||
2026-10-11T20:02:13Z keep
|
||||
2026-10-18T20:04:51Z keep
|
||||
== tester-1 --keep-weekly 2 (PBS dry-run)
|
||||
2026-10-04T19:57:18Z REMOVE
|
||||
2026-10-11T20:02:13Z keep
|
||||
2026-10-18T20:04:51Z keep
|
||||
== Tester-2 --keep-last 2 (PBS dry-run)
|
||||
2026-10-04T16:31:13Z keep
|
||||
== Tester-2 --keep-weekly 2 (PBS dry-run)
|
||||
2026-10-04T16:31:13Z keep
|
||||
@@ -0,0 +1,32 @@
|
||||
# Offline model of PBS's prune selection (pbs-datastore prune.rs compute_prune_info / mark_selections),
|
||||
# run over ep0's snapshot list. Never touches ep0. ep0's clock is UTC (timedatectl), and PBS keys
|
||||
# weekly buckets by local time "%G/%V" (ISO year/week).
|
||||
import sys, datetime
|
||||
snaps = {}
|
||||
for line in open(sys.argv[1]):
|
||||
line=line.strip()
|
||||
if line.startswith('ns/') and '/ct/' in line:
|
||||
ns = line.split('/')[1]; t = line.rsplit('/',1)[1]
|
||||
snaps.setdefault(ns, []).append(t)
|
||||
def sel_last(t): return t
|
||||
def sel_week(t):
|
||||
d = datetime.datetime.strptime(t, '%Y-%m-%dT%H:%M:%SZ'); y,w,_ = d.isocalendar(); return f"{y}/{w:02d}"
|
||||
def compute(lst, opts):
|
||||
lst = sorted(lst, reverse=True); mark = {}
|
||||
for keep, sel in opts:
|
||||
already = {sel(t) for t in lst if mark.get(t)=='keep'}; inc=set()
|
||||
for t in lst:
|
||||
if t in mark: continue
|
||||
s = sel(t)
|
||||
if s in already: continue
|
||||
if s not in inc:
|
||||
if len(inc) >= keep: break
|
||||
inc.add(s); mark[t]='keep'
|
||||
else: mark[t]='remove'
|
||||
return {t: mark.get(t,'remove') for t in lst}
|
||||
jobs = {'demo-hp':'prune-demo-hp','demo-felhom':'prune-demo-felhom'}
|
||||
print("box | job | snapshots today | keep-last 2 keeps / removes | keep-weekly 2 keeps / removes")
|
||||
for ns in sorted(snaps):
|
||||
a = compute(snaps[ns], [(2, sel_last)]); b = compute(snaps[ns], [(2, sel_week)])
|
||||
f = lambda m,k: ', '.join(f"{t[:10]} (w{sel_week(t)[5:]})" for t,v in m.items() if v==k) or 'none'
|
||||
print(f"{ns} | {jobs.get(ns,'NO JOB')} | {len(snaps[ns])} | {f(a,'keep')} / {f(a,'remove')} | {f(b,'keep')} / {f(b,'remove')}")
|
||||
@@ -0,0 +1,24 @@
|
||||
=== RUN TestR1159_Measure
|
||||
r1159_measure_test.go:85: GET /config/c1 (passphrase) -> 200; hub fields: [api_key enabled url]; carries the controller key: true
|
||||
r1159_measure_test.go:86: GET /config/c1 (wrong passphrase) -> 401
|
||||
r1159_measure_test.go:90: POST /host-enroll (passphrase) -> 200; returns the EXISTING host's key: true
|
||||
r1159_measure_test.go:105: POST /offsite/register-key/c1 controller key -> 200, wrong key -> 401
|
||||
r1159_measure_test.go:105: POST /offsite/confirm-key/c1 controller key -> 200, wrong key -> 401
|
||||
r1159_measure_test.go:105: POST /offsite/move-aside/c1 controller key -> 200, wrong key -> 401
|
||||
r1159_measure_test.go:105: POST /offsite/window-open/c1 controller key -> 200, wrong key -> 401
|
||||
r1159_measure_test.go:105: POST /offsite/abandon-request/c1 controller key -> 200, wrong key -> 401
|
||||
r1159_measure_test.go:105: POST /offsite/abandon-cancel/c1 controller key -> 200, wrong key -> 401
|
||||
r1159_measure_test.go:105: POST /offsite/abandon-status/c1 controller key -> 200, wrong key -> 401
|
||||
r1159_measure_test.go:121: GET /hosts/h1/escrow host key -> 200, wrong key -> 401
|
||||
r1159_measure_test.go:121: GET /hosts/h1/escrow/retained host key -> 200, wrong key -> 401
|
||||
r1159_measure_test.go:121: PUT /hosts/h1/escrow host key -> 400, wrong key -> 401
|
||||
r1159_measure_test.go:121: POST /hosts/h1/wg host key -> 400, wrong key -> 401
|
||||
r1159_measure_test.go:121: POST /hosts/h1/pbs/consume-token host key -> 404, wrong key -> 401
|
||||
r1159_measure_test.go:121: GET /hosts/h1/desired-state host key -> 200, wrong key -> 401
|
||||
r1159_measure_test.go:121: GET /hosts/h1/jobs host key -> 200, wrong key -> 401
|
||||
r1159_measure_test.go:121: PUT /hosts/h1/recovery-credential host key -> 400, wrong key -> 401
|
||||
r1159_measure_test.go:121: POST /host-report host key -> 200, wrong key -> 401
|
||||
r1159_measure_test.go:123: off-site service calls reached with the passphrase-derived key: RegisterKey, ConfirmKey, MoveAside, OpenWindowFor, RequestAbandon, CancelAbandon, AbandonStatusFor
|
||||
--- PASS: TestR1159_Measure (0.04s)
|
||||
PASS
|
||||
ok gitea.dooplex.hu/admin/felhom-hub/internal/api 0.065s
|
||||
@@ -0,0 +1,124 @@
|
||||
package api
|
||||
|
||||
// R-1159 MEASUREMENT HARNESS (temporary, not committed): what a holder of the household's retrieval passphrase ALONE
|
||||
// can reach through the real hub router. Field names and status codes only.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/offsitekeys"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
type r1159Tpl struct{}
|
||||
|
||||
func (r1159Tpl) Template() string { return "hub: {}\n" }
|
||||
|
||||
type r1159Keys struct{ calls []string }
|
||||
|
||||
func (f *r1159Keys) RegisterKey(_ context.Context, c, _ string) (offsitekeys.InstallResult, error) {
|
||||
f.calls = append(f.calls, "RegisterKey")
|
||||
return offsitekeys.InstallResult{Fingerprint: "SHA256:x"}, nil
|
||||
}
|
||||
func (f *r1159Keys) ConfirmKey(_ context.Context, c, _ string) (int, error) {
|
||||
f.calls = append(f.calls, "ConfirmKey")
|
||||
return 1, nil
|
||||
}
|
||||
func (f *r1159Keys) MoveAside(_ context.Context, c string) (string, error) {
|
||||
f.calls = append(f.calls, "MoveAside")
|
||||
return "repo.orphaned-x", nil
|
||||
}
|
||||
func (f *r1159Keys) OpenWindowFor(_ context.Context, c string, _ int) (offsitekeys.WindowGrant, error) {
|
||||
f.calls = append(f.calls, "OpenWindowFor")
|
||||
return offsitekeys.WindowGrant{}, nil
|
||||
}
|
||||
func (f *r1159Keys) CloseWindowFor(_ context.Context, c string, _ offsitekeys.WindowResult) error {
|
||||
f.calls = append(f.calls, "CloseWindowFor")
|
||||
return nil
|
||||
}
|
||||
func (f *r1159Keys) RequestAbandon(_ context.Context, c, _ string) (offsitekeys.AbandonStatus, error) {
|
||||
f.calls = append(f.calls, "RequestAbandon")
|
||||
return offsitekeys.AbandonStatus{State: "pending"}, nil
|
||||
}
|
||||
func (f *r1159Keys) CancelAbandon(c, _ string) (int, error) {
|
||||
f.calls = append(f.calls, "CancelAbandon")
|
||||
return 1, nil
|
||||
}
|
||||
func (f *r1159Keys) AbandonStatusFor(c string) (offsitekeys.AbandonStatus, error) {
|
||||
f.calls = append(f.calls, "AbandonStatusFor")
|
||||
return offsitekeys.AbandonStatus{State: "none"}, nil
|
||||
}
|
||||
|
||||
func TestR1159_Measure(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
h.templateProvider = r1159Tpl{}
|
||||
fk := &r1159Keys{}
|
||||
h.SetOffsiteKeyService(fk)
|
||||
if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ckey-1159", RetrievalPassword: "owner-pass"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY-1159"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Step 1: the passphrase alone.
|
||||
rr := r879Get(h, "/config/c1", "owner-pass")
|
||||
var y map[string]any
|
||||
_ = yaml.Unmarshal(rr.Body.Bytes(), &y)
|
||||
var hubFields []string
|
||||
if hm, ok := y["hub"].(map[string]any); ok {
|
||||
for k := range hm {
|
||||
hubFields = append(hubFields, k)
|
||||
}
|
||||
}
|
||||
sort.Strings(hubFields)
|
||||
key := ""
|
||||
if hm, ok := y["hub"].(map[string]any); ok {
|
||||
key, _ = hm["api_key"].(string)
|
||||
}
|
||||
t.Logf("GET /config/c1 (passphrase) -> %d; hub fields: %v; carries the controller key: %v", rr.Code, hubFields, key == "ckey-1159")
|
||||
t.Logf("GET /config/c1 (wrong passphrase) -> %d", r879Get(h, "/config/c1", "nope").Code)
|
||||
er := doEnroll(h, "c1", "owner-pass")
|
||||
var e enrollResp
|
||||
_ = json.Unmarshal(er.Body.Bytes(), &e)
|
||||
t.Logf("POST /host-enroll (passphrase) -> %d; returns the EXISTING host's key: %v", er.Code, e.APIKey == "HKEY-1159")
|
||||
|
||||
// Step 2: the controller key taken from the config.
|
||||
routes := []struct{ path, body string }{
|
||||
{"/offsite/register-key/c1", `{"public_key":"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl x"}`},
|
||||
{"/offsite/confirm-key/c1", `{"fingerprint":"SHA256:x"}`},
|
||||
{"/offsite/move-aside/c1", `{}`},
|
||||
{"/offsite/window-open/c1", `{"count_before":10}`},
|
||||
{"/offsite/abandon-request/c1", `{"path":"repo.orphaned-x"}`},
|
||||
{"/offsite/abandon-cancel/c1", `{}`},
|
||||
{"/offsite/abandon-status/c1", `{}`},
|
||||
}
|
||||
for _, r := range routes {
|
||||
ok := do(h, http.MethodPost, r.path, key, r.body)
|
||||
bad := do(h, http.MethodPost, r.path, "ckey-wrong", r.body)
|
||||
t.Logf("POST %-30s controller key -> %d, wrong key -> %d", r.path, ok.Code, bad.Code)
|
||||
}
|
||||
hk := e.APIKey
|
||||
for _, r := range []struct{ m, path, body string }{
|
||||
{"GET", "/hosts/h1/escrow", ""},
|
||||
{"GET", "/hosts/h1/escrow/retained", ""},
|
||||
{"PUT", "/hosts/h1/escrow", `{}`},
|
||||
{"POST", "/hosts/h1/wg", `{}`},
|
||||
{"POST", "/hosts/h1/pbs/consume-token", `{}`},
|
||||
{"GET", "/hosts/h1/desired-state", ""},
|
||||
{"GET", "/hosts/h1/jobs", ""},
|
||||
{"PUT", "/hosts/h1/recovery-credential", `{}`},
|
||||
{"POST", "/host-report", validReportBody("h1")},
|
||||
} {
|
||||
ok := do(h, r.m, r.path, hk, r.body)
|
||||
bad := do(h, r.m, r.path, "HKEY-wrong", r.body)
|
||||
t.Logf("%-4s %-32s host key -> %d, wrong key -> %d", r.m, r.path, ok.Code, bad.Code)
|
||||
}
|
||||
t.Logf("off-site service calls reached with the passphrase-derived key: %s", strings.Join(fk.calls, ", "))
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
ARP watch for 192.168.0.104 (demo-hp vmbr0 7c:d3:0a:77:d9:76), from 09:58:07 UTC
|
||||
09:58:07 mac=7c:d3:0a:77:d9:76 ssh=open
|
||||
10:02:18 mac=b8:06:0d:60:45:58 ssh=shut
|
||||
10:02:58 mac=7c:d3:0a:77:d9:76 ssh=open
|
||||
10:03:38 mac=b8:06:0d:60:45:58 ssh=shut
|
||||
10:08:59 mac=7c:d3:0a:77:d9:76 ssh=open
|
||||
10:10:19 mac=b8:06:0d:60:45:58 ssh=shut
|
||||
10:12:50 mac=7c:d3:0a:77:d9:76 ssh=open
|
||||
10:17:10 mac=b8:06:0d:60:45:58 ssh=shut
|
||||
10:17:50 mac=7c:d3:0a:77:d9:76 ssh=open
|
||||
10:19:21 mac=b8:06:0d:60:45:58 ssh=shut
|
||||
stopped by CC at 10:20:37 UTC (enough to show the flips)
|
||||
@@ -0,0 +1,4 @@
|
||||
FileBrowser 1.5.6-stable on demo-hp 9201, 2026-10-19
|
||||
index.html links: <link rel="manifest" href="/public/static/site.webmanifest" crossorigin="use-credentials">
|
||||
manifest: 200 application/manifest+json, display=standalone
|
||||
PWA config keys in the binary: disablePWAInstall only (strings | grep -i pwa/manifest)
|
||||
@@ -0,0 +1,10 @@
|
||||
jwt alg: HS256 | claim names: ['Permissions', 'belongsTo', 'exp', 'iat', 'iss']
|
||||
nested Permissions -> ['admin', 'api', 'create', 'delete', 'download', 'modify', 'realtime', 'share']
|
||||
exp-iat/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/649cb06d-b4ca-49f2-9e2c-16c8bc36531f/scrat seconds: 7200
|
||||
identity claims: {'belongsTo': 3}
|
||||
signed in: cookie-token alone -> HTTP chpad/jwt_probe3.sh: line 23: T3: unbound variable
|
||||
/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/649cb06d-b4ca-49f2-9e2c-16c8bc36531f/scrat401; bearer alone -> HTTP
|
||||
dashboard sign-out: 302
|
||||
after sign-out: coochpad/jwt_probe3.sh: line 23: T3: unbound variable
|
||||
/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/649cb06d-b4ca-49f2-9e2c-16c8bc36531f/scratkie-token alone -> HTTP 401; bearer alone -> HTTP
|
||||
chpad/jwt_probe3.sh: line 28: T3: unbound variable
|
||||
@@ -0,0 +1,5 @@
|
||||
with the files cookie (the household's normal path): HTTP 200
|
||||
token alone, still signed in: cookie 401 · bearer 401 · basic 401 · ?auth= 401
|
||||
dashboard sign-out: 302
|
||||
token alone, after sign-out: cookie 401 · bearer 401 · basic 401 · ?auth= 401
|
||||
FileBrowser's own sign-in page, no token: HTTP 200
|
||||
@@ -0,0 +1,27 @@
|
||||
=== demo-hp (enkisfelhom.hu) 10:15:18
|
||||
containers: 24 running; not healthy/running: traefik Up 3 days;
|
||||
unhealthy: 0
|
||||
2026/10/19 10:13:21 updater.go:855: [INFO] [selfupdate] Post-update startup: update successful (0.333.0-rc.2 → 0.333.0)
|
||||
allow-list refusals since start: 0
|
||||
2026/10/19 10:13:21 sync.go:142: [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
|
||||
installed apps (stacks with app.yaml): 12
|
||||
public https://felhom.enkisfelhom.hu/: 302
|
||||
public https://files.enkisfelhom.hu/: 200
|
||||
=== demo-felhom (demo-felhom.eu) 10:15:20
|
||||
containers: 9 running; not healthy/running: traefik Up 3 days;
|
||||
unhealthy: 0
|
||||
2026/10/19 09:59:39 [INFO] [selfupdate] Post-update startup: update successful (0.332.0 → 0.333.0)
|
||||
allow-list refusals since start: 0
|
||||
2026/10/19 09:59:40 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
|
||||
installed apps (stacks with app.yaml): 5
|
||||
public https://felhom.demo-felhom.eu/: 302
|
||||
public https://files.demo-felhom.eu/: 200
|
||||
=== tester-1 (enkicsifelhom.hu) 10:15:22
|
||||
containers: 10 running; not healthy/running: traefik Up 3 days;
|
||||
unhealthy: 0
|
||||
2026/10/19 09:59:48 [INFO] [selfupdate] Post-update startup: update successful (0.332.0 → 0.333.0)
|
||||
allow-list refusals since start: 0
|
||||
2026/10/19 09:59:49 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
|
||||
installed apps (stacks with app.yaml): 3
|
||||
public https://felhom.enkicsifelhom.hu/: 302
|
||||
public https://files.enkicsifelhom.hu/: 200
|
||||
@@ -0,0 +1,33 @@
|
||||
control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -)
|
||||
== tester-1 (enkicsifelhom.hu): 6 public name(s)
|
||||
OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1
|
||||
== demo-hp (enkisfelhom.hu): 14 public name(s)
|
||||
OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1
|
||||
== demo-felhom (demo-felhom.eu): 8 public name(s)
|
||||
OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
|
||||
== every public name redirects http -> https
|
||||
@@ -0,0 +1,26 @@
|
||||
== demo-hp
|
||||
sec: cannot run /tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/649cb06d-b4ca-49f2-9e2c-16c8bc36531f/scratchpad/sso_check.sh: Permission denied
|
||||
== tester-1
|
||||
sec: cannot run /tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/649cb06d-b4ca-49f2-9e2c-16c8bc36531f/scratchpad/sso_check.sh: Permission denied
|
||||
== tester-1 R-1155
|
||||
with the files cookie (the household's normal path): HTTP 200
|
||||
token alone, still signed in: cookie 401 · bearer 401 · basic 401 · ?auth= 401
|
||||
dashboard sign-out: 302
|
||||
token alone, after sign-out: cookie 401 · bearer 401 · basic 401 · ?auth= 401
|
||||
FileBrowser's own sign-in page, no token: HTTP 200
|
||||
== demo-hp
|
||||
dashboard sign-in: 302
|
||||
the Fajlok button goes to: https://files.enkisfelhom.hu/_felhom/sso
|
||||
the pass answer goes to: https://files.enkisfelhom.hu/files/; files cookie set: 1
|
||||
FileBrowser with the files cookie: felhom-household admin: False modify: True
|
||||
FileBrowser without it, a fresh browser: HTTP 401
|
||||
dashboard sign-out: 302
|
||||
after the sign-out, the files cookie sends the browser to: https://felhom.enkisfelhom.hu/__files/open?to=%2Ffiles%2F
|
||||
== tester-1
|
||||
dashboard sign-in: 302
|
||||
the Fajlok button goes to: https://files.enkicsifelhom.hu/_felhom/sso
|
||||
the pass answer goes to: https://files.enkicsifelhom.hu/files/; files cookie set: 1
|
||||
FileBrowser with the files cookie: felhom-household admin: False modify: True
|
||||
FileBrowser without it, a fresh browser: HTTP 401
|
||||
dashboard sign-out: 302
|
||||
after the sign-out, the files cookie sends the browser to: https://felhom.enkicsifelhom.hu/__files/open?to=%2Ffiles%2F
|
||||
@@ -0,0 +1,5 @@
|
||||
page bytes: 53890
|
||||
new text, shown by default (id=live-after-check, ASCII fragment 'lent a k'): 1
|
||||
old text, hidden until the claim says no confirm (id=live-now display:none): 1
|
||||
the switch in the page script: 1
|
||||
negative control (a sentence that is not on the page): 0
|
||||
@@ -0,0 +1,13 @@
|
||||
## global floor 0.333.0 / MinAgent 0.158.1 — 10:16:22 UTC
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /configuration?flash=floor_set
|
||||
## clear per-box floors
|
||||
tester-1: HTTP/1.1 303 See Other Location: /customers/tester-1?flash=floor_set
|
||||
demo-hp: HTTP/1.1 303 See Other Location: /customers/demo-hp?flash=floor_set
|
||||
demo-felhom: HTTP/1.1 303 See Other Location: /customers/demo-felhom?flash=floor_set
|
||||
## read back (hub.db + WAL)
|
||||
min_controller_version|0.333.0
|
||||
min_controller_version_declared_min_agent|0.333.0=0.158.1
|
||||
demo-felhom|
|
||||
demo-hp|
|
||||
tester-1|
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user