Compare commits
89 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| dbfe5dac06 | |||
| 47fa1e621e | |||
| ef4dbcf984 | |||
| cd3d05981a | |||
| 9688e76efd | |||
| e07b077ac2 | |||
| 7ad17be6bd | |||
| eaff4d890c | |||
| 7a287e71fc | |||
| 6559aa6d2e | |||
| a2d2569719 | |||
| ebf142d921 | |||
| 6d73110cd0 | |||
| ff06ab52da | |||
| a3be7525cd | |||
| 3decdf94a8 | |||
| 7a773207f8 | |||
| 19064ef1a7 | |||
| 31d975c997 | |||
| 0d3f72e463 | |||
| c4262a6f6c | |||
| a600d68d50 | |||
| 830ed9714e | |||
| 16188df64c | |||
| 0d4bfe7ca6 | |||
| 938afb7416 | |||
| c0b70b15ac | |||
| ce8cf2c477 | |||
| 4a557f54a5 | |||
| 92de2d324e | |||
| 12090df2d9 | |||
| 434fc7cc02 | |||
| 53d962d95b | |||
| 836b47f1ec | |||
| bb187f3503 | |||
| 9db8aacdad | |||
| 78278502ac | |||
| 2228b9f6b4 | |||
| ee743477b4 | |||
| ad6397308c | |||
| d750a90d0e | |||
| a942ae90d6 | |||
| 20a77d367b | |||
| a57a8363df | |||
| bffe2d9dea | |||
| 242584af4e | |||
| 43e5a5496d | |||
| c701b2530b | |||
| 54d1ee8210 | |||
| 1d7071e8fd | |||
| 9ec5f76a88 | |||
| f84f94d987 | |||
| 2954b0be3a | |||
| 5c487313fa | |||
| bcaa9576b2 | |||
| 301f4e9e92 | |||
| b0ae2c574a | |||
| e1d349f100 | |||
| 7d7868e5e5 | |||
| 189eb82eb0 | |||
| f92fa5a8c8 | |||
| 5d0dfeac5f | |||
| 15f7428871 | |||
| d92a712ee4 | |||
| 5f31da04bc | |||
| a28f162c66 | |||
| 97f27a7528 | |||
| b6a911c3f9 | |||
| 224fe41e06 | |||
| 73f3899266 | |||
| e2c35007b5 | |||
| fd75003e17 | |||
| 5d3cb5c132 | |||
| a17db0c3c8 | |||
| 503c93fada | |||
| d5e3c8b0d9 | |||
| db2371c6f5 | |||
| 200ab6530b | |||
| 5622ce2735 | |||
| afc939bb3c | |||
| b505ce4ba0 | |||
| 5121536f10 | |||
| e4f022a0d9 | |||
| b9b56c47c0 | |||
| 1fe9f84893 | |||
| 7f7081d127 | |||
| e881115c43 | |||
| 37cb83c166 | |||
| 3332eb0919 |
@@ -29,5 +29,7 @@ ArgoCD's source of truth is the **manifest in git**:
|
|||||||
`main`, and `/scripts/` from the tag `installer-v<SCRIPT_VERSION>`. Changing either `--ref` changes
|
`main`, and `/scripts/` from the tag `installer-v<SCRIPT_VERSION>`. Changing either `--ref` changes
|
||||||
what real machines download — the init container **and** the sidecar both carry it, and both must
|
what real machines download — the init container **and** the sidecar both carry it, and both must
|
||||||
move together. `hostinstall_gates.py` gate 6 fails if the manifest stops naming an `installer-v…`
|
move together. `hostinstall_gates.py` gate 6 fails if the manifest stops naming an `installer-v…`
|
||||||
tag or if the website stops tracking `main`. The full publish/rollback procedure is the R-110 fence
|
tag or if the website stops tracking `main`. All four git-sync containers (two sidecars, two init containers) read
|
||||||
|
Gitea **in-cluster** (`http://gitea.gitea-system.svc.cluster.local:3000/…`, since 2026-10-19) and must name the same
|
||||||
|
repo — gate 6 fails if they differ; the sidecars carry `--max-failures=-1` (retry, never exit). The full publish/rollback procedure is the R-110 fence
|
||||||
in the repo core `CLAUDE.md`.
|
in the repo core `CLAUDE.md`.
|
||||||
|
|||||||
@@ -184,6 +184,10 @@ jobs:
|
|||||||
# which prints at the moment a release is committed, where someone can actually act on it.
|
# which prints at the moment a release is committed, where someone can actually act on it.
|
||||||
# Those are the compensating controls that make this green honest. Every other gate still
|
# Those are the compensating controls that make this green honest. Every other gate still
|
||||||
# fails this job on any push, and golden-currency still fails it on a push touching code.
|
# fails this job on any push, and golden-currency still fails it on a push touching code.
|
||||||
|
env:
|
||||||
|
# R-986: the security-scan gate fetches the PINNED gitleaks here (sha256-checked); no Go, so govulncheck says
|
||||||
|
# NOT CHECKED in CI, out loud. Warn-first either way: the scan lists, it never fails this job.
|
||||||
|
FELHOM_SCAN_FETCH: "1"
|
||||||
run: python3 scripts/repo_gates.py --fast --scope="${PUSH_SCOPE:-code}"
|
run: python3 scripts/repo_gates.py --fast --scope="${PUSH_SCOPE:-code}"
|
||||||
|
|
||||||
- name: Manual proof - alarm key present, optional forced failure (dispatch only, R-1129)
|
- name: Manual proof - alarm key present, optional forced failure (dispatch only, R-1129)
|
||||||
@@ -228,8 +232,16 @@ jobs:
|
|||||||
|
|
||||||
repo = os.environ.get("GITHUB_REPOSITORY", "?")
|
repo = os.environ.get("GITHUB_REPOSITORY", "?")
|
||||||
sha = os.environ.get("GITHUB_SHA", "?")
|
sha = os.environ.get("GITHUB_SHA", "?")
|
||||||
run = os.environ.get("GITHUB_RUN_NUMBER", "?")
|
# R-1135: the link is for a PERSON, so it is the public address and the run ID.
|
||||||
srv = os.environ.get("GITHUB_SERVER_URL", "https://gitea.dooplex.hu")
|
# NOT GITHUB_SERVER_URL: the runner registers at the in-cluster address, so that variable
|
||||||
|
# reads http://gitea.gitea-system.svc.cluster.local:3000, which nobody outside the cluster
|
||||||
|
# can open (forced-test mail, job 1997). NOT GITHUB_RUN_NUMBER: Gitea 1.26 reads
|
||||||
|
# /actions/runs/<n> as a run id first, so a run number that is also an older run's id
|
||||||
|
# opens THAT run (measured 2026-10-18: felhom.eu /actions/runs/1116 opened the run whose
|
||||||
|
# id is 1116 = run number 613). The link is printed too, so the job log carries it.
|
||||||
|
run = os.environ.get("GITHUB_RUN_ID", "?")
|
||||||
|
srv = "https://gitea.dooplex.hu"
|
||||||
|
print("ALARM-LINK %s/%s/actions/runs/%s" % (srv, repo, run))
|
||||||
|
|
||||||
# R-265: how long the run took, so a reap is self-identifying. An honest gate failure
|
# R-265: how long the run took, so a reap is self-identifying. An honest gate failure
|
||||||
# lands in well under a minute; a multi-minute figure means the job hit its own timeout
|
# lands in well under a minute; a multi-minute figure means the job hit its own timeout
|
||||||
|
|||||||
@@ -0,0 +1,37 @@
|
|||||||
|
# .security-accepted — hits security_scan_gate.py (felhom.eu/scripts, R-986) lists as ACCEPTED, not WARN.
|
||||||
|
# One per line: <tool> <id> <where> # reason. govulncheck: OSV id + module dir; gitleaks: rule@<sha256[:12] of the line> + file.
|
||||||
|
# The first scan's verdicts (2026-10-19): felhom.eu documentation/audits/night-2026-10-19/P7/triage.md
|
||||||
|
gitleaks generic-api-key@0275c9c51957 documentation/audits/usenet-2026-10-12/sabnzbd-bench/13-compose-resolved.txt # accepted: a bench app's generated admin password and API key (LXC 9401, stopped; app removed from 9202)
|
||||||
|
gitleaks generic-api-key@0a189ae60ea4 documentation/tests/campaign2-evidence-2026-07-07/state/runner.log # accepted: the same dry-run deploy values in the campaign runner's log
|
||||||
|
gitleaks generic-api-key@2a27df76f661 documentation/audits/usenet-2026-10-12/sabnzbd-bench/02-telemetry-and-net.txt # accepted: a bench app's generated API key (LXC 9401, stopped; app removed from 9202 at teardown)
|
||||||
|
gitleaks generic-api-key@3a4c7b17e349 documentation/audits/usenet-2026-10-12/nzbget-bench/03-process-and-entrypoint.txt # accepted: a bench app's generated control password (LXC 9401, stopped; app removed from 9202 at teardown)
|
||||||
|
gitleaks generic-api-key@3c180a9f957c documentation/tests/campaign2-evidence-2026-07-07/state/runner.log # accepted: the same dry-run deploy values in the campaign runner's log
|
||||||
|
gitleaks generic-api-key@486819f82f58 documentation/audits/update-night-2026-09-21/apps/actualbudget/log.txt # accepted: a demo app's session token, demo-hp 2026-09-21 (Tier 0); no running demo app has it (2026-10-12 review)
|
||||||
|
gitleaks generic-api-key@486819f82f58 documentation/audits/update-night-2026-09-21/batchA.log # accepted: the same demo app session token as actualbudget/log.txt
|
||||||
|
gitleaks generic-api-key@5a64c6ffd6df documentation/audits/signup-lock-2026-09-29/B/wd.sh # accepted: a stranger-probe test password in an audit script (signup lock, Tier 0)
|
||||||
|
gitleaks generic-api-key@6e2e899b7d6e documentation/audits/offsite-lock-build-2026-10-03/partD/demo-hp-migration.txt # false: an SSH key FINGERPRINT in a log line, not a key
|
||||||
|
gitleaks generic-api-key@70380348263d hub/internal/web/bind_language_test.go # false: a test constant or a decoy value, no live secret
|
||||||
|
gitleaks generic-api-key@7134caf03160 documentation/audits/evidence-rehearsal-2026-08-09/GATE0-demo-hp-before.txt # false: a WireGuard PUBLIC key
|
||||||
|
gitleaks generic-api-key@7134caf03160 documentation/audits/evidence-rehearsal-2026-08-09/P1-demo-hp-after.txt # false: a WireGuard PUBLIC key
|
||||||
|
gitleaks generic-api-key@71ba98258279 hub/internal/api/r1140_accesslog_test.go # false: a test constant or a decoy value, no live secret
|
||||||
|
gitleaks generic-api-key@7aa746ddb210 documentation/audits/offsite-lock-build-2026-10-03/partD/demo-hp-migration.txt # false: an SSH key FINGERPRINT in a log line, not a key
|
||||||
|
gitleaks generic-api-key@7ce547845b1e documentation/audits/night-rulings-2026-09-30/D/D3-wanderer-routes-probe.txt # accepted: a throwaway app's session cookie (wanderer probe, 2026-09-30, Tier 0)
|
||||||
|
gitleaks generic-api-key@8044eef94360 documentation/tests/campaign2-evidence-2026-07-07/state/runner.log # accepted: the same dry-run deploy values in the campaign runner's log
|
||||||
|
gitleaks generic-api-key@85b662612937 hub/internal/store/r879_box_seal_test.go # false: a test constant or a decoy value, no live secret
|
||||||
|
gitleaks generic-api-key@9530a022fde8 documentation/audits/usenet-2026-10-12/sabnzbd-bench/13-compose-resolved.txt # accepted: a bench app's generated admin password and API key (LXC 9401, stopped; app removed from 9202)
|
||||||
|
gitleaks generic-api-key@9b3a9c2be48a documentation/tests/campaign2-evidence-2026-07-07/evidence/T-DEPLOY-SET/values_bookstack.json # accepted: dry-run deploy values of a 2026-07-07 campaign (generated, demo boxes since rebuilt)
|
||||||
|
gitleaks generic-api-key@a9538cfd7063 documentation/tests/campaign2-evidence-2026-07-07/evidence/T-DEPLOY-SET/values_paperless-ngx.json # accepted: dry-run deploy values of a 2026-07-07 campaign (generated, demo boxes since rebuilt)
|
||||||
|
gitleaks generic-api-key@b0414c6f59b9 documentation/audits/offsite-lock-build-2026-10-03/partD/demo-hp-migration.txt # false: an SSH key FINGERPRINT in a log line, not a key
|
||||||
|
gitleaks generic-api-key@b1db45e463b6 documentation/audits/family-gate-2026-10-02/tools/fg_items.py # false: test strings in an audit tool (websocket probe labels)
|
||||||
|
gitleaks generic-api-key@b1db45e463b6 documentation/audits/permanent-gate-2026-10-01/tools/items12.py # false: test strings in an audit tool (websocket probe labels)
|
||||||
|
gitleaks generic-api-key@bd228c851720 documentation/tests/walk5-r201-2026-08-07/journal.md # false: sha256 digests compared in a walk journal, not secrets
|
||||||
|
gitleaks generic-api-key@c1be98b7b703 documentation/tests/walk5-r201-2026-08-07/journal.md # false: sha256 digests compared in a walk journal, not secrets
|
||||||
|
gitleaks generic-api-key@c3b2b5665c9c documentation/audits/evidence-fixes-first-tester-2026-09-30/part0/cloudflare-dns.txt # false: a tunnel ID (a UUID), not the tunnel token
|
||||||
|
gitleaks generic-api-key@cf453385b707 documentation/tests/campaign2-evidence-2026-07-07/state/runner.log # accepted: the same dry-run deploy values in the campaign runner's log
|
||||||
|
gitleaks generic-api-key@e59eaeb4aee7 documentation/audits/new-apps-2026-10-01/tools/newapp_box.py # accepted: a throwaway test user's password in an audit tool for a fresh-app walk on a Tier-0 box
|
||||||
|
gitleaks generic-api-key@eada8358f62a hub/internal/web/render_test.go # false: a test constant or a decoy value, no live secret
|
||||||
|
gitleaks generic-api-key@efd4cc86d8d9 documentation/audits/power-cut-fixes-2026-10-14/E/demo-hp-journal-boot.txt # false: the id of HP's public UEFI certificate in a boot journal
|
||||||
|
gitleaks generic-api-key@f28c207293c1 documentation/tests/campaign2-evidence-2026-07-07/evidence/T-DEPLOY-SET/values_vaultwarden.json # accepted: dry-run deploy values of a 2026-07-07 campaign (generated, demo boxes since rebuilt)
|
||||||
|
gitleaks generic-api-key@f399eb9e3059 documentation/audits/night-2026-09-23/chaos/round-01-run.txt # accepted: the same demo app session token as round-01.log
|
||||||
|
gitleaks generic-api-key@f399eb9e3059 documentation/audits/night-2026-09-23/chaos/round-01.log # accepted: a demo app's session token, demo-hp 2026-09-23 (Tier 0); the 2026-10-12 review found no running demo app with it
|
||||||
|
gitleaks generic-api-key@fdabf6a76b48 documentation/audits/r890-instructions-2026-10-06/tools/wgerwalk.py # accepted: a throwaway wger walk's API token in an audit tool (Tier 0)
|
||||||
@@ -138,7 +138,14 @@ something, not only sessions that touch `documentation/` — which is why it is
|
|||||||
and the eleven names head `OPEN-ITEMS.md`, and `register_shape_gate.py` refuses anything else.
|
and the eleven names head `OPEN-ITEMS.md`, and `register_shape_gate.py` refuses anything else.
|
||||||
- **Root `STATUS.md`** — at the end of every session in which something shipped, broke or was
|
- **Root `STATUS.md`** — at the end of every session in which something shipped, broke or was
|
||||||
decided. It is a **view** of `OPEN-ITEMS.md`; nothing may exist only there. One screen, written for
|
decided. It is a **view** of `OPEN-ITEMS.md`; nothing may exist only there. One screen, written for
|
||||||
the operator in plain language, and deliberately **not** `CONTEXT.md`.
|
the operator in plain language (STE English), and deliberately **not** `CONTEXT.md`. How to update it:
|
||||||
|
1. **Replace** the „Updated" paragraph (at most 3 sentences, the newest change only) — never prepend; the old one
|
||||||
|
moves to `documentation/backlog/STATUS-HISTORY.md` under a dated heading.
|
||||||
|
2. A morning list goes under a dated sub-heading in „Waiting on you — do".
|
||||||
|
3. Remove what you finished: a done „do" item, a fixed „broken" line, a decided row (its ruling is in its row first).
|
||||||
|
4. A decide row: the question in one answerable sentence, at most 2 options, CC's pick first with the reason, the
|
||||||
|
other option's cost, what happens if the operator does nothing, and who is blocked.
|
||||||
|
5. `scripts/status_view_gate.py` checks it on every push (warn-first until 2026-10-27, then it refuses; R-1174).
|
||||||
- **The golden, weekly** (operator ruling 2026-10-15 11:26, `09` §3 decision 250; decision 7 stands): **a controller
|
- **The golden, weekly** (operator ruling 2026-10-15 11:26, `09` §3 decision 250; decision 7 stands): **a controller
|
||||||
release needs no golden** — the floor carries it, MinAgent declared, after the three test boxes (`09` §3 decision 250
|
release needs no golden** — the floor carries it, MinAgent declared, after the three test boxes (`09` §3 decision 250
|
||||||
item 2). The golden is baked **every Sunday night** when the floor is above it (`runbooks/nightly-rotation.md`) and
|
item 2). The golden is baked **every Sunday night** when the floor is above it (`runbooks/nightly-rotation.md`) and
|
||||||
@@ -178,6 +185,12 @@ something, not only sessions that touch `documentation/` — which is why it is
|
|||||||
rows — a run can sit several pages earlier. **Scan every page** and match on `head_sha`; with a
|
rows — a run can sit several pages earlier. **Scan every page** and match on `head_sha`; with a
|
||||||
few hundred jobs that is a handful of requests. Guessing the page produced three consecutive
|
few hundred jobs that is a handful of requests. Guessing the page produced three consecutive
|
||||||
false "no CI job for this commit" readings in one session.
|
false "no CI job for this commit" readings in one session.
|
||||||
|
3. **A rerun REPLACES the attempt the `jobs` endpoint shows** (R-1137, measured 2026-10-18): the job
|
||||||
|
keeps its id, `run_attempt` becomes 2, `started_at`, `conclusion` and the log are the rerun's — the
|
||||||
|
red first attempt is gone from `jobs` and its log (controller jobs 1944, 1959: failure mails, now
|
||||||
|
green). **A matched job with `run_attempt` > 1 is not a plain green:** list `actions/tasks` (same
|
||||||
|
paging, match `head_sha`) — one task per attempt, the verdict in `status`, so it reads
|
||||||
|
„failure, then rerun success". The first attempt's log is not reachable by the API.
|
||||||
|
|
||||||
An unchecked green is an assumption, not an observation — and so is a green read off a field the
|
An unchecked green is an assumption, not an observation — and so is a green read off a field the
|
||||||
API never populates.
|
API never populates.
|
||||||
|
|||||||
+76
@@ -16,6 +16,82 @@
|
|||||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||||
|
|
||||||
|
|
||||||
|
> **2026-10-21 (day brief, before Tester 2) — hub 0.163.0 DEPLOYED (operator yes, 11:46 UTC); controller 0.337.0 released
|
||||||
|
> (three test boxes 14:06 UTC, global floor after the freeze proof); agent `286a7e3` committed, NOT released.** Operator rulings
|
||||||
|
> today: **D24 = yes** (the hub's DST double run fixed: `nextDailyRun`, calendar day+1, `05` §16.5); **D25** (a second-drive
|
||||||
|
> copy whose SOURCE is gone is kept until the household releases it — `07` §8.2a; controller `tier2_kept.go`, the restore
|
||||||
|
> page's „Megőrzött másolatok"); the **R-1176 shape** = the hub CARRIES the household's retained sealed package to the
|
||||||
|
> rebuilt host (`store.ReattachRetainedEscrow`: one live host, no package held, the newest escrow-ACKED deletion; moved in
|
||||||
|
> one tx at the agent's first report and the controller's report; never opened/logged; two live hosts → refused + operator
|
||||||
|
> mail) — `03` §8a. Proven live on demo-hp (Tester4 from the public ISO): no fresh key minted, recovery offered, wrong code
|
||||||
|
> refused, right code opened the history, Nextcloud 3 031/3 033 files byte-identical, off-site appended 63 → 64
|
||||||
|
> (`audits/fixes-2026-10-21/P1-live/`). R-1182: retained rows carry `repo_tag` (NULL legacy / 'live' / the set-aside path
|
||||||
|
> tagged at the hub's move-aside); the abandonment purges only rows tagged with a path the hub DELETED; the ACK says
|
||||||
|
> `abandon_purge_done`. R-1203: the new-link page tells the holder of a real spent link that the old box is still linked;
|
||||||
|
> operator mail `bind_old_host_linked`. **R-1192 diagnosed with numbers:** Proxmox's LXC default swap cap (512 MB) — the
|
||||||
|
> agent never set `swap` — turned Immich's model load (+1.15 GB) into a code-page refault thrash at the 4096 MB guest limit
|
||||||
|
> (150 000–600 000 refaults and ~1 GB of reads per 10 s, memory „full" 92–95 %, load 155, dashboard dead 6½ min, 0 OOM
|
||||||
|
> kills); `pct set --swap 4096` live ended it within a minute. Agent fix (swap = RAM at bring-up, at RAM resize, once at
|
||||||
|
> start) committed; its release is the operator's. The controller's „Normál használat mellett ez nem okoz problémát" is
|
||||||
|
> gone (0.337.0). Observed and filed: R-1205 (PBS re-issue after an un-acked second install), R-1206 (future-dated
|
||||||
|
> snapshots). Report: `REPORT-fixes-2026-10-21.md`.
|
||||||
|
|
||||||
|
> **2026-10-19 (night) — controller 0.335.0 is the floor (MinAgent 0.158.1, 18:39 UTC); hub 0.162.0 BUILT, not deployed;
|
||||||
|
> agent `56aabc7` and installer 1.35.0 committed, not released/published.** R-921 (operator answer „add one field"): the
|
||||||
|
> agent serves its host-wide heavy-operation gate as `host_busy` on `GET /backup/status`; the controller reads it before
|
||||||
|
> any stop (held → contention, no stop). R-1070's cause: Docker 29 prints „error: no such object" in lower case, so a
|
||||||
|
> guest without cloudflared read `unknown` and failed the host OS-update rule; new state `absent` (passes when absent
|
||||||
|
> before the step too); `11` §8.2 had claimed `unknown` does not fail — corrected. R-1010: the janitor's forced snapshot
|
||||||
|
> delete is the API's `force=1`. Hub 0.162.0: R-1001 (restart seeding from the DB), R-1011 (exact-file sha). R-953 is
|
||||||
|
> gone: ArgoCD tracks none of the three Secrets now. **R-986: `scripts/security_scan_gate.py`, shared, in every repo's
|
||||||
|
> fast set and CI, WARN-FIRST**; govulncheck pin v1.1.4 → v1.8.0 (v1.1.4 cannot read Go 1.27); CI has no Go, so
|
||||||
|
> govulncheck runs only in the hook; a depth-1 CI clone scans the whole tree (the root commit), all accepted. R-893
|
||||||
|
> measured on 9202 (`audits/night-2026-10-19/P3/`) → D20. 9202 is on DHCP (.121 now).
|
||||||
|
|
||||||
|
> **2026-10-19 (late afternoon) — controller 0.334.0 is the floor (MinAgent 0.158.1, 16:31 local); hub 0.161.0 DEPLOYED;
|
||||||
|
> tenantsync v1.3.0 on ep0 (operator yes).** R-1159 (D19 „box key first, then lock"): the passphrase is used AFTER enroll
|
||||||
|
> (installer `/config` for its git login, the agent bakes it into `bootstrap.json`, the guest's first-boot pull, and —
|
||||||
|
> until 0.334.0 — every config refresh), so "keys only before a host exists" was not buildable. Now: `GET /box-config`
|
||||||
|
> with the controller key; the hub refuses passphrase keys (409, `passphrase_keys_refused`) once a ≥ 0.334.0 controller
|
||||||
|
> has reported since the host record was made — an rc (`0.334.0-rc.N`) counts as older. **A reinstall over a live box
|
||||||
|
> needs the operator's delete first.** R-1165: every new ep0 namespace gets `prune-<id>`; the hub alarms on a namespace
|
||||||
|
> with none. demo-hp's .104 clean after the router reservation. Register 163 → 161. `REPORT-day3-2026-10-19.md`.
|
||||||
|
|
||||||
|
> **2026-10-19 (afternoon) — controller 0.333.0 is the floor (MinAgent 0.158.1, 12:16 local); hub 0.160.0 DEPLOYED
|
||||||
|
> (operator yes); ep0 prune `keep-weekly 2` (operator yes, D18).** ep0 change: `prune-demo-hp`/`prune-demo-felhom` only,
|
||||||
|
> after PBS's own `--dry-run true` (API `proxmox-backup-debug api create …/prune`; it creates no task) showed nothing
|
||||||
|
> removed; households with no prune job (tester-1, Tester-2) = R-1165. R-1155: FileBrowser 1.5.6 reads its token from the
|
||||||
|
> cookie, `?auth=`, Bearer AND the Basic-auth password (`extractToken`) — the forwardAuth refusal must read all four (the
|
||||||
|
> security review caught Basic). R-1159 measured: passphrase → `/config` key and `/host-enroll` existing host key → every
|
||||||
|
> off-site key route; D19 open (CC picks A: keys only while no host exists). R-1158: the reinstall's off-site re-issue only
|
||||||
|
> rotates the hub's own registrar login; kept. gofmt gates in hub and agent runners. demo-hp's .104 is taken by another
|
||||||
|
> device (R-1166, the router). Register 167 → 163. `REPORT-day2-2026-10-19.md`.
|
||||||
|
|
||||||
|
> **2026-10-19 (day) — controller 0.332.0 is the floor (MinAgent 0.158.1, 05:24 local); agent 0.160.0 released and on the
|
||||||
|
> three test hosts, NOT vouched; hub 0.159.0 DEPLOYED (operator yes in advance).** One sign-in for the file manager (R-1100,
|
||||||
|
> D16): dashboard pass → host-only `felhom_files` cookie → traefik forwardAuth `/__felhom_gate/files` sets the per-box
|
||||||
|
> SECRET header (`<data>/filebrowser-sso-header`, never logged) naming FileBrowser user `felhom-household`; traefik makes a
|
||||||
|
> RELATIVE Location in a forwardAuth answer absolute against the auth server — always answer absolute URLs. R-1151: the
|
||||||
|
> escrow package uploads at `POST /escrow/ceremony/confirm` (two typed words), an old controller keeps the claim order.
|
||||||
|
> R-1149 (D17): local tier `keep-daily=N`. R-1154 rule (after a security review of the helper's first draft): the
|
||||||
|
> host-enroll reuse path NEVER re-issues — the passphrase must not reach the operator's re-issue powers; it mails the
|
||||||
|
> operator. R-1153 was the test file (FileBrowser previews .docx with MuPDF before OnlyOffice). R-887 closed (381 jobs, 0
|
||||||
|
> lost since S21). The old-restic check after demo-hp's first real 0.19.1 prune passed. Register 168 → 166.
|
||||||
|
> `REPORT-day-2026-10-19.md`.
|
||||||
|
|
||||||
|
> **2026-10-18 (evening) — controller 0.331.0 is the floor (MinAgent 0.158.1): OnlyOffice opens and saves from FileBrowser
|
||||||
|
> and Nextcloud, FileBrowser back at once after a restart (R-1018), a removal deletes a volume the definition dropped
|
||||||
|
> (R-1117); catalog `30eea97`: Immich/Nextcloud Redis without a volume; golden 0.331.0 baked, NOT vouched; a fresh install
|
||||||
|
> from the ISO proved installer 1.34.0 + vouched golden/agent + auto-floor (R-1148); agent `main` carries R-978, R-973,
|
||||||
|
> R-1093 unreleased. *Decided by CC unattended — operator may reverse:* the R-973 cap numbers (3 successful daily-tier runs
|
||||||
|
> per 24 h, one weekly off-site run per 6 days; `07` §6.1). Tester4 is kept as the standing fresh-install customer
|
||||||
|
> (operator). `REPORT-evening-2026-10-18.md`.**
|
||||||
|
|
||||||
|
> **2026-10-18 (day) — hub 0.158.0 DEPLOYED (access log live, agent-only vouch, R-895 first verdict 2026-10-20);
|
||||||
|
> controller 0.330.0 is the floor (R-1144); agent 0.159.2 released and on the three test hosts, NOT vouched; installer
|
||||||
|
> 1.34.0 published (vault-first step 4b, R-1145; fresh install unproven, R-1148); estimator Immich peak 3,983 MB
|
||||||
|
> (R-1147).** Independent review of the night: `audits/day-2026-10-18/P2/`. `REPORT-day-2026-10-18.md`.
|
||||||
|
|
||||||
> **2026-10-18 (night) — controller 0.329.0 is the floor (MinAgent 0.158.1); golden 0.329.0 baked, NOT vouched; hub 0.158.0
|
> **2026-10-18 (night) — controller 0.329.0 is the floor (MinAgent 0.158.1); golden 0.329.0 baked, NOT vouched; hub 0.158.0
|
||||||
> built, NOT deployed.** Each app's own `appdata/<folder>` (R-1142), empty allow-list (R-981), drive retirement asks the
|
> built, NOT deployed.** Each app's own `appdata/<folder>` (R-1142), empty allow-list (R-981), drive retirement asks the
|
||||||
> agent first (R-1063), plain no-durable-id refusal (R-1069), `:latest` for releases only (R-1143), Immich 3072M/2048M
|
> agent first (R-1063), plain no-durable-id refusal (R-1069), `:latest` for releases only (R-1143), Immich 3072M/2048M
|
||||||
|
|||||||
@@ -0,0 +1,139 @@
|
|||||||
|
# REPORT — day 2026-10-18
|
||||||
|
|
||||||
|
## In plain words
|
||||||
|
|
||||||
|
- **The hub update is live.** Its own access log works and writes no secret (tested live with a fake token). You can
|
||||||
|
now vouch an agent without a new golden. The off-site snapshot check gives its first verdict on 20 October.
|
||||||
|
- **A fresh reviewer read last night's code.** One medium finding: a stale host key could have left the emergency
|
||||||
|
password unknown. Today's installer change (save in the hub first, then set) already closes it. Three low findings:
|
||||||
|
two fixed today, one is a known design limit, now written in its row.
|
||||||
|
- **„Retire anyway":** when the host says no, the drive's apps start again. Where the host's disk list already says no,
|
||||||
|
nothing is stopped at all. Controller 0.330.0 is the floor for every box.
|
||||||
|
- **Agent 0.159.2** runs on demo-hp, Tester 1 and demo-felhom (your yes). A whole-box backup ran with no pool check.
|
||||||
|
**The vouch is yours** (STATUS).
|
||||||
|
- **Installer 1.34.0 is published** (your yes): the emergency password is saved in the hub first, then set, then
|
||||||
|
checked with a login. Proven on demo-hp. A fresh install with it has not run yet (row R-1148).
|
||||||
|
- **The calculator** shows Immich's bigger figure (your yes). Immich alone on an 8 GB machine now reads „does not fit".
|
||||||
|
- **The old-restic check has not run.** demo-hp's clean-up comes only after tonight's backup (2026-10-19 about 04:16
|
||||||
|
Budapest). The STATUS item stays open.
|
||||||
|
|
||||||
|
## 1. Hub 0.158.0 deploy
|
||||||
|
|
||||||
|
ArgoCD: only `Deployment/hub` was OutOfSync; synced that resource alone → Synced/Healthy, image `felhom-hub:0.158.0`,
|
||||||
|
startup log `felhom-hub 0.158.0 starting` + `access log: /data/access-log …, kept 90 days`. Floor read back from the hub
|
||||||
|
DB with its WAL: 0.329.0 / 0.158.1 at deploy time (`audits/day-2026-10-18/P1/hub-deploy-readback.txt`). Reports from all
|
||||||
|
three boxes in the access log within two minutes (sources .101, .149, .155).
|
||||||
|
- **Access log (R-1140, closed):** today's file `access-2026-10-18.log`, mode 0600. A fresh 48-hex fake token sent in a
|
||||||
|
query, a `/bind/<token>` path, a `session=` query, a Bearer header, a Cookie and a POST body: found 0 times; control
|
||||||
|
(the same file with the token appended) 1. Secret shapes (Bearer value, a 32+ character run, `password=`, `token=`,
|
||||||
|
`session=`): 0; control 1. `/bind/<token>` is written `/bind/[redacted]` (`P1/access-log-live-test.txt`).
|
||||||
|
- **Agent-only vouch:** read-only — `/configuration` loads with CC's token (Day-0 section, golden 0.329.0, agent 0.159.1);
|
||||||
|
the save (`/configuration/artifacts`) answers 403 to the token, by design. No manifest change by CC.
|
||||||
|
- **Off-site alarm (R-895, NARROWED):** no fixture route exists in the hub for it; the unit tests are the proof so far.
|
||||||
|
Tomorrow's 07:10 run is the baseline (no verdict by design); the first verdict is 2026-10-20 07:10 — a DUE-CHECKS row.
|
||||||
|
|
||||||
|
## 2. The independent review
|
||||||
|
|
||||||
|
A fresh subagent with no context from the night reviewed controller v0.328.0→v0.329.0, hub 0.157.0→0.158.0, the agent's
|
||||||
|
orphan-check change and the installer's `--rotate-recovery-only`. Its report was written to a file and read back
|
||||||
|
(`audits/day-2026-10-18/P2/independent-review.md`, 54 lines). Verdicts (`P2/verdicts.md`):
|
||||||
|
|
||||||
|
| # | Finding | Verdict |
|
||||||
|
|---|---|---|
|
||||||
|
| F1 medium | rotate-only: unauthenticated healthz, then chpasswd, then a vault that a stale key fails → an unvaulted password | REAL in the night's code → **fixed** by R-1145 (vault first); harness `R1145-e`, fails against HEAD |
|
||||||
|
| F2 low–medium | `offsite_snapshots_unexplained` is explained by the box's own reported run time | REAL, by design (decision 68) → written into R-895; the custody alarm `_vanished` is not affected |
|
||||||
|
| F3 low | „anyway" stops apps before the agent is asked | REAL, was R-1144 → **fixed** (controller 0.330.0) |
|
||||||
|
| F4 low | host key in curl's argv during the vault PUT | REAL, pre-existing → **fixed** (header file) |
|
||||||
|
|
||||||
|
## 3. R-1144
|
||||||
|
|
||||||
|
`decommissionAnyway`: reads the agent's disk list first (a role other than user-data, or the backup target → the plain
|
||||||
|
refusal, nothing stopped); after a refusal past the stops (policy refusal or a failed agent call) starts exactly the apps
|
||||||
|
that were running; a drive the agent let go keeps its apps stopped. Household text HU + EN. Seam `decomStacks`, so no
|
||||||
|
test reaches docker. Three tests, two red-proofs (`P3/redproof-r1144.txt`). Not pressed live: no test box has a drive
|
||||||
|
that is safe to retire.
|
||||||
|
|
||||||
|
## 4. Agent 0.159.2
|
||||||
|
|
||||||
|
Released with `release-agent.sh` (code pushed first, CHANGELOG after — the `release-complete` hook refuses the other
|
||||||
|
order): sha256 `4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9`, bundle
|
||||||
|
`477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9` (read back from the registry). **demo-hp proof:** a
|
||||||
|
whole-guest backup of 9201 through the dashboard (`/api/guest-backup/trigger`, snapshot mode, completed in 7.5 min):
|
||||||
|
**0** `felhom-thin-check` sudo lines; control — the night's backup on 0.159.1 at 04:33 had **1**
|
||||||
|
(`P4/demo-hp-backup-journal.txt`). The failure path stays covered by the agent's fixture tests. **Operator: yes** →
|
||||||
|
Tester 1 and demo-felhom by signed jobs (agent_update + agent_config_update): both `signed op COMPLETED`, capability
|
||||||
|
probe 75/75, `update committed` 0.159.2, and a host report on 0.159.2 from each (`P4/`). **Not vouched** — STATUS.
|
||||||
|
|
||||||
|
## 5. The password order (R-1145, R-1146)
|
||||||
|
|
||||||
|
Step 4b (install and rotate-only): generate → PUT to the vault (200 = the hub's DB write is done) → chpasswd (one retry)
|
||||||
|
→ a PVE API login with the new value. **The states that can remain, and why each is safe or not:**
|
||||||
|
|
||||||
|
| State | When | Safe? |
|
||||||
|
|---|---|---|
|
||||||
|
| nothing changed | the vault refuses or is unreachable (a stale key included) | safe — root and the vault keep their old, matching pair |
|
||||||
|
| root = vault = new, PVE login 200 | the normal end | safe |
|
||||||
|
| vault = new, root = previous | chpasswd fails twice (or the run is killed between vault and set) | **not fully safe:** the vaulted value does not open the console; root keeps its previous password (the shadow line is fingerprinted before/after to prove it). The host key cannot read the vault back (by design), so the previous value cannot be re-vaulted; the log says the state in words, and the next run fixes it (it vaults first again). |
|
||||||
|
| root = vault = new, PVE login refused | pveproxy trouble | safe for the shell; the run dies and says the web login refused |
|
||||||
|
|
||||||
|
The old order's dangerous state (root = new, vault = old) is no longer reachable. The hub cannot tell the installer its
|
||||||
|
„last vaulted" time (no host-readable route); the installer trusts the PUT's 200, and CC checked the time moved in the
|
||||||
|
live proof. Live on demo-hp: vault `set_at` 2026-07-21 → 2026-10-18T10:43:33Z, the new value gets a PVE ticket (200),
|
||||||
|
the old one 401 (`P5/demo-hp-rotation.txt`). Harness: 52 passed / 0 failed (was 45/2); R1145-a..e red-proofed against
|
||||||
|
HEAD. Installer **1.34.0 published** on the operator's yes: tag `installer-v1.34.0`, both webpage pins, ArgoCD synced
|
||||||
|
the webpage Deployment only; the served file's sha256 = the tag's (`df722ee4…`). R-1146: SC2066 and GL8-F6 fixed, the
|
||||||
|
harness is gate `hostinstall-harness` with two decoys.
|
||||||
|
|
||||||
|
## 6. Calculator (R-1147)
|
||||||
|
|
||||||
|
Before/after (`audits/day-2026-10-18/P6-before-after.md`): Immich peak 3,290 → 3,983 MB (typical unchanged 1,557). The
|
||||||
|
one changed verdict: **Immich alone on 8 GB: fits, but tight → does not fit**. The family starter set is unchanged.
|
||||||
|
**Operator: yes** → published (cache-bust v5); live `sizing-data.json?v=5` serves 3,983. SIZING.md's table and the
|
||||||
|
pinned estimator tests moved with it. R-1036 closed (D14 = no).
|
||||||
|
|
||||||
|
## 7. The old-restic check
|
||||||
|
|
||||||
|
**Not run — the prune has not happened yet.** The hub allows demo-hp a clean-up window 6 days 12 hours after the last
|
||||||
|
one (2026-10-12 02:16:53 UTC → from 2026-10-18 14:16:53 UTC). But the box asks for a window only after its own nightly
|
||||||
|
off-site run, and that runs at 04:15 CEST. So the first real 0.19.1 prune comes **2026-10-19 at about 02:16 UTC**
|
||||||
|
(04:16 Budapest). Read at 14:33 UTC: no `[offbox]` line since the 11:09 restart, the scheduler's next run 2026-10-19
|
||||||
|
04:15 CEST, the hub's last window still id 9 (`P8/cleanup-window-not-yet.txt`, the hub DB read with its WAL). CC did not
|
||||||
|
grant a window by hand (that is the operator's). The STATUS item stays open, with the corrected time; the check script
|
||||||
|
is ready (`rx.sh` + a wrapper that lists and restores the newest, the oldest and the newest 0.14-made snapshot with both
|
||||||
|
restics and compares tree hashes).
|
||||||
|
|
||||||
|
## 8. Versions, commits, fixtures
|
||||||
|
|
||||||
|
- Hub **0.158.0** deployed (felhom.eu `4c319dde`).
|
||||||
|
- Controller **0.330.0** — `6c6bd87`; floor 0.330.0 / MinAgent 0.158.1 at 11:10:59 UTC (303 `floor_set`, read back with
|
||||||
|
the WAL); per-box floors set and cleared; box walk green on the three test boxes (0 refusals, all healthy, login and
|
||||||
|
files 200, https-only OK).
|
||||||
|
- Agent **0.159.2** — `65c1994` (code), tag `v0.159.2`, CHANGELOG `e25454c`.
|
||||||
|
- Installer **1.34.0** — felhom.eu `88aecf55` (vault-first + harness), `624bcdc3` (1.34.0, tagged), `3332eb09` (pins).
|
||||||
|
- Estimator — `37cb83c1`. Fixtures: none changed (the new texts travel in JSON).
|
||||||
|
|
||||||
|
## 9. Register — 173 before, 168 after; opened 1, closed 6
|
||||||
|
|
||||||
|
Closed: R-1140, R-1144, R-1145, R-1146, R-1147, R-1036. Opened: R-1148. Narrowed: R-895 (dated check 2026-10-20).
|
||||||
|
|
||||||
|
## 10. Observations
|
||||||
|
|
||||||
|
1. Installer 1.34.0's default path has not run in a fresh install. FILED: R-1148.
|
||||||
|
2. The review's F2 (box-reported run times explain new snapshots). NOT-A-FINDING as a new row: a known design limit,
|
||||||
|
recorded in R-895.
|
||||||
|
3. `release-agent.sh` vs the `release-complete` hook: the CHANGELOG must follow the release. NOT-A-FINDING: the documented
|
||||||
|
order (memory and skill); I tried the other order first and the hook refused it, as designed.
|
||||||
|
4. The estimator's new Immich figure is `memory.current` (with cache), not the `used` measure it normally takes.
|
||||||
|
NOT-A-FINDING: on the safe side, said in SIZING.md, and approved by the operator with the table in front of them.
|
||||||
|
5. A safety check refused a `sh -c` with an unguarded `rm "$H"`. NOT-A-FINDING: rewritten with `${H:?}` in a helper.
|
||||||
|
|
||||||
|
## Teardown — three layers
|
||||||
|
|
||||||
|
- **Machines:** demo-hp 9201 — one whole-guest backup taken (kept, like any backup); no app installed or removed today.
|
||||||
|
demo-hp host — the installer copy shredded after the rotation; root password rotated on purpose (new value in the
|
||||||
|
vault). Tester 1, demo-felhom — agent 0.159.2 and controller 0.330.0 through their normal paths; nothing else.
|
||||||
|
- **Hub:** hub 0.158.0; global floor 0.330.0 / 0.158.1; no per-box floor; signed jobs consumed. No vouch by CC.
|
||||||
|
- DooPlex: builds only; ArgoCD synced two Deployments (hub, webpage), each alone.
|
||||||
|
|
||||||
|
Instruction files: none edited. System poster: none of the three (the poster shows backup retentions; the hub's access
|
||||||
|
log is not on it). Architecture documents read: `02`, `03`, `05`, `07`, `09` §3 decision 250; `break-glass.md`.
|
||||||
@@ -0,0 +1,156 @@
|
|||||||
|
# REPORT — day 2026-10-19
|
||||||
|
|
||||||
|
The operator answered the two STOPs in advance („YES to the agent and the hub release") and earlier: golden 0.331.0
|
||||||
|
vouched; D16 (one sign-in) yes; D17 „one per day, 3 days". No question was asked during the session.
|
||||||
|
|
||||||
|
## In plain words
|
||||||
|
|
||||||
|
- **Agent 0.160.0 runs on all three test hosts.** A new recovery code replaces the old one only after the household types
|
||||||
|
two of its words back; a broken-into box cannot flood the whole-box backups; the local copies are one per day for 3
|
||||||
|
days; an empty partition does not ask „data will be erased". **It is not vouched — that is yours.**
|
||||||
|
- **One sign-in for the file manager is live on every box** (controller 0.332.0, the floor). „Fájlok ↗" opens the app's
|
||||||
|
folder with no second login. A sign-out on the dashboard asks again. No app on the box can pretend to be the household.
|
||||||
|
- **Hub 0.159.0 is live.** A refused backup has its own alarm text. A reinstall over an old host record now mails you; it
|
||||||
|
does not re-issue the codes by itself (a security review stopped the first draft that did).
|
||||||
|
- **The preview error (R-1153) was a broken test file**, not the box.
|
||||||
|
- **The old backup program still reads demo-hp's off-site copies** after the first real clean-up with the new one.
|
||||||
|
|
||||||
|
## 1. Agent 0.160.0 (R-978, R-973, R-1093, R-1149, R-1151) — `audits/day-2026-10-19/P1/`
|
||||||
|
|
||||||
|
- **Content:** R-978 (an abandoned second ceremony keeps the first code), R-973 (the cadence cap: daily tier 3 per 24 h,
|
||||||
|
off-site 1 per 6 days), R-1093 (an empty partition is not data), **R-1149** (local tier `keep-daily=N`, operator D17),
|
||||||
|
**R-1151** (the package uploads only at `POST /escrow/ceremony/confirm` with two typed words; a claim without
|
||||||
|
`defer_upload` keeps the R-978 order, so an old controller still completes a ceremony). Commits `1b6e316`, `c11e3e2`,
|
||||||
|
`124164d` (CHANGELOG), `4545819` (two texts, after the release, unreleased). Released with `release-agent.sh`: sha256
|
||||||
|
`7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1`, bundle
|
||||||
|
`6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f`, round trip verified, tag `v0.160.0`.
|
||||||
|
- **Red-proofs:** keep-last for the primary → „primary tier prune = "keep-last=3"" and „holds 0 copies made before the
|
||||||
|
compromise"; the claim's defer flag ignored → „a code that was shown but never typed back replaced the package: code 1
|
||||||
|
no longer opens it (uploads=2)" (all three abandon paths); the word check skipped → „one wrong word: got 200".
|
||||||
|
- **demo-hp proof** (signed jobs 01:22 UTC, running 01:23, „update committed" 01:24; probe 75/75):
|
||||||
|
- a normal „Mentés most" of 9201: green, 6.9 GB in 443 s; yesterday's copy pruned (keep-daily=1, the same copy keep-last
|
||||||
|
would prune); the cap's history on disk holds both successes;
|
||||||
|
- escrow through the wizard's own calls: ceremony 1 — one wrong word 422, the right words 200; code 1 opens the hub's
|
||||||
|
package, code 0 is a kept earlier one (422, superseded at the confirm second 01:47:47). Ceremony 2 shown and abandoned —
|
||||||
|
code 1 still opens it (200), code 2 opens nothing (400). demo-hp's stored code is now code 1 (CC's secret store);
|
||||||
|
- R-1093 on the real route: a 64 MB RAM disk (`scsi_debug`, `/dev/sdb`), one empty GPT partition → `POST /disks/format`
|
||||||
|
→ 200 formatted, no confirmation; control: the same partition, now ext4 → 403 `needs_confirmation`. RAM disk removed.
|
||||||
|
**A loop device cannot be used:** the route and the probe accept only sd/nvme/vd/hd names (by design).
|
||||||
|
- **The answer:** the operator's „yes" (in advance). Tester 1 and demo-felhom took 0.160.0 by signed jobs (01:52 UTC →
|
||||||
|
01:54 / 01:55, probe 75/75, `keep-daily=3` armed on both); the hub's host rows read 0.160.0 for all three, and fresh
|
||||||
|
reports after the hub restart. **Old controller with the new agent:** pinned by `TestEscrowCeremony_R1151_OldControllerKeepsTheClaimOrder`; **new controller with agent 0.159.2:** pinned by `TestR1151_OldAgentClaimCompletesAtOnce` and the agentapi test — no host runs 0.159.2 any more, so not walked live.
|
||||||
|
|
||||||
|
## 2. One sign-in (R-1100) — `audits/day-2026-10-19/P2/`, `P4/05-*`, `P4/06-*`
|
||||||
|
|
||||||
|
- **Built** exactly the design note's recommendation (controller `9bdd8f8`, `601bdba`): the dashboard pass, the host-only
|
||||||
|
files cookie (derived from the session fingerprint, the R-976 shape), the per-box secret header name, the forwardAuth
|
||||||
|
answer; header user `felhom-household` with the household's file rights from userDefaults; `logoutRedirectUrl` = the
|
||||||
|
dashboard; `admin` stays a password user; the data tab's „asks for its own password" note only without the sign-in.
|
||||||
|
- **Tests, each red-proved:** replayed pass, expired pass, pass for another host (two guards: the host check and the
|
||||||
|
host-bound session key — red only with both removed), wrong X-Forwarded-Host, no session → no pass, signed out → asks
|
||||||
|
again (and after a password change), cross-site → no pass, dot-segments in `to`, the absolute redirect, the config
|
||||||
|
trusts only the secret name, the header user's rights, the route asks the controller and strips the header, the note.
|
||||||
|
- **Live, found and fixed before release:** on rc.2 the pass answer's relative redirect became
|
||||||
|
`http://felhom-controller:8080/files/` — traefik resolves a forwardAuth answer's Location against the auth server. Fixed
|
||||||
|
(absolute), red-proved, rc.3. A background security review then flagged CSRF and revocation: `/__files/open` now refuses
|
||||||
|
a cross-site request and dot-segments; the revocation residual is R-1155.
|
||||||
|
- **Live proof (rc.3, 9201, headless Chrome through the public names mapped to the box):** signed in → calibre-web's
|
||||||
|
„Fájlok ↗" → FileBrowser at `NVME 1TB/media/books`, no login form, user `felhom-household` (admin false, modify true,
|
||||||
|
login method proxy); signed out → `files.<domain>` went to the dashboard login. Screenshots `browser/01-…`, `02-…`,
|
||||||
|
`03-…`. Forged headers (`forged-headers.txt`): through traefik with no cookie — the old public name, a random name and the
|
||||||
|
REAL secret name all 401; from the bookstack container straight to `filebrowser:80` — the old name and a random name 401;
|
||||||
|
control: the real name from the same container 200 (the protection is the name's secrecy). The brief named Nextcloud;
|
||||||
|
bookstack was used (an app on 9201's network with curl).
|
||||||
|
|
||||||
|
## 3. Preview check (R-1153) — `audits/day-2026-10-19/P3/`
|
||||||
|
|
||||||
|
On 9202 (controller 0.331.0): OnlyOffice installed, the link on (FileBrowser back in 4 s). The hand-made `teszt.docx`
|
||||||
|
(912 bytes) → 500 „fitz: cannot open document"; OnlyOffice's own `sample.docx` → **200, a 256×256 JPEG, 0.3 s**. Same with
|
||||||
|
the link OFF. FileBrowser 1.5.6 previews .docx with its own MuPDF before OnlyOffice (`preview.go` `determinePreviewType`).
|
||||||
|
**Closed: not a product defect.** OnlyOffice removed through the product (both volumes gone), test files deleted.
|
||||||
|
|
||||||
|
## 4. Controller release 0.332.0 — `audits/day-2026-10-19/P4/`
|
||||||
|
|
||||||
|
Image `0.332.0` from `601bdba` (digest `sha256:245266a4…`), 03:16 UTC. Per-box floors 03:16:43 → Tester 1 and demo-felhom
|
||||||
|
on 0.332.0 at 03:16:45–55, demo-hp (from rc.3) at 03:17. Box walk on all three: 0 unhealthy, 0 allow-list refusals,
|
||||||
|
FileBrowser re-published 6 s after its re-create, public pages 200; https-only 28 OK, 0 failures, both controls right.
|
||||||
|
„Fájlok ↗" with no login through the public internet on demo-hp and Tester 1 (pass → cookie → `felhom-household`; a fresh
|
||||||
|
browser 401; sign-out → back to the dashboard). demo-felhom: CC holds no dashboard password for the lived-in box, so its
|
||||||
|
check is the files (secret 0600, config and route carry it) and the refusals (forged secret name through traefik 401, old
|
||||||
|
name from outside 401, a forged pass 403). **Global floor 0.332.0 / MinAgent 0.158.1 at 03:24:18 UTC**, read back with the
|
||||||
|
WAL; per-box floors cleared (read back empty). MinAgent 0.158.1 ≤ the vouched agent 0.159.2. i18n fixtures changed (named):
|
||||||
|
`escrow_first`, `escrow_receremony`; all others byte-identical. No golden (decision 250); the floor is above golden 0.331.0
|
||||||
|
until Sunday's bake.
|
||||||
|
|
||||||
|
## 5. Hub 0.159.0 (R-1150, R-1154) — `audits/day-2026-10-19/P5/`
|
||||||
|
|
||||||
|
- Written by a helper (`5121536f`, `b505ce4b`). **R-1154's rule, and why:** the helper's first draft re-issued the claim
|
||||||
|
code and the off-site provider password automatically when the old host was stale or switched off. A background
|
||||||
|
security review flagged it; the helper then read what the retrieval passphrase already grants (`/config` → the
|
||||||
|
controller API key; the enroll → the host key) and found the re-issues are the operator's session-only powers
|
||||||
|
(decision 252). So the passphrase must not reach them: **the reuse path re-issues nothing; when the old box looks gone it
|
||||||
|
mails the operator** („delete the old host and run the installer again, or re-issue …"), a live box gets an info record.
|
||||||
|
The operator's delete stays the gate. **R-1150:** `whole_guest_backup_refused` (operator-only, hub-origin; NOT in
|
||||||
|
`allowedEventTypes`, against the brief's wording — the suspected controller must not post or pre-empt it, R-895's
|
||||||
|
shape). Tests red-proved (helper's list in `hub/CHANGELOG.md`).
|
||||||
|
- **A push blocker on the way:** R-887's dated check was due today. Measured: 381 CI jobs since the S21 change, 0 lost (the
|
||||||
|
three re-runs' first attempts ran 47–49 s; control: the known lost attempt of 2026-10-10 ran 11 min 48 s and the attempt
|
||||||
|
count finds it). Closed.
|
||||||
|
- **Deploy:** image from a clean clone (the shared checkout held my uncommitted evidence), manifest `200ab653`, ArgoCD
|
||||||
|
synced the hub Deployment only (the only OutOfSync resource), Synced/Healthy, no ERROR, the floor read back, then fresh
|
||||||
|
reports from all three hosts (02:23–02:25 UTC) and all three controllers (02:29–02:31 UTC).
|
||||||
|
|
||||||
|
## 6. The old-restic check — `audits/evening-2026-10-18/P8/old-restic-check.txt`
|
||||||
|
|
||||||
|
The night watcher ran it: window 12 at 02:18 UTC removed 63 of demo-hp's 324 copies (→ 261); the old restic (0.14.0) and
|
||||||
|
the new (0.19.1) both list 261 and restore a 0.19.1-made copy and an August copy to identical content hashes. STATUS item
|
||||||
|
closed; `REPORT-evening-2026-10-18.md` §9 filled.
|
||||||
|
|
||||||
|
## 7. Versions, commits, fixtures, rows
|
||||||
|
|
||||||
|
- Agent **0.160.0** (released, not vouched) — `1b6e316`, `c11e3e2`, `124164d`, `4545819`. Controller **0.332.0** (floor) —
|
||||||
|
`ba4a6f0`, `9bdd8f8`, `601bdba`; rc.1–rc.3 on 9201 only. Hub **0.159.0** (deployed) — `5121536f`, `b505ce4b`, manifest
|
||||||
|
`200ab653`. felhom.eu `afc939bb` (R-887), `5622ce27`, this commit. Catalog: none.
|
||||||
|
- Fixtures: `escrow_first`, `escrow_receremony` (controller); no other page changed.
|
||||||
|
- **Register: 168 before, 166 after; closed 10** (R-887, R-978, R-973, R-1093, R-1149, R-1151, R-1100, R-1153, R-1150,
|
||||||
|
R-1154), **opened 8** (R-1155 … R-1162).
|
||||||
|
- `unproven.py --summary`: not walked 35 of 55 — unchanged. Capability map: not changed. System poster: text only — the
|
||||||
|
local whole-guest „Kept" cell („one copy per day, up to 3 days") edited in both the facts file and the drawing.
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
1. FileBrowser's own 2-hour token can outlive the dashboard session if only the files cookie is lost. FILED: R-1155.
|
||||||
|
2. A reinstall within 45 min of the old box's last report reads „live" (no mail). FILED: R-1156.
|
||||||
|
3. Repeated cap refusals inside the 24 h hold raise no new alarm. FILED: R-1157.
|
||||||
|
4. The off-site leg of the clean-slate re-issue may be obsolete since hub v0.127.0. FILED: R-1158.
|
||||||
|
5. The retrieval passphrase yields the controller API key, which registers and confirms off-site keys. FILED: R-1159.
|
||||||
|
6. Hub and agent have no gofmt gate (~20 and 14 files drift). FILED: R-1160.
|
||||||
|
7. The escrow wizard calls the code „live" at the reveal; since R-1151 it is live only after the typed-back words.
|
||||||
|
FILED: R-1161.
|
||||||
|
8. R-973's off-site half (ep0 prunes by count) outlives the closed row. FILED: R-1162.
|
||||||
|
9. traefik resolves a forwardAuth answer's relative Location against the auth server. NOT-A-FINDING: found on rc.2 and
|
||||||
|
fixed before the release (red-proved); recorded as a rule in CONTEXT and memory.
|
||||||
|
10. The ceremony log line and two R-973 comments were stale. NOT-A-FINDING: fixed in agent `4545819` (the size rule).
|
||||||
|
11. My first OnlyOffice link call sent `{}` and switched the link off. NOT-A-FINDING: my call; the API wants
|
||||||
|
`{"enabled": true}`; it was already off, nothing restarted.
|
||||||
|
12. The R-1093 route refuses loop devices. NOT-A-FINDING: by design (only real disk names are allowed); a RAM disk was used.
|
||||||
|
13. CC holds no dashboard password for demo-felhom, so its „Fájlok ↗" was not walked as a household. NOT-A-FINDING: the
|
||||||
|
lived-in box's household password is not CC's (decision 254); its files and refusals were checked instead.
|
||||||
|
14. A helper's commits sat unpushed in the shared felhom.eu clone for ~40 minutes (blocked by R-887's due check).
|
||||||
|
NOT-A-FINDING: the gate did its job; pushed with my R-887 commit.
|
||||||
|
15. Two shell scripts of mine were refused by Claude Code's removal check (a cleanup `rm` it could not read behind
|
||||||
|
`bash -c`). NOT-A-FINDING: rewritten without the removal; nothing was deleted.
|
||||||
|
|
||||||
|
## Teardown — three layers
|
||||||
|
|
||||||
|
- **Machines:** 9202 — OnlyOffice removed through the product (volumes gone), test documents deleted; 9201 — on 0.332.0
|
||||||
|
(the floor); the escrow test left demo-hp on code 1 (stored); demo-hp host — the RAM disk module removed, helper scripts
|
||||||
|
deleted, no file left in `/root` (an older helper, `/root/opens.sh`, removed by name at the end). Cookie jars and
|
||||||
|
password files on DooPlex shredded or emptied. One debugging sign-in on demo-hp's dashboard (curl, 03:03 UTC) was not
|
||||||
|
signed out; its cookie file is shredded, so the session expires unused.
|
||||||
|
- **Hub:** global floor 0.332.0 / 0.158.1, no per-box floor; hub 0.159.0; no vouch.
|
||||||
|
- **DooPlex:** builds (agent, controller rc.1–3 + 0.332.0, hub 0.159.0 from a clean clone) and the hub deploy; no Docker
|
||||||
|
cleanup.
|
||||||
|
|
||||||
|
Architecture read: `01`, `02` „Logins and folders", `03` §8a, `04`, `05`, `07` §6/§6.1; the design note. Documents changed:
|
||||||
|
`02`, `03` §8a, `07` §6.1, the poster facts + drawing text, `05` §8 (helper). Instruction files: none changed.
|
||||||
@@ -0,0 +1,120 @@
|
|||||||
|
# REPORT — second day brief, 2026-10-19 (ep0 one copy per week; the passphrase measured; hub 0.160.0; controller 0.333.0; gofmt gates)
|
||||||
|
|
||||||
|
## In plain words
|
||||||
|
|
||||||
|
- **ep0 keeps one whole-box copy per week, for 2 weeks** (your „yes"). Before the change, PBS's own dry run showed it
|
||||||
|
removes nothing today. The first real run is 2026-10-20 03:30 UTC; CC reads it then (row R-1162 stays open until then).
|
||||||
|
- **The household passphrase:** alone it gets the box's keys, and with them every off-site key action. Two options are
|
||||||
|
in STATUS (D19). CC picks A: the passphrase gives keys only while the customer has no box record.
|
||||||
|
- **Controller 0.333.0 is the floor for every box** (12:16 local): a dashboard sign-out now also ends the file manager's
|
||||||
|
own 2-hour pass; the recovery-code page says the code becomes live after the two words.
|
||||||
|
- **Hub 0.160.0 is live** (your „yes"): a reinstall within 45 minutes mails you too; a repeated refused backup is counted
|
||||||
|
and the alarm repeats at most once a day; the old off-site re-issue on a reinstall is kept, with true words.
|
||||||
|
- **gofmt:** the hub and the agent are formatted, and both now have the controller's gate.
|
||||||
|
- **New for you:** another device keeps taking demo-hp's network address (R-1166) — fix it in the router.
|
||||||
|
|
||||||
|
## 1. ep0 (R-1162; `07` Plane-2 row and §6.1)
|
||||||
|
|
||||||
|
- **Before** (`audits/day-2026-10-19b/P1/ep0-settings-before.txt`): `prune-demo-hp` and `prune-demo-felhom` —
|
||||||
|
`keep-last 2`, 03:30, ns-scoped, `max-depth 0`; `prune-operator-hubdb` — keep-daily 14 + keep-weekly 8, 03:45.
|
||||||
|
`tester-1` (3 copies) and `Tester-2` (1) have **no job**. ep0's clock is UTC.
|
||||||
|
- **DooPlex's copy of ep0:** `prune-ep0-copy`, `keep-weekly 8`, 07:30 (decision 71) — already by time; not changed.
|
||||||
|
- **Dry run** (PBS's own: `proxmox-backup-debug api create /admin/datastore/felhom-offsite/prune --dry-run true …`, no
|
||||||
|
task created; cross-checked with an offline model of PBS's selection, `P1/prune_sim.py`; 8 snapshots before and after):
|
||||||
|
|
||||||
|
| Box | Snapshots | keep-last 2 removes | keep-weekly 2 keeps | keep-weekly 2 removes |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| demo-hp | 2 | none | 2026-10-08, 2026-10-15 | none |
|
||||||
|
| demo-felhom | 2 | none | 2026-10-06, 2026-10-13 | none |
|
||||||
|
| tester-1 (no job) | 3 | (2026-10-04) | 2026-10-11, 2026-10-18 | (2026-10-04) |
|
||||||
|
| Tester-2 (no job) | 1 | none | 2026-10-04 | none |
|
||||||
|
|
||||||
|
- **Answer:** „Yes, the two rules" (not the tester-1 job). **After** (07:49 UTC, `P1/ep0-settings-after.txt`): both jobs
|
||||||
|
`keep-weekly 2`, schedule 03:30 unchanged, nothing run by hand. Check for 2026-10-20: the 03:30 `prunejob` tasks OK with
|
||||||
|
`keep-weekly 2` and two snapshots per namespace (the next step of R-1162).
|
||||||
|
|
||||||
|
## 2. The household passphrase (R-1159)
|
||||||
|
|
||||||
|
Measured with the hub's real router (`P2/passphrase-reach.txt`; a made-up customer, a fake off-site service). A live
|
||||||
|
measurement on Tester4 was not possible: CC holds no Tester4 passphrase, and reading it from the hub database was
|
||||||
|
refused by CC's own safety check (correctly).
|
||||||
|
|
||||||
|
- Passphrase → `GET /config/{id}` 200 with `hub.api_key` (wrong passphrase 401); `POST /host-enroll` 200 with the
|
||||||
|
EXISTING host's key.
|
||||||
|
- Controller key → `register-key`, `confirm-key` (removes every other line, the box's own key included), `move-aside`,
|
||||||
|
`window-open`, `abandon-request` (the hub deletes a set-aside copy after 7 days unless cancelled), `abandon-cancel`,
|
||||||
|
`abandon-status`: all past auth (wrong key 401 each).
|
||||||
|
- Host key → escrow GET/PUT, recovery-credential PUT, wg, desired-state, jobs, host-report: all past auth.
|
||||||
|
- The passphrase is used only at install time (the installer, the controller's setup wizard).
|
||||||
|
- **Option A (CC's pick):** keys only while the customer has no host record (`/config` leaves out the key, `/host-enroll`
|
||||||
|
refuses once a host exists). Closes the chain at its source; a reinstall needs the operator's delete first — already the
|
||||||
|
rule since R-1154. **Option B:** a second factor on the destructive off-site routes. Every reinstall waits for the
|
||||||
|
operator, and the host key stays reachable by the passphrase, so B alone does not close it. No change in this task.
|
||||||
|
|
||||||
|
## 3. Controller 0.333.0 — R-1155, R-1161, the Install chip
|
||||||
|
|
||||||
|
- **R-1155:** first measured on demo-hp (0.332.0): the household's FileBrowser token opened FileBrowser after the
|
||||||
|
dashboard sign-out (200, cookie and Bearer). Fix: the forwardAuth answer refuses a request with no valid files cookie
|
||||||
|
that carries a non-admin FileBrowser token, everywhere FileBrowser reads one (cookie, `?auth=`, Bearer, Basic
|
||||||
|
password). **The commit's security review found Basic missing** in the first version (`c809dd3`); fixed in `a6fa3c0`
|
||||||
|
before any release. Red-proofs: without the check → 200 after the sign-out; without the Basic branch → 200; with every
|
||||||
|
token refused → admin's own token 302. Live (rc.2 demo-hp, 0.333.0 Tester 1): files cookie 200; token alone 401 in all
|
||||||
|
four forms, before and after the sign-out; FileBrowser's sign-in page 200.
|
||||||
|
- **R-1161:** new key `backups_escrow.ez_a_kod_az_ellenorzes` (hu + en); the old sentence only when the claim answer says
|
||||||
|
no confirm. Fixtures re-captured: `escrow_first`, `escrow_receremony` (reason: R-1151 made the old sentence untrue);
|
||||||
|
all others byte-identical. Live on demo-hp (`P6/06-escrow-page-text.txt`): new text shown, old hidden, switch present,
|
||||||
|
negative control 0.
|
||||||
|
- **The Chrome „Install" chip: NOT-A-FINDING.** FileBrowser 1.5.6 links `/public/static/site.webmanifest` on every page;
|
||||||
|
its only PWA setting is `disablePWAInstall` (already set). Not patched (`P3/install-chip.txt`).
|
||||||
|
|
||||||
|
## 4. Hub 0.160.0
|
||||||
|
|
||||||
|
- **R-1156:** a live re-enroll is a warning mailed to the operator, same instruction; a `--resume` mails too and says it
|
||||||
|
needs nothing. **R-1157:** further refusals in one episode counted (`refusals_in_episode`), alarm at most once a day,
|
||||||
|
total on the cleared record. **R-1158:** the reinstall's off-site re-issue changes nothing a box uses (only the hub's
|
||||||
|
own registrar reads that password; no box has received it since v0.127.0); kept because it repairs a broken stored
|
||||||
|
login; comments and the `offsite_reissued` text corrected (`05` §8). Each red-proved.
|
||||||
|
- **Answer:** „Yes, deploy now". Image `0.160.0` digest `sha256:e9a360bc00d00a15014e272c4d023fef118a8b043956f3fc928dd26c00b34230`
|
||||||
|
(clean clone at `a17db0c3`; the first push hit a DNS timeout, the retry pushed it). Manifest `5d3cb5c1`; only
|
||||||
|
Deployment/hub was OutOfSync and only it was synced; Synced/Healthy; running image digest = built digest; 0 `[ERROR]`
|
||||||
|
lines; floor read back from hub.db + WAL (0.332.0 then, 0.333.0 now); host and controller reports from demo-hp, Tester 1
|
||||||
|
and demo-felhom after the restart.
|
||||||
|
|
||||||
|
## 5. gofmt (R-1160)
|
||||||
|
|
||||||
|
Hub `d5e3c8b0` (22 files) and agent `1edfd8e` (14 files), formatting only, tests green before each. Gates: `hub-gofmt`
|
||||||
|
(`scripts/hub_gofmt_gate.py`, `repo_gates.py`) and `gofmt` (felhom-agent `scripts/gofmt_gate.py`, `agent_gates.py`), fast
|
||||||
|
set, decoys on scratch trees, red-proved. No agent release.
|
||||||
|
|
||||||
|
## 6. Controller release (decision 250)
|
||||||
|
|
||||||
|
Per-box floors 0.333.0 at 09:59:38 UTC → all three on 0.333.0 within about a minute (demo-hp from rc.2 at 10:13). Box
|
||||||
|
walk (`P6/`): 0 unhealthy, 0 allow-list refusals, catalog in sync; https-only 28 OK, 0 failures; „Fájlok ↗" with no second
|
||||||
|
login on demo-hp and Tester 1, sign-out asks again. **Global floor 0.333.0 / MinAgent 0.158.1 at 10:16:22 UTC**
|
||||||
|
(303 `floor_set`, read back with the WAL); per-box floors cleared.
|
||||||
|
|
||||||
|
## 7. Versions, commits, fixtures, rows
|
||||||
|
|
||||||
|
- **Controller 0.333.0** (`c809dd3`, `a6fa3c0`, REPORT `e1a2abf`), image digest
|
||||||
|
`sha256:f150a8d8cd329f5cde9e34249e34edc59b6b5db29db1e3200900f3869a0de11b`; rc.1/rc.2 on demo-hp only. Fixtures:
|
||||||
|
`escrow_first`, `escrow_receremony`.
|
||||||
|
- **Hub 0.160.0** (`d5e3c8b0`, `503c93fa`, `a17db0c3`, manifest `5d3cb5c1`).
|
||||||
|
- **Agent:** no release (`1edfd8e`, `0cbd500`). 0.160.0 stays the vouched one.
|
||||||
|
- **Rows: 167 before, 163 after; closed 6** (R-1155, R-1156, R-1157, R-1158, R-1160, R-1161), **opened 2** (R-1165,
|
||||||
|
R-1166); R-1159 → READY (D19), R-1162 → NARROWED (one run to read).
|
||||||
|
- **System poster:** text only — `felhom-system-poster.facts.md` and the matching text in `felhom-system-poster.html`
|
||||||
|
(„last 2" → „one per week, 2 weeks").
|
||||||
|
- **Teardown:** machine — rc images only on demo-hp 9201, now on the release; no test app. Host — none. Hub — no test
|
||||||
|
customer; the per-box floors cleared. ep0 — the two prune jobs changed on the operator's yes, nothing else.
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
1. Two household namespaces on ep0 have no prune job; nothing creates one for a new household. FILED: R-1165.
|
||||||
|
2. Another device takes demo-hp's address 192.168.0.104 every few minutes; demo-hp also lost its link to ep0 for 8 minutes. FILED: R-1166.
|
||||||
|
3. A config download with the passphrase is a log line only, and the host key it yields can rewrite the box's escrow and recovery credential. FILED: R-1159.
|
||||||
|
4. The first R-1155 version missed FileBrowser's Basic-auth token source (the commit's security review). NOT-A-FINDING: fixed before any release in `a6fa3c0` and red-proved (the size rule).
|
||||||
|
5. Chrome offers „Install" for the file manager. NOT-A-FINDING: FileBrowser 1.5.6 has no setting that stops its manifest; the brief forbids patching it.
|
||||||
|
6. One of CC's probe scripts on DooPlex passed the household's FileBrowser token to curl on its command line (visible to local processes for about a second; never printed or saved). NOT-A-FINDING: CC's own tooling on DooPlex, not the product; every probe now passes tokens through 0600 header files, and the token expired two hours later.
|
||||||
|
7. Two copies of CC's hub-database snapshot ran into one folder at once and one read failed („malformed"). NOT-A-FINDING: CC's helper, a local copy only; the live hub and its database were untouched, and the next read was clean.
|
||||||
|
8. The first hub image push failed on a DNS timeout at DooPlex. NOT-A-FINDING: a short DNS outage (the website session saw it too); the build's own check found the tag absent and the retry pushed it.
|
||||||
|
9. FileBrowser's `admin` password login was not walked live. NOT-A-FINDING: CC holds no FileBrowser admin password; admin's token passing is unit-pinned and the sign-in page answers 200 live.
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
# REPORT — third day brief, 2026-10-19 (the passphrase locks once the box uses its own key; every ep0 household gets a prune job; demo-hp's address)
|
||||||
|
|
||||||
|
## In plain words
|
||||||
|
|
||||||
|
- **The household passphrase no longer gives a live box's keys** (your D19). Each box now fetches its config with its own
|
||||||
|
key. The passphrase still works for a new install. A reinstall over a live box needs you to delete the old host first.
|
||||||
|
- **The brief's first shape could not be built as written.** "Keys only before a box exists" would have broken every
|
||||||
|
install and every live box's config updates: both use the passphrase after the box record exists. CC stopped and asked;
|
||||||
|
you chose "box key first, then lock".
|
||||||
|
- **ep0:** every new household gets its weekly clean-up rule automatically, and the hub alarms on a household without
|
||||||
|
one. tester-1 and Tester-2 got theirs (your yes).
|
||||||
|
- **demo-hp's address** stayed demo-hp's for a 30-minute watch after your router change.
|
||||||
|
- **Live now:** hub 0.161.0, controller 0.334.0 (the floor), tenant-sync v1.3.0 on ep0.
|
||||||
|
|
||||||
|
## 1. The passphrase (R-1159; `05` §8 [DESIGN R-1159])
|
||||||
|
|
||||||
|
- **Install-order proof** (read before building, the brief's STOP): the passphrase is used AFTER `POST /host-enroll`:
|
||||||
|
- `scripts/felhom-host-install.sh:2631` → `:480`, and `:3525`: `step_agent_install` and `step_golden` fetch the
|
||||||
|
download login from `GET /config`;
|
||||||
|
- `felhom-agent/cmd/felhom-agent/main.go:2835`: the provision bakes the passphrase into the guest's `bootstrap.json`;
|
||||||
|
- `felhom-controller/controller/internal/bootstrap/bootstrap.go:142`: the guest's first boot pulls `GET /config` with it
|
||||||
|
(that is how the box gets its own key);
|
||||||
|
- `cmd/controller/main.go:1176` (`report.ConfigRefresher`): every live box re-pulled with it on each config change;
|
||||||
|
- `:2317`: the preflight checks it with `/config` on every run.
|
||||||
|
- **Your answer:** „Box key first, then lock".
|
||||||
|
- **Built:**
|
||||||
|
- Hub 0.161.0: `GET /box-config/{id}` (the controller's own key) and `passphraseKeysClosed`. Passphrase keys are
|
||||||
|
refused (409, no key, operator event `passphrase_keys_refused`) once a ≥ 0.334.0 controller has reported since the
|
||||||
|
host record was made. `/config`, `/recovery` and the `/host-enroll` reuse path are covered; `/artifacts` has no key
|
||||||
|
and is unchanged.
|
||||||
|
- Controller 0.334.0: the refresh pulls with its own key (`report.PullConfigPreferKey`). The passphrase is used only
|
||||||
|
against an older hub or before the box has a key, and never as a retry after a refused key.
|
||||||
|
- **Tests:**
|
||||||
|
- Hub: `TestR1159_PassphraseGivesKeysOnlyUntilTheBoxUsesItsOwn` (5 cases), `TestR1159_BoxPullsItsConfigWithItsOwnKey`,
|
||||||
|
`TestR1159_BoxConfigMinControllerIsTheReleaseThatPullsWithItsKey`.
|
||||||
|
- Controller: `TestR1159_RefreshUsesTheBoxKey`, `TestR1159_MainWiresTheKeyFirstRefresh`.
|
||||||
|
- Red-proofs: lock off → all three keys served; the old pull → 0 key pulls.
|
||||||
|
- **Live on demo-hp** (`audits/day-2026-10-19c/P4/`; the probe ran on the host from the box's own `bootstrap.json`,
|
||||||
|
printing status codes only):
|
||||||
|
- on 0.333.0, the passphrase → 200;
|
||||||
|
- after its first 0.334.0 report (14:30:13 UTC), the passphrase → **409, no key**, and `passphrase_keys_refused` was
|
||||||
|
raised (the operator got the mail — CC's probe);
|
||||||
|
- its own key → 200; a wrong key → 401.
|
||||||
|
|
||||||
|
## 2. ep0 prune jobs (R-1165; `07` Plane-2 row)
|
||||||
|
|
||||||
|
- **Automatic:** `felhom-tenantsync` v1.3.0 creates `prune-<id>` (keep-weekly 2, 03:30, max-depth 0) with a household's
|
||||||
|
namespace unless a job already names it; it removes the job with the household. Its `usage` answer lists namespaces no
|
||||||
|
job covers.
|
||||||
|
- The hub alarms on that list (`pbsdr_namespace_without_prune_job`). An unread list never reads as "all covered".
|
||||||
|
- Tests: 12 script cases (`scripts/test_tenantsync_prune.py`, run on every push), the checker and the client; each
|
||||||
|
red-proved.
|
||||||
|
- **Dry run before the question** (PBS's own, `P2/ep0-dry-run-tester.txt`; 32 snapshots before and after):
|
||||||
|
- tester-1: keep 2026-10-11, 2026-10-18; remove 2026-10-04.
|
||||||
|
- Tester-2: keep 2026-10-04.
|
||||||
|
- **Answer:** „Yes, add both". **After** (13:35 UTC, `P2/ep0-jobs-after.txt`): `prune-tester-1` and `prune-Tester-2`
|
||||||
|
(keep-weekly 2, 03:30) beside the demo-hp, demo-felhom and operator jobs.
|
||||||
|
- **Script on ep0** (14:07 UTC, the same yes): the installed v1.2.0 matched the repo byte for byte; v1.3.0 is installed
|
||||||
|
(hash = the repo's `36412c6a…`), the old copy kept as `.bak-1.2.0`. Live `usage` → `unpruned: []`; `fingerprint` still
|
||||||
|
answers.
|
||||||
|
|
||||||
|
## 3. demo-hp's address (R-1166)
|
||||||
|
|
||||||
|
- A watch every 10 s for 30 minutes (10:59–11:29 UTC): **180 samples, 0 with another MAC**, SSH open throughout.
|
||||||
|
- The host's own MAC (`cat /sys/class/net/vmbr0/address`): **`7c:d3:0a:77:d9:76`** — the same as your router entry.
|
||||||
|
- Note: your DHCP pool is .120–.160, so .104 was never in it. The other device (`b8:06:0d:60:45:58`) most likely had .104
|
||||||
|
set by hand. If it comes back, look there.
|
||||||
|
|
||||||
|
## 4. Hub deploy and the controller release
|
||||||
|
|
||||||
|
- **Hub 0.161.0**
|
||||||
|
- Image digest: `sha256:d56c96a59c86ce2d006938d0980803a12949bcbf027cea2834cabd64118838b8` (clean clone at `b6a911c3`).
|
||||||
|
- Manifest `97f27a75`; only Deployment/hub was synced. Result: Synced/Healthy; the running digest is the built one;
|
||||||
|
0 `[ERROR]` lines; the ep0 read at 14:13 UTC was OK, with no alarm (every namespace has a job).
|
||||||
|
- Checks: the floor was read back; host and controller reports from demo-hp, Tester 1 and demo-felhom arrived after
|
||||||
|
the restart.
|
||||||
|
- **Controller 0.334.0**
|
||||||
|
- Image digest: `sha256:deeabe59d932cd60f8939a37bccf94f145555101fc7be64b6c5cc15f3557031a` (from `546eef3`); rc.1 ran
|
||||||
|
on demo-hp only.
|
||||||
|
- Per-box floors at 14:16:11 UTC. Tester 1 and demo-felhom moved at once; demo-hp moved at 14:30 (from rc.1).
|
||||||
|
- Box walk (`P5/`): 0 unhealthy, 0 refusals, catalog in sync; https-only 28 OK, 0 failures; „Fájlok ↗" with no second
|
||||||
|
login on demo-hp and Tester 1.
|
||||||
|
- **Global floor 0.334.0 / MinAgent 0.158.1 at 14:31:44 UTC.** The per-box floors were cleared.
|
||||||
|
|
||||||
|
## 5. Versions, commits, rows
|
||||||
|
|
||||||
|
- **Hub 0.161.0** (`b6a911c3`, manifest `97f27a75`); **controller 0.334.0** (`546eef3`); **tenantsync v1.3.0** (in
|
||||||
|
`b6a911c3`, on ep0). No agent change. No i18n fixture changed (no page changed).
|
||||||
|
- **Rows: 163 before, 161 after; closed 3** (R-1159, R-1165, R-1166), **opened 1** (R-1167).
|
||||||
|
- **System poster:** facts file only (`felhom-system-poster.facts.md`: new households get their prune job); the drawing's
|
||||||
|
text („one per week, 2 weeks") is still true.
|
||||||
|
- **Teardown:**
|
||||||
|
- Machine: rc.1 only on demo-hp 9201, now on the release; the probe scripts removed from demo-hp.
|
||||||
|
- Host: none.
|
||||||
|
- Hub: per-box floors cleared; one `passphrase_keys_refused` event on demo-hp (CC's probe).
|
||||||
|
- ep0: two jobs added and the script replaced, both on the operator's yes; nothing else.
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
1. The installer prints „unexpected hub status 409" instead of the hub's sentence when a re-run meets a live box. FILED: R-1167.
|
||||||
|
2. The brief's „keys only while no host exists" would have broken every install and every live box's config refresh. NOT-A-FINDING: measured before building and put to the operator, who chose the two-step shape (D19); recorded in `05` §8.
|
||||||
|
3. A reinstall over a live box now gets two operator mails (R-1154's reinstall warning and the refusal). NOT-A-FINDING: both are true and give the same instruction; the 1-hour cooldown limits repeats.
|
||||||
|
4. An rc build (`0.334.0-rc.N`) counts as older than 0.334.0, so the lock stays open on an rc. NOT-A-FINDING: by design (an rc runs on demo-hp only, and the floor moves it to the release); proven on rc.1 (200) and the release (409).
|
||||||
|
5. demo-hp's .104 lies outside the router's DHCP pool, yet another MAC held it this morning. NOT-A-FINDING: the router reservation fixed it (30 minutes clean); the other device's source is noted in R-1166's closing text in case it returns.
|
||||||
@@ -0,0 +1,153 @@
|
|||||||
|
# REPORT — evening → night 2026-10-18
|
||||||
|
|
||||||
|
## How I read the brief
|
||||||
|
|
||||||
|
- **Part 5, R-1135:** the brief says the mail should use the public Gitea URL — the register row says the same, and that
|
||||||
|
is what was done (the CI helper first read the brief's wording the other way round and followed the row).
|
||||||
|
- **R-1052** was already closed (2026-10-15); Part 1 proves R-1148 and re-proves R-1052's path with tonight's golden.
|
||||||
|
- **Part 6, Immich:** 9202's Immich lives on `/mnt/sizing-bench`, a hand-made bench folder outside `/mnt/felhom-drives/`,
|
||||||
|
so the allow-list refuses its definition (see Observations); the restore half of the measurement ran on Nextcloud.
|
||||||
|
- **The operator kept Tester4** (asked at 21:20): not deleted; its old host must go before its next install (R-1154).
|
||||||
|
|
||||||
|
## In plain words
|
||||||
|
|
||||||
|
- **The morning list:** vouch golden 0.331.0; say yes or no to the agent release with three fixes; two decisions (one
|
||||||
|
sign-in, local backup retention).
|
||||||
|
- **A brand-new box from nothing works end to end** with installer 1.34.0: emergency password saved in the hub first,
|
||||||
|
then set, then the web login took it; the approved golden and agent; the box moved itself to the floor.
|
||||||
|
- **OnlyOffice now really works** from FileBrowser and from Nextcloud. Before tonight FileBrowser could not open a
|
||||||
|
document and Nextcloud lost every save while saying „saved". Fixed in controller 0.331.0, now the floor for every box.
|
||||||
|
- **One sign-in for the file manager can be built safely**, but only with two safeguards — measured, design note written.
|
||||||
|
- **Three agent fixes are written and tested, not released.**
|
||||||
|
- **CI:** the failure mail's link opens from outside; the 5-minute CI timeout never worked (filed); a rerun hides the
|
||||||
|
first try's log, and the check now looks for it.
|
||||||
|
- **Redis for Immich and Nextcloud has no disk volume any more**, after a test; the test found that removing an app left
|
||||||
|
an old volume behind — fixed too.
|
||||||
|
- **Golden 0.331.0 is baked** (not vouched).
|
||||||
|
- **The old-restic check:** §9.
|
||||||
|
|
||||||
|
## 1. The morning list
|
||||||
|
|
||||||
|
1. **Vouch golden 0.331.0** — sha256 `3881de62b0986ccbe5e55580e3f1bb9765d65b8d898f67c6339bac105244f84f` (registry round
|
||||||
|
trip = the bake), agent 0.159.2 unchanged, **Min agent 0.158.1**.
|
||||||
|
2. **Agent release (R-978, R-973, R-1093) — may I?** If yes: release, prove on demo-hp, ask for the vouch.
|
||||||
|
3. **Decide D16** (R-1100: build one sign-in with the two safeguards?) and **D17** (R-1149: local whole-box backups
|
||||||
|
„last 3" or „one per day, 3 days"?). STATUS has both.
|
||||||
|
4. Before Tester4's next install: delete its old host `Tester4-31fa0e` (Hosts page).
|
||||||
|
|
||||||
|
## 2. Fresh install from nothing (R-1148, R-1052) — `audits/newbox-2026-10-18/`
|
||||||
|
|
||||||
|
| UTC | Step | Result |
|
||||||
|
|---|---|---|
|
||||||
|
| 17:42:21 | Tester4 created by the operator | bind mail sent |
|
||||||
|
| 17:43:15 | VM 9302 `newbox-r1148` on demo-hp, ISO 1.29.0 (sha = published) | 8 GB / 4 cores / 120 GB on `/mnt/hdd_1` root |
|
||||||
|
| 17:50:19 | Install pressed (graphical, Hungarian keyboard kept) | installed, rebooted, pairing code on the console |
|
||||||
|
| 18:02:58 | bind: mailed link + pairing code + owner phrase (operator's 0600 file, shredded) | „Sikeres összekötés" |
|
||||||
|
| 18:03:43 | step 4b: vault → chpasswd → PVE login | `[OK] … vaulted, set, and the PVE login takes it`; hub `set_at` same second |
|
||||||
|
| 18:03:44 | step 5: the hub manifest | agent **0.159.2**, golden **0.329.0** — the vouched pair, both sha-verified |
|
||||||
|
| 18:05:37 → 18:05:51 | first controller reports | 0.329.0, then **0.330.0** (`initiated_by: auto-floor` at 18:05:38) — 2 min 47 s after the bind |
|
||||||
|
| 18:17:14 | CC's PVE login from DooPlex | vaulted value **200**, installer value **401** |
|
||||||
|
| 18:18:32 → 18:20:05 | claim (setup code mail), BentoPDF | claimed; BentoPDF healthy |
|
||||||
|
| 18:24:15 | household „turn off" | hub `deliberate_off_at` set; VM stopped |
|
||||||
|
| — | teardown | name read twice, VM 9302 destroyed, disk gone; other VMs untouched; Tester4 kept (operator) |
|
||||||
|
|
||||||
|
## 3. OnlyOffice (R-1018) — `audits/onlyoffice-2026-10-18/`
|
||||||
|
|
||||||
|
On 0.330.0: FileBrowser → the editor loaded, the document never did (OnlyOffice fetched `https://files.<domain>` → 403);
|
||||||
|
Nextcloud → edit and „All changes saved", but the file kept the old text (`http://onlyoffice:80cache/…`). Switching the
|
||||||
|
link on/off left files.<domain> 502 for 4 min 8 s / 3 min 38 s. **Fixed (controller 0.331.0):** FileBrowser
|
||||||
|
`server.internalUrl`, Nextcloud's slash, the gate after a FileBrowser restart. On rc.1: both open, edit, save, read back
|
||||||
|
(„… RC1-FB-OK", „… RC1-NC-OK"); the stop revokes both (no editor; Nextcloud 404); the stop's outage one 2-s sample.
|
||||||
|
|
||||||
|
## 4. One sign-in spike (R-1100 → READY) — `audits/proxyauth-spike-2026-10-18/DESIGN-NOTE.md`
|
||||||
|
|
||||||
|
Traefik forwardAuth strips a client's header (any case, twice) and its own value wins — measured. But an app container
|
||||||
|
reaching FileBrowser directly logs in with a forged header (200), and the dashboard cookie never reaches `files.`. Verdict:
|
||||||
|
build only with a files-only ticket cookie and a per-box secret header name. Everything on 9202 removed (users, routes,
|
||||||
|
stand-in container, config).
|
||||||
|
|
||||||
|
## 5. Agent fixes (committed, NOT released) — felhom-agent `REPORT.md`
|
||||||
|
|
||||||
|
R-1093 `f7b7308` + `0963bae` (and a fail-open hole closed after a background review), R-978 `8561dc8`, R-973 `f17e197`;
|
||||||
|
tests and red-proofs for each; CI job 2062 success. R-1093 live: a loop disk on demo-hp showed the probe's condition holds.
|
||||||
|
|
||||||
|
## 6. CI (R-1135, R-1136, R-1137 closed)
|
||||||
|
|
||||||
|
Link public and by run id (forced mail 18:46:01Z, job 2067, one mail). The runner's real limit is 3 h; the job-level
|
||||||
|
5-minute timeout is not enforced (R-265 corrected; R-1152 for the step-level one). A rerun replaces the job's attempt; the
|
||||||
|
pull-check recipe in `CLAUDE.md` now reads `actions/tasks` (instruction-file edit by the CI helper: felhom.eu `CLAUDE.md`
|
||||||
|
CI recipe, item 3 added — a proven fact, stricter; `b9b56c47`).
|
||||||
|
|
||||||
|
## 7. Redis volumes (R-1117 closed) — `audits/redis-volumes-2026-10-18/`
|
||||||
|
|
||||||
|
(a) Both apps healthy on the new shape; Immich's metadata job ran (0 → 11, 0 failed); Nextcloud `cron.php` rc 0, WebDAV
|
||||||
|
200. (b) The dump skips the dropped volume (the old tar gone from the unit); a second-drive restore from a copy made WITH
|
||||||
|
the volume: 3/3 volumes, 1/1 database, healthy — it brings back that copy's compose (by design). (c) The old volume stays
|
||||||
|
attached under the tmpfs; **a removal left it behind** — fixed (controller 0.331.0, red-proofed). Templates changed
|
||||||
|
(catalog `30eea97`); Nextcloud on 9202 took it through the sync + restart (tmpfs, `save ""`).
|
||||||
|
|
||||||
|
## 8. Controller release and golden — `audits/evening-2026-10-18/P7/`, `tests/golden-0.331.0-2026-10-18/`
|
||||||
|
|
||||||
|
0.331.0 (digest `1b11d25a…`): per-box floors 20:44:00 → all three on 0.331.0 within 20 s; box walk green (0 refusals,
|
||||||
|
0 unhealthy, FileBrowser re-published in 5–6 s, public pages 200 after one 20-s blip each on two names, https-only 28 OK);
|
||||||
|
**global floor 0.331.0 / 0.158.1 at 20:48:56 UTC**, read back with the WAL; per-box floors cleared. Golden 0.331.0 baked
|
||||||
|
after it, round trip = bake sha, token checks 0 with a working control, teardown clean. No controller push after the bake.
|
||||||
|
|
||||||
|
## 9. The old-restic check (Part 8)
|
||||||
|
|
||||||
|
**Done 2026-10-19 by the watcher** (filled in by the day session, `audits/evening-2026-10-18/P8/old-restic-check.txt`):
|
||||||
|
demo-hp's clean-up window 12 ran at 02:17:53–02:18:24 UTC after the night run — „63 of 193 planned snapshot(s) removed,
|
||||||
|
324 -> 261" (the hub's row: `pruned`, 324 → 261). Then `rx.sh` (read-only): the new restic (0.19.1) and the OLD one (image
|
||||||
|
0.322.0, restic 0.14.0) both list **261**; both restore `c0b6299c` (made by 0.19.1, 02:17:48) to the same content hash
|
||||||
|
`d808e6537c7bcdb6` (7 files) and `05c3346a` (2026-08-31) to `8df241f4cf889265` (10 files). The old restic still reads and
|
||||||
|
restores demo-hp's store after its first real 0.19.1 prune. (Pre-check at 20:57 UTC: 310 and 310.)
|
||||||
|
|
||||||
|
## 10. Versions, commits, fixtures, rows
|
||||||
|
|
||||||
|
- Controller **0.331.0** — `772ef36`, `8ae9178` (+ `615940d` CI); rc.1 on 9202 only. Agent `main` `f7b7308` `8561dc8`
|
||||||
|
`0963bae` `f17e197` `79ba884` `1693da4` — unreleased. Catalog `c97eb1c`, `30eea97`. felhom.eu `1fe9f848`, `b9b56c47`,
|
||||||
|
this commit. Golden **0.331.0** baked. Hub unchanged (0.158.0).
|
||||||
|
- Fixtures: none changed — no page changed.
|
||||||
|
- **Register: 168 before, 168 after; closed 6** (R-1148, R-1018, R-1117, R-1135, R-1136, R-1137), **opened 6**
|
||||||
|
(R-1149, R-1150, R-1151, R-1152, R-1153, R-1154). Updated: R-1100 (READY), R-978, R-973, R-1093 (fixed in agent
|
||||||
|
`main`), R-265 (corrected).
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
1. Installer password, golden and agent all proven in one fresh install. — closes R-1148 (not a finding).
|
||||||
|
2. FileBrowser's office preview answered 500 with the link on. FILED: R-1153.
|
||||||
|
3. A reinstall reuses an old host and skips two re-issues; now recurring with the kept Tester4. FILED: R-1154.
|
||||||
|
4. The R-973 cap leaves the local keep-last 3 floodable within a day. FILED: R-1149.
|
||||||
|
5. The cap's refusal reaches the operator with a misleading event text. FILED: R-1150.
|
||||||
|
6. R-978's typed-back confirmation is browser-only. FILED: R-1151.
|
||||||
|
7. The CI job timeout is inert. FILED: R-1152.
|
||||||
|
8. 9202's Immich sits on `/mnt/sizing-bench`, outside `/mnt/felhom-drives/`, so the allow-list refuses its definition every
|
||||||
|
15 minutes since 0.328.0 and a restore of it is refused. NOT-A-FINDING: the check works as designed; the folder is a
|
||||||
|
hand-made bench artifact (R-994), not a shape a real box has.
|
||||||
|
9. A restore brings back the backup's own compose (the old Redis volume with it). NOT-A-FINDING: by design (R-47: the
|
||||||
|
unit's compose becomes the live one); the older shape works.
|
||||||
|
10. My commit `7f7081d1` (day session) carries a „Co-Authored-By" trailer, against both briefs' rule 10. NOT-A-FINDING:
|
||||||
|
my slip; it is pushed and cannot be rewritten without a force-push (forbidden); no later commit has one.
|
||||||
|
11. I started the catalog's FULL gate run on DooPlex, whose third gate starts containers. NOT-A-FINDING: stopped at gate 2
|
||||||
|
(a network check), no container was made (checked), the fast run used instead.
|
||||||
|
12. One OnlyOffice „Connection is lost" on the first rc.1 attempt, after two blind clicks. NOT-A-FINDING: not reproduced;
|
||||||
|
the next attempt saved; no container restarted in between.
|
||||||
|
13. Two public names answered 000 once in the box walk. NOT-A-FINDING: 18 of 18 retries answered 200 in < 0.2 s.
|
||||||
|
14. The R-1100 row named the config key `proxyAuth`; it is `auth.methods.proxy`. NOT-A-FINDING: corrected in the row.
|
||||||
|
|
||||||
|
## Teardown — three layers
|
||||||
|
|
||||||
|
- **Machines:** VM 9302 destroyed (its disk gone). 9202: OnlyOffice and Nextcloud removed through the product (their
|
||||||
|
volumes, data and backup copies), the test documents deleted, the spike's route file, container, FileBrowser users and
|
||||||
|
proxy setting removed, helper files and the Immich scratch password shredded; Immich keeps the new Redis shape by hand
|
||||||
|
(= the template); 9202 on controller 0.331.0. `nextcloud_nextcloud_redis_data` (the orphan the pre-fix
|
||||||
|
removal left) removed by name afterwards; no Nextcloud volume remains. demo-hp host: the loop-disk file removed. Drill VM: CT 9100 purged, VM off,
|
||||||
|
disk `virgin`.
|
||||||
|
- **Hub:** global floor 0.331.0 / 0.158.1, no per-box floor; Tester4 and its host `Tester4-31fa0e` kept (operator); no
|
||||||
|
vouch, no deploy.
|
||||||
|
- **DooPlex:** builds and the golden bake only; the headless browser as a test client; no Docker cleanup.
|
||||||
|
|
||||||
|
Architecture read: `01`, `02`, `03` §8a, `04`, `05`, `07` §6/§8, `09` §3 decisions 162/250; `target-selection.md`,
|
||||||
|
`nightly-rotation.md`, `RUNBOOK-manual-build.md` §4. Documents changed: `03` §8a, `07` §6.1 and §8 row 10,
|
||||||
|
`target-selection.md` (Tester4). System poster: none of the three (no listed fact changed). `unproven.py --summary`:
|
||||||
|
not walked 35 of 55 — unchanged.
|
||||||
@@ -0,0 +1,131 @@
|
|||||||
|
# REPORT — the day brief of 2026-10-21: fix the four serious problems from the rehearsal, and the Sunday clock change
|
||||||
|
|
||||||
|
Separate from the shared `REPORT.md` on purpose: another session works in this clone (repo rule: a second session writes
|
||||||
|
`REPORT-<topic>.md`). Evidence: `documentation/audits/fixes-2026-10-21/`. Design read first: `03` §8a, `07` §7.0/§6/§8.2,
|
||||||
|
`05` (scheduler, escrow store), `04` (self-scoped reads), `09` §3 decisions D19, 78, R-241.
|
||||||
|
|
||||||
|
## 1. Five lines
|
||||||
|
|
||||||
|
1. **R-1176 (a rebuilt box gets its off-site backups back) — FIXED AND LIVE.** Hub 0.163.0 (operator yes, 11:46 UTC);
|
||||||
|
proven end to end on demo-hp: no fresh key, recovery offered, wrong code refused, right code opened the history,
|
||||||
|
Nextcloud 3 031 of 3 033 files byte-identical, the next off-site runs appended (63 → 64).
|
||||||
|
2. **R-1175 (shares on two drives) — FIXED AND LIVE.** Controller 0.337.0 (global floor 15:54 UTC), D25: a gone source
|
||||||
|
keeps its copy until the household removes it.
|
||||||
|
3. **R-1191 (an imported app keeps its family lock) — FIXED AND LIVE.** Controller 0.337.0; proven on 9202: a stranger
|
||||||
|
gets 401 after export → remove → import (it was 200).
|
||||||
|
4. **R-1192 (the 8 GB freeze) — CAUSE MEASURED, FIX COMMITTED, NOT RELEASED.** The guest's swap was capped at Proxmox's
|
||||||
|
512 MB default; the agent fix (swap = RAM) is committed (`286a7e3`); its release and vouch are the operator's (D26).
|
||||||
|
Proven by applying the same setting by hand for 2 hours of the same uploads.
|
||||||
|
5. **Ready for Tester 2: NO — one step left:** the agent release with the swap fix. Everything else on the P1 list is live.
|
||||||
|
|
||||||
|
## 2. R-1192 — the mechanism, the numbers, the fix
|
||||||
|
|
||||||
|
**One sentence:** the guest's swap is capped at 512 MB (Proxmox's LXC default — the agent never set `swap`), so when
|
||||||
|
Immich loaded its analysis models (+1.15 GB) on top of 8 apps, the 4096 MB guest could evict only program code, and
|
||||||
|
~25 containers re-read their code from disk without end.
|
||||||
|
|
||||||
|
| | the freeze (13:26–13:34 UTC, cap 512 MB) | the proof (13:43–15:43 UTC, cap = RAM by hand) |
|
||||||
|
|---|---|---|
|
||||||
|
| guest private memory (anon), max | 3 585 MB | 2 782 MB |
|
||||||
|
| guest swap used, max | 511 MB of 512 (full) | 2 555 MB |
|
||||||
|
| refaults per 10 s, max | 2 741 774 | 187 462 |
|
||||||
|
| disk reads per 10 s, max | 12 GB | 0.9 GB |
|
||||||
|
| memory „full" pressure (avg10), max | 96 % | 12 % |
|
||||||
|
| IO pressure, max | 97 % | 53 % |
|
||||||
|
| load, max | 162 | 33 |
|
||||||
|
| guest „limit hit" events per 10 s, max | 3 889 | 0 |
|
||||||
|
| OOM kills | 0 | 0 |
|
||||||
|
| dashboard (DooPlex probe every 10 s) | dead 13:28:18–13:34:48 | every probe answered, slowest 1.26 s |
|
||||||
|
|
||||||
|
The experiment between them: `pct set 9201 --swap 4096` at 13:34:06, uploads still running → memory „full" 96 % → 1.7 %
|
||||||
|
within 2 minutes, the dashboard back 42 s after the change. The host had 2.2 GB RAM and 7.4 GB swap free throughout.
|
||||||
|
Load for the proof: the same uploads as fresh work, repeated — Paperless 110 cycles × 53 PDFs (emptied each time), Immich
|
||||||
|
111 new users × 306 photos (305 accepted each time; one photo is refused every cycle), two Nextcloud trees (WebDAV small
|
||||||
|
files, ~20 files/min under this load — slow, nothing lost). Method: a recorder on the box (`tools/rec.py`, 10 s, copied to
|
||||||
|
DooPlex every minute) and the second channel on DooPlex (`tools/outside.sh`: the dashboard's `/api/health` and QEMU's
|
||||||
|
disk counters from demo-hp).
|
||||||
|
|
||||||
|
**Fix:** agent `286a7e3` (commit only): `swap` = memory at bring-up, at a RAM resize, and once at agent start for every
|
||||||
|
pool guest whose swap is below its RAM (live, no restart, never lowers). Tests red-proved. The controller half (0.337.0):
|
||||||
|
„Normál használat mellett ez nem okoz problémát." is gone; the install page now says **„Az alkalmazások csúcsterhelése
|
||||||
|
meghaladhatja a rendelkezésre álló memóriát. Ilyenkor a doboz nagy feltöltések alatt lelassulhat, és akár percekig sem
|
||||||
|
válaszol. Ha ez gyakran előfordul, távolíts el egy alkalmazást."** No catalog change and no admission-ceiling change:
|
||||||
|
the measured cause is the swap cap, and the order of preference (catalog → controller → agent) gave way to the proof.
|
||||||
|
R-1193 (an earlier, plainer alarm): a design note in its row; build it only if a freeze is seen after the agent release.
|
||||||
|
|
||||||
|
## 3. R-1176 — the shape, the order check, the live proof
|
||||||
|
|
||||||
|
**Shape (operator ruling):** `store.ReattachRetainedEscrow` moves the newest retained row of the customer's newest
|
||||||
|
escrow-ACKED host deletion into `host_escrow` for its one live host, in one transaction (no other live host, no live
|
||||||
|
package, the deleted host not live again, an identity blob present). Called on the agent's host report and before the
|
||||||
|
controller ACK's escrow status. Never opened, re-sealed or logged; audit `escrow_carried` (host ids + hash); two live
|
||||||
|
hosts → `escrow_carry_refused` mailed once. Self-scope unchanged: the new host reads only rows attached to it (its
|
||||||
|
retained GET for the old host is 403).
|
||||||
|
|
||||||
|
**Order check (measured, not assumed):** the hub carried at the agent's FIRST report (12:04:21 UTC); the controller's
|
||||||
|
settle gate went GO at 12:06:25 after its first ACK; at 12:06:28 it logged „NOT minting a repository password: the hub
|
||||||
|
holds a sealed recovery package". So no controller change was needed; 0.337.0 still records the escrow before
|
||||||
|
`SetFloor` (a pinned belt for a hub that carries later).
|
||||||
|
|
||||||
|
**Live proof (demo-hp VM 9321, Tester4, public ISO 1.29.0, no operator press before the install):** bind (one OCR misread
|
||||||
|
of the pairing code cost one of five tries — harness), claim, the dashboard offered „A korábbi, házon kívüli mentéseid
|
||||||
|
visszaszerezhetők"; a wrong code: „A megadott helyreállítási kódot nem fogadtuk el … Semmi nem változott"; the right code
|
||||||
|
unlocked 9 groups; Nextcloud reinstalled („előbb telepítsd újra" is the page's own order), „Teljes visszaállítás" with
|
||||||
|
the placement confirm → „3113 fájl és 2 adatkötet és az adatbázis visszaállítva"; **sha256 through Nextcloud's own WebDAV
|
||||||
|
against the rehearsal's A5 manifest: 3 031 of 3 033 byte-identical** — the 2 missing never reached the old box (the 1.5 GB
|
||||||
|
file Nextcloud refused, a symlink rclone skips); 4 extra files are the rehearsal's later uploads. Off-site: two runs, 63 → 64
|
||||||
|
snapshots in the same 1.4 GB store, `orphaned: false`; the hub's full log has no set-aside and no key change.
|
||||||
|
|
||||||
|
## 4. Versions, commits, tests
|
||||||
|
|
||||||
|
| Component | Version | State | Commits |
|
||||||
|
|---|---|---|---|
|
||||||
|
| hub | 0.163.0 | **deployed** 11:46 UTC (operator yes), Synced/Healthy | `9688e76e`, manifest `cd3d0598` |
|
||||||
|
| controller | 0.337.0 (`sha256:264af313…`) | **released**: rc.1 on 9202; per-box floors 13:59 → three test boxes on 0.337.0 by 14:06; box walk green; **global floor 0.337.0 / MinAgent 0.158.1 at 15:54:29 UTC**, read back; per-box floors cleared | `8f5bea0`, `5929b8e`, `7848d4a` |
|
||||||
|
| agent | (0.162.0 when released) | **committed, not released** | `286a7e3` |
|
||||||
|
| catalog, installer | — | unchanged | — |
|
||||||
|
|
||||||
|
Tests (all pass; each red-proved — seen failing first): hub `TestR1183_*` (3), `TestR1176_*` (store 7, api 3),
|
||||||
|
`TestR1182_*` (store 2, api 1), `TestR1203_*`; controller `TestR1175_*` (3), `TestR1181_*` (2), `TestD25_*` (3),
|
||||||
|
`TestR1179_*`, `TestR1188c_*`, `TestR1191_*` (2), `TestR1176_EscrowIsRecordedBeforeTheFloorIsKnown`, `TestR1182_*` (2);
|
||||||
|
agent `TestR1192_*` (2) + the resize test's swap assertion. CI: green by job id for every push (see §5).
|
||||||
|
|
||||||
|
## 5. Morning list, teardown, rows, observations
|
||||||
|
|
||||||
|
**Morning list (operator):** D26 — release + vouch the agent with the swap fix; delete host **Tester4-f6b67c** (asked
|
||||||
|
today); R-1205 (the whole-box tier waits for „Re-issue PBS credentials" after a failed second install); the
|
||||||
|
`REHEARSAL_2026_10_20_*` / `FIXES_2026_10_21_*` names in sec.
|
||||||
|
|
||||||
|
**Operator acts today:** the hub-deploy yes; the root credential reveal for the freeze test (~2 min); the host delete
|
||||||
|
(done 16:49 UTC — the delete appears 6 minutes after a box goes off, R-30's rule).
|
||||||
|
|
||||||
|
**Teardown (three layers):**
|
||||||
|
- **Machine (demo-hp, Tier 0):** VM 9321 `rh1021-box2` and helper CT 9331 `rh1021-loader` destroyed with `--purge` at
|
||||||
|
16:27:40 UTC (names checked first); no image left on `nvme-scratch`; 341, 9201, 9202, 9401 untouched. 9202 moved from
|
||||||
|
`0.337.0-rc.1` to `0.337.0` by hand (it does not take the hub's floor). The public ISO 1.29.0 was only read.
|
||||||
|
- **Hub:** the operator deleted host Tester4-f6b67c (16:49:37 UTC, escrow acknowledged): no Tester4 host, no current
|
||||||
|
package, the household's package retained under Tester4-f6b67c with the same hash, `repo_tag` 'live' — a future
|
||||||
|
rebuild carries it again; the WireGuard peer gone (`P6-release/05-teardown-hub.txt`).
|
||||||
|
- **ep0 / Storage Box (read only, nothing removed):** Tester4's off-site store gained 2 snapshots (63 → 64, normal
|
||||||
|
traffic); the hub's registrar pinned the rebuilt box's key and removed the dead box's line (normal enrollment). No PBS
|
||||||
|
descriptor was made (R-1205), so nothing new on ep0's PBS.
|
||||||
|
|
||||||
|
**Rows:** before 182 (181 after a parallel session closed R-1201); after 176; opened 2 (R-1205, R-1206); closed 7 (R-1176,
|
||||||
|
R-1203, R-1175, R-1179, R-1181, R-1182, R-1191). R-1183 → VERIFY (due 2026-10-26); R-1192 → WAITING-ON-OPERATOR;
|
||||||
|
R-1188 (c) fixed. Capability map A12, A13 → PROVEN-LIVE. `unproven.py --summary`: unchanged, 35 of 55 not walked.
|
||||||
|
|
||||||
|
**Observations:**
|
||||||
|
- PBS auto re-issue refused for the rebuilt Tester4 (latest deletion un-acked) — **FILED: R-1205**.
|
||||||
|
- Future-dated snapshots from the clock test show as „Legutóbbi mentés: 2026. 10. 28." — **FILED: R-1206**.
|
||||||
|
- The pairing-code OCR misread one character — **NOT-A-FINDING:** harness (a human reads the screen); corrected against the hub's record.
|
||||||
|
- Immich's first admin needs the household's dashboard session (the setup gate) — **NOT-A-FINDING:** designed.
|
||||||
|
- The full restore asks to confirm a new place (backup on `hdd_1`, now the internal disk) — **NOT-A-FINDING:** designed (R-351).
|
||||||
|
- Nextcloud's small-file WebDAV upload ~20 files/min under heavy concurrent load — **NOT-A-FINDING:** slow, not lost; the new install-page sentence says big uploads can be slow.
|
||||||
|
- `kubectl logs -l` reads only 10 lines — **NOT-A-FINDING** (a tooling trap; saved to memory, two checks redone over the full log).
|
||||||
|
- Tester 2 last reported 2026-10-04 on 0.292.0; the floor reaches it only when it reports — **NOT-A-FINDING** (R-1139 covers the box).
|
||||||
|
|
||||||
|
**Secrets:** names only — `FIXES_2026_10_21_PAIRING_CODE`, `FIXES_2026_10_21_BIND_TOKEN`, `FIXES_2026_10_21_SETUP_CODE`
|
||||||
|
(one-time, spent), `FIXES_2026_10_21_ROOTPW` (the operator's reveal; the box is torn down). The bind link and the setup code
|
||||||
|
were read from the mailbox by the mail connector, so both one-time values entered this session's record (spent within
|
||||||
|
minutes, as in the rehearsal). A scan of all 45 evidence files for every secret used today: 0 hits. Nothing secret in any
|
||||||
|
commit.
|
||||||
@@ -0,0 +1,289 @@
|
|||||||
|
# REPORT — the night briefs of 2026-10-19 and 2026-10-20 (part 2 run by day): one stop less per night, the safe undo for a failed off-site restore, hub 0.162.0 / agent 0.161.0 / installer 1.35.0 delivered, downloads out of the backups, a scanner on every push
|
||||||
|
|
||||||
|
# PART 2 — the night brief part 2 (run by day, 2026-10-20): your four answers, the safe undo, downloads out of the backups, Immich, the update test
|
||||||
|
|
||||||
|
**Safest readings taken alone (written first, as the brief asks):**
|
||||||
|
1. **R-1083 is not in agent 0.161.0.** An agent key would make 9202's controller book the agent's whole-guest backups,
|
||||||
|
one of which writes to ep0 — a change on ep0 the brief forbids; 9202 is also outside the `felhom` pool.
|
||||||
|
2. **All five downloaders' folders are `excluded`, not only the two named** (qBittorrent, Transmission; also NZBGet,
|
||||||
|
SABnzbd, MeTube): the brief's gate rule — a download folder is never optional or mandatory — covers all five.
|
||||||
|
3. **No update-test edge was run:** the bench (guest 9401) and the drill catalog are outside the brief's machines and
|
||||||
|
repos. The list done and the list left are written; the ask is on the morning list.
|
||||||
|
4. The brief was written for a night run and ran by day; its „stop by 05:00" was read as a time box, not a clock.
|
||||||
|
|
||||||
|
## Part 2 — in plain words
|
||||||
|
|
||||||
|
- **Morning list:** vouch agent 0.161.0 · decide D21 (Immich) · MeTube's videos out of the second-drive copy — keep? ·
|
||||||
|
the bench and the drill catalog for the update test — yes?
|
||||||
|
- **Hub 0.162.0 is live.** Its own deploy was the real restart test: Tester-2 (down since 4 October) got no repeat mail
|
||||||
|
— read in the hub and, separately, in your mailbox.
|
||||||
|
- **Agent 0.161.0 runs on demo-hp, Tester 1 and demo-felhom.** On demo-hp the agent said „busy" during a backup and
|
||||||
|
„free" before and after. **Not vouched — that is yours.**
|
||||||
|
- **Installer 1.35.0 is published.** The served file equals the tag's file.
|
||||||
|
- **A failed off-site restore now puts the app back exactly as it was** (your D20 = A, controller 0.336.0, the floor).
|
||||||
|
Proven on the scratch box: after a forced failure the database matched its fingerprints from before, the app ran,
|
||||||
|
nothing was held; a normal restore then worked.
|
||||||
|
- **Downloads are out of the backups** (torrent, usenet, MeTube): the controller's own check copies 0 of them now.
|
||||||
|
- **Immich's off-site copy:** 1.33 GB, of which 102 MB must survive. Two options; pick B (leave out the AI model files).
|
||||||
|
- **ep0's first weekly clean-up:** done this morning (Part 10 of part 1 below).
|
||||||
|
|
||||||
|
## Part 2 — each part
|
||||||
|
|
||||||
|
| Part | Done? | Why / how |
|
||||||
|
|---|---|---|
|
||||||
|
| 1a hub deploy | **done** | `manifests/hub.yaml` → 0.162.0 (`e1d349f1`), only Deployment/hub synced (the only OutOfSync); image `sha256:69285e64…` = the build; Synced/Healthy; reports from all three boxes; floor read 0.335.0; Day-0 agent list has no `-step1`. R-1001 + R-1011 closed. |
|
||||||
|
| 1b agent release | **done** | R-1083 left out (above). `release-agent.sh 0.161.0` from a clean clone; signed `agent_update` per box: demo-hp 06:38, Tester 1 06:54, demo-felhom 06:55 UTC, each COMPLETED, 76/76, committed. `host_busy` live; tunnel `absent` = the fixture row (9202's real Docker 29 answer); R-1010 by its tests. R-1070 + R-1010 closed. |
|
||||||
|
| 1b R-921 live | **not done** | The no-stop case happens only on the night schedule (a press backs up the local tier only). Row R-1173 with a dated check after demo-hp's next off-site night. |
|
||||||
|
| 1c installer | **done** | tag `installer-v1.35.0`, both webpage pins (`b0ae2c57`), webpage Deployment synced alone; served sha256 `f7a196fd…` = tag. R-1167 closed. |
|
||||||
|
| 2 safe undo | **done** | below. R-893 + R-1169 closed. |
|
||||||
|
| 3 downloads | **done, one read left** | catalog `c50adfc`, gate `download-binds`; classifier proof. demo-felhom (qBittorrent) runs its next second-drive copy 2026-10-21 03:30 — dated check on R-1019 (NARROWED). |
|
||||||
|
| 4 Immich | **done (plan only)** | below. R-49 → READY, D21. |
|
||||||
|
| 5 update test | **not done** | fixture side complete for every installed app; the edges need the bench / drill catalog (above). R-462 NARROWED. |
|
||||||
|
| 6 ep0 clean-up | **done** | part 1, Part 10. |
|
||||||
|
| 7 controller release | **done** | 0.336.0 below. |
|
||||||
|
|
||||||
|
## Part 2 — the safe undo, the 9202 proof (`07` §6.3; `audits/night2-2026-10-20/P2/`)
|
||||||
|
|
||||||
|
- Code `93fb251`: the fit check before the stop, the copy right after it (`backups/restore-undo/<app>/`), the put-back on
|
||||||
|
every failure branch, the hold only when the put-back fails, the copy removed on success. 11 tests; nine mutations red.
|
||||||
|
- **9202, rc.1** (a helper with the fences in full; Rallly + postgres 18; the same forced failure as on 19 October):
|
||||||
|
|
||||||
|
| | Live before | After the forced failure |
|
||||||
|
|---|---|---|
|
||||||
|
| public tables | 36 | 36 |
|
||||||
|
| markers | A, B, C, D | A, B, C, D |
|
||||||
|
| `r893_only_old` (backup only) | absent | **absent** (on 19 October: present) |
|
||||||
|
| schema / data / row-count md5 | `05828c77` / `e5c1a72f` / `d8bcbc4b` | **equal** |
|
||||||
|
| app | running | **running**, no hold |
|
||||||
|
| the copy | — | 69 MB in 0.9 s, removed after |
|
||||||
|
|
||||||
|
The sentence: „…visszaállítása sikertelen. Nem változott semmi: az alkalmazás adatai visszakerültek a visszaállítás
|
||||||
|
előtti állapotba, és az alkalmazás újra fut". Then a normal restore of the same snapshot: the snapshot's content
|
||||||
|
(A–C, the backup-only table present), the app running, the copy removed. 9202 back on 0.331.0 as found.
|
||||||
|
- Not measured live (each pinned by a red-proved test): the version-change branch, the `_files` sentence, the refusals
|
||||||
|
before the stop, a failing put-back, the operator event (9202 has no hub).
|
||||||
|
|
||||||
|
## Part 2 — download binds (`15` §7; `audits/night2-2026-10-20/P3/`)
|
||||||
|
|
||||||
|
| App | Bind (userdata) | Class now | Copied by a tier |
|
||||||
|
|---|---|---|---|
|
||||||
|
| qBittorrent | media/torrents | excluded (was optional) | none (was the second drive) |
|
||||||
|
| Transmission | media/transmission, media/transmission/watch | excluded, excluded | none |
|
||||||
|
| NZBGet | media/nzbget | excluded (was optional) | none |
|
||||||
|
| SABnzbd | media/usenet | excluded (was optional) | none |
|
||||||
|
| MeTube | media/metube | excluded (was optional) | none |
|
||||||
|
| Sonarr, Radarr | media (rw) | excluded | none |
|
||||||
|
| Jellyfin, Plex, Emby | media (:ro) | excluded | none |
|
||||||
|
| Prowlarr | no userdata bind | — | — |
|
||||||
|
|
||||||
|
## Part 2 — Immich (`audits/night2-2026-10-20/P4/immich-offsite-plan.md`)
|
||||||
|
|
||||||
|
Model cache 824 MB (62 %) · raw database folder 283 MB (21 %) · Immich's own backups 115 MB (9 %) · logical dump 54 MB
|
||||||
|
· originals 49 MB · thumbnails 2 MB. **A** (leave out all three re-fetchable parts, −92 %) needs a never-run restore
|
||||||
|
from the dump alone on the vectorchord image; **B** (only the model cache, −62 %) needs one restore with an empty cache.
|
||||||
|
**Pick B now**, A's other parts after their proof.
|
||||||
|
|
||||||
|
## Part 2 — update-test edges
|
||||||
|
|
||||||
|
None run (above). Installed apps: 16; with a fixture: 15; without: bentopdf (keeps no data). Upstream within a major,
|
||||||
|
waiting for a tested step: PrivateBin 2.0.6→2.1.0, Home Assistant 2026.9.4→2026.10.2, RomM 5.3.1→5.4.0, Plex
|
||||||
|
1.41.4→1.43.4, Kimai 2.67.0→2.70.0 (`audits/night2-2026-10-20/P5/r462-list.txt`).
|
||||||
|
|
||||||
|
## Part 2 — versions, commits, rows
|
||||||
|
|
||||||
|
- Hub 0.162.0 deployed (`e1d349f1`). Agent 0.161.0 released (`5152123`, tag `v0.161.0`; notes `4e1621c`). Installer
|
||||||
|
1.35.0 published (`b0ae2c57`). Controller 0.336.0 the floor (`93fb251`; notes `aa9e09e`, `7ca59e2`), image
|
||||||
|
`sha256:93cd67d8…8d73`. Catalog `c50adfc`. Evidence `301f4e9e`, `bcaa9576`.
|
||||||
|
- i18n: no page template changed; seven Go-side keys (HU + EN), one rewritten (R-1169); no fixture changed.
|
||||||
|
- **System poster:** no listed fact changed — none of the three.
|
||||||
|
- **Register 160 before, 156 after. Closed 7:** R-1001, R-1011, R-1167, R-1070, R-1010, R-893, R-1169. **Opened 3:**
|
||||||
|
R-1171 (demo-hp's local copy never restore-tested), R-1172 (the put-back logs at ERROR; an empty folder stays),
|
||||||
|
R-1173 (R-921's no-stop not yet seen live). Re-stated: R-1019 (NARROWED, dated 2026-10-21), R-1083, R-462 (NARROWED),
|
||||||
|
R-49 (READY, D21). **Both briefs together: 161 before, 156 after; closed 11, opened 6.**
|
||||||
|
|
||||||
|
## Observations — part 2
|
||||||
|
|
||||||
|
1. demo-hp's local whole-guest copy is never restore-tested: one copy kept, a 24 h settle, replaced at 24 h. FILED: R-1171
|
||||||
|
2. A successful put-back is logged at ERROR, and an empty `backups/restore-undo/` folder stays after a restore. FILED: R-1172
|
||||||
|
3. R-921's no-stop branch has no live observation yet (it needs a night schedule). FILED: R-1173
|
||||||
|
4. R-462's „only three apps are tested" was out of date: 58 apps have fixtures on both venues. NOT-A-FINDING: the row is re-stated with today's list.
|
||||||
|
5. The hub deploy at 06:03 UTC removed the old pod's log, which held the 07:10 Budapest off-site audit lines of today (R-895). NOT-A-FINDING: R-895's dated check is 2026-10-21; that run happens in the new pod.
|
||||||
|
6. In step 4 of the 9202 drill the data-only md5 differed from a read taken before snapshot 2, while the schema and every row count were equal. NOT-A-FINDING: the two reads were at different moments (the read, then the snapshot's own dump); the claim under test — before and after the forced failure — matched on all three fingerprints.
|
||||||
|
7. The helper's brief expected 37 tables after the normal restore; snapshot 2 has 36. NOT-A-FINDING: last night's 37 was the mixed state, not the snapshot (my arithmetic, corrected).
|
||||||
|
8. The first background wait loop for demo-hp's agent update printed a time and stopped before the box had updated. NOT-A-FINDING: the version was read directly and a foreground loop used instead.
|
||||||
|
9. A Python edit of the controller CHANGELOG failed on a quote character, and the commit carried only REPORT.md. NOT-A-FINDING: fixed by the next commit (`7ca59e2`).
|
||||||
|
|
||||||
|
## Part 2 — teardown
|
||||||
|
|
||||||
|
- Machine: 9202 back on 0.331.0 as found (Rallly removed through the product, the SFTP target, images and files removed
|
||||||
|
by name, settings and the off-box folder put back); demo-hp 9201's probe script removed.
|
||||||
|
- Host: demo-hp `/tmp` copies removed; nothing on `pve/data`; no whole-guest backup of 9202.
|
||||||
|
- Hub: per-box floors set and cleared; the global floor raised; nothing else.
|
||||||
|
- ep0: read only.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# PART 1 — the evening/night brief of 2026-10-19
|
||||||
|
|
||||||
|
## In plain words
|
||||||
|
|
||||||
|
- **Morning list:** deploy hub 0.162.0? · release agent 0.161.0? · publish installer 1.35.0? · decide D20 (R-893).
|
||||||
|
- **One app stop less per night** (R-921): the controller (0.335.0, the floor) now asks the agent „are you busy?" before
|
||||||
|
it stops the apps. The agent's answer is written but not released, so it starts working with the agent release.
|
||||||
|
- **No false error for a box without a tunnel** (R-1070): the real cause was Docker 29 writing its „not found" message
|
||||||
|
in small letters. Fixed in the agent (not released).
|
||||||
|
- **A failed off-site restore, measured** (R-893): the box puts the newer rows over the backup's older volume; a table
|
||||||
|
only the backup had stays. The app is held stopped. Two options written; pick A (save the app's volumes first).
|
||||||
|
- **Hub fixes built, not installed:** no repeat mail after a restart; the agent list offers only real agent packages.
|
||||||
|
The ArgoCD danger to three hub secrets is already gone (ArgoCD no longer tracks them).
|
||||||
|
- **Installer message written, not published:** a re-run on a live box says „delete the old host first", EN and HU.
|
||||||
|
- **A scanner on every push, in all four repos** (warn-first). First scan: no real secret, no called weakness.
|
||||||
|
- **The scratch box cannot get its agent key without an agent change** (R-1083): the exact gap is written down.
|
||||||
|
- **ep0's first weekly clean-up ran as predicted:** only tester-1's 4 October copy went; every household has a rule.
|
||||||
|
R-1162 closed.
|
||||||
|
|
||||||
|
Decisions CC took: none beyond the brief. The operator answered one question (R-921, 19:2x): „add one field".
|
||||||
|
|
||||||
|
## Part 1 — R-921 (`07` §6.4; `03` local API) — DONE (live once the agent ships)
|
||||||
|
|
||||||
|
- No existing agent answer served the host-wide gate (read: `/backup/due`, `/backup/status`, the 409 shape). Operator:
|
||||||
|
add a read-only field to the existing route, no new route.
|
||||||
|
- Agent `56aabc7`: `host_busy` `{busy, what}` on `GET /backup/status`. Controller 0.335.0: `precheckSkips` → held →
|
||||||
|
no stop, contention for each due tier (15-minute skip, one „BLOCKED" alarm after 3 h); free / older agent / failed
|
||||||
|
read → the old order; the window's own tier in flight → attach as before.
|
||||||
|
- **The race** (free at the read, busy at the start): one short stop, resumed at once — today's behaviour, pinned.
|
||||||
|
- Tests, red-proved: agent `TestBackupStatus_R921_HostBusy` (assignment removed → `free gate: host_busy = <nil>`);
|
||||||
|
controller `TestR921_NoStopWhileTheHostIsBusy` (branch removed → 3 apps stopped, the start refused),
|
||||||
|
`…SkipsStillAlarmAfterTheCeiling`, `…OtherAnswersKeepTheOldOrder` (4 rows), `TestHostBusy_ReadsTheAgentsGate`.
|
||||||
|
- Live: demo-hp's agent answers no `host_busy` (read inside 9201) — the old order, as designed. A live busy state was
|
||||||
|
not made (it needs the new agent on the box). R-921 closed per the brief.
|
||||||
|
|
||||||
|
## Part 2 — R-1070 (`11` §8.2) — DONE in the agent (not released)
|
||||||
|
|
||||||
|
- Where: `felhom-agent/internal/osupdate/leg.go` `HostHealthVerdict` (the tunnel check) and
|
||||||
|
`internal/hub/cloudflared.go` `ClassifyTunnel`. Measured on demo-hp: `docker inspect` of a missing name prints
|
||||||
|
`error: no such object: …` (Docker 29); the match wanted `No such object` → `unknown` → the host rule failed.
|
||||||
|
- Fix: case-insensitive match → new state `absent`; the host rule passes `absent` only when the reading from before
|
||||||
|
the step (Docker answered) also had no cloudflared; `not_running` and `unknown` still fail. Hub 0.162.0 shows
|
||||||
|
`absent` as „none set up". `11` §8.2 had said `unknown` does not fail — corrected (the code always failed it).
|
||||||
|
- Tests, red-proved: `TestHostHealthVerdict_R1070_NoTunnelSetUp` (7 rows), three `TestClassifyTunnel` rows,
|
||||||
|
`TestSystemRows_R1070_AbsentTunnelIsNotAWarning`.
|
||||||
|
|
||||||
|
## Part 3 — R-893: measured and designed, no restore code changed
|
||||||
|
|
||||||
|
Helper on 9202 (fences carried in full). Evidence and the note: `documentation/audits/night-2026-10-19/P3/`.
|
||||||
|
|
||||||
|
- Rallly 4.15.3 + postgres 18; a scratch SFTP off-site on 9202's own disk; one failing SQL line in the prepared copy
|
||||||
|
of the dump, so the replay failed AFTER the volume was replaced.
|
||||||
|
- **What it left:** the newest rows (the undo dump) over the backup's volume; a table only the backup had stayed
|
||||||
|
(37 tables against 36); definition unchanged (same version); app held (`restore_mixed`), a start → 409.
|
||||||
|
- The restore page's sentence is false on both halves for this app → R-1169.
|
||||||
|
- **Options:** A — copy the app's live volumes before the replay, put them back on failure, start the app (one copy's
|
||||||
|
disk for minutes; Rallly 69 MB in ~1 s; a fit check before the stop). B — empty the DB volume before the undo load
|
||||||
|
(no disk; a bad undo copy leaves an empty database; usually still a support call). **Pick: A** (the operator's D8
|
||||||
|
direction). Not measured: the version-change branch (needs a drill-catalog commit).
|
||||||
|
- Teardown: 9202 back on 0.331.0 as found, every throwaway removed (checked again by CC); host: `/tmp` only; hub: nothing.
|
||||||
|
|
||||||
|
## Part 4 — hub 0.162.0, built, NOT deployed (`05`)
|
||||||
|
|
||||||
|
- R-1001: start-up seeding from the DB for `host_stale`/`host_down`; `restore_test_stale` edge from the DB. Tests
|
||||||
|
red-proved (each restart = a new checker over the same DB).
|
||||||
|
- R-1011: exact-file sha (`ErrNoFile`); step bundles leave the list; a vouch of one → `artifact_file_missing`. Registry
|
||||||
|
read: 6 step-bundle versions, every release holds `felhom-agent`. Tests red-proved.
|
||||||
|
- R-953: **no change needed** — read back 18:21 UTC: ArgoCD tracks none of the three Secrets (no tracking mark), so the
|
||||||
|
next deploy does nothing with them. `P4/r953-readback.txt`.
|
||||||
|
- Image `sha256:69285e640a9c5e4ce59b1fbb5cd94623fa47a71798e9d86fa0767ee6c69a6ac8` from a clean clone at `d92a712e`.
|
||||||
|
|
||||||
|
## Part 5 — agent, committed, NOT released (`03`)
|
||||||
|
|
||||||
|
R-1010 (`56aabc7`): on „Failed to find logical volume …_vzdump" one retry with the API's `force=1` (`pvesh usage`
|
||||||
|
lists `--force`; the same token privilege; no sudoers line, no root command); never a second force. Red-proved. Plus
|
||||||
|
R-921's field and R-1070. CHANGELOG „Unreleased". Not exercised live (it needs a half-removed snapshot on `pve/data`).
|
||||||
|
|
||||||
|
## Part 6 — installer 1.35.0, written, NOT published
|
||||||
|
|
||||||
|
R-1167: `die_live_box` in EN and HU at the preflight, host-enroll and the controller.yaml fetch. Harness: a static case
|
||||||
|
and two behaviour cases (the real functions against a local fake hub), red-proved on a mutated copy. **CI red twice**
|
||||||
|
(2106, 2108): the CI runner has no curl; the behaviour cases now SKIP out loud there (`5d0dfeac`).
|
||||||
|
|
||||||
|
## Part 7 — the scanner (R-986), warn-first — DONE
|
||||||
|
|
||||||
|
- Shared `scripts/security_scan_gate.py` in all four runners' fast sets (hook + CI); `.security-accepted` per repo;
|
||||||
|
8 decoys in scratch repos, red-proved; CI fetches the pinned gitleaks (both sha256 checked — seen working in CI 2108).
|
||||||
|
- govulncheck pin v1.1.4 → **v1.8.0** (v1.1.4 cannot read the Go 1.27 code). CI has no Go: govulncheck NOT CHECKED
|
||||||
|
there, out loud; the hook runs it.
|
||||||
|
- **First scan of today's `main`** (`P7/triage.md`): govulncheck 0 called in all three modules. gitleaks history:
|
||||||
|
controller 12, agent 15, catalog 1 (all as on 2026-10-12), felhom.eu 50 (+7: the usenet bench 5, a boot journal 1,
|
||||||
|
a test 1). Today's tree: controller 4, felhom.eu 42, agent 15, catalog 0 — every one triaged: **0 real**; accepted =
|
||||||
|
throwaway demo/bench app values (the bench LXC 9401 stopped, the apps gone from 9202); false = test constants, public
|
||||||
|
keys, an SSH fingerprint, a tunnel ID, hashes.
|
||||||
|
|
||||||
|
## Part 8 — R-1083: the exact gap (no agent change)
|
||||||
|
|
||||||
|
The agent mints a per-guest token only inside `--selftest=provision`, for a guest it brings up from a golden in the same
|
||||||
|
call. No product way exists for an existing guest. Row stays OPEN with the gap and two options.
|
||||||
|
|
||||||
|
## Part 9 — controller 0.335.0, the floor
|
||||||
|
|
||||||
|
rc.1 on demo-hp 18:08 UTC (0 errors) → `0.335.0` `sha256:262831f9…` → per-box floors 18:22 → all three moved by
|
||||||
|
18:37 → box walk green (0 unhealthy, 0 refusals, catalog in sync; https-only 28 OK, 0 FAIL; „Fájlok ↗" one sign-in on
|
||||||
|
demo-hp and Tester 1) → **global floor 0.335.0 / MinAgent 0.158.1 at 18:39:26 UTC**, read back; per-box floors cleared.
|
||||||
|
i18n: no page changed, no fixture changed. Evidence `P9/`.
|
||||||
|
|
||||||
|
## Part 10 — ep0's first weekly clean-up (`07` §plane-2 retention) — DONE, R-1162 closed
|
||||||
|
|
||||||
|
Read only on ep0, 03:45–03:47 UTC. Evidence `P10/01-before.txt` (19:49 UTC), `P10/02-after.txt`.
|
||||||
|
|
||||||
|
| Household | Task 03:30 UTC | Removed | Copies after |
|
||||||
|
|---|---|---|---|
|
||||||
|
| demo-hp | OK, keep-weekly 2 | nothing | 2 (10-08, 10-15) |
|
||||||
|
| demo-felhom | OK, keep-weekly 2 | nothing | 2 (10-06, 10-13) |
|
||||||
|
| Tester-2 | OK, keep-weekly 2 | nothing | 1 (10-04) |
|
||||||
|
| tester-1 | OK, keep-weekly 2 | 2026-10-04 | 2 (10-11, 10-18) |
|
||||||
|
|
||||||
|
- Exactly the dry runs' prediction. The operator job (03:45, keep-daily 14 + keep-weekly 8) removed one hub copy
|
||||||
|
(10-06) — its own rule.
|
||||||
|
- **No „household without a rule" alarm.** Positive check from a second channel: ep0's read-only `usage` op, run
|
||||||
|
directly, answers `unpruned_checked: true, unpruned: []` (tenantsync v1.3.0, sha256 same as the repo file). The hub
|
||||||
|
(0.161.0) logs its ep0 read every 15 minutes (03:31 UTC the last before my read) and no „no prune job" line since its
|
||||||
|
start (2026-10-19 14:10 UTC). The alarm is operator-channel only, so it is not in the events table (none there).
|
||||||
|
|
||||||
|
## Versions, commits
|
||||||
|
|
||||||
|
- Controller 0.335.0 (`15ea002`, REPORT `c4b3b0a`, scanner `a03c25a`). Hub 0.162.0 built (`5f31da04`). Agent
|
||||||
|
`56aabc7`, `84ef581` (not released). Installer 1.35.0 (`d92a712e`, `5d0dfeac`, not published). felhom.eu scanner
|
||||||
|
`15f74288`, docs `f92fa5a8`. Catalog `87f5588`, `57ddb20`.
|
||||||
|
- **System poster:** no listed fact changed tonight — none of the three.
|
||||||
|
- **Instruction/doc corrections:** `11-os-updates.md` §8.2 (the `unknown` tunnel sentence was false); `nodes.md` and
|
||||||
|
`nightly-rotation.md` (9202 is DHCP, .121 now).
|
||||||
|
|
||||||
|
## Rows
|
||||||
|
|
||||||
|
Register **161 before, 160 after.** **Closed 4:** R-921, R-953, R-986, R-1162. **Opened 3:** R-1168 (warn →
|
||||||
|
fail), R-1169 (the false restore sentence), R-1170 (a vanished tunnel raises no alarm). Re-stated: R-1001, R-1011,
|
||||||
|
R-1010, R-1070, R-1167 (READY, waiting on the release/deploy/publish), R-893 (READY, D20), R-1083 (the gap).
|
||||||
|
**Dates moved** (the due-checks gate fired on the new date): R-895 and R-435, 2026-10-20 → 2026-10-21. Both are reads
|
||||||
|
this brief did not name; R-895's first verdict lands at 07:10 hub time, after this session. The reason is in each row.
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
1. The restore page's held-mixed sentence is false for the measured app. FILED: R-1169
|
||||||
|
2. A tunnel container that disappears on a box with a tunnel configured raises no alarm. FILED: R-1170
|
||||||
|
3. The scanner warns only. FILED: R-1168
|
||||||
|
4. `11` §8.2 said an `unknown` tunnel does not fail the host rule; the code always failed it. NOT-A-FINDING: corrected in the session (the size rule); it is R-1070's cause.
|
||||||
|
5. govulncheck v1.1.4 (the 2026-10-12 pin) cannot read Go 1.27 code. NOT-A-FINDING: the pin moved to v1.8.0 tonight, recorded in `SCANNERS.md` §5.
|
||||||
|
6. The CI runner has no curl; two new harness cases failed CI 2106 and 2108. NOT-A-FINDING: fixed the same night (`5d0dfeac`), they skip out loud there.
|
||||||
|
7. R-953's tracking marks went after 2026-10-12 with no record of who removed them. NOT-A-FINDING: the danger the row named is gone, nothing to fix.
|
||||||
|
8. 9202's DHCP address moved (.114 → .121) and the walk named the old one. NOT-A-FINDING: fixed (`box_walk.py` reads it; two docs corrected).
|
||||||
|
9. The brief said the installer already prints HU and EN; it printed English only. NOT-A-FINDING: the HU lines are new, named in the scripts CHANGELOG.
|
||||||
|
10. The agent's backup cap (R-973) can also refuse a POST after the stop. NOT-A-FINDING: it answers 409, so the apps resume at once (the BUSY path), and it fires only after a flood of presses that each stopped the apps anyway.
|
||||||
|
11. The manual off-site run dumped and restarted every app, not only the one switched on. NOT-A-FINDING: by design, the coherent app-data pair (`07`).
|
||||||
|
12. On CI's depth-1 clone, gitleaks scans the whole tree (the root commit), ~100 s for felhom.eu. NOT-A-FINDING: a stronger check than the diff, all current hits accepted; recorded in `CONTEXT.md`.
|
||||||
|
13. My 19:49 before-file listed the operator job's keep-weekly but not its keep-daily 14. NOT-A-FINDING: my read printed only some fields; the job is unchanged.
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
- Machine: 9202 back as found (0.331.0, no throwaway left — checked); 9201 on the release 0.335.0; probe scripts removed.
|
||||||
|
- Host: `/tmp` copies only, removed. Nothing on `pve/data`.
|
||||||
|
- Hub: per-box floors cleared; nothing else created.
|
||||||
|
- ep0: read only (task list, task logs, snapshot list, prune-job list, the read-only `usage` op). Nothing changed.
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
# REPORT — system poster refresh check + OMV guide fixes (2026-10-21)
|
||||||
|
|
||||||
|
Baseline `main` = `7a773207`. Own clone; documentation only; no machine, box or release touched.
|
||||||
|
Binding: `documentation/architecture/felhom-system-poster.facts.md`.
|
||||||
|
|
||||||
|
## Plain words
|
||||||
|
|
||||||
|
- **The new poster is in the repository**, the web page and the PDF. It took three exports.
|
||||||
|
- The first export was a folder of 6 files, and Claude Design had changed four texts. Nothing was committed then
|
||||||
|
(commit `3decdf94`, with the OMV guide fixes and R-1201).
|
||||||
|
- The second export was one file; two texts were still short. The third export passed every check.
|
||||||
|
- The STATUS line „System poster needs a refresh" is gone. R-1201 is closed.
|
||||||
|
|
||||||
|
## Part A — checks
|
||||||
|
Full table, all three rounds: `documentation/audits/poster-refresh-2026-10-21/CHECKS.md`.
|
||||||
|
Round 3: file PASS · shape: plain static HTML (better than the old loader) · two changes PASS · nothing lost PASS ·
|
||||||
|
secrets PASS · look PASS (PDF; HTML render not checked, no browser on DooPlex).
|
||||||
|
|
||||||
|
## Part B — committed (second commit)
|
||||||
|
- `documentation/architecture/felhom-system-poster.html` replaced; `felhom-system-poster.pdf` added (6.25 MB, A1 size).
|
||||||
|
- Facts file: one History line.
|
||||||
|
- STATUS: the poster line removed. R-1201 moved to `CLOSED-ITEMS.md`.
|
||||||
|
- Evidence: round-3 hashes, text diff, three PNG crops.
|
||||||
|
|
||||||
|
## Part C — `documentation/pilot/GUIDE-omv-kvm-hu.md` (first commit, `3decdf94`)
|
||||||
|
1. „Ha valami nem megy" item 7 (HDD_PATH error) removed; 8 → 7.
|
||||||
|
2. Jellyfin: „Ha a géped egy korábbi telepítővel…" sentence removed.
|
||||||
|
3. Memory bullet: „Az első napokban kevés alkalmazással indulunk; a nagy fotógyűjtemény feltöltését együtt csináljuk." added.
|
||||||
|
4. Operator note: dated line, point (4) obsolete since installer 1.33.0; history kept.
|
||||||
|
|
||||||
|
Re-checked: `scripts/iso/felhom-bootstrap.sh:165` fetches `https://felhom.eu/scripts/felhom-host-install.sh`, which
|
||||||
|
serves `SCRIPT_VERSION="1.35.0"` (≥ 1.33.0). `https://felhom.eu/letoltes` lists `felhom-installer-1.29.0-pve9.2-1.iso`
|
||||||
|
and `dceacae5…4829`, the same as the guide's step 3.
|
||||||
|
|
||||||
|
## Rows
|
||||||
|
Opened R-1201 (first commit), closed R-1201 (second commit). Register 178 → 179 → 178.
|
||||||
|
|
||||||
|
## Claims in the task that were wrong
|
||||||
|
- The PDF is not A2: 2400×1696.5 pt ≈ A1 size.
|
||||||
|
- `claude/DESIGN-poster-refresh-2026-10-21.md` is not in this repository: `claude/` is the planning project's folder
|
||||||
|
(as for `claude/MONTHLY-security-retest.md`).
|
||||||
|
- felhom.eu has no root `CHANGELOG.md`; this repo records sessions in `REPORT-<topic>.md` files, so this is one.
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
- The pre-push hook refuses a clone outside `/mnt/5_hdd/felhom.eu`; the clones were moved under it. No `--no-verify`.
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
Provisioned nothing. Scratch copies and clones deleted. `~/temp/` untouched.
|
||||||
@@ -0,0 +1,158 @@
|
|||||||
|
# REPORT — Tester-2 rehearsal with a pretend household + an independent data-loss hunt (2026-10-20 → 21)
|
||||||
|
|
||||||
|
> Run 2026-10-20 09:30 UTC → 2026-10-21 ~09:50 UTC. Drafted before the night, finished in the morning. This repo's
|
||||||
|
> shared `REPORT.md` was not touched: a parallel session works in the same clone, and the repo rule says the second
|
||||||
|
> session writes `REPORT-<topic>.md` only (the brief asked for both; this line is why one was skipped).
|
||||||
|
> Evidence: `documentation/audits/rehearsal-2026-10-20/` (journal, A0–B10, screens) and
|
||||||
|
> `documentation/audits/dataloss-hunt-2026-10-20/` (C1–C3). Architecture read for the claims: `07-backup-architecture.md`,
|
||||||
|
> `10-localisation.md`, `00-capability-map.md` (A15).
|
||||||
|
|
||||||
|
## Five sentences
|
||||||
|
|
||||||
|
A pretend household installed a box from the public ISO, linked it, claimed it, put eight apps and about 2 GB of family
|
||||||
|
data on it, used it for a day, and every customer restore path brought its data back. The first night's backups, the
|
||||||
|
clock change, three days off and power cuts in the middle of installs and catch-ups all worked, and the morning data was
|
||||||
|
byte-identical to the evening's. But the 8 GB minimum box froze twice on its first afternoon and only a plug pull helped
|
||||||
|
(R-1192), and the household's warning mails went to the operator instead (R-1199). When the box died, the household could
|
||||||
|
not even start again without the operator, and after the operator's press the rebuilt box could not reach its off-site
|
||||||
|
history at all — proven live (R-1176, P1). Nothing was released; 29 new register rows carry the rest.
|
||||||
|
|
||||||
|
## The ten places a household gets stuck
|
||||||
|
|
||||||
|
`documentation/audits/rehearsal-2026-10-20/B10-stuck/README.md`. In short:
|
||||||
|
|
||||||
|
1. The box froze twice on the first afternoon (≈1 h and ≈35 min); only pulling the plug helped (R-1192, P1).
|
||||||
|
2. The bind page needs an owner passphrase the household does not have — 2 h 4 min waiting for the operator.
|
||||||
|
3. A power cut during a full off-site restore leaves the app on the backup's older data; the page only says „interrupted" (R-1195).
|
||||||
|
4. A pulled drive breaks the last uploads (0 bytes or gone) although each was confirmed (R-1196).
|
||||||
|
5. The family's warning mails went to the operator, not the family (R-1199).
|
||||||
|
6. „Ödön" and „Írisz" are refused as family names (W12).
|
||||||
|
7. A new drive needs a „mount name" nobody knows; the button silently does nothing (W13).
|
||||||
|
8. Sharing waits for ever for a password the page never asks for (W21).
|
||||||
|
9. Errors arrive as raw technical English (W22, W25).
|
||||||
|
10. After the box died, the household was promised a mail that never came, and after the operator's press the rebuilt box
|
||||||
|
could not reach the family's off-site data (R-1203, R-1176). (Close behind: three apps' wrong „first steps", W17–W19.)
|
||||||
|
|
||||||
|
## Data-loss findings, ranked
|
||||||
|
|
||||||
|
| row | Sev | harm | reproduced | design / defect |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| R-1175 | P1 | shares on two drives sharing one second drive: every night one drive's share copy is pruned; the night a drive dies its copy is deleted | unit test on production code (`C3-D1-shares-tier2/`) | defect |
|
||||||
|
| R-1176 | P1 | a box rebuilt after „delete the old host" cannot reach its off-site history: no recovery code asked, a new key, the old history set aside on night one | hub store unit test (`C3-K1-…`) + **LIVE in B7** (2026-10-21) | defect |
|
||||||
|
| R-1191 | P1 | a `.fab` import makes a family-only app open to anyone | LIVE on scratch 9202 | defect (security) |
|
||||||
|
| R-1192 | P1 | the 8 GB box froze under first-day use; two plug pulls | LIVE on Tester4 (twice) | defect — cause not diagnosed (no shell on the box) |
|
||||||
|
| R-1177 | P2 | a failed local restore destroys the live data, keeps no copy | LIVE on 9202 | design question (D22 in STATUS) |
|
||||||
|
| R-1195 | P2 | a power cut mid full off-site restore: app on the backup's data, page says only „interrupted" | LIVE on Tester4 (B6 #2) | defect |
|
||||||
|
| R-1178 | P2 | second-drive „whole restore" can change files, refuse the DB, leave the app stopped and say „untouched" | unit test | defect |
|
||||||
|
| R-1180 | P2 | the nightly mirror (`rsync --delete`) can run during a restore from it | unit test | defect |
|
||||||
|
| R-1181 | P2 | a missing app folder deletes its second-drive copy in one night (the only copy for optional folders) | unit test | defect (D23 in STATUS) |
|
||||||
|
| R-1179 | P2 | a drive marked „away" can still be the second-drive target | code + unit test | defect |
|
||||||
|
| R-1182 | P2 | abandoning one set-aside off-site copy deletes every retained key package | code | defect |
|
||||||
|
| R-1199 | P2 | warning mails reach the operator, not the household | LIVE (two mailboxes) | defect |
|
||||||
|
| R-1204 | P2 | a box that ran once with its clock ahead loses its WireGuard tunnel to ep0 (PBS copies + operator door) until that date | LIVE (B6 #8, ep0 `wg show`) | defect |
|
||||||
|
| R-1203 | P2 | after a box death, the household is promised a link mail that never comes until the operator deletes the old host | LIVE (B7) | defect |
|
||||||
|
| R-1185 | P3 | a dump that fails its own check has already replaced the good one | unit test (docker stub) | defect (order) |
|
||||||
|
| R-1186 | P3 | the nightly off-site copy is skipped silently when another backup holds the flag | code | defect |
|
||||||
|
| R-1184 | P3 | a cut-off shares restore can be reported restored with files missing | code | defect |
|
||||||
|
| R-1196 | P3 | uploads confirmed just before a drive pull are 0 bytes or gone | LIVE (B6 #4) | defect |
|
||||||
|
| R-1183 | P3 | on 2026-10-25 every hub daily job runs twice (DB snapshot keep-2 pruned a day early) | replay of the hub's arithmetic | defect (D24 in STATUS) |
|
||||||
|
| R-1200 | P3 | an afternoon install gets no whole-box backup on night one | LIVE (B5) | design question |
|
||||||
|
| R-1202 | P3 | a box with its clock behind stops an app silently and only says „Hub nem elérhető" | LIVE (B6 #8) | defect |
|
||||||
|
| R-1187 … R-1190 | P3 | the unproven candidates of the four reviewers, grouped | code reading | mixed |
|
||||||
|
|
||||||
|
## Failure scenarios (B6)
|
||||||
|
|
||||||
|
| # | scenario | result | evidence |
|
||||||
|
|---|---|---|---|
|
||||||
|
| ★1 | power cut mid app install | PASS — clean rollback, honest event, reinstall in 15 s | `B6-failures/s1-…` |
|
||||||
|
| ★2 | power cut mid off-site restore | 6 s: safe; 14 s: app on the backup's data, page „interrupted" → R-1195 | `s2-…` |
|
||||||
|
| ★3 | data disk 100 % during backup | PASS — warned at 89 %, refused before writing, healed; SMB error unclear; system disk 97 % not run | `s3-…` |
|
||||||
|
| ★4 | data drive pulled, put back | apps stopped in 42/34 s, back by themselves; last uploads 0 B/missing → R-1196 | `s4-…` |
|
||||||
|
| ★5 | slow lossy line, first big off-site | run failed under 1 Mbit + 5 % loss, raw English; second press OK 3 m 38 s | `s5-…` |
|
||||||
|
| ★6 | off-site blocked | failed in 2.5 min, raw English; catch-up OK 2 m 37 s (shortened, not a night) | `s6-…` |
|
||||||
|
| ★7 | DST | PASS — the database copy ran once, at the second 02:30; W30 (no summer/winter mark in the repeated hour) | `s7-dst/` |
|
||||||
|
| ★8 | wrong clock at boot ±2 y | FINDINGS — −2 y became −6 months (the OS floor): „Hub nem elérhető" only, Paperless stopped silently (R-1202); +2 y: a whole-box archive dated 2028 (R-1189 a), then the ep0 tunnel dead (R-1204) | `s8-wrong-clock/` |
|
||||||
|
| ★9 | two actions at once | PASS — second action refused with a clear sentence | `s9-…` |
|
||||||
|
| ★10 | box off 3 days (+#15 restart in the catch-up) | PASS — ONE catch-up, a clear note to the household, the interrupted catch-up recovered | `s10-three-days-off/` |
|
||||||
|
| 11 | hard reset mid app update | NOT RUN — no update was offered to the box (nothing was released during the run) | — |
|
||||||
|
| 12 | DNS dead 1 h | PASS for data; operator told after ~37 min, household never | `s12-…` |
|
||||||
|
| 13 | second drive added later | not offered as the backup place → R-1198; safe removal OK; blank texts (W29) | `s13-…` |
|
||||||
|
| 14 | remove keeping data → reinstall „use kept data" | PASS — 47/47 documents, 47/47 originals | `s14-…` |
|
||||||
|
| 16 | English browser / English dashboard | follows the household's setting (by design); a few Hungarian strings → fixed W26 (unreleased) | `s16-…` |
|
||||||
|
|
||||||
|
## Restore paths (B8)
|
||||||
|
|
||||||
|
All seven customer paths PASS on the live box (`B8-restores/README.md`): A.1.1 Mealie unit 17 s; A.1.2 Nextcloud from
|
||||||
|
the second drive 3/3 sha256; A.1.3 Radicale verify copy 5 s; A.1.4 Nextcloud missing-only 3/3 sha256 (NFC/NFD: both
|
||||||
|
names come back on the share; Nextcloud itself folds NFD into NFC); A.1.5 BookStack full off-site 30 s; A.1.6 shares
|
||||||
|
2 files; A.1.7 `.fab` Mealie. Gaps: the messages do not say which point in time or how many files.
|
||||||
|
|
||||||
|
## Night chain and DST
|
||||||
|
|
||||||
|
**The first night (B5), no hands:** database copy 00:30Z PASS (3 m 27 s); second-drive copy 01:30Z PASS (8 apps + the
|
||||||
|
share, 58 s); off-site 02:15Z PASS (44 snapshots, 3 m 6 s, nothing deleted; the hub's own listing agrees: 44 files,
|
||||||
|
newest 02:18Z); whole-box backup NOT RUN — not due under the 24 h rule, because the install's own backup at 12:09Z
|
||||||
|
started the clock (R-1200, P3: the family's first day reaches a whole-box copy only on the second night). At 02:46Z
|
||||||
|
demo-hp's own kernel update rebooted the test host and shut the rehearsal VM down (harness, not a Tester4 defect; a BYO
|
||||||
|
host skips that step by design). Health 200 all night before that; memory steady. **The morning data snapshot is
|
||||||
|
identical to the evening one, byte for byte** (`B5-night/README.md`).
|
||||||
|
|
||||||
|
**DST (#7), live:** the box was started with its clock at 2026-10-25 02:20 CEST and left alone: no database copy at the first 02:30, one at the second (01:30:00Z, 1 m 55 s). The code replay agrees for the box; the hub runs every daily job twice on that day (R-1183, D24) and spring 2027 puts the 02:30 dump on the 03:30 copy (`B6-failures/dst-replay/`).
|
||||||
|
|
||||||
|
## Interventions (operator-only acts)
|
||||||
|
|
||||||
|
| # | when (UTC) | what | minutes the household waited |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 1 | 09:57 → 12:00:57 | the owner passphrase handed over (by design, out of band) | 124 |
|
||||||
|
| 2 | asked 09:35, done 12:01–12:15 | off-site switched on for Tester4 | 0 (done before the household needed it) |
|
||||||
|
| 3 | asked 08:59, done ~09:11 (2026-10-21) | delete host Tester4-816e21 before the reinstall (B7) | ~12 (and 17 min of a dead end before it, R-1203) |
|
||||||
|
|
||||||
|
Not counted as interventions: the two plug pulls (14:11, 16:13) — a household can do that, and did.
|
||||||
|
|
||||||
|
## Versions
|
||||||
|
|
||||||
|
controller 0.336.0 (`7ca59e2`, floor), agent 0.161.0 (`4e1621c`), hub 0.162.0, golden 0.331.0, min_agent 0.158.1,
|
||||||
|
installer 1.35.0, ISO 1.29.0 (sha256 `dceacae5…4829`), catalog `c50adfc6` (65 templates). Nothing was released, deployed,
|
||||||
|
vouched, baked or published during the run. One text fix was committed, not released: controller `01c967e` (W26).
|
||||||
|
|
||||||
|
## Browser spike (A4)
|
||||||
|
|
||||||
|
Works: Chromium 155 + Playwright 1.52 in a throwaway LXC on demo-hp, phone and computer, HU and EN, in ~12 minutes.
|
||||||
|
121 screenshots. Limits: no real DNS/TLS for the household's address; not a human. `A4-browser/README.md`.
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
Machine: VM 9320, CT 9330 and the ISO removed from demo-hp by their own IDs; no drill firewall rules left; memory back
|
||||||
|
to the start. DooPlex: three watchers stopped by PID. Hub: **host Tester4-648b80 is left for the operator to delete**
|
||||||
|
(the customer Tester4 stays). ep0: one PBS snapshot `ns/Tester4/ct/9201/2026-10-20T12:17:26Z` and the WireGuard peer
|
||||||
|
10.77.0.6 remain; the Storage Box keeps the household's restic history — listed, nothing removed (`D-teardown/`).
|
||||||
|
|
||||||
|
## Secrets (names only)
|
||||||
|
|
||||||
|
All in `sec`, prefix `REHEARSAL_2026_10_20_`: `9202_PW`, `9202_RESETCODE`, `APP_BOOKSTACK`, `APP_RADICALE`,
|
||||||
|
`APP_VAULTADMIN`, `BIND_TOKEN`, `BOOKSTACK_TOKEN`, `DASH_PW`, `FAMILY_IRISZ`, `FAMILY_KATA`, `FAMILY_ODON`,
|
||||||
|
`INSTALL_ROOTPW`, `IRISZ_PW`, `KATA_PW`, `ODON_PW`, `OLD_BIND_TOKEN`, `PAIRING_CODE`, `PASSPHRASE`, `RECOVERY_CODE`,
|
||||||
|
`SETUP_CODE`, `SMB_PW`. Kept (the operator decides); none written anywhere else. Evidence scanned for their values:
|
||||||
|
0 hits.
|
||||||
|
|
||||||
|
## Register
|
||||||
|
|
||||||
|
Before 153 open rows; now 182. Opened by this run: 29 (R-1175 … R-1200, R-1202, R-1203, R-1204). Closed: R-1019 (a dated
|
||||||
|
check read on demo-felhom, read-only). One row (R-1201) came from a parallel session. Updated: R-1176 (proven live),
|
||||||
|
R-1189 (a half-live), R-895 (first verdict), R-435 (moved). Capability map: A15 still PARTIAL with new evidence; A12 and
|
||||||
|
A13 PROVEN-LIVE → PARTIAL (B7). `unproven.py --summary`: unchanged (35 of 55 not walked).
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
FILED: everything in the tables above. NOT-A-FINDING: the dashboard ignores an English browser (by design: the
|
||||||
|
household's setting wins); Nextcloud folding NFD into NFC (upstream behaviour; both forms survive on the share); the
|
||||||
|
films' off-site size (CBR filler compresses; nothing missing); demo-hp's own kernel reboot at 02:46 (a harness effect: a
|
||||||
|
BYO host skips that step by design, `osupdate/leg.go:674`). FILED: everything in the tables above.
|
||||||
|
|
||||||
|
**Harness slips (mine), each recorded in the journal:** a screenshot step blanked a form before submit (15 min lost); the
|
||||||
|
two watcher logs froze at 20:24 when `git pull --autostash` replaced their files (recovered in full from the open files);
|
||||||
|
my first #10 start used a date the #7 records had made meaningless (restarted, recorded before any result). **Secrets:**
|
||||||
|
the nightly DooPlex sweep found three rehearsal secret values in my session record (the old and new bind tokens and a
|
||||||
|
setup code — I had read their mails in full and typed one into a `sec put`); in B7 a second bind link and setup code
|
||||||
|
entered the record the same way. All are one-time values, spent within minutes; nothing secret is in any committed file
|
||||||
|
(323 files scanned, 0 hits; pictures checked by eye; the B7 pairing code was blacked out).
|
||||||
@@ -0,0 +1,120 @@
|
|||||||
|
# REPORT — the status page back to one screen, and a gate that keeps it there (2026-10-20, day)
|
||||||
|
|
||||||
|
## In plain words
|
||||||
|
|
||||||
|
The status page was 107 lines and 31,695 bytes; now it is 50 lines and about 5,500 characters outside the decide table,
|
||||||
|
which holds the 8 decisions that are still open. The „do" list went from about 25 items to 7 „still needed? (yes/no)"
|
||||||
|
questions, 2 morning-list items and 5 other lines (plus one „Later" line of row ids); CC checked the rest in the hub
|
||||||
|
and on DooPlex, and 7 were already done. **You answer:** the „Still needed? (yes/no)" list below, one word per line.
|
||||||
|
|
||||||
|
## Still needed? (yes/no) — copied from STATUS.md
|
||||||
|
|
||||||
|
1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). Since 2026-10-16.
|
||||||
|
2. The N100 sheet, about 30 minutes: Plex, the admins in Prowlarr, Sonarr and Radarr, your download sources
|
||||||
|
(`documentation/audits/n100-lived-in-2026-10-15/SHEET.md`). Since 2026-10-15.
|
||||||
|
3. Look at these pages on demo-hp when you have time: Calibre-Web („Adatok és mentés"), the file manager, AdventureLog,
|
||||||
|
Plex, the app page, the Plex and Sonarr guides („Útmutató"). Since 2026-10-16.
|
||||||
|
4. Remove the test server „plex-k6" from your Plex account (plex.tv → Settings → Authorized Devices). Since 2026-10-15.
|
||||||
|
5. Ask Google to index `https://felhom.eu/sajat-felho-tarhely` in Search Console (R-1035). Since 2026-10-14.
|
||||||
|
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable.
|
||||||
|
Since 2026-10-14.
|
||||||
|
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12.
|
||||||
|
|
||||||
|
## What CC checked, and how (method: the hub's GET pages with CC's scoped token, from the running pod; read-only local checks on DooPlex)
|
||||||
|
|
||||||
|
| „Do" item | Verdict | Evidence |
|
||||||
|
|---|---|---|
|
||||||
|
| Vouch agent 0.161.0 | **done** — the Day-0 manifest names 0.161.0 with the asked sha256 | `documentation/audits/status-view-2026-10-20/hub-reads.txt` |
|
||||||
|
| Delete the test customer Tester3 | **done** — not in the customer overview | same |
|
||||||
|
| Discard registrations 40 and 41 | **done** — the Hosts page has no „Unclaimed appliances" section (only `registered`/`bound` rows render) | same |
|
||||||
|
| Delete Tester4's old host `Tester4-31fa0e` | **done** — not on the Hosts page | same |
|
||||||
|
| Say „tear it down" for Tester 1's media tests | **done** — R-1022 closed 2026-10-17 | `CLOSED-ITEMS.md` |
|
||||||
|
| Hub master key into the password manager, then shred the file | **done** — `new.key` is gone; its folder is there (control) | `…/local-checks.txt` |
|
||||||
|
| Hub 0.153.0 „no token can …" | not a task — a statement; moved to history | — |
|
||||||
|
| Pull the failed disk Z6A07P2G | **not done** — still in DooPlex (lsblk; its twin is the control) | `…/local-checks.txt` |
|
||||||
|
| The Facebook `first-post/` leftovers | **not done** — 15 files still there | same |
|
||||||
|
| The `deny`-list edit in homelab-manifests | **not done** — still uncommitted | same |
|
||||||
|
| Old „broken" line: a Longhorn volume at 91 % | **gone** — Alertmanager lists only the two R-934 disk alarms | same |
|
||||||
|
| Old „broken" line: CI jobs get lost (R-887) | **gone** — R-887 closed 2026-10-19 (381 jobs, 0 lost) | `CLOSED-ITEMS.md` |
|
||||||
|
|
||||||
|
Every decided row (D1, D2, D10–D20) left the decide table only after its ruling was found in its closed row in
|
||||||
|
`CLOSED-ITEMS.md`. The whole old page is in `STATUS-HISTORY.md` under „Moved from STATUS on 2026-10-20" (headings
|
||||||
|
moved down two levels, otherwise unchanged).
|
||||||
|
|
||||||
|
## Register
|
||||||
|
|
||||||
|
- **Four numbers:** rows before **156**, after **153**, opened **1** (R-1174), closed **4**.
|
||||||
|
- **Closed:** R-1121 (the rotation file is gone; control `~/.bashrc`), R-1118 (its last part, R-1134, closed
|
||||||
|
2026-10-17; `sec doctor` all OK), R-231 (scripts in git and installed from it, the first run green; its same-disk
|
||||||
|
facts live on in R-232) — evidence in `CLOSED-ITEMS.md`, `audits/status-view-2026-10-20/local-checks.txt`,
|
||||||
|
`audits/answers-2026-10-12/s20-backup-scripts.txt`.
|
||||||
|
- **Merged:** R-993 into R-991 (one decision, D5; the proposal and the „no host floor at install" fact copied into R-991).
|
||||||
|
- **Narrowed:** R-132 (the hub password was rotated 2026-10-15; left: the break-glass hash field in the public repo);
|
||||||
|
R-1049 → WATCHING (the fix is live since hub 0.151.0; it no longer waits on the operator).
|
||||||
|
- **Owner cell fixed:** R-919 `CC` → `operator (upload + the look), then CC (close)` — its next action is the operator's.
|
||||||
|
- **Operator-owned rows (43), still waiting on the operator?** yes for all, except: R-933 — **CC first** (the size
|
||||||
|
measurement), then the operator's go; R-388 — **partly** (slice 1 is built; the scope decision is open);
|
||||||
|
R-920 — parked by the operator's own choice (c). On the page: decide D3–D9, D21; do (still needed, the rest);
|
||||||
|
the „Later" line for the remaining 23.
|
||||||
|
- **Rows over 4.5 KB (not rewritten, §N.7):** R-336 7.6 KB, R-516 7.0, R-462 6.8, R-433 6.7, R-264 5.8, R-919 5.0,
|
||||||
|
R-440 4.6, R-925 9.0.
|
||||||
|
|
||||||
|
## The gate — `scripts/status_view_gate.py` (R-1174)
|
||||||
|
|
||||||
|
- **Checks:** size outside the decide table (≤ 50 lines, ≤ 6,000 characters); one „Updated" paragraph (≤ 3 sentences,
|
||||||
|
no „Earlier" chain); no Hungarian letter outside a quote; every cited R-n open; no decided/closed/done row in the
|
||||||
|
decide table; the P1 row under „What is broken" or „decide"; every open row whose Owner names the operator under
|
||||||
|
„decide" or „do". Exit 0 clean, 1 convicted, 2 inconclusive (no sections, missing section, no or empty register).
|
||||||
|
- **Switch:** warn-first until **2026-10-27** (`SWITCH_DATE`), then a refusal; the date is also a due check for
|
||||||
|
R-1174, so the next push on or after that date names it.
|
||||||
|
- **Tests** (`scripts/test_status_view_gate.py`, 39 cases, all PASS): the genuine fixture before and after the switch;
|
||||||
|
the clean run's „what it looked at" line; per check, decoys that warn before and refuse from the switch — size (50
|
||||||
|
more lines; few lines and too many characters), updated (a prepended paragraph, an Earlier chain, four sentences, no
|
||||||
|
paragraph), english (a Hungarian do-line), open-ids (a closed row cited), decide (a DECIDED row; a „done" pick), p1
|
||||||
|
(the P1 line deleted; moved to What works), operator (the Later line deleted; a row only in „broken"); genuine
|
||||||
|
controls that must pass (a long decide table, version numbers in a sentence, Hungarian inside a quote); four
|
||||||
|
inconclusive cases; and the real `STATUS.md` against the real register.
|
||||||
|
- **Red-proof:** each check removed in turn → exactly its own decoys FAIL (size 4, updated 8, english 2, open-ids 2,
|
||||||
|
decide 4, p1 4, operator 4), then put back → 39/39. The switch moved to 2099 → the 14 „refuses" cases FAIL; moved
|
||||||
|
one day earlier → the 14 „warns on 10-26" cases FAIL. Put back → 39/39.
|
||||||
|
- **Run against the old page** (as it was at `2954b0be`): 75 convictions across all seven checks (81 lines, 25,472
|
||||||
|
characters outside decide; 13 decided rows; 24 operator rows missing; the P1 only in „do").
|
||||||
|
- Wired: `repo_gates.py` (fast set, docstring entry 19), `test_gate_decoys.py` (COVERS + runs the suite), found by
|
||||||
|
`script_tests_gate.py`; `decoy_coverage_gate.py` reports it COVERED.
|
||||||
|
|
||||||
|
## The CLAUDE.md edit (repo `felhom.eu/CLAUDE.md`, end-of-session checklist, line 139–141)
|
||||||
|
|
||||||
|
- **Before:** „… One screen, written for the operator in plain language, and deliberately **not** `CONTEXT.md`."
|
||||||
|
- **After:** the same, plus „(STE English) … How to update it:" and five numbered lines — replace the „Updated"
|
||||||
|
paragraph (never prepend; the old one to `STATUS-HISTORY.md`); a morning list under a dated sub-heading in „do";
|
||||||
|
remove what you finished; the decide-row rule; the gate checks it (R-1174).
|
||||||
|
- **Why allowed:** a format rule, not a loosened safety rule (`PROMPT-TEMPLATE.md` §9 rule 9, decision 150).
|
||||||
|
|
||||||
|
## Commits and the running session
|
||||||
|
|
||||||
|
`2954b0be` register pass · `9ec5f76a` the page · `1d7071e8` the gate · `54d1ee82` the instruction · this report.
|
||||||
|
**The rehearsal session shares this clone.** Its commit `f84f94d9` (audit evidence only) landed between my parts 1
|
||||||
|
and 2; it touched neither `STATUS.md` nor the register, so there was nothing to merge. **Its `git push` carried my
|
||||||
|
parts 2–4 to `origin` with its own** (the shared branch); its hook ran with `--scope=docs`. My own `--fast` run of the
|
||||||
|
same content was green (rc 0), and `scripts/push_scope.py` classifies the full range correctly as CODE. Its decoy
|
||||||
|
suite briefly planted changes in `website/gyik.html` and `scripts/felhom-host-install.sh` while I worked (R-928's
|
||||||
|
shape); I staged only my own paths, and both files were clean again afterwards. `unproven.py --summary`: unchanged
|
||||||
|
(NOT WALKED 35 of 55).
|
||||||
|
|
||||||
|
**Gate runs.** Every push: `repo_gates.py --fast`, rc 0 (the last one on the final tree, this report included). **Not
|
||||||
|
followed: „the full run before the last push".** The full run adds exactly one gate, `iso-bootstrap`, which starts a
|
||||||
|
container from an ISO image; on DooPlex that is a Docker run on production (workspace rule, and a standing feedback
|
||||||
|
memory), and it checks only `scripts/iso/`, which this session did not touch.
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- „My commits were pushed by the other session's push, under a docs-scope hook" — **NOT-A-FINDING:** CI re-runs the
|
||||||
|
whole fast set on every push whatever the hook's scope, and CI was green for every push of this session (by `head_sha` on the `jobs` endpoint, attempt 1 each): `2954b0be` job 2131, `54d1ee82` job 2132 (it carried parts 2–4), `c701b253` job 2133 — all `success`; the
|
||||||
|
shared-clone behaviour is already recorded (R-928, the shared-worktree rule).
|
||||||
|
- Tester4 shows DOWN with no host on the hub — **NOT-A-FINDING:** the rehearsal session reinstalls Tester4 today.
|
||||||
|
- „A Longhorn volume at 91 %" had no row — **NOT-A-FINDING:** the alarm is no longer active (Alertmanager, with the two
|
||||||
|
R-934 alarms as the control).
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
Provisioned nothing: no machine, no host, no hub write (GET only, CC's scoped token).
|
||||||
@@ -0,0 +1,87 @@
|
|||||||
|
# REPORT — the website keeps syncing during an internet outage (2026-10-19)
|
||||||
|
|
||||||
|
## In plain words
|
||||||
|
|
||||||
|
1. **It worked.** The website's copy helpers now read Gitea inside the cluster, with no internet hop, and they wait
|
||||||
|
and try again instead of stopping. The new website pod runs with 0 restarts, and the site and the installer
|
||||||
|
script both answer normally.
|
||||||
|
2. **Note for later:** this ties the website to the DooPlex cluster. If it moves (for example to Hetzner), the
|
||||||
|
address must go back to the public one — register row R-1164.
|
||||||
|
|
||||||
|
## Part 1 — anonymous in-cluster fetch (before any commit)
|
||||||
|
|
||||||
|
Throwaway pod in a scratch namespace (`ccscratch-gitsync`, deleted after), no credentials:
|
||||||
|
`git ls-remote http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git main installer-v1.34.0`
|
||||||
|
|
||||||
|
| ref | in-cluster | public (anonymous) |
|
||||||
|
|---|---|---|
|
||||||
|
| `refs/heads/main` | `5d3cb5c1…` | `5d3cb5c1…` — equal |
|
||||||
|
| `refs/tags/installer-v1.34.0` | `add58eec…` (annotated tag object) | `add58eec…` — equal; peels to `624bcdc3…` |
|
||||||
|
|
||||||
|
Evidence: `audits/website-gitsync-incluster-2026-10-19/gitsync-part1-*.txt`.
|
||||||
|
|
||||||
|
## Part 2 — the change
|
||||||
|
|
||||||
|
Commit **`fd75003e`** (no Co-Authored-By, per the brief):
|
||||||
|
- `manifests/webpage.yaml`: all four git-syncs `--repo=http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git`;
|
||||||
|
the two sidecars `--max-failures=-1` (not the `--one-time` inits); one comment above the website sidecar. Refs and
|
||||||
|
`nginx-config-rev` unchanged.
|
||||||
|
- `scripts/hostinstall_gates.py` gate 6: fails unless there are four `--repo=` values and all are identical. Existing
|
||||||
|
ref checks unchanged.
|
||||||
|
|
||||||
|
Gates: `python3 scripts/repo_gates.py` → `all felhom.eu gates OK`; `python3 scripts/hostinstall_gates.py` → rc 0
|
||||||
|
(`ok: manifest: all four git-syncs read the same repo (…)`).
|
||||||
|
|
||||||
|
## Part 4 — deploy and proof
|
||||||
|
|
||||||
|
**Deploy:** hard refresh showed only `Deployment/felhom-webpage` OutOfSync; synced that resource alone at `fd75003e`
|
||||||
|
(the skill's Deployment-only sync). New pod `felhom-webpage-869ff88bf7-fswbc` ready at 09:42 UTC; the old pod
|
||||||
|
terminated. ArgoCD: Synced, operation Succeeded. (The first `rollout status` returned at once — it ran before the new
|
||||||
|
ReplicaSet existed; re-run after the sync it waited for the old replica and finished.)
|
||||||
|
|
||||||
|
**Scenario A** (`gitsync-scenarioA.txt`): restarts `0,0,0`; both sidecars started with the in-cluster `--repo` and
|
||||||
|
`--max-failures=-1`, then `updated successfully` (`main` → `fd75003e`, `installer-v1.34.0` → `624bcdc3`); both init
|
||||||
|
containers `updated successfully`; `https://felhom.eu/` **200**, `https://felhom.eu/scripts/felhom-host-install.sh`
|
||||||
|
**200**, served `SCRIPT_VERSION="1.34.0"` = manifest `--ref=installer-v1.34.0`.
|
||||||
|
|
||||||
|
**Scenarios B and C** (`gitsync-scenarioBC.txt`): scratch namespace `ccscratch-gitsync-bc` (deleted after),
|
||||||
|
`git-sync:v4.4.0 --repo=http://192.0.2.1/x.git --period=10s`:
|
||||||
|
- **B, with `--max-failures=-1`:** `error syncing repo, will retry` (`failCount:1`), restarts **0**.
|
||||||
|
- **C, without:** `too many failures, aborting`, restarts **1**.
|
||||||
|
- The first read at 95 s showed nothing for either: one fetch to an address that never answers takes **~133 s** to
|
||||||
|
time out. The re-read at 09:48 recorded both results. Instead of 90 s, a later run of this test needs about 150 s.
|
||||||
|
|
||||||
|
**Scenario D — gate red-proof:** one `--repo` changed to the public address → gate 6 `FAIL … the git-sync --repo
|
||||||
|
values differ`, rc 1; restored byte-identical → rc 0.
|
||||||
|
|
||||||
|
## Part 3 — docs changed
|
||||||
|
|
||||||
|
- `documentation/architecture/01-topology-and-trust.md` §7: a `[DESIGN]` paragraph — the website pulls `website/`
|
||||||
|
(main) and `scripts/` (installer tag) from Gitea in-cluster over plain HTTP, sidecars retry forever, tied to the
|
||||||
|
DooPlex cluster.
|
||||||
|
- `.claude/rules/manifests.md` „The two git-syncs": the fact that all four containers read Gitea in-cluster, must name
|
||||||
|
one repo (gate 6), and the sidecars carry `--max-failures=-1`.
|
||||||
|
- `scripts/CHANGELOG.md`: one entry.
|
||||||
|
|
||||||
|
## Register
|
||||||
|
|
||||||
|
Rows before **166**, after **167**. Closed **R-1163** (finding + fix, with evidence). Opened **R-1164** (P4, Hub &
|
||||||
|
operator: move the website to Hetzner; then the git-syncs must use the public address again).
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- The 95-second wait in the brief is too short for an address that never answers (~133 s TCP timeout per fetch) —
|
||||||
|
NOT-A-FINDING: recorded here as the correct wait for this test; nothing in the product depends on it.
|
||||||
|
- The skill says a whole-app sync would push git's view over de-gitted Secrets; today only the webpage Deployment was
|
||||||
|
OutOfSync — NOT-A-FINDING: the Deployment-only sync was used anyway, as the skill says.
|
||||||
|
|
||||||
|
- Around 09:50–09:55 UTC, DooPlex itself briefly could not resolve names (`Temporary failure in name resolution`;
|
||||||
|
`curl https://felhom.eu/` → `000`). By 09:56 DNS, the internet and the site (200) were back. The website pod kept
|
||||||
|
restarts `0,0,0` and synced `e2c35007` in-cluster at 09:54 with no error. NOT-A-FINDING: the same class of short
|
||||||
|
outage that triggered this task; it is not proof that the fix worked (whether the blip reached the cluster is unknown).
|
||||||
|
- CI: `fd75003e` job 2092 success, `e2c35007` job 2094 success.
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
Scratch namespaces `ccscratch-gitsync` and `ccscratch-gitsync-bc` deleted (`kubectl get ns | grep -c ccscratch` → 0).
|
||||||
|
No hub, controller, agent, customer box, demo box or ep0 touched.
|
||||||
@@ -1,93 +1,66 @@
|
|||||||
# STATUS — what works, what's broken, what's next
|
# STATUS — what works, what's broken, what's next
|
||||||
|
|
||||||
**Ready for the first real tester (Tester-2): yes. Tester 2 (a laptop) is off; nothing was sent to it.**
|
**Ready for the first real tester (Tester 2)? No — one step left:** release the agent with the swap fix (decide D26); the 8 GB box can still freeze under big uploads until then. Tester 2 (a laptop) is off; nothing was sent to it.
|
||||||
|
|
||||||
**Updated 2026-10-18 (night).** Controller 0.329.0 is the floor for every box (21:36 UTC, MinAgent 0.158.1), after the three test boxes: each app reaches only its own data folder on the drive; homepage no longer controls Docker, so no app has a special right; when the host refuses to retire a drive, nothing changes and the household reads why; a virtual disk without a serial gets a plain sentence. Immich gets 3 GB (it was killed twice at 2 GB on a 4,000-photo import). Golden 0.329.0 is baked, NOT vouched. Hub 0.158.0 is built, NOT deployed — `REPORT-night-2026-10-18.md`. **Earlier:** **Updated 2026-10-17 (evening).** Controller 0.328.0 is the floor for every box (17:22 local): the box refuses a catalog change that asks for dangerous container rights (your D2 = A). Agent 0.159.1 (a gone, decommissioned drive is forgotten; an orphan thin snapshot is named in the backup failure; new Go) — runs on demo-hp, Tester 1 and demo-felhom; **your vouch in the hub is still open**. New emergency root passwords on demo-hp, Tester 1, demo-felhom (your D1 = A; read them in the hub). Hub 0.157.0 deployed on your yes — `REPORT-agent-allowlist-rotation-2026-10-17.md`. **Earlier:** **Updated 2026-10-17 (afternoon, later).** Controller 0.327.1 is the floor for every box since 12:06 UTC: FileBrowser is published only after its default password is checked (your D11 = A), and the install check uses real memory amounts with a clear message (your D10 = A) — `REPORT-filebrowser-memory-ci-2026-10-17.md`. **Earlier:** Controller 0.326.0 is the floor for every box since 09:55 UTC: FileBrowser's default password is checked at every FileBrowser start. Tester 1 is back on the live app list (test apps and drives removed); six memory limits raised — `REPORT-tester1-limits-filebrowser-2026-10-17.md`. **Earlier today:** controller 0.325.0 was the floor since 08:21 UTC (after the three test boxes): two more password doors count a try before checking it, family sign-ins are stored as fingerprints, a file-share rename always applies; five apps' Redis saves snapshots now, which a full disk cannot damage — `REPORT-burndown-3-2026-10-17.md`. **Earlier (2026-10-17 night):** controller 0.324.0 was the floor for every box since 02:46 UTC (all three test boxes made tonight's off-site backup on it first): a restore now brings the database password back with the data (your 1A), restic 0.19.1 (your 2A), dashboard hardening — `REPORT-night-2026-10-17.md`. **Earlier (2026-10-16 evening):** 0.323.0, Debian 13 — `REPORT-debian13-restic-2026-10-16.md`. **Earlier (midday):** Controller 0.319.0 (the app page in panels, the login in one panel) is the floor, MinAgent 0.158.1 — `REPORT-app-page-tuning-2026-10-16.md`. **Earlier (morning):** Storage identity build B shipped: agent 0.158.1 and golden 0.318.0 vouched by you, controller 0.318.0 is the floor (MinAgent 0.158.1), installer 1.33.0 published — new one-disk boxes get their own data area. Open items: see `documentation/backlog/OPEN-ITEMS.md`. Older notes, day by day:
|
**Updated 2026-10-21 (evening).** A rebuilt box now gets the household's off-site backups back (hub 0.163.0, proven live: 3 031 files byte-identical), and shared folders, imported apps and away drives keep their copies and locks (controller 0.337.0, the floor). The 8 GB freeze is explained — the guest's swap was capped at 512 MB — and the agent fix is ready but not released. Report: `REPORT-fixes-2026-10-21.md`.
|
||||||
`documentation/backlog/STATUS-HISTORY.md`. Reports: `REPORT-night-2026-10-14.md`, `REPORT-immich-rollout-2026-10-14.md`, `REPORT-seo-felho-tarhely-2026-10-14.md`, `REPORT-immich-repair-2026-10-14.md`, `REPORT-power-cut-fixes-2026-10-14.md`, `REPORT-night-power-2026-10-13.md`, `REPORT-poweroff.md`.
|
|
||||||
|
|
||||||
## What works
|
## What works
|
||||||
- **A full disk can no longer leave an app's Redis broken** (catalog, 2026-10-17): RomM, Docmost, paperless-ngx, Immich and Nextcloud save their small Redis store as snapshots. Proven first on the scratch box: 100 of 100 test keys kept through the switch and through a backup and restore. A box takes it at the app's next start (normally that night's backup). Tester 1 waits: it still follows the drill catalog (R-1022).
|
- A box installs from nothing with the public installer, links to the hub, and moves to the floor by itself (map A1, A3, A4).
|
||||||
- **Two more password doors count a guess before checking it** (controller 0.325.0): the share-link password and the off-site wizard's password. Family sign-ins are stored as fingerprints; signed-in family members stay signed in.
|
- 65 catalog apps install, update one tested step at a time, and a failed update undoes itself (map B1, B3, B4, B6).
|
||||||
- **A restore brings the database password back with the data** (controller 0.324.0, your decision 1A): remove an app, install it again, restore it — the app opens its own data (proven on demo-hp with Rallly and Grimmory, off-site and on-box, and through „use my kept data"). A backup without the password is refused before anything changes, with a plain sentence. Backup program restic 0.19.1 (your 2A): old and new copies read both ways; scanner findings in it 50 → 19.
|
- Backups: database dumps, a second-drive copy, an off-site copy and whole-box copies; restore tests run by themselves (map C1–C4, A17, A18).
|
||||||
- **The dashboard is harder to attack** (0.324.0): no sending the browser to another site after sign-in, no framing, parallel password guesses counted, 12-character minimum, reset-code requests limited, sign-out needs a token (the old sign-out link did nothing), a password change ends every family and gate session.
|
- A restore brings an app back with its data and its database password; a failed off-site restore puts the app back as it was (map C5, D6).
|
||||||
- **The box's backup program is now chosen, not inherited** (controller 0.323.0, 2026-10-16): restic 0.18.1, checked by its signature and fingerprint at every build, on Debian 13. Every way the box reads restic was compared old vs new with real outputs; old copies stay readable by the new program and new copies by the old one (on demo-hp's real store). The file-sharing image has 0 known high holes (was 48), and the same household password still works after the change.
|
- After a power cut the box starts by itself, cleans up a half-written backup, and Immich repairs broken photos (map C10, F12).
|
||||||
- **The hub's key can no longer open anything on a box** (controller 0.322.0, 2026-10-16): the settings route is gone, and every former door answers 401 even with the box's own key. Every box that reports answers only over https (35 names). The controller and the hub (0.156.0, deployed at your yes) are built on a supported Go (1.27.2): known Go weaknesses in each 62 → 1; the hub has 0 high findings.
|
- A second drive is offered as the backup target; media apps see every drive; a one-disk box gets its own data area (map D1, D13, D15).
|
||||||
- **RomM game saves are in the backup now** (2026-10-16, catalog): a save survived a backup, a removal and a restore on demo-hp, byte for byte. Eight apps no longer claim „a known, shared password" (your D10 = A). If someone changes the file manager's password in the file manager, the box notices and offers a new one (controller 0.321.0).
|
- The three test boxes answer only over https; the hub's key opens nothing on a box; password guesses are counted (map E12, E13, E17).
|
||||||
- **Every installed app says where its login is, and has a „Fájlok ↗" button to its folder** (controller 0.320.0, 2026-10-16). The file manager is shown as part of the box again, with its password; on a box that does not know that password, one press sets a new one (done on demo-hp and, at your yes, on the N100).
|
- The hub mails alarms and recoveries, and keeps one whole-box copy per week on ep0; CC uses its own limited token, not your password (map F1, F11, G15).
|
||||||
- **Every app but three has a user guide and an FAQ, in Hungarian and English** (catalog, 2026-10-15 night): the „Útmutató” tab shows it on boxes with controller 0.315.0 or newer. Every button name in a guide was checked in the app's own files at the version we ship; a guide that could not be checked stays hidden (plant-it, home-assistant), and Emby has none (R-1092).
|
|
||||||
- **The N100 is the lived-in box** (since 2026-10-15): used like a household's box, changed only through the product. qBittorrent, Prowlarr, Sonarr and Radarr are installed and linked by the box; films never go off-site (checked before any download). Its diary: `documentation/lived-in/n100.md`.
|
|
||||||
- A restored app comes back exactly as safe as it was at the backup: closed to strangers until its first account is made (0.314.0, proven on the scratch box). A box on a test build now moves with the floor (Tester 1 did, by itself). The box claims Plex again (demo-hp's Plex claimed through the dashboard, 2026-10-15). A brand-new box from the public installer updated itself to the floor 3 minutes after it was linked (test box, since removed).
|
|
||||||
- Scripts use the hub with a scoped token, not your password (hub 0.153.0): CC sets the floor with its own token; no token can see a secret, delete anything, or send, make or show a household's password or link. Your new hub password is yours alone (changed 2026-10-15).
|
|
||||||
- Every public name of the three test boxes answers only over https: plain http gets a 301 to the same https address, and browsers are told to remember it for one day (29 names checked, 2026-10-14 night).
|
|
||||||
- Media apps work together on one drive: a download reaches Sonarr/Radarr as one file, no copy; the download apps see only their own folder (live catalog, proven on demo-hp).
|
|
||||||
- A power cut in the middle of a backup no longer blocks later backups: the box removes the half-written backup at start-up (proven with a real cut on demo-hp, on all three boxes now).
|
|
||||||
- After a power cut, Immich repairs the photos the cut left broken, by itself, and the app page lists any lost photo (proven with a real cut on demo-hp: 41 of 41 repaired; on all three boxes now). An Immich update cannot switch it off quietly: the catalog refuses an untested Immich version.
|
|
||||||
- A household can switch the machine off; the hub holds alarms for 14 days and clears the state by itself (N100).
|
|
||||||
- A real power cut on demo-hp: it started by itself after all 8 cuts of the night (with the BIOS set), every app back in ~4 min; stale and recovered mails true and paired.
|
|
||||||
- A page for people searching „felhő tárhely" is live (/sajat-felho-tarhely, /en/own-cloud-storage): what cloud storage is, what to do when Google or iCloud fills up, a five-year cost table with prices read 2026-10-14, and who should NOT buy this.
|
|
||||||
- The machine estimator is public (/gepigeny, /en/sizing), with figures from two machines (N100 and HP t740; raised only, 2026-10-14).
|
|
||||||
- 65 apps in the catalog. New today: Prowlarr — you enter your download sources once, and Sonarr and Radarr get them, linked by the box with no button (the website lists 63 apps — R-1036). OS updates, the kernel lane too, run at night.
|
|
||||||
- The hub is closed again after the 2026-10-12 security review; one box cannot read another's data.
|
|
||||||
|
|
||||||
- Felhom runs in a virtual machine on OpenMediaVault 8, tried on an exact copy of the first outside tester's box: install, bind, claim, apps, backup, and back by itself after an OMV restart. The Hungarian guide for him is `documentation/pilot/GUIDE-omv-kvm-hu.md`; open: one-disk Jellyfin (R-1068), NAS films for Jellyfin (R-1071, yours), no disk health in a VM (R-1073, yours). Waits on you: send the guide; delete test customer Tester4 in the hub.
|
|
||||||
- A second local copy counts only when it is on a different physical disk, and the box can prove it. Inside a virtual machine the box makes the copy but says „A Felhom nem tudja ellenőrizni…" (Tester 1). On real disks nothing changes (demo-hp, N100). A NAS never hides the „only one copy" notice. A box with no drive refuses a media app in words, with a link to the Meghajtók page. Live on the three test boxes; the NAS and no-drive parts are unit-tested only. Next: build B (the internal place on one-disk boxes, R-1079). Your rulings D1–D6 are in the design (`architecture/16-storage-physical-identity.md`).
|
|
||||||
- **One-disk boxes get their own data area** (2026-10-16): a new install makes a „Belső tárhely" on the disk, outside every whole-box copy, and media apps install there with no extra step (proven on a fresh 512 GB box: 213 GiB area, Jellyfin with no extra step). A disk split into parts can be used part by part. The backup offer no longer promises disk-failure safety when it cannot check the disk, and `/mnt/sys_drive` is no longer offered.
|
|
||||||
|
|
||||||
## What is broken
|
## What is broken
|
||||||
- Two root-level weaknesses wait on your decisions below (R-982, R-985). Tester 2 still has its old emergency password (R-1139, when it is back).
|
- **P1:** the 8 GB box can freeze for minutes under big uploads until the agent with the swap fix is released (R-1192; cause measured, fix proven by hand for 2 hours).
|
||||||
- DooPlex: a failed disk with its alarms silenced (R-934); CI jobs still get lost (R-887); something sets the shared
|
- **P1:** about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours).
|
||||||
checkout to „bare" (R-935); a Longhorn volume at 91 %.
|
- Two root-level weaknesses wait on your decisions D3 and D4 (R-985, R-982).
|
||||||
- Tester 2 has never made an off-site check (R-862). The scratch box cannot read the drill catalog (R-992).
|
- Tester 2 is off since 2026-10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862).
|
||||||
|
- DooPlex: the failed disk is still in the box, its alarms silenced (R-934); something sets the shared checkout to „bare" (R-935).
|
||||||
|
- The scratch box cannot use agent features (R-1083) or read the drill catalog (R-992).
|
||||||
|
|
||||||
## What we work on
|
## What we work on
|
||||||
- **CC's secrets** (R-1118): done 2026-10-17 — the store, `sec`, the guards in every CC session (option B: `bash -x` and `curl -v` allowed inside `sec run`), the nightly check (linger on). The old credentials file and its three copies are deleted.
|
- Dated checks, run by CC. 2026-10-21 done: the hub's own off-site check found nothing wrong (R-895); downloads stay out of the second-drive copy (closed). 2026-10-23: no app stop when the host is busy (R-1173). 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the weekly clean-up window (R-435; the logs of the last one were lost to restarts).
|
||||||
- **After 2026-10-18 14:16 UTC** (demo-hp's next clean-up window): check that the old restic (image 0.322.0) still lists and restores demo-hp's store after its first real 0.19.1 prune (`audits/night-restore-2026-10-17/tools/rx.sh`). Everything else of R-1105/R-1107 is done.
|
- A gate keeps this page short and true: it warns now and refuses from 2026-10-27 (R-1174). CC.
|
||||||
- The power-cut findings still open (R-1004, R-1005, R-1010, R-1013). R-1012 closed: agent 0.159.1 names an orphan thin snapshot in the backup failure.
|
|
||||||
- Media apps on every drive (R-1023): live with the 0.313.0 floor — every drive in every media app, a new drive made ready by the box, a pulled drive loses nothing, a safe removal lets go first and the drive comes back by itself.
|
|
||||||
|
|
||||||
## Waiting on you — decide
|
## Waiting on you — decide
|
||||||
|
|
||||||
| # | Row | Question | Pick | The other option costs | If you do nothing | Blocked |
|
Newest first. CC's pick is in the „Pick" column. Each row also has its full text in the register.
|
||||||
|
|
||||||
|
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|
||||||
|---|---|---|---|---|---|---|
|
|---|---|---|---|---|---|---|
|
||||||
| D1 | R-954 | **DECIDED 2026-10-17: A** — new random emergency root passwords on demo-hp, Tester 1 and demo-felhom (your answer: random); each new one works, each old one is refused. Tester 2: R-1139. | done | — | — | — |
|
| D26 | R-1192 | Release agent 0.162.0 with the swap fix (every guest may swap as much as its RAM), and vouch it? | **Yes, tomorrow morning** (measured: the freeze ended within a minute of the change; 2 hours of the same uploads stayed answering) | Wait: a household's 8 GB box can freeze for minutes, up to an hour, under its first uploads | Tester 2 stays „not ready"; the controller already warns honestly | Tester 2 |
|
||||||
| D2 | R-971 | **DECIDED 2026-10-17: A** — the box refuses a catalog change that asks for dangerous container rights (controller 0.328.0, every box). Only homepage's read-only Docker socket is allowed. A refusal keeps the app running and mails you. | done | — | — | — |
|
| D22 | R-1177 | A LOCAL restore that fails destroys the live data and keeps no copy. Give it the off-site restore's „put back exactly" (your D20)? | **Yes** (proven live today: a failed restore left the app dead and the newest data gone) | No: a failed local restore can lose the household's newest data | the main restore button can lose data when a backup copy is damaged | — |
|
||||||
| D3 | R-985 | „Zero-knowledge": change the wording, or remove the operator's root reach to the key? | **Change the wording now**; revisit after the first sale | Remove the reach: a design change to break-glass | the promise stays stronger than the product | the website text |
|
| D21 | R-49 | Immich's off-site copy: leave out only the AI model files now (B), or also the raw database folder and Immich's own backups (A)? | **B now** (−62 %, no new risk); A's other parts after their proof | A: −92 %, but it needs a restore path we have never run | every Immich box sends about 12 times its real data off-site each night | — |
|
||||||
| D4 | R-982 | Root SSH with a password from the LAN? | **No — key only**; the password stays for the console | Keep it: anyone on the household network can try it | the host keeps accepting root password logins from the LAN | — |
|
| D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — |
|
||||||
| D5 | R-993 (R-991) | Approve the minimum machine (`audits/sizing-2026-10-13/SIZING.md` §5: x86-64, 4 cores, 8 GB, 256 GB SSD)? | **Approve** | Change it: one edit | no hardware page; a buyer with a Raspberry Pi can still download the installer | the hardware page |
|
| D8 | R-206 | Install DooPlex's Docker settings from git (Docker restarts once), with a weekly build-cache prune? | **Yes, in a quiet window** (a rebuilt DooPlex then keeps the settings) | No: the live settings stay hand-made, and a rebuild loses them | git and the host can drift apart | — |
|
||||||
| D6 | R-992 | The scratch box cannot read the private drill catalog: give it a read-only login, or make that copy readable? | **A read-only login** | Readable copy: the drill catalog becomes public | every new-app test needs hand-staging | new-app tests |
|
| D7 | R-990, R-989 | Rule on the forum ideas and the selfh.st ideas now, or park them? | **Park them** (P4; nothing waits on them) | Rule now: about one hour of reading | nothing changes; the research gets old | — |
|
||||||
| D7 | R-990, R-989 | Rule yes / no / later on the forum ideas (`audits/forum-research-2026-10-12/FELHOM.md` §5) and the selfh.st ideas (`audits/selfhst-survey-2026-10-12/FELHOM.md` §6) | — | — | nothing changes; the research ages | — |
|
| D6 | R-992 | The scratch box cannot read the private drill catalog: give it a read-only login, or make that copy public? | **A read-only login** (the drill catalog stays private) | Public copy: anyone can read the drill catalog | every new-app test needs staging by hand | new-app tests |
|
||||||
| D8 | R-206 | Install DooPlex's Docker settings from git (restarts Docker), and prune the build cache weekly: yes or no? | **Yes, in a quiet window** | — | a rebuild of DooPlex still has the settings in git | — |
|
| D5 | R-991 | Approve the minimum machine: x86-64, 4 cores, 8 GB RAM, 256 GB SSD (recommended: 16 GB, 512 GB)? | **Approve** (measured on the N100 with 61 apps) — **but wait for R-1192**: an 8 GB box froze on its first afternoon, 2026-10-20 | Change it: one edit | no hardware page; a buyer with a Raspberry Pi can still download the installer | the hardware page |
|
||||||
| D13 | R-1140 | **DECIDED 2026-10-17: A** — the hub keeps its own access log, 90 days. Built in hub 0.158.0 (≈ 19 MB for 90 days: the hub serves ≈ 1,870 requests a day, not 500); it waits for your deploy yes (morning list). | done | — | — | — |
|
| D4 | R-982 | Stop root SSH with a password from the household network (key only)? | **Yes — key only** (the password stays for the console) | Keep it: anyone on the household network can try the password | the host keeps accepting root password logins from the LAN | — |
|
||||||
| D14 | R-1036 | The apps page lacks Plant-it and wger. Both are closed to new installs (wger hidden, Plant-it abandoned by its makers): add their cards anyway? | **No — leave them off** and close the row; the site shows only what a visitor can install | Yes: first open them for install again (wger's login throttle, Plant-it has no upstream), then the cards | nothing changes; the site lists 62 catalog apps, all installable | — |
|
| D3 | R-985 | Our „zero-knowledge" wording is stronger than the product: change the wording, or remove the operator's root reach to the key? | **Change the wording now**; look again after the first sale | Remove the reach: a design change to break-glass | the website promises more than the product does | the website text |
|
||||||
| D15 | R-1147 | The public machine estimator sizes Immich from a 200-photo test. Tonight's 4,000-photo test needed more. May the public figures move up? | **Yes** — CC re-runs the estimator with the new figures and shows you the before/after first | No: the page keeps the old Immich figure | a buyer with a big photo library may pick too small a machine | — |
|
|
||||||
| D10 | R-1125 | **DECIDED 2026-10-17: A** — the 23 memory requests now match real use (catalog `1af6ea7`); the install refusal and page say what to do (controller 0.327.0). An 8 GB box no longer installs the whole household set. | done | — | — | — |
|
|
||||||
| D11 | R-1126 | **DECIDED 2026-10-17: A** — FileBrowser is published only after its default password is checked; it never answers on the default (controller 0.327.1). The tunnel still runs before the claim, by design (`01` §5). | done | — | — | — |
|
|
||||||
| D12 | R-1129 | **CLOSED 2026-10-17: not a defect** — CC misread the CI log; the failure mail always worked. A forced test mail went to admin@felhom.eu at 10:48:54 UTC (CC saw it arrive). Your key change did no harm. | done | — | — | — |
|
|
||||||
| D9 | R-1062 | The N100's USB drive is also its whole-box backup drive. Split it in two anyway? | **No — keep one drive**; the lived-in box keeps its nightly whole-box backup | Split: the box loses its local whole-box backup (or add a second drive first) | it stays one drive; multi-drive tests stay on Tester 1 | — |
|
|
||||||
|
|
||||||
## Waiting on you — do
|
## Waiting on you — do
|
||||||
- **Vouch golden 0.329.0 and agent 0.159.1 together** (Configuration → Day-0 artifacts): golden sha256 `58dc40df555676d87200e5cb24cdc65f39cbacb2417e15a1955013526de4f68c`, agent sha256 `de8f6ad96860ec399757a88d8bbb0191c089c9bb99db8212482c12daab41aff9` (read from the registry tonight: matches), bundle `2af3d27d686fe611f3f44e89ba3d5123a5572f51eaf87e0ac63df00a41e67cd6`, Min agent **0.158.1**. If you do not: a new box installs golden 0.318.0 with agent 0.158.1 and then updates itself to the floor.
|
|
||||||
- **May I deploy hub 0.158.0** (you can vouch an agent without a new golden; the alarm for missing off-site copies; the hub's own 90-day access log)? Built tonight, not installed (night rule). The next daytime session asks you. If you say no: the hub stays 0.157.0 and nothing changes.
|
**Still needed? (yes/no)** — CC could not check these. Answer yes or no per line.
|
||||||
- **Sunday night's golden:** a golden at the floor's version (0.329.0) already exists, unvouched. The Sunday rotation does not bake a duplicate (runbook updated); it repeats the vouch line above if it is still open, and bakes only if the floor has moved above 0.329.0.
|
1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). Since 2026-10-16.
|
||||||
- **Nézd meg a demo-hp-n a Calibre-Web „Adatok és mentés" fülét és a Fájlkezelőt** (`felhom.enkisfelhom.hu`, controller 0.321.0, minden dobozon kiadva). A mentés egy részben, egyszerű szavakkal; a Fájlkezelő nem kér frissítést és nem ajánl „Install app"-ot.
|
2. The N100 sheet, about 30 minutes: Plex, the admins in Prowlarr, Sonarr and Radarr, your download sources (`documentation/audits/n100-lived-in-2026-10-15/SHEET.md`). Since 2026-10-15.
|
||||||
- **Look at `roms/roms/` on the N100's drive in the file manager (R-1099).** RomM now scans `roms/<platform>/`. A game uploaded through RomM before today sits one folder deeper: if there are platform folders in `roms/roms/`, move them up into `roms/` and press „Scan” in RomM. If not: those games show under a platform called „roms”.
|
3. Look at these pages on demo-hp when you have time: Calibre-Web („Adatok és mentés"), the file manager, AdventureLog, Plex, the app page, the Plex and Sonarr guides („Útmutató"). Since 2026-10-16. Today's rehearsal screenshots: `documentation/audits/rehearsal-2026-10-20/screens/`.
|
||||||
- **Nézd meg a Calibre-Web oldalát a demo-hp-n** (`felhom.enkisfelhom.hu`, controller 0.319.0, minden dobozon kiadva). A fejléc, a fülsor és minden rész most külön panel (az „A” stílus). A belépési adatok egy panelben vannak, egymás mellett. A Calibre-Web jelszaváról a doboz nem tud, mert az alkalmazás előbb települt, mint hogy a sablonban jelszó lett volna — ezt most egy mondat mondja ki. Ha nem tetszik, írd meg, mit változtassak. Ha nem nézed meg: semmi nem romlik el.
|
4. Remove the test server „plex-k6" from your Plex account (plex.tv → Settings → Authorized Devices). Since 2026-10-15.
|
||||||
- **Discard two unclaimed boxes and delete Tester4** (hub, browser only): the build B scratch VMs registered as unclaimed boxes — registration ids 40 (`b43afa10…`) and 41 (`9c78ea30…`); both VMs are destroyed. Tester4 (the throwaway customer) is switched off the household way. If not: two dead entries stay in the Hosts list and Tester4 stays in Customers.
|
5. Ask Google to index `https://felhom.eu/sajat-felho-tarhely` in Search Console (R-1035). Since 2026-10-14.
|
||||||
- System poster needs a refresh: one-disk boxes now have a data area („Belső tárhely") on the system disk, outside the whole-guest backups (facts file updated 2026-10-16).
|
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. Since 2026-10-14.
|
||||||
- **Olvasd el a Plex és a Sonarr útmutatót a demo-hp-n** (az alkalmazás oldalán az „Útmutató” fül). Ha más formát szeretnél, egy menetben átírom mind a 62-t. A robot 2026-10-16-án bejelentkezve megnézte: mindkét oldal megjelenik magyarul és angolul, 4 rész és a GYIK, minden sor egyezik a katalógussal. Ha nem nézed meg: csak a forma marad emberi szemmel nem látott.
|
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12.
|
||||||
- **Nézd meg az AdventureLog és a Plex oldalát a demo-hp-n, a rendes címén** (`felhom.enkisfelhom.hu`). A túl nagy keret javítva: a tartalom most 32 képpontra van a bal oszloptól és 36-ra a tetejétől (előtte 80 és 64 volt). Az alkalmazás oldala mostantól egy hasábban van: egy szélesség, vékony vonalak a szakaszok között, aláhúzott fül, és doboz csak ott, ahol az alkalmazás vár rád (kék) vagy ahol valami kockázatos (sárga). A „sikeres frissítés" sáv kis szürke sorrá vált a név alatt. Ha nem nézed meg: a felület él, csak senki nem látta még böngészőben.
|
|
||||||
- **Nézd meg az új alkalmazás oldalt a demo-hp-n (egy telepített és egy nem telepített app).** Egy alkalmazás mostantól egy oldal négy füllel (Beállítások · Adatok és mentés · Útmutató · Napló), az Indítás/Újraindítás/Leállítás a „Működés" kártyán van, és a „Vissza" az alkalmazáshoz visz, nem a listához. Mondd meg, jó-e így az elrendezés. Ha nem nézed meg: a felület él, csak senki nem látta még böngészőben — a robot a DooPlexen nem tud kattintani.
|
**Morning list, 2026-10-21 (fix day).** No decision was taken by CC.
|
||||||
- **The N100 sheet (about 30 minutes):** `documentation/audits/n100-lived-in-2026-10-15/SHEET.md` — install Plex and claim it, make the admins in Prowlarr/Sonarr/Radarr, add your download sources in Prowlarr, add one show and one film. Then say „done". If not: the media apps wait, safe behind their gate.
|
- D26: the agent release with the swap fix. Also new: after a box death with a failed second install, the whole-box backup waits for your „Re-issue PBS credentials" press (R-1205).
|
||||||
- **Delete the test customer „Tester3” in the hub (browser only).** Its test box is already gone and switched off on purpose, so no alarm is coming. If not: a dead test customer stays in your list.
|
- Secrets: keep or remove the `REHEARSAL_2026_10_20_*` and `FIXES_2026_10_21_*` names in sec (the bind, pairing and setup codes are spent; the root password belongs to a deleted box).
|
||||||
- **Hub 0.153.0 (your rulings 13:20 + R-1059 „A"): no token can resend a claim code, create or edit a customer, make a passphrase or a link, or re-issue an off-site password.** Live since 13:39 (checked with CC's token: all six refused).
|
|
||||||
- **Remove the test server „plex-k6” from your Plex account** (plex.tv → Settings → Authorized Devices). The claim test passed and the server is gone from Tester 1. If not: a dead server stays listed in your Plex apps.
|
**The rest**
|
||||||
- System poster needs a refresh: every public name answers only over https (http → 301).
|
- Pull the failed disk Z6A07P2G out of DooPlex (R-934); it is still in. If not: its alarms return on 11 November. Since 2026-10-12.
|
||||||
- **Leftovers moved out of the shared checkout** to `/mnt/5_hdd/felhom.eu/leftovers/2026-10-14/` (nothing deleted): the Facebook `first-post/` folder is not in git — keep, commit or delete it. The rest is already in git. If not: it just stays there.
|
- Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). Since 2026-10-09.
|
||||||
- **The media tests on Tester 1 are finished** — results in `REPORT-night-2026-10-14.md`. Say „tear it down" when you want Tester 1 back to normal (R-1022; now three test drives and a throwaway Plex).
|
- Two leftovers on DooPlex: keep, commit or delete the Facebook `first-post/` folder in `/mnt/5_hdd/felhom.eu/leftovers/2026-10-14/`; commit or drop the `deny`-list edit in `~/git/homelab-manifests/.claude/settings.json`. If not: they stay. Since 2026-10-14.
|
||||||
- **Ask Google to index the new page:** Search Console → URL-ellenőrzés → `https://felhom.eu/sajat-felho-tarhely` → „Indexelés kérése". If not: the page may sit unseen, and the 11 November / 9 December checks (R-1035) read zero for a reason that is not the page.
|
- Before the first paying customer: Tandoor's permission (R-789); a lawyer for the licence list and the terms (R-802, R-813); SparkyFitness by e-mail (R-784); print the break-glass sheet and the signing keys (R-923, R-924).
|
||||||
- **N100 BIOS:** read „After Power Loss" (the HP's „Previous State" made it start after 8 of 8 cuts). **HP BIOS:** „Bypass F1 Prompt on Configuration Changes" → Enable.
|
- Later (yours, not urgent): R-862 (Tester 2's bootstrap, when it is back), R-504, R-1071, R-1073, R-1077, R-1042, R-132, R-908, R-779, R-904, R-232, R-433, R-933, R-213, R-832, R-388, R-882, R-883, R-814, R-1056, R-1164, R-930, R-920, R-902.
|
||||||
- **Pull the failed disk** Z6A07P2G (R-934; the twin Z6A070XZ is `/mnt/3_hdd`). If not: its alarms return on 11 November.
|
|
||||||
- **Hub master key** into the password manager, then `shred -u` the key file (security review 2026-10-12, in the history).
|
|
||||||
- **12 reused homelab logins** to rotate, and the repo-visibility decision (R-925, P1).
|
|
||||||
- **Before the first paying customer:** Tandoor permission (R-789); a lawyer for the licence list (R-802) and the terms (R-813); SparkyFitness (R-784 — its author replied on 2026-10-13); print the break-glass sheet (R-923).
|
|
||||||
- **Smaller:** pin the first Facebook post (R-917); Tester 2's bootstrap (R-862); look at the uncommitted `deny`-list edit in `homelab-manifests/.claude/settings.json`; the rest of „Your own tasks" (T6–T19) is in the history, 2026-10-10.
|
|
||||||
|
|||||||
@@ -85,10 +85,10 @@
|
|||||||
| <a id="a9"></a>[A9](capability-map-detail/A.md#a9) | One name per secret on every surface — „Beállító kód" (the setup code) and „Tulajdonosi jelmondat" (the owner passphrase) — and the re-enrol mail points at a page a rebuilt box shows | controller v0.211.0, hub v0.104.0 + v0.105.0 (R-295, R-323) | **IMPLEMENTED** | written BUILT: source-read; no customer has typed the passphrase | R-295; R-323 |
|
| <a id="a9"></a>[A9](capability-map-detail/A.md#a9) | One name per secret on every surface — „Beállító kód" (the setup code) and „Tulajdonosi jelmondat" (the owner passphrase) — and the re-enrol mail points at a page a rebuilt box shows | controller v0.211.0, hub v0.104.0 + v0.105.0 (R-295, R-323) | **IMPLEMENTED** | written BUILT: source-read; no customer has typed the passphrase | R-295; R-323 |
|
||||||
| <a id="a10"></a>[A10](capability-map-detail/A.md#a10) | The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE** | demo-felhom, real endpoints | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04); `audits/RECON-offsite-dr-chain-2026-08-04.md` (2026-08-04) |
|
| <a id="a10"></a>[A10](capability-map-detail/A.md#a10) | The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE** | demo-felhom, real endpoints | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04); `audits/RECON-offsite-dr-chain-2026-08-04.md` (2026-08-04) |
|
||||||
| <a id="a11"></a>[A11](capability-map-detail/A.md#a11) | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** | demo/drill fleet only | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
|
| <a id="a11"></a>[A11](capability-map-detail/A.md#a11) | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** | demo/drill fleet only | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
|
||||||
| <a id="a12"></a>[A12](capability-map-detail/A.md#a12) | A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PROVEN-LIVE** | the success path; the failure path is not claimed (R-379, R-380) | `audits/DRILL-r201-night-run-2026-08-04.md` (2026-08-04); `audits/DRILL-r356-hot-only-restore-2026-08-22/` (2026-08-22); `audits/DRILL-r356b-driveless-db-restore-2026-08-22/` (2026-08-22) |
|
| <a id="a12"></a>[A12](capability-map-detail/A.md#a12) | A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PROVEN-LIVE** | 2026-10-21 (fix brief, hub 0.163.0): PASSES — the rebuilt Tester4 box (public ISO) got the household's sealed package from the hub, asked for the recovery code, opened the history; Nextcloud came back with 3 031 of 3 033 manifest files byte-identical (the 2 never reached the old box) and the next off-site runs appended (63 → 64). Earlier the same day it FAILED (R-1176, now closed); the failure paths are not claimed (R-379, R-380) | `audits/fixes-2026-10-21/P1-live/` (2026-10-21); `audits/rehearsal-2026-10-20/B7-died/README.md` (2026-10-21); `audits/DRILL-r201-night-run-2026-08-04.md` (2026-08-04); `audits/DRILL-r356-hot-only-restore-2026-08-22/` (2026-08-22); `audits/DRILL-r356b-driveless-db-restore-2026-08-22/` (2026-08-22) |
|
||||||
| <a id="a13"></a>[A13](capability-map-detail/A.md#a13) | The customer's own UNAIDED recovery journey, end to end | controller v0.206.0, hub v0.98.0, agent v0.127.0 | **PROVEN-LIVE** | fifth walk, scoped — read what it does not claim | `tests/walk5-r201-2026-08-07/journal.md` (2026-08-07); `audits/CAMPAIGN-11-recovery-journey-2026-08-05.md` (2026-08-05) |
|
| <a id="a13"></a>[A13](capability-map-detail/A.md#a13) | The customer's own UNAIDED recovery journey, end to end | controller v0.206.0, hub v0.98.0, agent v0.127.0 | **PROVEN-LIVE** | 2026-10-21 (fix brief, hub 0.163.0): PASSES with ONE operator act by design (D19: delete the dead host) — install, bind, claim, the recovery offer, a wrong code refused, the right code, „Teljes visszaállítás", all through the household's own pages; the new-link page now tells a household whose old box is still linked to write to Felhom (R-1203). Earlier the same day it stopped at the rebuilt dashboard (R-1176) | `audits/fixes-2026-10-21/P1-live/` (2026-10-21); `audits/rehearsal-2026-10-20/B7-died/README.md` (2026-10-21); `tests/walk5-r201-2026-08-07/journal.md` (2026-08-07); `audits/CAMPAIGN-11-recovery-journey-2026-08-05.md` (2026-08-05) |
|
||||||
| <a id="a14"></a>[A14](capability-map-detail/A.md#a14) | A random night of household actions while random things go wrong — twelve rounds, unattended | controller v0.245.0, agent v0.131.0, hub v0.116.0, golden 0.245.0, ISO 1.28.0 | **PROVEN-LIVE** | nested box, twelve chaos rounds | `audits/DRILL-chaos-night-2026-09-17.md` (2026-09-17) |
|
| <a id="a14"></a>[A14](capability-map-detail/A.md#a14) | A random night of household actions while random things go wrong — twelve rounds, unattended | controller v0.245.0, agent v0.131.0, hub v0.116.0, golden 0.245.0, ISO 1.28.0 | **PROVEN-LIVE** | nested box, twelve chaos rounds | `audits/DRILL-chaos-night-2026-09-17.md` (2026-09-17) |
|
||||||
| <a id="a15"></a>[A15](capability-map-detail/A.md#a15) | A stranger's FIRST HOUR on a fresh install — download → install → claim → two apps → use → backup → remove → restore, plus a power cut and a mistyped code | controller v0.242.0, agent v0.130.0, hub v0.112.0, golden 0.242.0, installer ISO 1.26.1 | **PARTIAL** | every mechanism passed; the unaided journey failed before the first app | `audits/DRILL-fresh-install-0242-2026-09-14.md` (2026-09-14); `audits/DRILL-new-household-2026-09-30.md` (2026-09-30) |
|
| <a id="a15"></a>[A15](capability-map-detail/A.md#a15) | A stranger's FIRST HOUR on a fresh install — download → install → claim → two apps → use → backup → remove → restore, plus a power cut and a mistyped code | controller v0.336.0, agent v0.161.0, hub v0.162.0, golden 0.331.0, installer 1.35.0, ISO 1.29.0 | **PARTIAL** | 2026-10-20: download → install → bind → claim → 8 apps → use → backup → remove → every restore path → power cut → mistyped codes all worked; the 8 GB box FROZE twice in the first afternoon and only a plug pull brought it back (R-1192), and the owner passphrase needed the operator | `audits/rehearsal-2026-10-20/journal.md` (2026-10-20); `audits/DRILL-fresh-install-0242-2026-09-14.md` (2026-09-14); `audits/DRILL-new-household-2026-09-30.md` (2026-09-30) |
|
||||||
| <a id="a16"></a>[A16](capability-map-detail/A.md#a16) | Off-site app-data capture covers the paths an app declares MANDATORY — on BOTH drive layouts | controller v0.197.0 | **PROVEN-LIVE** | both drive layouts | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04) |
|
| <a id="a16"></a>[A16](capability-map-detail/A.md#a16) | Off-site app-data capture covers the paths an app declares MANDATORY — on BOTH drive layouts | controller v0.197.0 | **PROVEN-LIVE** | both drive layouts | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04) |
|
||||||
| <a id="a17"></a>[A17](capability-map-detail/A.md#a17) | Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE** | demo-hp | `audits/SPIKE-r82-phase0-2026-07-26.md` (2026-07-26) |
|
| <a id="a17"></a>[A17](capability-map-detail/A.md#a17) | Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE** | demo-hp | `audits/SPIKE-r82-phase0-2026-07-26.md` (2026-07-26) |
|
||||||
| <a id="a18"></a>[A18](capability-map-detail/A.md#a18) | Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard | agent v0.104.0 → v0.121.0, hub v0.77.0 → v0.91.0 | **PROVEN-LIVE** | — | *(evidence not on file — not re-verifiable)*: the row cites per-repo CHANGELOGs and the spec `backlog/SPEC-r85-phase4-5-2026-07-26.md` — a spec, not a run record |
|
| <a id="a18"></a>[A18](capability-map-detail/A.md#a18) | Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard | agent v0.104.0 → v0.121.0, hub v0.77.0 → v0.91.0 | **PROVEN-LIVE** | — | *(evidence not on file — not re-verifiable)*: the row cites per-repo CHANGELOGs and the spec `backlog/SPEC-r85-phase4-5-2026-07-26.md` — a spec, not a run record |
|
||||||
|
|||||||
@@ -271,6 +271,13 @@ app. Controller down → the gated app answers an error, never the app. Measured
|
|||||||
tunnel. Since controller v0.286.0 it sits ALONE on the `felhom-tunnel` network at the fixed address `172.16.253.2`
|
tunnel. Since controller v0.286.0 it sits ALONE on the `felhom-tunnel` network at the fixed address `172.16.253.2`
|
||||||
(traefik at `.3`), so traefik can believe forwarded headers from it and from nothing else (§5). Geo-restriction WAF
|
(traefik at `.3`), so traefik can believe forwarded headers from it and from nothing else (§5). Geo-restriction WAF
|
||||||
is **hub-enforced** (the hub holds the CF API token; the controller only reports geo desired-state).
|
is **hub-enforced** (the hub holds the CF API token; the controller only reports geo desired-state).
|
||||||
|
- **[DESIGN] The website's content comes from Gitea IN-CLUSTER** (2026-10-19). The `felhom-webpage` pod pulls
|
||||||
|
`website/` (from `main`) and `scripts/` (from the `installer-v…` tag, R-110) with four git-syncs that read
|
||||||
|
`http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git` — plain HTTP, safe only because it never
|
||||||
|
leaves the cluster (the hub's own Gitea client does the same). No internet hop: the pod can start, and keep its copy
|
||||||
|
fresh, during an internet outage. The two sidecars run with `--max-failures=-1`, so an unreachable Gitea is retried,
|
||||||
|
not a restart. **This ties the website to the DooPlex cluster:** moved elsewhere, it must read the public HTTPS
|
||||||
|
address again. `scripts/hostinstall_gates.py` gate 6 keeps the four `--repo` values identical.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -539,8 +539,28 @@ state (both demo boxes) the panel says „Ezt a jelszót te állítottad be a F
|
|||||||
the press. The box uses FileBrowser's own CLI (`set -u admin,<pw>`) with the server STOPPED (the CLI opens the
|
the press. The box uses FileBrowser's own CLI (`set -u admin,<pw>`) with the server STOPPED (the CLI opens the
|
||||||
database with no lock timeout), verifies new=200 and admin/admin refused, stores the value encrypted and records
|
database with no lock timeout), verifies new=200 and admin/admin refused, stores the value encrypted and records
|
||||||
`generated`; a failed verify leaves `operator`. Not the `FILEBROWSER_ADMIN_PASSWORD` env: it re-applies on every
|
`generated`; a failed verify leaves `operator`. Not the `FILEBROWSER_ADMIN_PASSWORD` env: it re-applies on every
|
||||||
start. Spike: `audits/logins-folders-2026-10-16/P1b`. FileBrowser still asks for its own login the first time; one
|
start. Spike: `audits/logins-folders-2026-10-16/P1b`. Since v0.332.0 the folder links open FileBrowser without its own
|
||||||
sign-in for both is R-1100.
|
login (the next paragraph); `admin`'s password is the fallback and the operator's way in.
|
||||||
|
|
||||||
|
**[DESIGN] One sign-in for the file manager (v0.332.0, R-1100, operator decision D16 = yes 2026-10-19; spike
|
||||||
|
`audits/proxyauth-spike-2026-10-18/DESIGN-NOTE.md`).** Every dashboard link to a FileBrowser folder goes to
|
||||||
|
`/__files/open?to=<path>` on the dashboard host, which — for a signed-in, not-cross-site request — mints a pass (60 s,
|
||||||
|
one use, bound to `files.<domain>` and to the session) and sends the browser to `files.<domain>/_felhom/sso`. traefik
|
||||||
|
asks the controller before every FileBrowser request (forwardAuth `felhom-files-sso` → `/__felhom_gate/files`, the setup
|
||||||
|
gate's handshake): the pass becomes the host-only `felhom_files` cookie (a value derived from the session's fingerprint
|
||||||
|
for that host — the R-976 shape); a valid cookie → 200 with the box's **secret header** naming FileBrowser user
|
||||||
|
`felhom-household`; no cookie → 200 without it (FileBrowser's own password); a stale cookie (sign-out, password change)
|
||||||
|
→ both cookies cleared, back to the dashboard. **Why a secret header name:** FileBrowser trusts the header blindly and
|
||||||
|
every app container reaches `filebrowser:80` directly — measured: a forged fixed-name header from an app logged in. The
|
||||||
|
name (`X-Felhom-User-<32 hex>`, per box, 0600 in the data dir) is written only into FileBrowser's config and traefik's
|
||||||
|
route (`authResponseHeaders`, which strips a client's copy); never a log, a page or a report. **Why not a parent-domain
|
||||||
|
cookie:** every app subdomain is third-party code and would receive the dashboard session. **What stays:** `admin` is a
|
||||||
|
password user (R-513's reveal and reset, the R-1126 gate unchanged); the header user is a separate FileBrowser user whose
|
||||||
|
rights come from the box's userDefaults (change, add, delete, share, download — not admin, not API); FileBrowser's logout
|
||||||
|
goes to the dashboard. Residual: FileBrowser's own 2-hour token for the header user is cleared with the files cookie,
|
||||||
|
but a browser that lost only the files cookie keeps it until it expires. Measured live on all three test boxes
|
||||||
|
(`audits/day-2026-10-19/P2/`, `P4/`): opened at the folder with no login; forged headers refused through traefik and
|
||||||
|
from an app container; a sign-out asks again.
|
||||||
|
|
||||||
**[DESIGN] A stored FileBrowser password is checked before it is offered (v0.321.0, R-1101).** In `generated`, the
|
**[DESIGN] A stored FileBrowser password is checked before it is offered (v0.321.0, R-1101).** In `generated`, the
|
||||||
page asks FileBrowser whether the stored value still logs in; only FileBrowser's own 401/403 counts as refused (a
|
page asks FileBrowser whether the stored value still logs in; only FileBrowser's own 401/403 counts as refused (a
|
||||||
@@ -562,8 +582,9 @@ role, else the first declared folder, else (a media app) the first present drive
|
|||||||
`movies`, the players both, a download client its `media_library.folder`) with NO backup sentence — the bind's class
|
`movies`, the players both, a download client its `media_library.folder`) with NO backup sentence — the bind's class
|
||||||
there means „not this app's data", which is not the class-driven „temporary, not backed up". **A folder whose drive
|
there means „not this app's data", which is not the class-driven „temporary, not backed up". **A folder whose drive
|
||||||
is not bound into FileBrowser now is text, never a link** (`drivePresent` = the same gates the FileBrowser sync
|
is not bound into FileBrowser now is text, never a link** (`drivePresent` = the same gates the FileBrowser sync
|
||||||
applies). The button never promises one click: FileBrowser's router brings a cold visitor back to the folder after its
|
applies). Since v0.332.0 the button opens the folder signed in (one sign-in, above); on a box without it, FileBrowser's
|
||||||
login (measured in the headless browser, `audits/logins-folders-2026-10-16/browser/`).
|
router brings a cold visitor back to the folder after its login (measured in the headless browser,
|
||||||
|
`audits/logins-folders-2026-10-16/browser/`).
|
||||||
|
|
||||||
**[DESIGN] „Cím és kulcsok" is rows, not the install form.** The settings tab does not re-render the
|
**[DESIGN] „Cím és kulcsok" is rows, not the install form.** The settings tab does not re-render the
|
||||||
install wizard's fields: the address is the „Cím" row, the storage is a „Tárhely: <név>" row, every
|
install wizard's fields: the address is the „Cím" row, the storage is a „Tárhely: <név>" row, every
|
||||||
|
|||||||
@@ -232,7 +232,11 @@ The controller (in its LXC) reaches the agent (on the host) over the local bridg
|
|||||||
- `POST /rollback` — roll *this* guest back to a named snapshot (post-deploy failure recovery).
|
- `POST /rollback` — roll *this* guest back to a named snapshot (post-deploy failure recovery).
|
||||||
- `POST /backup` — request a backup-now of *this* guest (enqueued; non-destructive).
|
- `POST /backup` — request a backup-now of *this* guest (enqueued; non-destructive).
|
||||||
- `GET /backup/due` — whether a policy-scheduled backup is due for *this* guest, so the controller can quiesce then call `POST /backup` (the app-consistent path, §8).
|
- `GET /backup/due` — whether a policy-scheduled backup is due for *this* guest, so the controller can quiesce then call `POST /backup` (the app-consistent path, §8).
|
||||||
- `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI.
|
- `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI. **`host_busy`
|
||||||
|
(R-921, agent Unreleased `56aabc7`):** `GET /backup/status` also says whether the host-wide heavy-operation gate is
|
||||||
|
held (`{busy, what}` — another guest's backup, the restore test, an OS step, the stale-lock sweep, a power-off), the
|
||||||
|
state in which `POST /backup` answers 409; absent when the gate is not wired. The controller (≥ 0.335.0) reads it
|
||||||
|
before it stops any app. A read, never a reservation.
|
||||||
- **Crash-boot fact (R-856, agent v0.149.0):** `GET /host/crash-guard` — the host crash guard's last-boot record
|
- **Crash-boot fact (R-856, agent v0.149.0):** `GET /host/crash-guard` — the host crash guard's last-boot record
|
||||||
(`present`, `last_boot_at`, `last_boot_unclean`, `tripped`) read from `/var/lib/felhom-crash-guard/state.json`; a missing
|
(`present`, `last_boot_at`, `last_boot_unclean`, `tripped`) read from `/var/lib/felhom-crash-guard/state.json`; a missing
|
||||||
or garbled file answers 200 `present:false`, an older agent 404 — both read as a normal boot. The controller waits
|
or garbled file answers 200 `present:false`, an older agent 404 — both read as a normal boot. The controller waits
|
||||||
@@ -399,7 +403,10 @@ per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a ba
|
|||||||
> space on …" — so the hub raises `restore_test_failed`; it is never a pass and never dropped.
|
> space on …" — so the hub raises `restore_test_failed`; it is never a pass and never dropped.
|
||||||
> - **Leftovers on a timer.** A failed scratch teardown and the stale-lock sweep (R-673) run every 10
|
> - **Leftovers on a timer.** A failed scratch teardown and the stale-lock sweep (R-673) run every 10
|
||||||
> minutes, not only at agent start; the sweep holds the one-heavy-operation gate; after 3 failed
|
> minutes, not only at agent start; the sweep holds the one-heavy-operation gate; after 3 failed
|
||||||
> teardown tries the operator is told.
|
> teardown tries the operator is told. **A half-removed `vzdump` snapshot (R-1010, agent Unreleased
|
||||||
|
> `56aabc7`):** when the sweep's delete fails ONLY with „Failed to find logical volume …_vzdump" (one disk's
|
||||||
|
> snapshot already gone), it retries once with Proxmox's own `force=1` on the same API call (no sudoers line, no
|
||||||
|
> root command); any other error is returned unchanged, and a failed forced delete is never forced again.
|
||||||
> - **A thin pool ≥ 90 % requests an immediate report**; the hub judges a thin pool on the worse of data
|
> - **A thin pool ≥ 90 % requests an immediate report**; the hub judges a thin pool on the worse of data
|
||||||
> and metadata, critical at 90 %, one alarm per pool per 6 hours (`08` §6.2).
|
> and metadata, critical at 90 %, one alarm per pool per 6 hours (`08` §6.2).
|
||||||
>
|
>
|
||||||
@@ -474,6 +481,18 @@ box. The escrow is how `K` comes back **without** Felhom ever being able to read
|
|||||||
(hub v0.8.0, `PUT /api/v1/hosts/{host_id}/escrow`). Validated end-to-end on a throwaway in
|
(hub v0.8.0, `PUT /api/v1/hosts/{host_id}/escrow`). Validated end-to-end on a throwaway in
|
||||||
`documentation/tests/slice7-escrow-spike-findings.md`. Restore-mode *serving/consumption* is slice 10.
|
`documentation/tests/slice7-escrow-spike-findings.md`. Restore-mode *serving/consumption* is slice 10.
|
||||||
|
|
||||||
|
#### The rebuilt box gets the household's package back — the CARRY [DESIGN, operator ruling 2026-10-21, hub v0.163.0, R-1176]
|
||||||
|
|
||||||
|
After the operator deletes a dead box's host (escrow acknowledged), the package is demoted to retained custody under
|
||||||
|
the OLD host id. The hub now **re-attaches** it: when the customer has exactly one live host, no live host holds a
|
||||||
|
package, and its newest escrow-acknowledged deletion left a retained row with an identity blob, that row becomes the new
|
||||||
|
host's current escrow row — moved, in one transaction, on the agent's first report (and again on the controller's
|
||||||
|
report, idempotently). **Zero-knowledge is unchanged:** the hub moves the same ciphertext from one row to another; it
|
||||||
|
never opens, re-seals or logs it; the audit (`escrow_carried`) carries host ids and the non-reversible hash. Two live
|
||||||
|
hosts → no carry, `escrow_carry_refused` to the operator. The box then sees „identity present", its mint guard (R-241)
|
||||||
|
holds, and it offers the recovery screen. Proven live 2026-10-21 (`audits/fixes-2026-10-21/P1-live/`: the right code
|
||||||
|
opened the household's history, Nextcloud's 3 031 files came back byte-identical, the next off-site run appended).
|
||||||
|
|
||||||
#### The separation principle (the rule that governs every posture)
|
#### The separation principle (the rule that governs every posture)
|
||||||
Reading customer data needs **BOTH** the encrypted chunks **AND** a usable key. **Zero-knowledge
|
Reading customer data needs **BOTH** the encrypted chunks **AND** a usable key. **Zero-knowledge
|
||||||
holds for exactly as long as Felhom never holds both at once.** Every posture below is just a
|
holds for exactly as long as Felhom never holds both at once.** Every posture below is just a
|
||||||
@@ -544,6 +563,19 @@ nicety.
|
|||||||
**loses Felhom's backup guarantees** — no PBS verify / monitoring on storage we can't reach. An
|
**loses Felhom's backup guarantees** — no PBS verify / monitoring on storage we can't reach. An
|
||||||
honest signup-time tradeoff, not a hidden one.
|
honest signup-time tradeoff, not a hidden one.
|
||||||
|
|
||||||
|
**When the new package reaches the hub (R-978; agent 0.160.0, 2026-10-19):** a ceremony started again no
|
||||||
|
longer replaces the hub's package when it starts. The package travels with the pending ceremony and is uploaded when its
|
||||||
|
code is claimed, before the code is handed out (an upload failure is a 502 and the code stays claimable). An abandoned
|
||||||
|
ceremony — TTL, a new start, a restart — leaves the first code opening the hub's package. **[DESIGN] Only after the
|
||||||
|
household typed it back (R-1151; agent 0.160.0 + controller 0.332.0, 2026-10-19):** a controller ≥ 0.332.0 claims with
|
||||||
|
`{"defer_upload": true}`; the agent hands out the code with two word positions it drew, keeps the package (and only those
|
||||||
|
two words, never the code) for 30 minutes, and uploads it only when `POST /escrow/ceremony/confirm` brings the two words
|
||||||
|
back — so a code opened and not written down never replaces the package. Wrong words 422 (still waiting), window gone 410,
|
||||||
|
nothing waiting 404, hub failure 502 (still waiting). A controller that has not updated claims with `{}` and gets the
|
||||||
|
R-978 order. Proven on demo-hp 2026-10-19: ceremony 1 typed back → code 1 opens the hub's package, code 0 is a kept
|
||||||
|
earlier one (422); ceremony 2 shown and abandoned → code 1 still opens it, code 2 opens nothing
|
||||||
|
(`audits/day-2026-10-19/P1/escrow-*.txt`).
|
||||||
|
|
||||||
#### Custody across generations — what the hub retains when a blob is superseded (hub v0.60.0; **contract corrected hub v0.93.0, R-198**)
|
#### Custody across generations — what the hub retains when a blob is superseded (hub v0.60.0; **contract corrected hub v0.93.0, R-198**)
|
||||||
|
|
||||||
A ceremony that seals a *different* passphrase does not destroy the blob it replaces: the hub copies
|
A ceremony that seals a *different* passphrase does not destroy the blob it replaces: the hub copies
|
||||||
|
|||||||
@@ -90,7 +90,11 @@ Evolves the existing staleness checker (60s **cadence**, a **configured** thresh
|
|||||||
2026-09-17, **45 m** since operator ruling A on R-549; OK under it, down at 2× = 90 m; today: controller-report recency → `node_stale`/`down`/`recovered`):
|
2026-09-17, **45 m** since operator ruling A on R-549; OK under it, down at 2× = 90 m; today: controller-report recency → `node_stale`/`down`/`recovered`):
|
||||||
|
|
||||||
- **Primary = host-report recency → `host_stale` / `host_down`.** The agent heartbeat is the box's
|
- **Primary = host-report recency → `host_stale` / `host_down`.** The agent heartbeat is the box's
|
||||||
liveness signal; a silent agent = the box is gone (the critical alert).
|
liveness signal; a silent agent = the box is gone (the critical alert). **A hub restart does not re-mail a known
|
||||||
|
silence (R-1001, hub 0.162.0 — built 2026-10-19, not yet deployed):** a host whose newest `host_stale`/`host_down`
|
||||||
|
event is newer than its last report is seeded with that state at start, silently; `restore_test_stale` reads its edge
|
||||||
|
from the DB on the first sweep (a signal for the tier inside the read window and newer than its last proof). A failed
|
||||||
|
lookup lets the alarm through.
|
||||||
- **Guest up/down comes from the host report's per-guest status** — authoritative, every poll, faster
|
- **Guest up/down comes from the host report's per-guest status** — authoritative, every poll, faster
|
||||||
than waiting for a guest report to go stale.
|
than waiting for a guest report to go stale.
|
||||||
- **Guest-report recency = secondary** app-level signal.
|
- **Guest-report recency = secondary** app-level signal.
|
||||||
@@ -134,7 +138,11 @@ walk is green; the golden is baked weekly and vouched by the operator; a new box
|
|||||||
itself to the floor (R-1052: not yet proven end to end). Setting the global floor today needs the operator page with the
|
itself to the floor (R-1052: not yet proven end to end). Setting the global floor today needs the operator page with the
|
||||||
operator's password — there is no scoped route for CC (R-1051). **An agent can be vouched mid-week; the golden stays
|
operator's password — there is no scoped route for CC (R-1051). **An agent can be vouched mid-week; the golden stays
|
||||||
weekly** (hub v0.158.0): the R-120 golden guard judges only a CHANGED golden selection, so a Day-0 save that keeps the
|
weekly** (hub v0.158.0): the R-120 golden guard judges only a CHANGED golden selection, so a Day-0 save that keeps the
|
||||||
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet.
|
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet. **The agent list
|
||||||
|
offers real agent packages only (R-1011, hub 0.162.0, not yet deployed):** the sha lookup asks for exactly the
|
||||||
|
`felhom-agent` file (the golden: `golden.tar.zst`) and never falls back to another file, so a step bundle
|
||||||
|
(`0.156.1-step1`, only the config bundle) is left out of the list, and a vouch of one is refused
|
||||||
|
(`artifact_file_missing`).
|
||||||
|
|
||||||
**The controller floor and its agent requirement (hub v0.112.0, R-472).** The managed controller floor
|
**The controller floor and its agent requirement (hub v0.112.0, R-472).** The managed controller floor
|
||||||
rides the report ACK. `store.ResolveManagedFloor` decides per box whether to serve it, from three
|
rides the report ACK. `store.ResolveManagedFloor` decides per box whether to serve it, from three
|
||||||
@@ -222,6 +230,51 @@ Today: `GET /config/{id}` with an `X-Retrieval-Password` (Hungarian passphrase)
|
|||||||
The existing `configgen` deep-merge + Hungarian-passphrase machinery is the base; it grows the
|
The existing `configgen` deep-merge + Hungarian-passphrase machinery is the base; it grows the
|
||||||
agent-first + key-pinning + restore-mode steps.
|
agent-first + key-pinning + restore-mode steps.
|
||||||
|
|
||||||
|
**A reinstall over an existing host record is made loud, never acted on [DESIGN, 2026-10-19 — R-1154, brief
|
||||||
|
2026-10-19; hub v0.159.0].** `POST /host-enroll` is the installer's step 4, run on every pass (a `--resume` too), with
|
||||||
|
the customer's retrieval passphrase; an existing host record is REUSED (same id and key — the reinstalled box takes
|
||||||
|
it over). The clean-slate re-issues (a new claim code mailed to the household, the off-site provider password reset)
|
||||||
|
run only on the MINT path, i.e. after the operator deleted the old host — **the operator's delete is their gate,
|
||||||
|
and stays so.** A reinstall that kept the old record used to skip them in silence. Now the reuse path classifies the
|
||||||
|
old box and re-issues NOTHING: **gone** (no controller wait connection open now, and either `deliberate_off_at`
|
||||||
|
inside its hold or BOTH the host report and the controller report older than `alerting.stale_threshold` or absent —
|
||||||
|
the `hostStatus` threshold and comparison) → the operator-only WARNING `host_reinstall_over_old_host`, mailed to the
|
||||||
|
operator: delete the old host (Hosts page) and run the installer again, or re-issue the claim code and the off-site
|
||||||
|
login from the customer page; **live** → the operator-only `host_reenroll_while_live` — since hub v0.160.0 (R-1156) a
|
||||||
|
WARNING mailed with the same instruction (a reinstall within 45 minutes of the old box's last report reads as live; an
|
||||||
|
installer `--resume` or a repair re-run mails too, and the text says those need nothing).
|
||||||
|
**Why not re-issue automatically** (a security review of the first draft, the same day): the passphrase is a
|
||||||
|
household-held secret typed into the installer, and both re-issues are operator SESSION-ONLY actions — not even an
|
||||||
|
operator token may run them (§17.5; `09` §3 decision 252, R-1055/R-1059: an off-site re-issue „stops its backup for a
|
||||||
|
while"). Re-issuing on the reuse path would give those powers to anyone holding the passphrase whenever the box is
|
||||||
|
switched off or silent. The cost of the loud shape is two operator acts per reinstall that kept its old record. Both
|
||||||
|
event types are hub-origin only. `hub/internal/api/reenroll.go`, pinned by `hub/internal/api/r1154_reenroll_test.go`.
|
||||||
|
|
||||||
|
**What the household's retrieval passphrase can get [DESIGN, 2026-10-19 — R-1159, operator decision D19 „box key
|
||||||
|
first, then lock"; hub v0.161.0 + controller v0.334.0].** The passphrase is the household's install secret: a NEW box
|
||||||
|
needs it (the installer's preflight, download login and enroll; the guest's first boot, `bootstrap.json` → `GET
|
||||||
|
/config`, which is how the box gets its own hub key). Measured the same day, it also gave anyone who held it the box's
|
||||||
|
keys for good, and through them every off-site key action (`audits/day-2026-10-19b/P2/`). **The rule:** the passphrase
|
||||||
|
gives keys only until the box uses its own — `GET /config`, `GET /recovery` and the reuse path of `POST /host-enroll`
|
||||||
|
answer **409 with no key** (and the operator gets `passphrase_keys_refused`) once a host record exists AND a controller
|
||||||
|
report arrived since it was made AND that controller pulls its config with its own key (≥ 0.334.0, `GET /box-config`,
|
||||||
|
`boxConfigMinController`). Open for a first install, the install in progress, a reinstall after the operator deleted
|
||||||
|
the old host, and a box on an older controller (which still refreshes with the passphrase — the lock arms box by box
|
||||||
|
as the floor moves). A reinstall over a live box therefore needs the operator's delete first, which R-1154 already
|
||||||
|
made the rule. **Why this shape and not „before a host exists":** the install itself uses the passphrase AFTER enroll
|
||||||
|
(`felhom-host-install.sh` step_agent_install → `/config`; the agent bakes it into `bootstrap.json`; the guest pulls with
|
||||||
|
it), and so did every live box's config refresh — measured from the code before building, and put to the operator.
|
||||||
|
`hub/internal/api/passphrase_scope.go`; pinned by `r1159_passphrase_scope_test.go`. Proven live on demo-hp
|
||||||
|
(2026-10-19 14:30 UTC): before 0.334.0 → 200; after its first 0.334.0 report → 409, no key, the event raised; the box's
|
||||||
|
own key → 200.
|
||||||
|
|
||||||
|
**[FACT, 2026-10-19 — R-1158, hub v0.160.0] What the MINT path's off-site re-issue still does.** It resets the Storage
|
||||||
|
Box sub-account password and stores the new one — a password only the hub itself uses (`offsitekeys.TargetFor`, the key
|
||||||
|
registrar's login). No box receives it since v0.127.0 (`consume-password` answers 410; `ConsumeOneTimeSecret` has no
|
||||||
|
production caller); the fresh box registers its own key with the controller API key. So it changes nothing a box uses.
|
||||||
|
It is kept because it hands the hub a fresh, working login at each clean reinstall, which repairs a broken stored one;
|
||||||
|
the `offsite_reissued` record now says the password is the hub's and the backups do not change.
|
||||||
|
|
||||||
## 9. DR model
|
## 9. DR model
|
||||||
|
|
||||||
The headline: the **old heavy infra-backup push retires** — not because the hub authors everything
|
The headline: the **old heavy infra-backup push retires** — not because the hub authors everything
|
||||||
@@ -522,6 +575,14 @@ are deleted at start-up and at each new day (`TestAccessLog_Keeps90DaysThenDelet
|
|||||||
ingress log: ≈ 1,900 hub requests a day at ≈ 110 bytes a line → ≈ 0.2 MB a day, ≈ 19 MB for 90 days. A logging
|
ingress log: ≈ 1,900 hub requests a day at ≈ 110 bytes a line → ≈ 0.2 MB a day, ≈ 19 MB for 90 days. A logging
|
||||||
failure never fails or delays a request: lines go through a non-blocking buffer to one writer (`internal/accesslog`).
|
failure never fails or delays a request: lines go through a non-blocking buffer to one writer (`internal/accesslog`).
|
||||||
|
|
||||||
|
### 16.5 Daily jobs and the clock change [DESIGN — hub v0.163.0, operator ruling D24 2026-10-21, R-1183]
|
||||||
|
|
||||||
|
Every daily hub job (`scheduleDaily`: db-snapshot 02:00, prune, deadline-check 05:00, offsite-key-audit 07:10) runs at
|
||||||
|
its Budapest wall-clock time **once per calendar day**: the next run is the calendar day after (`time.Date`, day+1),
|
||||||
|
never „now + 24 h", and the slot that ran last never runs again. On a 25-hour day (the last Sunday of October) the old
|
||||||
|
arithmetic ran every job after 02:00 twice; on a 23-hour day it ran them an hour late. Pinned by `TestR1183_*` for
|
||||||
|
2026-10-25 and 2027-03-28.
|
||||||
|
|
||||||
## 17. Operator access tokens [DESIGN — hub v0.152.0, operator ruling 2026-10-15 11:38 (`09` §3 decision 251)]
|
## 17. Operator access tokens [DESIGN — hub v0.152.0, operator ruling 2026-10-15 11:38 (`09` §3 decision 251)]
|
||||||
|
|
||||||
**The problem.** Scripts — Claude Code's floor step first — drove the hub with the operator's own password (HTTP Basic,
|
**The problem.** Scripts — Claude Code's floor step first — drove the hub with the operator's own password (HTTP Basic,
|
||||||
|
|||||||
File diff suppressed because one or more lines are too long
@@ -724,7 +724,11 @@ Evidence: `audits/os-host-lane-2026-10-04/` (parts A–G).
|
|||||||
skips the host step.
|
skips the host step.
|
||||||
- **The host health rule** (`HostHealthVerdict`, pinned in `internal/osupdate`): `felhom-agent`, `pveproxy`,
|
- **The host health rule** (`HostHealthVerdict`, pinned in `internal/osupdate`): `felhom-agent`, `pveproxy`,
|
||||||
`pvedaemon`, `pvestatd` and `pve-cluster` are `active`; the customer guest runs; the guest health rule (§8.1)
|
`pvedaemon`, `pvestatd` and `pve-cluster` are `active`; the customer guest runs; the guest health rule (§8.1)
|
||||||
passes; the tunnel is `running` — an `unknown` tunnel does not fail it, `not_running` does. Same 5-minute wait.
|
passes; the tunnel is `running` — `not_running` and `unknown` fail it (CORRECTED 2026-10-19: this line said
|
||||||
|
„an `unknown` tunnel does not fail it"; the code always failed it, which is what Tester4 hit, R-1070). **Since agent
|
||||||
|
Unreleased `56aabc7` (R-1070):** a guest with NO cloudflared container reads `absent` (Docker 29's lower-case
|
||||||
|
„no such object" used to fall through to `unknown`), and `absent` passes when the reading from before the step also
|
||||||
|
listed no cloudflared container (Docker answered) — a box with no tunnel set up. Same 5-minute wait.
|
||||||
- **Separate approved sets:** host and guest releases are separate (`os-host-…`, `os-guest-…`), each by the same rule
|
- **Separate approved sets:** host and guest releases are separate (`os-host-…`, `os-guest-…`), each by the same rule
|
||||||
(24 h, 1 night of THAT layer, every ring-0 box). Ring 1 receives `host_release` beside `release`.
|
(24 h, 1 night of THAT layer, every ring-0 box). Ring 1 receives `host_release` beside `release`.
|
||||||
- **Reboot needed:** reported when PID 1 or `lxc-start` maps a replaced file, with the date of the first scanned
|
- **Reboot needed:** reported when PID 1 or `lxc-start` maps a replaced file, with the date of the first scanned
|
||||||
|
|||||||
File diff suppressed because one or more lines are too long
@@ -16,6 +16,8 @@
|
|||||||
> renamed here the same day, with the operator's three Claude Design fixes folded in (marked *fix
|
> renamed here the same day, with the operator's three Claude Design fixes folded in (marked *fix
|
||||||
> (a)/(b)/(c)* below) and one measured correction: felhom.eu is served from DooPlex with
|
> (a)/(b)/(c)* below) and one measured correction: felhom.eu is served from DooPlex with
|
||||||
> **Cloudflare DNS only**, not through Cloudflare.
|
> **Cloudflare DNS only**, not through Cloudflare.
|
||||||
|
> 2026-10-21: poster regenerated in Claude Design (one-disk data area, https-only); PDF print version added next to it
|
||||||
|
> (`felhom-system-poster.pdf`).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -87,8 +89,8 @@ Title: **"Where a household's data is copied"**
|
|||||||
| **Tier 1 — recovery unit** | each app: its settings, database dump, volumes | data drive 1, on the box | nightly at **02:30** (W) | 1 point per app | not encrypted (it never leaves the home) |
|
| **Tier 1 — recovery unit** | each app: its settings, database dump, volumes | data drive 1, on the box | nightly at **02:30** (W) | 1 point per app | not encrypted (it never leaves the home) |
|
||||||
| **Tier 2 — cross-drive** | a mirror of Tier 1, plus the app's files | data drive 2, on the box (counted only on a different physical disk; in a VM made, not counted) | nightly **03:30** | mirror | not encrypted |
|
| **Tier 2 — cross-drive** | a mirror of Tier 1, plus the app's files | data drive 2, on the box (counted only on a different physical disk; in a VM made, not counted) | nightly **03:30** | mirror | not encrypted |
|
||||||
| **Tier 3 — off-site, per app** | Tier 1 + the app's must-keep files | **Hetzner Storage Box** (Germany), one folder per household | nightly **04:15** | 7 daily, 4 weekly, 6 monthly | the household's own restic password |
|
| **Tier 3 — off-site, per app** | Tier 1 + the app's must-keep files | **Hetzner Storage Box** (Germany), one folder per household | nightly **04:15** | 7 daily, 4 weekly, 6 monthly | the household's own restic password |
|
||||||
| **Whole guest — local** | the guest's system (not the data drives) | the box's own system disk | daily, in the window **04:30–08:30** | up to 3 copies (fewer on a small disk) | not encrypted |
|
| **Whole guest — local** | the guest's system (not the data drives) | the box's own system disk | daily, in the window **04:30–08:30** | one copy per day, up to 3 days (fewer on a small disk) | not encrypted |
|
||||||
| **Whole guest — off-site** | the same | **ep0** (Hetzner, Germany), one namespace per household | weekly, same window | last 2 | the household's own key, escrowed with a recovery code |
|
| **Whole guest — off-site** | the same | **ep0** (Hetzner, Germany), one namespace per household | weekly, same window | one per week, 2 weeks | the household's own key, escrowed with a recovery code |
|
||||||
|
|
||||||
Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–3 follow at fixed offsets so they cannot run out of order. A missed night is caught up 15 minutes after the box starts."
|
Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–3 follow at fixed offsets so they cannot run out of order. A missed night is caught up 15 minutes after the box starts."
|
||||||
|
|
||||||
@@ -99,7 +101,7 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
|
|||||||
- one namespace **per household** (whole-guest copies, each household's own key),
|
- one namespace **per household** (whole-guest copies, each household's own key),
|
||||||
with a NEUTRAL note under it (not a red gap box): "By design: each household holds the key to its own copies; the operator cannot open them without their recovery code." — *fix (c), 2026-10-09: this is a property of the design, not a gap, and a red box said the opposite of what it means.*
|
with a NEUTRAL note under it (not a red gap box): "By design: each household holds the key to its own copies; the operator cannot open them without their recovery code." — *fix (c), 2026-10-09: this is a property of the design, not a gap, and a red box said the opposite of what it means.*
|
||||||
- the **`operator`** namespace: two groups — **"hub database"** and **"DooPlex copy"** (code, password manager, secrets, signing keys).
|
- the **`operator`** namespace: two groups — **"hub database"** and **"DooPlex copy"** (code, password manager, secrets, signing keys).
|
||||||
Prune jobs on ep0: households keep the last 2; `operator` keeps 14 daily + 8 weekly.
|
Prune jobs on ep0: households keep one copy per week for 2 weeks (`keep-weekly 2`, R-1162, 2026-10-19; every new household's job is made with its namespace, R-1165); `operator` keeps 14 daily + 8 weekly.
|
||||||
Mark ep0 with a small shield: "**Protected** — holds the only off-premises copy of real household data."
|
Mark ep0 with a small shield: "**Protected** — holds the only off-premises copy of real household data."
|
||||||
- **Hetzner Storage Box** (grey): the shared box for the households' per-app restic copies (Tier 3), one sub-account per household, add-only keys.
|
- **Hetzner Storage Box** (grey): the shared box for the households' per-app restic copies (Tier 3), one sub-account per household, add-only keys.
|
||||||
- **Mail service** (grey, "Resend"): every mail the hub sends — to the operator and to households.
|
- **Mail service** (grey, "Resend"): every mail the hub sends — to the operator and to households.
|
||||||
|
|||||||
File diff suppressed because one or more lines are too long
+17104
File diff suppressed because one or more lines are too long
@@ -0,0 +1,24 @@
|
|||||||
|
loop=/dev/loop2
|
||||||
|
--- lsblk p1 (the bug input)
|
||||||
|
{"blockdevices":[{"name":"loop2p1","fstype":null,"pttype":"gpt","mountpoint":null}]}
|
||||||
|
--- blkid -p p1 (empty part)
|
||||||
|
DEVNAME=/dev/loop2p1
|
||||||
|
PART_ENTRY_SCHEME=gpt
|
||||||
|
PART_ENTRY_UUID=2b13a20b-f5a4-4906-9e30-985c170a50ad
|
||||||
|
PART_ENTRY_TYPE=0fc63daf-8483-4772-8e79-3d69d8477de4
|
||||||
|
PART_ENTRY_NUMBER=1
|
||||||
|
PART_ENTRY_OFFSET=2048
|
||||||
|
PART_ENTRY_SIZE=1048576
|
||||||
|
PART_ENTRY_DISK=7:2
|
||||||
|
rc=0
|
||||||
|
--- blkid -p p2 (ext4)
|
||||||
|
TYPE=ext4
|
||||||
|
PART_ENTRY_SCHEME=gpt
|
||||||
|
PART_ENTRY_NUMBER=2
|
||||||
|
rc=0
|
||||||
|
Re-reading the partition table failed.: Invalid argument
|
||||||
|
--- blkid -p p1 (a dos table inside)
|
||||||
|
PTTYPE=dos
|
||||||
|
PART_ENTRY_SCHEME=gpt
|
||||||
|
rc=0
|
||||||
|
ls: cannot access '/root/r1093-scratch.img': No such file or directory
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# C1 reviewer 2 — delete and retention (fresh, read-only; clones at controller 7ca59e2)
|
||||||
|
|
||||||
|
| # | Sev guess | harm | mechanism |
|
||||||
|
|---|---|---|---|
|
||||||
|
| D1 | P1 (mech high, reach medium) | Shares second-drive copy: one drive group's reconcile deletes ANOTHER drive's share mirrors (on the night a drive is decommissioned/away; or every night when two groups target one drive, then a full re-copy) | `internal/backup/tier2_shares.go:135` groups by source; `:201-202` own destBase/legRels; `:230` `tier2ReconcileRoots(destBase, sharesTier2DestRoots(destBase), legRels)`; `:266-267` ReadDir lists every drive key on disk; `tier2.go:299-329` removes any child not in this group's legs; `shares_payload.go:233-251` away/decommissioned drives drop out |
|
||||||
|
| D2 | P1 optional legs / P2 mandatory; design question with a defect corner | A missing or emptied app folder on the drive deletes its second-drive copy within one night; optional legs (audiobooks, comics, ROMs, Immich external library) are in no other tier | `tier2_capture.go:55-61` missing leg skipped (optional silently) → `tier2.go:455` reconcile deletes its mirror; run recorded success `:465`; emptied → `rsync -a --delete` `tier2.go:751` via `:443`; catalog optional legs `audiobookshelf/.felhom.yml:24-26`, komga, romm, immich `:32-34`; 07 §8.2 „data legs are NOT guarded" covers a leg leaving the class set, not a declared-but-missing leg |
|
||||||
|
| D3 | P2 high (mech) | A host clock ahead during one whole-guest backup silences that tier until real time passes it; the hub reads the same clock and says OK | agent `internal/localapi/server.go:1178-1190` negative age never due; `internal/backup/runner.go:607-629` newest ctime unbounded; hub `monitor/deadline_tiers.go:155`, `deadline.go:243`; R-973 cap counts future entries `backup_cap.go:61`; restic: `offbox_window.go:133-136` pinned guard refuses every window → quota fills; NAS tier no guard `:209-216`; `offbox_abandon.go:248,287` set-aside rm on box clock |
|
||||||
|
| D4 | P2 | A failed LOCAL unit restore leaves the app running on mixed data, no undo copy (same as R1 of reviewer 1 — independent agreement) | `restore_unit.go:430-489`; `writeSafetyDump` only in `offbox_reconstitute.go`; 07 §6.3 ladder not scoped to off-site |
|
||||||
|
| D5 | P2 order defect / depth design | A dump that fails its own content check has ALREADY replaced last night's good dump; Tier-2 mirrors it an hour later | `appbackup/dbdump.go:388-397` size-only check, `:399` rename, `:419` `ValidateDump` after; `backup.go:688-690`; volume tars promoted with no content check `backup.go:902` |
|
||||||
|
| D6 | P3 med | „Megőrzött adatok" can list a RUNNING app's live folder as deletable leftover if its app.yaml is unreadable | `stacks/manager.go:672`; `kept.go:220-235` skips `!Deployed || AppConfig==nil`; `ListKept :330-358`; `DeleteKept :385-402` RemoveAll; `deploy.go:1180-1190` write+rename without fsync |
|
||||||
|
| D7 | P3 low-med | A unit restore can point the app back at an OLD drive; F5 then deletes the current drive's unit | `restore_unit.go:90-111, 507`; `cmd/controller/main.go:3252`; `deploy.go:820`; `backup.go:1517-1574` `pruneStalePrimaryDirs` even when this run's capture failed (`:726-733`) |
|
||||||
|
| D8 | P3/P4 design | local whole-guest keep=1 prunes the only good archive before the new one is proven | agent `runner.go:229`, `:139-161`, restore-test settle 24 h `:462` |
|
||||||
|
|
||||||
|
**Checked SAFE (reviewer):** hub off-site deletions by exact identity (`offsitekeys.go:366-515`); pinned off-site retention guard refuses future/too-new snapshots and big plans (`offbox_window.go:132-161`); dump and tar writes crash-safe (tmp+fsync+rename); R-403 hollow-unit guard; drive migration deletes source only after verify (`stacks/migrate.go:614-667`); app removal refusals and protected folders (`delete.go:136-153, 241-301, 839-846`); mirror/undo clean-ups allow-listed/prefix-guarded; agent prune never on PBS unless opted in; restore-test skips unsettled/future/foreign-key archives; `ForgetRetiredDrives` mount units only; SMB share removal deletes no folder. Not reviewed: agent `vzpartial` janitor, tenantsync/ep0 prune config, hub RESET beyond `PurgeRepos`.
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# C1 reviewer 3 — keys that gate recovery (fresh, read-only; controller 7ca59e2, hub 5c48731, agent 4e1621c)
|
||||||
|
|
||||||
|
| # | Sev guess | harm | mechanism |
|
||||||
|
|---|---|---|---|
|
||||||
|
| K1 | P2 (P1 with K2 / a RESET) high | **A reinstall through „delete the old host first" (D19/R-1154) cuts the new box off from the household's escrow: the recovery code is never asked, a NEW key is minted, and the old off-site history is set aside by itself on night one; typing the old code answers „wrong code"** | hub `api/passphrase_scope.go:58` 409 forces the delete; `store.go:3204-3210` delete demotes escrow under the OLD host_id; `handler.go:1261` new host_id; `GetEscrowStatusForCustomer` `store.go:3500-3527` joins on `hosts` → nil (superseded join on deleted host also gone) → `identity_blob_present`/`superseded_present` false; controller `offbox.go:542-555` mints new pw, `:1788` no recovery offer, `:905-906` firstNight → `resetOrphanedRepo`; `handleHostEscrowRetainedGet` 404 for a deleted host (`handler.go:1691`); contrast purge `store.go:4185-4190` DOES include `host_deletions` |
|
||||||
|
| K2 | P2 high | Abandoning ONE set-aside copy deletes EVERY retained recovery package of the customer — other set-aside copies become ciphertext nobody can open | hub `handler.go:626-627` → `PurgeSupersededEscrowForCustomer` `store.go:4185-4190` deletes all rows for the customer (incl. deleted hosts); the comment at `:4176-4179` states the invariant it breaks |
|
||||||
|
| K3 | P3 med-high | A ceremony while off-site is not configured seals a package with NO repo password and supersedes the one that had it | controller `escrow_handlers.go:217` re-stage only when configured; agent `main.go:2975-2981`; hub `store.go:3392`; `escrow_confirm.go:222-258` stale card local only |
|
||||||
|
| K4 | P3 med | The global „create your recovery code" bar links to the ceremony the backups page hides while a recovery offer is outstanding | `backups_remote.html:144-160` vs `escrow_banner.go:23-33` + hu.json:1682; no `recoveryOffer()` refusal in the wizard/start handlers; settle gate `main.go:876`, `offsiteapply.go:84` |
|
||||||
|
| K5 | P3 med-high | After a reinstall recovered on the reuse path, the NEW PBS key K is never escrowed and nothing compares fingerprints | `felhom-pbs-apply:99` autogen; `escrow_confirm.go:182-205` auto-confirm on hash; no compare of `host_escrow.key_fingerprint` vs live key (agent `internal/backup/runner.go:407-429` only) |
|
||||||
|
| K6 | P3/P4 design | Whole-guest off-site (PBS) copies run before any escrow exists; host lost before the ceremony → unreadable archives | controller gates Tier 3 (`offbox.go:781`); agent `internal/backup`, `internal/pbsdr` have no escrow gate |
|
||||||
|
| K7 | P4 low | escrow PUT split in two statements; a failure leaves a row whose hash claims a key its blob lacks | hub `handler.go:1388, 1412-1420`; `store.go:3402-3408` |
|
||||||
|
| K8 | P4 | operator off-site re-issue resets the provider password before saving it | hub `offsite.go:217-245` |
|
||||||
|
|
||||||
|
**Checked SAFE (reviewer):** mint guard while the hub holds a package (`offbox.go:542-555, 659-720`); orphaned repo moved aside, never re-inited/deleted (`offbox.go:886-943, 332-397`); supersession retains both blobs (`store.go:3343-3416`); host delete demotes not destroys (`store.go:3164-3229`); ceremony two-word confirm; `escrow.Create` self-verifies; recovery refuses to replace a different local key; `ClearOffboxTarget`; PBS wrapper has no delete verb; hub abandon delete limited to one path after a delay (`offsitekeys/service.go:464-496`).
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
# C1 reviewer 1 — restore and rollback (fresh, read-only; clones at controller 7ca59e2 / v0.336.0)
|
||||||
|
|
||||||
|
Raw candidates as returned by the reviewer, condensed; each is UNVERIFIED until C2 marks it. File:line are in the controller unless named.
|
||||||
|
|
||||||
|
| # | Sev guess | harm | mechanism |
|
||||||
|
|---|---|---|---|
|
||||||
|
| R1 | P2 high | A local / second-drive unit restore that fails halfway restarts the app on half-replaced data, no undo copy; the next night's capture then overwrites the only restore point with the broken state | `internal/backup/restore_unit.go:429-431` failed stop only warns; `restore.go:217-218` `docker volume rm -f` error ignored before untar (10-min limit `:237-241`); `restore_unit.go:466-486` DB replay failure → app started anyway; one restore point per app, promoted in place (`atomicPromoteTar`) |
|
||||||
|
| R2 | P2 high | A power cut / controller restart during a restore starts the app on a mixed state; the undo copy is never used at boot | `offbox_reconstitute.go:942` marker → `appstop_marker.go:307` StartStack with no restore check; `offbox_restore_undo.go:137` undo dir only dropped, never read at start; unit restore and .fab import set no marker → `bootrecon/bootrecon.go:210-213,269` starts the app; updates have a journal (`stacks/update.go:1420-1489`), restores do not |
|
||||||
|
| R3 | P2 high | A cut-off shared-folder restore can be „placed" and reported as restored while files are missing; scratch deleted | `shares_restore.go:138` readiness = `rErr == nil && len(entries) > 0` (R-358 class, not fixed for shares); 30-min ctx `web/offbox_handlers.go:716`; place reports N files and deletes scratch `:301`; free-space gate fails open and fixed 2 GiB `:107` |
|
||||||
|
| R4 | P2 high | A `.fab` export silently drops a whole drive folder that contains a symlink; import never sets owners | `appexport/fabplan.go:292` `tar.FileInfoHeader(info,"")` then `os.Open`+`io.Copy` on a symlink; `export.go:714` warn-and-continue; `assertBundleDataComplete` refuses only when no data at all; `restore.go:1157-1176` extracts only dirs/regular files, no chown |
|
||||||
|
| R5 | P2 med-high | Second-drive whole restore swaps files, then refuses the DB half, leaves the app STOPPED and says „the app is untouched" | `tier2_whole.go` stops + merges first, then `RestoreFromRecoveryUnitAtWith` whose refusals (`restore_unit.go:366-371, 386-389, 413-416, 420-423`) assume nothing happened; en.json:2667 „the restore did not start. The app is untouched" |
|
||||||
|
| R6 | P2 med | `.fab` import into a STOPPED app merges into existing volumes; the default export (stop box ticked) has no SQL dump | `appexport/restore.go:361-374` volumes removed only if running; `docker cp` merge `:721`; `templates/app_data.html:221` stop default; `export.go:262` stop before `dumpDatabase` `:366`; `appbackup/dbdump.go:142` discovers only running containers |
|
||||||
|
| R7 | P2/P3 high | `.fab` import has no concurrency guard, no undo, no record | `restore.go:181-186`; `web/handler_export.go:312-353` no `restoreOpBlocked()`; `RemoveStackVolumes` before anything else, no copy |
|
||||||
|
| R8 | P2 med | `.fab` import drops the family gate, sign-up lock and version pin → a family-only app opens to anyone | `cmd/controller/main.go:3517-3530` bare `AppConfig`; `stacks/family_gate.go:173-190` removes the gate file when `FamilyGate == nil`; R-773 fix only on the unit path (`stacks/deploy.go:840-870`) |
|
||||||
|
| R9 | P3 med | `.fab` import trusts the bundle's HDD_PATH; empty → relative path; older userdata overwrites newer | `restore.go:589-631` |
|
||||||
|
| R10 | P3 high | `.fab` DB import: plain psql without ON_ERROR_STOP; postgis not classified | `restore.go:1052-1053`, `:928-937` |
|
||||||
|
| R11 | P3 med | volume replace never checks the old volume is gone | `restore.go:218` |
|
||||||
|
| R12 | P3 high | interrupted second-drive whole restore leaves `<dst>.felhom-restoring` (O_EXCL) → every retry fails „file exists" | `tier2_whole.go:86` |
|
||||||
|
| R13 | P3 design | full off-site download capped at 30 min per press; household not told to retry | `web/offbox_handlers.go:455` |
|
||||||
|
| R14 | P4 med | place/full restore re-read the LATEST snapshot instead of the downloaded one | `offbox_restore.go:675-679`, `offbox_reconstitute.go:657-661` |
|
||||||
|
|
||||||
|
**Inventory drift (July → now):** three more customer restore entries exist: the Tier-2 unit restore (R-102), the Tier-2 whole restore (decision 26, `tier2_whole.go`), the kept-data Load (`kept_load.go`).
|
||||||
|
|
||||||
|
**Checked SAFE (reviewer):** `.fab` path confinement (`appexport/restore.go:1104-1107,1152-1155`; `validate.go:28-66`; `web/handler_export.go:385-394`); off-site placement map refuses `..`/outside/`backups/` (`offbox_restore.go:603-640`); R-358 marker after restic success; full off-site restore fit/safety-dump/undo checks before stop, fail closed (`offbox_reconstitute.go:719-818, 909-927`); Tier-1 refusals before change; main loader `ON_ERROR_STOP=1 --single-transaction` (`appbackup/dbdump.go:775-784`); Tier-2 file restore `rsync --ignore-existing`, never `--delete`; whole restore newer-wins + Lchown; **NFC/NFD: no normalisation anywhere — every lane compares bytes, so the missing-only compare sees TWO names (consistent on one filesystem)**; agent RestoreLXC force only when asked. Not reviewed: `kept_load.go`, hub operator restore paths.
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
# C1 reviewer 4 — time and concurrency (fresh, read-only; controller 7ca59e2, agent 4e1621c, felhom.eu 5c48731)
|
||||||
|
|
||||||
|
## (A) Schedulers and what each does on Sunday 2026-10-25 (03:00 CEST → 02:00 CET; 02:00–02:59 happens twice)
|
||||||
|
|
||||||
|
| owner | job | schedule | zone | file:line | 2026-10-25 | after 3 days off |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| controller | db-dump | daily W (default 02:30) | Budapest (`time.Date`, day+1) | main.go:1262, scheduler.go:499-518 | once, at the SECOND 02:30 (CET = 01:30Z) — Go normalises the ambiguous time to the later offset | not replayed by the scheduler; the catch-up runs it once 15 min after start (nightchain.go:277-345) |
|
||||||
|
| controller | tier2-backup | W+60m | Budapest | main.go:1352 | once, 03:30 CET | once, by the catch-up |
|
||||||
|
| controller | offbox-backup + app-update leg | W+105m | Budapest | main.go:1519 | once, 04:15 CET | catch-up runs the off-site leg only |
|
||||||
|
| controller | offsite-abandon-sweep 05:10, offsite-proof 05:30, offsite-integrity 06:00 | fixed | Budapest | main.go:1561, 1613, 1591 | once | **no catch-up** (finding T6) |
|
||||||
|
| controller | metrics-prune 04:00, fill-watch 03:30, asset-sync 05:00, selfupdate-auto 04:30 (off by default) | fixed | Budapest | main.go:1621,1671,1768,1741 | once | floor update runs on any hub reply |
|
||||||
|
| controller | whole-guest quiesce loop | poll 5 min; gate [W+2h, W+6h) by wall minute; valve when age > cadence+24h | Budapest wall | quiesce.go:150, 191-205, 790-818 | gate 04:30–08:30 CET (4 h real) | valve fires ~5 min after start, any time of day |
|
||||||
|
| controller | ~20 `Every` tickers | monotonic | n/a | main.go:898-2146 | unaffected | restart at boot |
|
||||||
|
| agent | whole-guest due | age ≥ cadence | UTC | localapi/server.go:1182-1192 | unaffected | due at once |
|
||||||
|
| agent | restore-test | 30 min after start, then 6 h | UTC | backup/schedule.go:148-177 | unaffected | after 30 min |
|
||||||
|
| agent | OS leg (+ kernel reboot if hub „Tonight") | after any non-manual primary backup +90 s, 20 h min gap | no wall clock | server.go:982-986, main.go:1272-1280, leg.go:655-664, kernel.go:101-114 | follows the backup | follows the valve backup = daytime (T8) |
|
||||||
|
| agent | fstrim | hourly; due Wed 10:00, only 10:00–20:59 | host local | fstrim/fstrim.go:39-50,110 | n/a | next eligible hour |
|
||||||
|
| hub | scheduleDaily: db-snapshot 02:00 (keep 2), prune 04:30, deadline-check 05:00, offsite-key-audit 07:10 | `next.Add(24*time.Hour)` | Budapest | hub cmd/hub/main.go:1049-1072 | **every job runs TWICE** (T9) | n/a |
|
||||||
|
| host timers | mgmt-watchdog, crash-guard-check | monotonic | n/a | felhom-agent/configs/*.timer | unaffected | after boot |
|
||||||
|
| DooPlex timers | hub-db-backup 02:30; dooplex-offsite 00:20; hub-db-restore-test Sun 04:30; offsite-restore-test Sun 05:30; ep0-copy-gc 08:00 | OnCalendar local | DooPlex | felhom.eu/scripts/*/*.timer | Oct 25 is a Sunday: both weekly restore tests run on the DST day; 02:30 is in the repeated hour (systemd behaviour not verified) | Persistent=true |
|
||||||
|
|
||||||
|
## (B) Candidates
|
||||||
|
|
||||||
|
| # | Sev guess | harm | mechanism |
|
||||||
|
|---|---|---|---|
|
||||||
|
| T1 | P2 med-high | The nightly second-drive mirror (`rsync --delete`) can run DURING a restore from that same mirror and delete the files not yet copied back | `tier2.go:479-520` RunAllTier2 checks held/version/disconnected only; `:443` mirror, `:744-751` `--delete`; restore `tier2_whole.go:301` takes only `acquireRunning`; Tier-2 leg takes only `LegLock` (main.go:1348-1352); `update_guard.go:595-601` isHeld ignores restores |
|
||||||
|
| T2 | P2 if true; low-med | The Debian 13 controller image may lack tzdata → every schedule on UTC silently | `Dockerfile:77-100` debian:13.7-slim, no tzdata, no `time/tzdata` import; fallbacks scheduler.go:19-23 (logs), quiesce.go:749-751 (silent), funcmap.go:62-66 |
|
||||||
|
| T3 | P3 med | Whole-guest start creeps ~5 min/night until a night is skipped silently | agent `server.go:1184` `age >= Cadence` no slack; age from archive start; controller poll 5 min `quiesce.go:150`; gate refusal `:807-818` |
|
||||||
|
| T4 | P3 med-high | The whole-guest quiesce stops apps in the middle of a restore or an update | `quiesce.go:213-275` no `IsRunning()`/restore check; `RunningAppStacks` (stacks/manager.go:620-636); `TriggerNow` (:419-445) same; `DumpAppVolumesSafe` (backup.go:1001-1045) can restart an app quiesce stopped |
|
||||||
|
| T5 | P3 high | The nightly off-site copy is skipped SILENTLY when anything else holds the backup flag, and the catch-up never retries | `offbox.go:981-996` INFO „skipped — another backup is running" return nil; `withLeg` MarkEnded unconditionally (main.go:1246); proof 05:30 / integrity 06:00 take the same flag |
|
||||||
|
| T6 | P3 high | Off-site check and proof never run on a box off at 05:30–06:00; the code comment says the opposite | main.go:1575-1577 comment vs 1591-1619; `catchUpLegs` nightchain.go:17-18; no hub alarm on integrity/proof age |
|
||||||
|
| T7 | P3 high | Hub raises a false „no DB dump" error every morning for W after ~04:45 or evening W | hub `monitor/deadline.go:416-419` midnight Budapest; `:556-565` → `expected_dbdump_missed`; hub has no idea of W |
|
||||||
|
| T8 | P3 design | The OS leg and kernel reboot follow ANY scheduled backup, including the daytime valve after power-on | agent server.go:982-986; main.go:1279 labels „night"; kernel.go:101-114; hub osupdates/kernel.go:46-48,181-185 |
|
||||||
|
| T9 | P4 high | Hub daily jobs run TWICE on 25 October (db-snapshot keep-2 prunes a day early; duplicate deadline alarms); once an hour late in spring | hub cmd/hub/main.go:1058-1062 `next.Add(24*time.Hour)`; the controller fixed this (scheduler.go:511-516) |
|
||||||
|
| T10 | P4 high | Spring forward 2027-03-28: db-dump and tier2 fire at the same instant; tier2 may mirror yesterday's unit | scheduler normalisation; `LegLock` main.go:1237 plain mutex |
|
||||||
|
| T11 | P4 design | A clock ahead stops whole-guest backups silently (same as D3) | server.go:1182-1192; quiesce.go:816-817; deadline.go:244; backup_cap.go:61 |
|
||||||
|
| T12 | P4 high | self-update busy check before the image pull, swap after it with no re-check | updater.go:532-545, :764 |
|
||||||
|
| T13 | P4 med | double-clicking a restore shows it as failed while it runs | handlers.go:1853, :1874 `BeginRestoreOp` after the check; second POST's `EndRestoreOp(false)` |
|
||||||
|
|
||||||
|
**Checked SAFE (reviewer):** controller DST day+1 arithmetic and late-fire skip (scheduler.go:342-348, 511-516); night ledger persisted, seeded, one catch-up for many nights; quiesce marker persisted and recovered; in-memory backup flag (no stale lock); flock-based OS-leg locks; restic stale locks self-heal; two restores blocked by `m.running`; Remove refused during restore/quiesce; catch-up and whole-guest wait for each other; agent due/cap UTC durations; off-site prune guard refuses future snapshots; host power-off re-checks; no cron/OnCalendar in the installer; hub staleness uses the hub's own receive clock.
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
# C2 register search, 2026-10-20T10:11:13Z, OPEN-ITEMS.md + CLOSED-ITEMS.md at c701b253; grep -ciE
|
||||||
|
1 open / 2 closed : R-893
|
||||||
|
0 open / 0 closed : tier2_shares|_shares.*tier|shares.*second.drive|Megoszt.*2\. ment
|
||||||
|
0 open / 2 closed : host_deletions|retained.*delet|deleted host|escrow.*reinstall
|
||||||
|
2 open / 7 closed : scheduleDaily|DST|daylight|oraallit|time.?zone
|
||||||
|
1 open / 8 closed : restore-undo|undo copy|safety dump|unit restore.*fail
|
||||||
|
1 open / 0 closed : ValidateDump|dump.*replac.*valid
|
||||||
|
0 open / 0 closed : SharesScratchReady|shares.*scratch|shares restore
|
||||||
|
0 open / 4 closed : tier2_whole|whole restore|Teljes vissza
|
||||||
|
0 open / 3 closed : optional leg|missing on disk|--delete
|
||||||
|
0 open / 0 closed : PurgeSuperseded|abandon.*purge
|
||||||
|
0 open / 1 closed : future.dated|clock ahead|clock.*future|negative age
|
||||||
|
0 open / 0 closed : another backup is running
|
||||||
|
1 open / 2 closed : expected_dbdump_missed
|
||||||
|
2 open / 1 closed : symlink
|
||||||
|
0 open / 1 closed : family.gate.*fab|fab.*family|\.fab
|
||||||
|
0 open / 0 closed : felhom-restoring
|
||||||
|
0 open / 0 closed : quiesce.*restore|restore.*quiesce
|
||||||
|
1 open / 4 closed : offsite-proof|integrity.*catch|proof.*catch
|
||||||
|
0 open / 0 closed : double.click|dupla
|
||||||
|
0 open / 1 closed : disconnected.*target|away.*target
|
||||||
|
Every non-zero hit was read: none describes the candidate (positive control R-893 found in both).
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
# C2 — the main session's verdict on every candidate (2026-10-20)
|
||||||
|
|
||||||
|
Method: each candidate's `file:line` read in LIVE source (controller `7ca59e2` = v0.336.0, agent `4e1621c`, felhom.eu
|
||||||
|
`5c48731`+); then (a) the register searched, open AND closed, with ASCII fragments — positive control `R-893` found in
|
||||||
|
both; (b) the design test against `07-backup-architecture.md` and `CONTEXT.md`. Verdict words:
|
||||||
|
|
||||||
|
- **CONFIRMED** — the mechanism read true in live source AND a proof ran (unit test against production code, or live).
|
||||||
|
- **CONFIRMED-CODE** — the mechanism read true in live source; no proof ran yet.
|
||||||
|
- **PLAUSIBLE** — the reviewer's reading, spot-checked only.
|
||||||
|
- **NOT-A-FINDING** — with the reason.
|
||||||
|
|
||||||
|
Register: none of the candidates below is in OPEN-ITEMS or CLOSED-ITEMS (searches in `C2-register-search.txt`).
|
||||||
|
|
||||||
|
| id | verdict | design or defect | note |
|
||||||
|
|---|---|---|---|
|
||||||
|
| **D1** shares second-drive copy pruned by another drive's group | **CONFIRMED** (unit test, production code, fake mirror seam — `C3-D1-shares-tier2/`) | defect | **Worse than reported:** with shares on two drives that pick the same target, EVERY night the second group's reconcile deletes the first group's copy (`removed stale dest dir hdd_2/b_share`) — one drive's shares never have a second copy while the run says ok. And the night a drive goes away, its copy is deleted (`removed stale dest dir hdd_3/csaladi_fotok`). |
|
||||||
|
| D1b an AWAY drive is still chosen as the second-drive target | CONFIRMED-CODE (+ the same unit test: `hdd_3/iskola => hdd_1/...` after hdd_1 was marked disconnected) | defect | `tier2.go:151-180` loops `GetSchedulableStoragePaths()` (`settings.go:1593-1603` excludes only Decommissioned); no Disconnected check. On a real box the away drive's empty mount point is on the guest's root disk |
|
||||||
|
| **K1** reinstall after „delete the old host" loses the route to the old escrow | **CONFIRMED** at the hub store (unit test against production code, `C3-K1-escrow-after-host-delete/`); the controller half (new key minted, history set aside on night one) CONFIRMED-CODE | defect (03 §8a / 07 §7.0 promise the route; D19+R-1154 removed it unrecorded) | hub `store.go:3164-3229` demotes into `host_escrow_superseded` under the OLD host_id and deletes the `hosts` row; `GetEscrowStatusForCustomer` `store.go:3499-3527` joins on `hosts` → nil for the new box; `customer_reset.go:48,155` and `store.go:4189` DO include `host_deletions`. Live proof needs Tester4 off-site ON (B7) |
|
||||||
|
| K2 abandoning one set-aside copy purges every retained package | CONFIRMED-CODE (`store.go:4185-4190`) → R-1182 | defect (breaks `store.go:4176-4179`'s own invariant) | `store.go:4185-4190` customer-wide delete read true |
|
||||||
|
| **R1/D4** a failed LOCAL unit restore: no copy of the live state, app started anyway | **CONFIRMED LIVE** on 9202 (`C3-R1-local-restore-9202/`): live row destroyed, DB crash-loops, no undo copy, raw English Docker text to the household | **design question** for the operator: decision 154 (R-638 A) chose „start anyway"; D20 (R-893 A) gave „put back exactly" to the OFF-SITE path only | `restore_unit.go:428-489` |
|
||||||
|
| R2 a crash mid-restore starts the app on a mixed state; the undo copy is never read at start | PLAUSIBLE | defect | `appstop_marker.go:307`, `offbox_restore_undo.go:137` |
|
||||||
|
| R3 a cut-off shares restore can be placed and reported restored | CONFIRMED-CODE | defect (R-358 class, fixed for apps, not shares) | `shares_restore.go:131-140` readiness = non-empty dir |
|
||||||
|
| R4 `.fab` export drops a drive folder holding a symlink | PLAUSIBLE | defect | `appexport/fabplan.go:292`, `export.go:714` |
|
||||||
|
| R5 second-drive whole restore: files swapped, then the unit half refuses → app left STOPPED, text says „untouched" | **CONFIRMED** (unit test, production code: `C3-R5-whole-restore/` — 1 file restored + 1 replaced, app stopped, no start, message „az alkalmazás és az adatai érintetlenek") | defect | `tier2_whole.go:301-340`: stop + merge, then `RestoreFromRecoveryUnitAtWith` whose early refusals return before its own stop/start; `hu.json` `err.backup.unit_versions_mixed` „Az alkalmazás érintetlen" |
|
||||||
|
| R6 `.fab` import into a stopped app merges volumes | PLAUSIBLE | defect | `appexport/restore.go:361-374` |
|
||||||
|
| R7 `.fab` import has no guard / undo / record | PLAUSIBLE | defect | |
|
||||||
|
| R8 `.fab` import drops the family gate | **CONFIRMED LIVE** on 9202 (`C3-R1187e-fab-family-gate/`: stranger 401 before, 200 after the import) → **R-1191 (P1)** | defect | `cmd/controller/main.go:3517-3530`, `stacks/family_gate.go:173-190` |
|
||||||
|
| R9–R11, R13, R14 | PLAUSIBLE | defect / design (R13) | low |
|
||||||
|
| R12 interrupted whole restore leaves `*.felhom-restoring`, every retry fails | **CONFIRMED** (unit test: `C3-R12-leftover-temp/`) → R-1188 (c) | defect | `tier2_whole.go:86-87` O_EXCL; removed only on an in-process error |
|
||||||
|
| **D2** a missing / emptied app folder deletes its second-drive copy within one night | **CONFIRMED** for a MISSING folder (unit test, production code + real copy seam: `C3-D2-missing-leg/` — mirror of `photos` deleted, run „ok", the warning says only „nem kerültek a másodlagos mentésbe"); the EMPTIED case CONFIRMED-CODE (`rsyncMirror` = `rsync -a --delete`, `tier2.go:744-751`; the test seam does not delete) | defect-shaped corner of a choice: 07 §8.2 „data legs are NOT guarded" is about a leg LEAVING the class set; this is a declared leg that is MISSING, which the code itself logs as a gap (`tier2_capture.go:55-61`) and then reconciles away (`tier2.go:454`) | P1 for optional legs (no other tier holds them) |
|
||||||
|
| D3 / T11 a host clock ahead silences the whole-guest tier; the hub reads the same clock | PLAUSIBLE (B6 #8 on the rehearsal VM is its live test) | defect | agent `localapi/server.go:1178-1190` |
|
||||||
|
| D5 a dump that fails its own check has already replaced the good one | **CONFIRMED** (unit test with a docker stub: `C3-D5-dump-replaced/`) → R-1185 | defect for the ORDER | `appbackup/dbdump.go:388-422`: size check, rename (`:399`), then `ValidateDump` |
|
||||||
|
| D6 kept-data page can list a running app's folder | PLAUSIBLE | defect | needs an unreadable app.yaml |
|
||||||
|
| D7 unit restore can point the app at an old drive | PLAUSIBLE (low) | defect | |
|
||||||
|
| D8 local whole-guest keep=1 | design (07 §6.1 R-684 chose it) | design | **NOT-A-FINDING: the operator chose keep=1 for demo-hp; recorded in 07** |
|
||||||
|
| T1 the nightly mirror (`rsync --delete`) can run during a restore from that mirror | **CONFIRMED** (unit test, production rsync mirror: `C3-T1-mirror-during-restore/`) → R-1180 | defect | `RunAllTier2` (`tier2.go:479-520`) takes no backup flag; the whole restore holds only `acquireRunning` (`tier2_whole.go:301`) |
|
||||||
|
| T2 controller image without tzdata → UTC schedules | **NOT-A-FINDING:** measured on 9202 (controller 0.331.0, Debian 13 base): `/usr/share/zoneinfo/Europe/Budapest` present, scheduler logs „scheduled for … CEST" | — | |
|
||||||
|
| T3 whole-guest start creeps ~5 min/night | PLAUSIBLE | defect | needs the agent journal over weeks |
|
||||||
|
| T4 whole-guest quiesce stops apps mid-restore | PLAUSIBLE | defect | `quiesce.go:213-275` |
|
||||||
|
| T5 nightly off-site copy skipped SILENTLY when the flag is held; catch-up never retries | CONFIRMED-CODE → R-1186 | defect (the code chose nil for the nightly path; the catch-up gap is not chosen) | `offbox.go:981-996`, `main.go:1246` |
|
||||||
|
| T6 proof / integrity never run on a box off at 05:30–06:00 | PLAUSIBLE | defect (comment `main.go:1575-1577` says the opposite) | |
|
||||||
|
| T7 hub false „no DB dump" for late or evening W | PLAUSIBLE | defect | hub `monitor/deadline.go:416-419, 556-565` |
|
||||||
|
| T8 the OS leg follows a daytime valve backup | design question | design | 07 §6.1 / 11 §8.1 set no time bound |
|
||||||
|
| **T9** hub daily jobs run TWICE on 2026-10-25 | **CONFIRMED** (replay of the hub's exact arithmetic, `C3-T9-hub-dst/replay.txt`) | defect | deadline-check fires 04:00 CET AND 05:00 CET; db-snapshot twice (keep-2 → the previous day's copy pruned a day early). **In five days, on the live hub.** |
|
||||||
|
| T10 spring forward collision | PLAUSIBLE | defect (P4) | |
|
||||||
|
| T12, T13 | PLAUSIBLE | defect (P4) | |
|
||||||
|
| K3–K8 | PLAUSIBLE | K6 design | |
|
||||||
|
|
||||||
|
**NFC/NFD (the brief's question for A.1.4):** reviewer 1 found no normalisation anywhere — rsync, restic and the Go
|
||||||
|
merge compare names byte for byte, so the missing-only compare sees **two names** for the NFC/NFD pair. The household
|
||||||
|
data carries the pair; B8 measures it live.
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
# go test at controller main 7ca59e2 (v0.336.0), scratch clone, fake mirror seam (no rsync/docker); 2026-10-20T10:04:31Z
|
||||||
|
=== RUN TestRehearsalD1_DeadDriveShareCopyIsPruned
|
||||||
|
zz_rehearsal_d1_test.go:46: drive keys: map[hdd_1:hdd_1 hdd_2:hdd_2 hdd_3:hdd_3]
|
||||||
|
zz_rehearsal_d1_test.go:47: POSITIVE CONTROL — copies after run 1 (both drives present): [hdd_1: hdd_3/iskola hdd_2: hdd_1/csaladi_fotok]
|
||||||
|
zz_rehearsal_d1_test.go:48: mirror calls run 1: [hdd_1/csaladi_fotok=>hdd_2/backups/secondary/_shares/hdd_1/csaladi_fotok data/shares-payload=>hdd_2/backups/secondary/_shares/_payload hdd_3/iskola=>hdd_1/backups/secondary/_shares/hdd_3/iskola data/shares-payload=>hdd_1/backups/secondary/_shares/_payload]
|
||||||
|
zz_rehearsal_d1_test.go:56: FAULT — copies after run 2 (hdd_1 away): [hdd_1: hdd_3/iskola hdd_2: hdd_1/csaladi_fotok]
|
||||||
|
zz_rehearsal_d1_test.go:57: mirror calls run 2: [hdd_3/iskola=>hdd_1/backups/secondary/_shares/hdd_3/iskola data/shares-payload=>hdd_1/backups/secondary/_shares/_payload]
|
||||||
|
--- PASS: TestRehearsalD1_DeadDriveShareCopyIsPruned (0.02s)
|
||||||
|
=== RUN TestRehearsalD1_TwoGroupsOneTarget
|
||||||
|
zz_rehearsal_d1_test.go:73: copies after two runs: [hdd_1: hdd_3/c_share hdd_2: hdd_1/a_share]
|
||||||
|
zz_rehearsal_d1_test.go:75: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_1 → hdd_2/backups/secondary/_shares (18 B)
|
||||||
|
zz_rehearsal_d1_test.go:75: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_3 → hdd_1/backups/secondary/_shares (36 B)
|
||||||
|
zz_rehearsal_d1_test.go:75: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_1 → hdd_2/backups/secondary/_shares (18 B)
|
||||||
|
zz_rehearsal_d1_test.go:75: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_3 → hdd_1/backups/secondary/_shares (36 B)
|
||||||
|
--- PASS: TestRehearsalD1_TwoGroupsOneTarget (0.01s)
|
||||||
|
=== RUN TestRehearsalD1_SameTargetThenDriveDies
|
||||||
|
zz_rehearsal_d1_test.go:89: POSITIVE CONTROL run 1: [hdd_1: hdd_3/csaladi_fotok]
|
||||||
|
zz_rehearsal_d1_test.go:91: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_2 → hdd_1/backups/secondary/_shares (18 B)
|
||||||
|
zz_rehearsal_d1_test.go:91: LOG [INFO] [backup] Tier 2 reconcile: removed stale dest dir hdd_2/b_share
|
||||||
|
zz_rehearsal_d1_test.go:91: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_3 → hdd_1/backups/secondary/_shares (42 B)
|
||||||
|
zz_rehearsal_d1_test.go:100: FAULT run 2 (hdd_3 away): [hdd_1: hdd_2/b_share]
|
||||||
|
zz_rehearsal_d1_test.go:102: LOG [INFO] [backup] Tier 2 reconcile: removed stale dest dir hdd_3/csaladi_fotok
|
||||||
|
zz_rehearsal_d1_test.go:102: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_2 → hdd_1/backups/secondary/_shares (18 B)
|
||||||
|
--- PASS: TestRehearsalD1_SameTargetThenDriveDies (0.01s)
|
||||||
|
PASS
|
||||||
|
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.049s
|
||||||
+104
@@ -0,0 +1,104 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate D1). Never committed.
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func listShareCopies(env *sharesEnv) []string {
|
||||||
|
var out []string
|
||||||
|
for name, d := range env.drives {
|
||||||
|
base := filepath.Join(NamespaceRoot(d, true), "backups", "secondary", SharesPseudoStack)
|
||||||
|
keys, _ := os.ReadDir(base)
|
||||||
|
for _, k := range keys {
|
||||||
|
if !k.IsDir() || k.Name() == filepath.Base(sharesPayloadDestRel) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
subs, _ := os.ReadDir(filepath.Join(base, k.Name()))
|
||||||
|
for _, s := range subs {
|
||||||
|
out = append(out, name+": "+k.Name()+"/"+s.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sort.Strings(out)
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func keyOwner(env *sharesEnv) map[string]string {
|
||||||
|
m := map[string]string{}
|
||||||
|
for name, d := range env.drives {
|
||||||
|
m[sharesDriveKey(d)] = name
|
||||||
|
}
|
||||||
|
return m
|
||||||
|
}
|
||||||
|
|
||||||
|
// A drive whose shares were mirrored goes away; another share group's run lands on the same target.
|
||||||
|
func TestRehearsalD1_DeadDriveShareCopyIsPruned(t *testing.T) {
|
||||||
|
env := newSharesEnv(t, "hdd_1", "hdd_2", "hdd_3")
|
||||||
|
env.addShare(t, "hdd_1", "csaladi_fotok", true)
|
||||||
|
env.addShare(t, "hdd_3", "iskola", true)
|
||||||
|
if err := env.m.RunSharesTier2(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Logf("drive keys: %v", keyOwner(env))
|
||||||
|
t.Logf("POSITIVE CONTROL — copies after run 1 (both drives present): %v", listShareCopies(env))
|
||||||
|
t.Logf("mirror calls run 1: %v", env.mirrored)
|
||||||
|
if err := env.sett.SetDisconnected(env.drives["hdd_1"], true, nil); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
env.mirrored = nil
|
||||||
|
if err := env.m.RunSharesTier2(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Logf("FAULT — copies after run 2 (hdd_1 away): %v", listShareCopies(env))
|
||||||
|
t.Logf("mirror calls run 2: %v", env.mirrored)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Shares on two drives that both pick the SAME target: does group A's reconcile remove group B's copy?
|
||||||
|
func TestRehearsalD1_TwoGroupsOneTarget(t *testing.T) {
|
||||||
|
env := newSharesEnv(t, "hdd_1", "hdd_2", "hdd_3")
|
||||||
|
env.addShare(t, "hdd_1", "a_share", true)
|
||||||
|
env.addShare(t, "hdd_3", "c_share", true)
|
||||||
|
var removed []string
|
||||||
|
env.m.logger = newCaptureLogger(&removed)
|
||||||
|
if err := env.m.RunSharesTier2(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := env.m.RunSharesTier2(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Logf("copies after two runs: %v", listShareCopies(env))
|
||||||
|
for _, l := range removed {
|
||||||
|
t.Logf("LOG %s", l)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Shares on hdd_2 and hdd_3 (both pick hdd_1 as target). Then hdd_3 dies.
|
||||||
|
func TestRehearsalD1_SameTargetThenDriveDies(t *testing.T) {
|
||||||
|
env := newSharesEnv(t, "hdd_1", "hdd_2", "hdd_3")
|
||||||
|
env.addShare(t, "hdd_2", "b_share", true)
|
||||||
|
env.addShare(t, "hdd_3", "csaladi_fotok", true)
|
||||||
|
var logs []string
|
||||||
|
env.m.logger = newCaptureLogger(&logs)
|
||||||
|
if err := env.m.RunSharesTier2(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Logf("POSITIVE CONTROL run 1: %v", listShareCopies(env))
|
||||||
|
for _, l := range logs {
|
||||||
|
t.Logf(" LOG %s", l)
|
||||||
|
}
|
||||||
|
logs = nil
|
||||||
|
if err := env.sett.SetDisconnected(env.drives["hdd_3"], true, nil); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := env.m.RunSharesTier2(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Logf("FAULT run 2 (hdd_3 away): %v", listShareCopies(env))
|
||||||
|
for _, l := range logs {
|
||||||
|
t.Logf(" LOG %s", l)
|
||||||
|
}
|
||||||
|
}
|
||||||
+18
@@ -0,0 +1,18 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
import (
|
||||||
|
"log"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
type capW struct{ dst *[]string }
|
||||||
|
|
||||||
|
func (c capW) Write(p []byte) (int, error) {
|
||||||
|
s := string(p)
|
||||||
|
if strings.Contains(s, "reconcile") || strings.Contains(s, "tier-2 copied") {
|
||||||
|
*c.dst = append(*c.dst, strings.TrimSpace(s))
|
||||||
|
}
|
||||||
|
return len(p), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func newCaptureLogger(dst *[]string) *log.Logger { return log.New(capW{dst}, "", 0) }
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
# go test at controller 7ca59e2 (v0.336.0), scratch clone; the real copyTree mirror seam (rsync --delete stands in by copyTree for the emptied case — see note); 2026-10-20T10:19:17Z
|
||||||
|
=== RUN TestRehearsalD2_MissingAndEmptiedLegDeleteTheirMirror
|
||||||
|
zz_rehearsal_d2_test.go:22: POSITIVE CONTROL after run 1: mirror photo=true doc=true
|
||||||
|
zz_rehearsal_d2_test.go:33: run 2 err=<nil>; status="ok" warning="Figyelmeztetés: a(z) app alkalmazás egyes adatmappái nem kerültek a másodlagos mentésbe: appdata/app/photos."
|
||||||
|
zz_rehearsal_d2_test.go:34: FAULT after run 2: mirror photo=false doc=true (the second-drive copy is the household's only local copy of these files)
|
||||||
|
--- PASS: TestRehearsalD2_MissingAndEmptiedLegDeleteTheirMirror (0.02s)
|
||||||
|
PASS
|
||||||
|
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.030s
|
||||||
+35
@@ -0,0 +1,35 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate D2). Never committed.
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRehearsalD2_MissingAndEmptiedLegDeleteTheirMirror(t *testing.T) {
|
||||||
|
m, src, target, prov := newTier2V2(t, "app")
|
||||||
|
prov.has["app"] = true
|
||||||
|
prov.binds["app"] = []ClassifiedBind{mHDD("appdata/app/photos"), mHDD("appdata/app/docs")}
|
||||||
|
mustWrite(t, filepath.Join(src, "appdata", "app", "photos", "family.jpg"), "PHOTO")
|
||||||
|
mustWrite(t, filepath.Join(src, "appdata", "app", "docs", "contract.pdf"), "PDF")
|
||||||
|
if err := m.RunTier2("app"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
destBase := filepath.Join(target, "backups", "secondary", "app")
|
||||||
|
photo := filepath.Join(destBase, "hdd", "appdata", "app", "photos", "family.jpg")
|
||||||
|
doc := filepath.Join(destBase, "hdd", "appdata", "app", "docs", "contract.pdf")
|
||||||
|
t.Logf("POSITIVE CONTROL after run 1: mirror photo=%v doc=%v", exists(photo), exists(doc))
|
||||||
|
|
||||||
|
// The household (or an app bug, or fsck → lost+found) loses the live photos folder; the docs folder is emptied.
|
||||||
|
if err := os.RemoveAll(filepath.Join(src, "appdata", "app", "photos")); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.Remove(filepath.Join(src, "appdata", "app", "docs", "contract.pdf")); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
err := m.RunTier2("app")
|
||||||
|
cd := m.settings.GetCrossDriveConfig("app")
|
||||||
|
t.Logf("run 2 err=%v; status=%q warning=%q", err, cd.LastStatus, cd.LastWarning)
|
||||||
|
t.Logf("FAULT after run 2: mirror photo=%v doc=%v (the second-drive copy is the household's only local copy of these files)", exists(photo), exists(doc))
|
||||||
|
}
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
# go test at controller 7ca59e2 (v0.336.0), scratch clone; a docker STUB on PATH (no real docker reached); 2026-10-20T11:07:38Z
|
||||||
|
=== RUN TestRehearsalD5_InvalidDumpReplacesTheGoodOne
|
||||||
|
zz_rehearsal_d5_test.go:26: POSITIVE CONTROL: last night's dump, 152 B, ValidateDump={Valid:true TableCount:1 Error: FileSize:152 ModTime:2026-10-20 13:07:39.022566077 +0200 CEST UserTableFound:false UserRows:0 LooksEmpty:false}
|
||||||
|
2026/10/20 13:07:39 [WARN] [backup] ValidateDump FAIL: /tmp/TestRehearsalD5_InvalidDumpReplacesTheGoodOne2769942191/002/unit/app-postgres.sql — dump file too small (< 100 bytes)
|
||||||
|
zz_rehearsal_d5_test.go:31: tonight's dump: err=<nil> validation={Valid:false TableCount:0 Error:dump file too small (< 100 bytes) FileSize:99 ModTime:2026-10-20 13:07:39.026566071 +0200 CEST UserTableFound:false UserRows:0 LooksEmpty:false}
|
||||||
|
zz_rehearsal_d5_test.go:32: FAULT: the file on disk now = 99 B, still holds CREATE TABLE photos: false
|
||||||
|
--- PASS: TestRehearsalD5_InvalidDumpReplacesTheGoodOne (0.00s)
|
||||||
|
PASS
|
||||||
|
ok gitea.dooplex.hu/admin/felhom-controller/internal/appbackup 0.008s
|
||||||
+35
@@ -0,0 +1,35 @@
|
|||||||
|
package appbackup
|
||||||
|
|
||||||
|
// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate D5 / R-1185). Never committed. A docker STUB on PATH
|
||||||
|
// (no real docker is reached) prints a header-only dump, as a database that was re-initialised empty would.
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"io"
|
||||||
|
"log"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRehearsalD5_InvalidDumpReplacesTheGoodOne(t *testing.T) {
|
||||||
|
bin := t.TempDir()
|
||||||
|
stub := "#!/bin/sh\ncase \"$1\" in\n inspect) echo true ;;\n exec) printf -- '-- PostgreSQL database dump\\nSET statement_timeout = 0;\\n\\n--\\n-- PostgreSQL database dump complete\\n--\\n' ;;\nesac\n"
|
||||||
|
if err := os.WriteFile(filepath.Join(bin, "docker"), []byte(stub), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Setenv("PATH", bin)
|
||||||
|
dumpDir := filepath.Join(t.TempDir(), "unit")
|
||||||
|
final := filepath.Join(dumpDir, "app-postgres.sql")
|
||||||
|
good := "-- PostgreSQL database dump\nCREATE TABLE public.photos (id int);\nCOPY public.photos (id) FROM stdin;\n1\n2\n\\.\n\n--\n-- PostgreSQL database dump complete\n--\n"
|
||||||
|
os.MkdirAll(dumpDir, 0o755)
|
||||||
|
os.WriteFile(final, []byte(good), 0o644)
|
||||||
|
t.Logf("POSITIVE CONTROL: last night's dump, %d B, ValidateDump=%+v", len(good), ValidateDump(final, DBTypePostgres))
|
||||||
|
|
||||||
|
db := DiscoveredDB{ContainerName: "app-db", ContainerID: "0123456789abcdef", DBType: DBTypePostgres, DBUser: "u", DBName: "d", StackName: "app"}
|
||||||
|
res := DumpOneTo(context.Background(), db, final, log.New(io.Discard, "", 0), false)
|
||||||
|
b, _ := os.ReadFile(final)
|
||||||
|
t.Logf("tonight's dump: err=%v validation=%+v", res.Error, res.Validation)
|
||||||
|
t.Logf("FAULT: the file on disk now = %d B, still holds CREATE TABLE photos: %v", len(b), contains(string(b), "CREATE TABLE public.photos"))
|
||||||
|
}
|
||||||
|
|
||||||
|
func contains(s, sub string) bool { return len(s) >= len(sub) && (func() bool { for i := 0; i+len(sub) <= len(s); i++ { if s[i:i+len(sub)] == sub { return true } }; return false })() }
|
||||||
+10
@@ -0,0 +1,10 @@
|
|||||||
|
# go test, hub store at felhom.eu 5c48731 (scratch clone), in-memory SQLite; 2026-10-20T10:12:12Z
|
||||||
|
=== RUN TestRehearsalK1_NewBoxAfterHostDeleteSeesNoEscrow
|
||||||
|
zz_rehearsal_k1_test.go:20: POSITIVE CONTROL (old host live): status=&{IdentityBlobPresent:true ResticPwSHA256:SHA_OLD CreatedAt:2026-10-18T00:00:00Z SupersededPresent:false SupersededAt:} err=<nil>
|
||||||
|
zz_rehearsal_k1_test.go:27: retained rows under the DELETED host_id: 1 (identity blob present: true)
|
||||||
|
zz_rehearsal_k1_test.go:34: FAULT (new box, what its ACK carries): status=<nil> err=<nil>
|
||||||
|
zz_rehearsal_k1_test.go:36: => GetEscrowStatusForCustomer returns nil: identity_blob_present=false AND superseded_present=false for the new box
|
||||||
|
zz_rehearsal_k1_test.go:39: retained rows reachable under the NEW host_id: 0
|
||||||
|
--- PASS: TestRehearsalK1_NewBoxAfterHostDeleteSeesNoEscrow (0.04s)
|
||||||
|
PASS
|
||||||
|
ok gitea.dooplex.hu/admin/felhom-hub/internal/store 0.048s
|
||||||
+40
@@ -0,0 +1,40 @@
|
|||||||
|
package store
|
||||||
|
|
||||||
|
// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate K1). Never committed.
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
func TestRehearsalK1_NewBoxAfterHostDeleteSeesNoEscrow(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
const cust = "Tester4"
|
||||||
|
// The household's first box, with an escrowed package (K blob + the identity blob that opens the off-site repo).
|
||||||
|
if err := s.UpsertHost(&Host{HostID: "Tester4-old", CustomerID: cust, APIKey: "k1"}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, _, err := s.SaveHostEscrow("Tester4-old", []byte("k-blob"), "fp", "zk", "2026-10-18T00:00:00Z", "SHA_OLD"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := s.SaveHostDRBundle("Tester4-old", []byte("identity-opening-the-old-repo")); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
st, err := s.GetEscrowStatusForCustomer(cust)
|
||||||
|
t.Logf("POSITIVE CONTROL (old host live): status=%+v err=%v", st, err)
|
||||||
|
|
||||||
|
// The operator deletes the old host (the installer's 409 tells them to), escrow acknowledged → demoted.
|
||||||
|
if err := s.DeleteHost("Tester4-old", true); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
ret, _ := s.ListSupersededEscrow("Tester4-old")
|
||||||
|
t.Logf("retained rows under the DELETED host_id: %d (identity blob present: %v)", len(ret), len(ret) == 1 && len(ret[0].IdentityBlob) > 0)
|
||||||
|
|
||||||
|
// The reinstalled box enrolls under a NEW host_id for the same customer.
|
||||||
|
if err := s.UpsertHost(&Host{HostID: "Tester4-new", CustomerID: cust, APIKey: "k2"}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
st, err = s.GetEscrowStatusForCustomer(cust)
|
||||||
|
t.Logf("FAULT (new box, what its ACK carries): status=%+v err=%v", st, err)
|
||||||
|
if st == nil {
|
||||||
|
t.Logf("=> GetEscrowStatusForCustomer returns nil: identity_blob_present=false AND superseded_present=false for the new box")
|
||||||
|
}
|
||||||
|
ret2, _ := s.ListSupersededEscrow("Tester4-new")
|
||||||
|
t.Logf("retained rows reachable under the NEW host_id: %d", len(ret2))
|
||||||
|
}
|
||||||
+306
@@ -0,0 +1,306 @@
|
|||||||
|
Tue Oct 20 10:09:14 UTC 2026
|
||||||
|
filebrowser Up About an hour (healthy)
|
||||||
|
felhom-controller Up About an hour (healthy)
|
||||||
|
paperless-webserver Up 3 hours (healthy)
|
||||||
|
paperless-postgres Up 3 hours (healthy)
|
||||||
|
paperless-redis Up 3 hours (healthy)
|
||||||
|
immich-server Up 3 hours (healthy)
|
||||||
|
immich-postgres Up 3 hours (healthy)
|
||||||
|
immich-machine-learning Up 3 hours (healthy)
|
||||||
|
immich-redis Up 3 hours (healthy)
|
||||||
|
traefik Up 4 days
|
||||||
|
0a4fb1a71c72cf704d1afd44e1ab24a8a68e11b8d82ef1dcb71a75d61a9dc1a3
|
||||||
|
0ce36c2e6db54e085f71a1436c4c89a9807a930c373198f451fa1665f90f0cab
|
||||||
|
0e2ca5e8738b95134285a666b1a7622bfcabc4728fb9269cc972dfaccbd59d45
|
||||||
|
0f4002ab75695fb312775a8203cc5213b256f1b01d255210cb5cdd0e50cd9070
|
||||||
|
1a8fe48409be3defe01713e2f621091630b965e1805cbb5872d1738043efd35c
|
||||||
|
1afe3420067df65e0d3e53baaec3debc39d2d42d333c7a3c266e4f6ff98e04f7
|
||||||
|
1b41940af5c4fa82af59320d6c81ba622d97f2e43d8ec51c50dcf65c2fdcb1e8
|
||||||
|
1f5120663962c7b03539bb993ab736be4800096e3d1f7147eb922d44b308ae6a
|
||||||
|
01feef94eb7e38a3679921b66cfe206e08adfd9b360ed4a60414cde606de1643
|
||||||
|
2a71e17dfa1e930ffe0c213cd1615c8d1adcbbf8d61615a38064d331c265c02e
|
||||||
|
2aaab84bf843817d9e6f32b673277fd1e0c2c8c2d038b8126df44927b1b41006
|
||||||
|
2ac49fc9cc2839f55c8abae76039950d15a5c301ea5b3918fa3e5d8715a15f42
|
||||||
|
2bbdb463b29493d60d352bafefccbe57856cf80bd2cc193b61f9bcc1310fb45e
|
||||||
|
2bc95b104191ffd31bdfdbe1e038d284d0624c04735094494c2c77d957e8e905
|
||||||
|
2dcb4bd2c3c618d71f582d33d5b1243e6ab075d3b00abc238d4430668a2b6f55
|
||||||
|
2e4c79db25c47436b44245376b1f6ff1991208ba081fddce1938c3897fc63c07
|
||||||
|
2ef8994a563ae88024ba29db4e349d26e6e3889e0683d30d1fb258208c9a10b4
|
||||||
|
2f08e882c5d23373e2df5486dce28d4728330f9f0131a2735ebc938bfb4d7b3a
|
||||||
|
02f42b55ee403ceef63a472c3624da0b48fab9e6aa6dc0c7800ec68b2f2041ef
|
||||||
|
3a3d7bc33a1ba21ae4a39c8cb345254cf66cd1dc60afae64e2299513b094b93b
|
||||||
|
3ac9820f84d0a50f233c63a62ee1b63b92acac026d68c60a8227f370b0be00b6
|
||||||
|
3d9a93769342e83a3c512d517b99cd64f8e794eb305c009e25c73a6f6627ad29
|
||||||
|
3d9c1f23abbcef771a4c38a75d75865a12d76a4359fe83c1828652dba3f91710
|
||||||
|
3f015f020f49d4967d2ac71b94060e6eaec86bc8852cf9a0b5f3dd8133dd528c
|
||||||
|
3fa030697dbaffb6809991bb69a335dc4e9accfafd4ed5ee953ed676673912e6
|
||||||
|
4a9ed2aed1fe75a0b02f688e29a20acc2bf7e081ba7518a7d75451be30fdc4e1
|
||||||
|
4bbaae5ae8cc8cc3330a73bb872b16e1420805197fa9bad903f7db9a1f358041
|
||||||
|
4c54a4cda3c5280d5b13facf3089f8a0710c4570876aaeb679025aabf92df99f
|
||||||
|
4e052b4ef5602a68eddcea049bbc4e79b306012cf8b7f0c9cfc3e6c39af64fce
|
||||||
|
4ea780981a54c3e87aa72eb80436e02f3c6b3aaeab96ecf869602f5f5b0b546e
|
||||||
|
4f2ebb779645de5cf9d24e93c6fbd9d95d358b390dec94eed6f0223016785e42
|
||||||
|
4f2266bb0b8a9b902eaa439ae9225963d6fe4359fe30569c2d8afa926b582585
|
||||||
|
004ae946f5997471f4554ab7158a3b982b72391b59c59be40ea79b501eb1dab9
|
||||||
|
5b84763ca63a28ae79c1393d700c534e7dd6e610f890f540981e49972e3854e9
|
||||||
|
5c7dd57f445d87a39eea1e7b299e1a0acbbc946d6bfc92ba165c4af8933b322a
|
||||||
|
5d47a8950cfd56492e0f8b4ed6bfbf470a2e22811206ac0da0b79e520890b41e
|
||||||
|
5def44013bf255d4cd97c3829ac7ae895c607333a3ee586a4090bda128d379fd
|
||||||
|
5ec44fe3223976727132521b3055187e88783716a1f5fd07d1625205acff9888
|
||||||
|
6a84846fd8e476d206eafb8f5a529ecbdf6b1021daab890429077e32ef4c2b6b
|
||||||
|
6aef4d0808bef53a15fc6d19614b641a5e36c941f2729572cfef8cfc8e36bbfd
|
||||||
|
6b5a55f08cc63514e70a93a3eb5d294e9b109e990521bb0ddcea4818a8df1456
|
||||||
|
6b8acdd640ae067c2ee5235ddea8ad4c0e1eec2aaef1277b426923f56303e07d
|
||||||
|
6bcaa587af8bf282f83605713213c20275b1c8178c28de97b154af25bb6a674f
|
||||||
|
6bdf1f89b902e5a7134a87e0269229531ec6c8391256ea1bf466d2d6b66dd8c9
|
||||||
|
6bf7012e27730a047bff7d69ceadef9c06433783d0b9582faefa9c9d0f966c86
|
||||||
|
6ce965b12082d21ed8ade00884de6477faea9f97cf7a85e4b5ab3302d3517fce
|
||||||
|
6ddb2a450b19b940661c677b20936e92e8de54b8bc76f8dbacc56398d7d13a38
|
||||||
|
6e21c6398b13ebac045a11b136fbb2d289e6abe99401eed9108b5464754a4c87
|
||||||
|
6e0521e4314af444375b56e348a5e3d34aceb88d2d5e2f48b6986c27250e395e
|
||||||
|
6f19aa563c4e3e2000ed997157bc7f5bc8a45ee1d77c2cdcd1fd239117a62705
|
||||||
|
7b5530f7f440ff6477f2ee5885b37ff9cfbf933d7908225998875b233b4d2a2a
|
||||||
|
7bfe063b0155297f5bca22c7ed02ae7ef9ce1b968670607415f7720e1e8649f0
|
||||||
|
7ef24d6bf9fb190db2572505b489e45731a326c5c9c4ae6e36ffd7874560bbab
|
||||||
|
7fc41ddc47424ea11d4940f7062cdae69b92af645e6759f8759f47b689a19cb7
|
||||||
|
8b44a3824395c930d0db784e5650ba7b58663090c3bd72a44f9dc716008a9642
|
||||||
|
8bdcbc26d645ae3bc60742debe32650510bcfb7c69ad4416688af5b1815e5645
|
||||||
|
8c2a7cf3f809218d20576368d4d419a63a07a89c3b12cc389fb2f3424fb45b73
|
||||||
|
8c4d45c90f336c0ffd5d619765d8d2a45e802d619548f187bb3f2672b3534969
|
||||||
|
8cae3c834dd9284d1d0069de042c2d50e49a138a78d93a59eb752272e4a5e4f4
|
||||||
|
8d2ab3e77a72ebd708630e22f2eefc0a66f73cd94ed2abbfc4f666c504a3f172
|
||||||
|
008dfd2a2e98c6b6673ee471d72b9e150cfe02859eb3a3bbd31a3c582b5a3a46
|
||||||
|
9a2e6958ba1ea4443ade7619d326c84c50dbc444810b47904ff625969725a091
|
||||||
|
9a86729a66d00d8be95f93062381b9bd91b16f94b654636681553fec6c7c7985
|
||||||
|
9abdd18e6a7b07800067237c93d48448d49d9ad2388a91d35653323c2bed4389
|
||||||
|
9c59fdcefb7ed258e615f8cb688334a69aebdb8e38fd1075a60d6fd98167065a
|
||||||
|
9c033896f0b00023816a49cea882a342c9b62ca4528640969a0d22c21f507505
|
||||||
|
9db05859a329f9544f5a48cbd987f02fd00e48f160dcf6ee8331d8accfb423b4
|
||||||
|
9e9fa98cb2344c5eda9bae7417557f2bc2503de1eed8219fdc2d2d378e1883fe
|
||||||
|
9e06437718bd8f9a27240eff7b425406aae50f061ea47a6991b9830ee2af9f59
|
||||||
|
9ee6f261789b97ea0887a6a905bf5bba61a7070cc91ff64cda0091923d9cdf25
|
||||||
|
09ea67268ce8c0442357ba316f6431cd96143da6bb54008d3c342be8b6ddb169
|
||||||
|
13b744ec13f00bd3e5e2ecaa37bf5385af38cf0ff94fd05e660cfed3339a0c05
|
||||||
|
13d76d014354ffb4e5aabfe00eeed5206b98c502dcfd515545fdc7210316bf61
|
||||||
|
18e51d43a77ac282e108fd8a8e55207282cdabd2219af1e47c2c214117179f51
|
||||||
|
20ee459190ea3442ab7cf0c288182033fb84b75a15bdfc14540e80ab60dec542
|
||||||
|
22ab14c474cdb6cf9f873dcb6c8ce85dfc588abb46e2bd0925cdeff384437c46
|
||||||
|
24a30fa240cec28f9a8b6d02a47ad1fda39674f094ae095b0d6ec3cbf099ed44
|
||||||
|
24da0da42dc2eb9b6d35dcb7cd513b1e5c7d2ee38b6d59d9a1d51f0d01f6f920
|
||||||
|
27a56be2953f040882ecd07fd8b91937f8cb17c25d1ceed56656ed2075518f86
|
||||||
|
31f3b426743b1cff7952948995915586480486ed7fd06eef3379cf54b8fb3d9a
|
||||||
|
38ba44b0ff38f8df925512a477b8b5b4cba216a8ec0bd51b7463a86a88c42fea
|
||||||
|
43d6928bc4bb7a1e097122424e35f22393a0ccf2e03863d7dc3cce44163c8ae8
|
||||||
|
49d0cebfc3199cef395ae27a6ffa479340c1c2f69da9248a777d7bff9b6b9ca9
|
||||||
|
49dbe15539d7170822958bb0699d140c4d493215236b32679340588710d5452b
|
||||||
|
054cfa9b87316799ecd24ecfb480a38aa04b5f85a478a3db95c41706e5b63523
|
||||||
|
57e9dc8d409d939d877a513023c95146400dd10078919ab9402d8144d5f30ca0
|
||||||
|
62e86d5d6aded4a8ba536696c0df0f0af89f2cd7b4b893a7383e3abc00eeadc5
|
||||||
|
63ddb5c60e6cf39dea412903cf1d6c0164a1146bcf4ab98ee94c9b1f9e45c6bb
|
||||||
|
63e2655474549ef03aaf7d9d741502287d2f01170c5ca008d7b09b8e00b1be66
|
||||||
|
67eff4c5c60d493dc2a9f7841f7f138b610dc389c32793a27a62ddc13d417a1c
|
||||||
|
68d18a8d3519b6ea6928bbca40f9db7500b700f66e6eab236e7c32a214de2cc7
|
||||||
|
68f83c9613471fe0e046a1f9252b71e86aa50d36315c675e92dc44a7cdd0267e
|
||||||
|
71cb6156444948894aec343c9a2c0c7674aca84d292d090c7f786d13146709ed
|
||||||
|
72ed18f5962e48dd5fb54e7d91d2fd0479017c27b2c60fc58a5d6b6e55242757
|
||||||
|
74c6b39b7b926dd56db16f175026c2bf9e80a912916f273c122227672f181d03
|
||||||
|
075b39bc7ae8c56a6d99923e4938ddbbb4174df6f0e653abf9c99b420b401a31
|
||||||
|
78c01e39d1997eb54344f02f14dc02c7230cfb14ba58abc1a11feab2e9af85be
|
||||||
|
78e7acb9228c1df4ca4c706d459b97c8e79e8399fe06476ce72bd919a3b2b77f
|
||||||
|
83a8287d1e8700aa7e207b2c617a68557ce6f83f373f2c3c6d1443ae50032826
|
||||||
|
83bc388f3ae110e72ba51a8f0b68f7bbdec3685907c1ff08095b63a1759502af
|
||||||
|
85d6f7309436e59cce9e373b0aa1626fa5e12d8d81163773f5f2c45da9d56543
|
||||||
|
88c6573974e81f1fff96999bf54b80c66bc6f1e9af6746b3cd6801616198e0fd
|
||||||
|
92e2e73e3aef6e5330d2ce8aed3c58e7e3632a6f191f2b03c997ccabd245bb65
|
||||||
|
93c2b9d0a1f54c2e6355d206d93bf2309f81076628aae9d868406c288b0f6bf4
|
||||||
|
98a80f5f7903ef45f92f74a91f34dc01d527f8a2fe98b4fbe76711f5259416a5
|
||||||
|
99f7aaac2646c66dd0eeeebc157b8563b56299c27ab4bf64e13920d089ca7156
|
||||||
|
104e16e4796c52fe212c04b06417161e0bab20fc01632bfcf71421b73b979fbc
|
||||||
|
122fe66bacc9f04b3f3e4f6d0a3b15abdef8321eed36637a1284e3bd145463b8
|
||||||
|
158c590779b033e48b5efcf895fd0bad4c6f277303d72b963cd0802ecac1f440
|
||||||
|
218a186ab13a38d514cf9f523e5798452ae4e5ec76ecf26fe1a46ae22b47a508
|
||||||
|
283e9d53742a13c471e5f59f53a57c179d461f0017ad5f8db27881cdd994cd04
|
||||||
|
305b75817cf5c06105b19dc8c56ff0d796dc6e267ae5bcfc6018aa3d12037744
|
||||||
|
325ec2aa2a6c307bb82d4dcb600fa3ff127aa6d1f35dc683ab61124ef5f0f5b3
|
||||||
|
338f7535ae9ef91754e76604238ca7652273ffce9e6f0a1300de0b1edb61da77
|
||||||
|
343d41a3b38b5a7eac10750eb0213b81780edd3075e9a1fdf3a2c479f4d2474c
|
||||||
|
346eaddf98e91d6d6e7b8c526074c27847e4a18be01b41027bec93abc2b7b2fb
|
||||||
|
355ba69bab94a95e00e87c6c4808587881b9675699083e087917ec8492bebc65
|
||||||
|
426ccc08685def1c9025c94d343de0e59713073984634ee226e69634d6a18fb8
|
||||||
|
552a4b340eb6be03c188cb17628aee0f916b544142465f8e52e7e9d2325206ab
|
||||||
|
556c805068afd3f6784bda2eb59642d7dc03756580e959e1c75262711a30d4d9
|
||||||
|
0642eb9eee222e235de9310b9bd86b42493e5f4d71d430619b5832484dd09cd6
|
||||||
|
645d70ab53013564fd243eae61db209e3793b826a9eeffd77395b18e27458789
|
||||||
|
670e6447721f667df5222cb06a68c2682eb92803184a44faead8aea0dff44df6
|
||||||
|
0717b5f7615c345fe47ab8212c06f506e08585551d93c6a8083b56a216adb71b
|
||||||
|
734e6b63f955d0b508dea39a8637755e75fbb353d54430956a0010f607eb727d
|
||||||
|
748a1560ef6216c8bf80545dbeadd29f99fd5c984b6f34bb678febf8b369adcc
|
||||||
|
748cce6aa136bdc0b3111c73a7c9434508200f8c231933b925d42b37b9cca17c
|
||||||
|
760cd2f9203dc3dc3f933e4a34bbaae857dee0afed66e727846a4c7f750ab5a8
|
||||||
|
762e15e05b2f84f035a7011e10c64558dc6637c9fdde1846dc77f4056f3962d2
|
||||||
|
787dc06078f3718da6faaba82269a7d21b213482c9e45025315bdb6cf59010b5
|
||||||
|
866fc46ac37407bbf74761cecd21fa6322c4e7621f3dfbe963a85671d5753e5c
|
||||||
|
921d92a1c692ea0fd42cbe500bbdb1f3defcd5fb9b51df216451aa260d95ad45
|
||||||
|
932bae2149e79ef113eb4929614a895c642960160d5f5cb2f2e9afac7dfa6af2
|
||||||
|
968a5d7a54fb1d8ef4163f287e128cbd541ab46a2cc54db721dd3eafe1397b65
|
||||||
|
991a5d344112325fb2c40700cd0ee96a784ce2150cba0a7b0be3ed6eb2411878
|
||||||
|
2469a8c88a452ed226dd6d1082f0cbfbe3107d76f919b930b6bc7ce3b839ed90
|
||||||
|
2634a60c3c37d5eba4be4827c08c0e99a951b934604c29d116fcddc27808e480
|
||||||
|
2788da1d8f917a9d5b8acab2c7fe7336b6a7810f1c7edab63ed327a6022ceddf
|
||||||
|
3000c7863b2e80a72329ca1fa0dd03d2fd3994a2391e82450a8c92199cbb56b6
|
||||||
|
3461d48cd9e88caf5363b0e4d96e799f6759ab8b8bc6ba4c148492a1637c01f0
|
||||||
|
4681b8c67ad20fdd3ad04db895084d37a9296ac76b74ea7bc9465bb3ac93df8f
|
||||||
|
4690cfe7c61625190732c028ad4b918930a65e295005171d48114aa8f27b567b
|
||||||
|
5206b60d49110c01e05dc24034e12b767efa739e2ad21c252ab83db47e56d526
|
||||||
|
5234fcace55ee7e44631499ed1f95fd0289918a84bd9f5cf6d9f761d43c34f17
|
||||||
|
07402fbe22b572302f3317958f8cc4c6ca30317e0ee5bbbe9ff51c60b3da5a05
|
||||||
|
7458b4c4b975debbd2bb705e9575d2d1ff5e9308ed9e70c302f5643136cd0cc0
|
||||||
|
8097f97f4ae00b654b1c0b4408ad585f5bfce7999e482f244bd698268e1f6061
|
||||||
|
8158fe1f757d26cadb484ed9690ad4d20d4cb382eeec7c9b0627b365afd6576f
|
||||||
|
9917d608ac0ad05856d31873432d641828ef519825a3f451e6a1ab2519dbf0d1
|
||||||
|
13175cc97740c834e2404c11ba9fa9418d0c390d27a87e6e869977d6a86c20c0
|
||||||
|
36571aa758bc335acc463c3ea074422a5123c309a37e1a7fc818a2069941c493
|
||||||
|
59438a5f6378d218a73d3106ea1984d1c160fe4fcbbb7ec1177067f17430dcb3
|
||||||
|
60496cd3a1e1ad1dcaed30427b70a57b8b14a122bae7f2cd882f8154a4549978
|
||||||
|
61544bee8e2c1cd65e999121bb63011a9c1b4139b20b04fa32e3a850ca8c82b7
|
||||||
|
84046f48a922cd337284bc2448a5d2682ea6a2e3479e71531122aff9ee4cbc93
|
||||||
|
86022b0cd1043facf7b1c933c3b6a6ff36277a76d50bdb2da6a5690d57381b9e
|
||||||
|
98129d711a1e807ab7c3ae4c97187ab8505cebf1012218e8530d0829a2ad436f
|
||||||
|
177451abc7ed0606468250a7793e018725e8c7f370ee8937d4941a11571dcb9d
|
||||||
|
0217869eb1be5181b7053b993443dac73d9cd0804cd5bedb604edeb43f911702
|
||||||
|
219118eb01fa7be03fcce2124b74708cacafe66aea617fd2bafa32f0b2d4fd80
|
||||||
|
394985d0c3db0791b8c7569fccabd36959d00deb285075f9852568c4c81d32eb
|
||||||
|
722380f945884d4dea921ffb958513e02a4802483ec0886712e5ed3fa28d751c
|
||||||
|
774878c58b6a8cae6fbecdf656f2c1f09f37a89190068663e923763dbd6969ce
|
||||||
|
925226a74657c7ed0fcdbcfa9b504a47f7e21d0982c44f7ed9576140fb417146
|
||||||
|
963079a9efb903a575b363b27e15927312569528b9066511b4c282e01e1ec931
|
||||||
|
3830913f0478ec59a7a31d538f858f8ef612e1b2ce3bb9058a2d18f7b43abe9c
|
||||||
|
8579968bcd362a1aa9dcf14e22956b2fd15fd74a2fc2e7369ffb1457c2ce6693
|
||||||
|
8678479fa55ea3871598eb6914eb8428ba599da4c619c62f231e6931f0cb1f54
|
||||||
|
13694424a3895e8e73cc1e06af29aae0c3179c3cb5587b9daadeadcdb29b7cc7
|
||||||
|
67711784cb23efbc0762e83f03365b837ec5216c38f3ba682a82eaff6a7863db
|
||||||
|
92236992ea73ecd9befe3330e88e96a7aedacd9cbca0bb8c32c31d0e2c914e5b
|
||||||
|
134995217d29dc47fb0ca1a274ceb5ea74225d36a98c4bbe6e07bf8c8d6898c7
|
||||||
|
481706314b835ab042583c3107e168932844a162a2adf3b9ef2fa27433a7fd1d
|
||||||
|
4098005900d38c71b152787ce4bdedfb8e2b2b8b061f87645b2b4a14e0004335
|
||||||
|
4699188665aa6eb965b7bb9f4938187f377e7a4ad55fc97e44802ea76f3612e9
|
||||||
|
04762469942ae24c353f929fced777f245bb9f2f888a9f2044ff27fc45933665
|
||||||
|
a6c297b9282b39caca7e513724219af096a3cf6fd453c3736ee7c4c833ef417d
|
||||||
|
a22cb584ba3444bc1dbd90b3723b1ad9424352251f2218a92b1884ef74e63b9f
|
||||||
|
a68fad1bcb1f1030af3811590cc18d74eb1da3c67190af4459fa69118aa8f035
|
||||||
|
a74b9b2333ee936c17edd0525b49fec1337768ccab68f5576a39486c875ff968
|
||||||
|
a74b41238ed9ded9e866679ab00f85813d800f92e4bb2642d3f9de28a1655b81
|
||||||
|
a84ff849188b2bab211fefe8bc61a70f58f48f6a2a6ca25c2645e0bb5c384cbf
|
||||||
|
a2815ec8af59e0a137aa0cdf2baa48ca6d1b86e291c5728fab9449280d15fa75
|
||||||
|
aa5cf2d08c68b8d3dfe3330861935fd97ff8fb9169645f6c632d970ad71b575c
|
||||||
|
aadb8f0be78890f1e08630b7a762a82853dc1402326bd16cd2a2844fa63fef1d
|
||||||
|
ac48d92438e8d19cf546764cc051172af82092223daad8ccfbfb4cb05100f5d6
|
||||||
|
ac78e0a5ac10c7d0239e0e1ba3b604cd6fd418d87a6f32b66f3ce98c674164d8
|
||||||
|
ad79ebffc7d8225cfbee6ad5fec4092ba7376f38dca912d6df8ea932f1eba20b
|
||||||
|
ad7084f002b17a1a27f70c796e3d7424a064b2ef5b71eadb7b3b08dcc9f59042
|
||||||
|
ae2a94eac2960e95f944c9a4a0649c94c405fb0e518adfba2e6fa0f80f2a5af2
|
||||||
|
ae9d5c15cba1adf881044a4449eaf72f5f2ed753c301edc1e43bcfa4560e8d31
|
||||||
|
ae5499adb73000b6f460e63d6e045fe492702bd73187f868d9988690cb72a453
|
||||||
|
b0e341cb8fb7f2fb42f7a8512a9b922591cfa40bd3522f31a422a9fdb4586008
|
||||||
|
b4bc9ff2d9b533e896dfc5082ba865a4ea7676d71259d73aa33d515a5a60dd91
|
||||||
|
b8d2fbe7e6afc931581bf5f1edacaefb9071bcc9f709e294a3f5ead104f73890
|
||||||
|
b8dd8697ad52f98dc44ed3c940752aa52e16d6ab7563d36b1c9e221db22d1e11
|
||||||
|
b21ec6b73a9f1f51623355f2bf8229b5f6f9da1484e1ce105cf8e0363475a5b8
|
||||||
|
b081c3329ed821e22b0ee9cecbc2f52a5e9be70be2b1cb83bb262c21c84640d1
|
||||||
|
b274ece5f6e1d506ca0646077964797c3e7b4413c9a9f1e453d9257e6b4d09f9
|
||||||
|
b676fcf4f5272015b942c8589135a18fa16204c5aec457f1d0898ce6c84638eb
|
||||||
|
b1591e2a2604b6404d0d18e4b2b7599aafbcfb2bbacba06b5de5ff14abf14be3
|
||||||
|
b7000ea518cd327f7dfb42858434fdb80c451aecf0a0325b6ccbff1f30545803
|
||||||
|
beb17a9d330ce361923600aac8e13f8b10e62d0b4067380ad7a95c4d77f0110e
|
||||||
|
bfb91f5666855fd57228801ed0a4294981bd2a6665a93d1093bf3c4a0e89cb2c
|
||||||
|
c0fb7358aabef2c38e6921e2ed06b7fdddddecd29d530814545968323eba964e
|
||||||
|
c1d4fc53203c4b65abb1ff76f9505a46175c57afcd9c95cf445c6a3c443839fb
|
||||||
|
c3fbcb924a928bca859b69c75e4df275eb5e0abf03029785247313b0fa1cdd27
|
||||||
|
c4bccf1c99f4990b0a23848b6fa58977dea0bdbf6ab69b7c0c2b90f7f0a2f5c3
|
||||||
|
c5e875c34f4e4f1fd4edb1c6dbf4862a6543e26387ee480eeba0f5e3f44af7b6
|
||||||
|
c8bb03639a4d7c41c233fa28d5a1c06dee874fe00614f764df9db3df14da86b5
|
||||||
|
c16aa92f949bb89a034c6ef22ea8ece859883ab309086a204e43cedced8dd059
|
||||||
|
c31f2070f92b36708a62c1d53a7aea495fd997440f2c01c851ad83c66357e379
|
||||||
|
c35e0b961694a882dcc4c7bdb8e657d4b0685687cb5c4469559ce5078c95302a
|
||||||
|
c56e78b468dfe3991fc787ab7fe30b4282a73d7fd2e79e4dfc9305bb064685ea
|
||||||
|
c67a888239bc5e11ed44ca31307f3b46f7f76698d706ffc764024110d1a69ceb
|
||||||
|
c424d546895a59e0f88e2296a665c03f4618ac304ef19742b6f1c543665b9bb3
|
||||||
|
c716ebc8ba8fa1556e1dddfaae98c7019618d39ece0e78dbd1b9f58dd0760d94
|
||||||
|
c797d99e68b0efce9b1337dc0e3e9cbf42806bef03021ade362cb4e4d1de4222
|
||||||
|
c5708e0a13ababf73203b1e297b3b73229cd6370e3fec1653b7cf1dc143a0d0a
|
||||||
|
c52417cfa3d923ce7168b7c2b082eb3ab31bc1750ff3f3edfc2c1cd59fb74f3d
|
||||||
|
c53655b36c7abc28db3020511a0ebdffe8fa37d86412753fca86f4caa7ae7a6b
|
||||||
|
c782734d8be184e0721d0a984b6ac3e8e0052206a9e46d22188a7fb2f923face
|
||||||
|
cb71ab036375acb25c69afd915261acf65f919008fbd59417989f6ebef110c7a
|
||||||
|
cb613ceec1bf86b718059ef3e864dce7ed60e5b6a7bd5ab9f7c633350df48f3a
|
||||||
|
cb10732ef556d77ec0a0a5d7b29c0789b47a9757b67bdb3fa9d5be22809debd0
|
||||||
|
cf799a4b43f87d63eba6042a73b9eba3f17b354be1fd2d94dfac2048e13f3fce
|
||||||
|
d3c941cd28b5a692c2532f1381afdc57740d603fbba5eddbe3539f561f26a0ca
|
||||||
|
d5bd7d95fbb1f56d7617d96a5a6deb65417b4448692cb45f4a0bedd824165fc1
|
||||||
|
d6a74fc39944848c79012bb06d915c609b92c1099881b90b4cda59074bd1124f
|
||||||
|
d58b42f643175b6272bb579d354a9510bc34fc88e4e70dda08346997ba3cec13
|
||||||
|
d256b9490d623a136093e459b9f466dedadaa0d00235d6934044e6591ee7e2c5
|
||||||
|
d352bb11678cf6cb47a16740ee9b79cfbf4c19d4da23f4fb95621bbdea6b91b2
|
||||||
|
d053441be51f569ac393c26c220b6369ae3455ae76f3627eba223e8b4e40eacd
|
||||||
|
daa717bf6363a3afe8dbcd3ab863a81ca737e0999242bd99c618daa0e4fa085b
|
||||||
|
db0c37c3ce9df2f2090c622eb4194e6860e26846398d4b4a092b9c4c21441af4
|
||||||
|
db2434b8bc6fb7858aa5eea51b23eb3fe17ed8e8ac025596ebb4f857d00f9ba9
|
||||||
|
db6910df9d6429a8ecffced74f22c26f45a11d48efea51e48e8097bf43331540
|
||||||
|
dbcfeebe1162c457fdf6f7cbfff66c971dfaca8a1a8ae2fb5472fb9007f83ddc
|
||||||
|
dc5c95c9a8f2f01a8eab63c7d591a7d008dbaf5f4b6157380faeeaeab168cd59
|
||||||
|
de81766fb8e806f93c97771ae1a0ff529831bc5e7f50e2187d41dd606b8e23f1
|
||||||
|
e0e25d3ed1797b0092c897ad904a4ac7a2bd77e12b96e1790a939aa009851803
|
||||||
|
e0ee75c206f1b9be41e608c6de05f8fad8532e3bda240cfad0640c673ae46395
|
||||||
|
e1ae949c5339eca43334dd6720b49df19082036b8f53151b8dfac97f2d2c0b6d
|
||||||
|
e2ca0579185aa6d7324f144357323b4d5521c1e94e8dc7929f0ffa9a73e1e4cd
|
||||||
|
e5f9a9a8f7e4b6d097aef1b0493180a547dbf19006920433e7b80aa8aac0026b
|
||||||
|
e6a40ba40f9ac5f43ba2276b3cb6110bb4c0e20003e866a55ca28c954ddd0e9a
|
||||||
|
e9e1b0601616187e8934c7212e2fde5b590b50308aa9c8c5dd12e419c6bebc18
|
||||||
|
e45c71e0680d2856e2945881fd01ad6320868e0a36427c6bca637c53dfecc625
|
||||||
|
e62af03dd8d599dfbdbecb8c6516d95e582ee803f2ddd3798479fbc769229030
|
||||||
|
e72bfb25201770edb1889f678f5a844a57c9f5308acd57a86c5f4936b800d441
|
||||||
|
e311fa91eca24a97e0c558e91d06dca353aa9348a7ea4b111e694e181868fc27
|
||||||
|
e339a495e0c048e30483e4fd3bef66186712975f3c33c90ef63a3c9686da3160
|
||||||
|
e668b1c1f44dae28ca34f530e313c1e544751b63bf0c480300be8e242683fecb
|
||||||
|
e4638cbb5b12548a76a2cb6e87c7f2ada75d61d3c7d1062535ef0bf2b3cde05d
|
||||||
|
e7945baeafe0ac6d9a1e6761786c35fddad76c81d86e544d369188c186a0389b
|
||||||
|
ea35c704202ee36415d4f7a0873bcb254d30268450d7c9ed2dc611c1e2f4d5d1
|
||||||
|
eaeb108b23d455d01ec0c9e44d39ff395522942794d26f3c149e29615be9bf1f
|
||||||
|
eb420ae07f9c459abc101c1264fa8cf779002ef20748ef97abe7c3764c24f032
|
||||||
|
ec30ca07d0b0af16afe60f217dedf15573045ed9adc45e21c3592ee6716fbe39
|
||||||
|
ed02b9c56e48d34910bc97ebff834f1cb082c23ebcae97b0f51f3a2633664466
|
||||||
|
edcc77bb3d279e3dfe7d687fe30e22f95f5d9391b9908e61b2206a6cf309ea91
|
||||||
|
eea963e8499939d36d09e42d8d5cd86798e81e264c4d4c012191e42926e508b8
|
||||||
|
f1b01ee5efb2117033ddcad5f5eafe8e6f65ad6fc3e933902bfc494c13a7d3c0
|
||||||
|
f01b151ee48f9a3a631457069ece3a2ac3e111df28110f3c06f221ee7c54eff8
|
||||||
|
f2ec30f9f836ed26cb13fdb80064d1dcabf953d1952fea2d85bb6410f4cb2271
|
||||||
|
f8a87cab6bc0890c7a2e36f73e533eac1a1cabb13b886b8ed71afa4c8b7d90be
|
||||||
|
f9b0fe025aedb49e94d80119a8a8dc0f85800bc592e568dc95b0f1e5439c54fc
|
||||||
|
f56a1ab986e83e118ff174e9805f6151e47ff693c17a2f475fcedd83cb98e051
|
||||||
|
f82a956615af08264b64e051e09c1441152d92371d4d2db8d5c0ac4ab47a7182
|
||||||
|
f287b03a768126d72af210dceab12e329573779e7aad56c123ac0dbb14fc4f4e
|
||||||
|
f347e779fb8c09daba2a757dfc3ee7c415f8f5a2eaace19535d363cc3ab3171a
|
||||||
|
f547c232f5e1785aa11200ac03dcfda6f056400bf1456a3388db6d73a8158ca0
|
||||||
|
f6822a239614fc82462535db5b2806dd616696c9552e1d469922f4b7cd2bf56e
|
||||||
|
fa453f6fb451a564d31e704bbca0bcbcc9ad1109a25a8e5ad5c704789a424c4b
|
||||||
|
fb5ad60698efb6ee9f45fd440e56c060ba01ba03a854f556e42f8090b2057c7e
|
||||||
|
fbc407de7b12a4975b60dac518ac287f076659213cce708871162342419ab383
|
||||||
|
fbf633eb758b7b66ab761c34eeebc759791e2f73020e839b3cfc2d692c9ba21f
|
||||||
|
fc1bddcbef8c02474db71bf11dace70e1e51d94f3c3fe6d9d45f4a669ac4a9fd
|
||||||
|
fd453dd47dc82ff7625bd4093c62140fe14a348814f094b43b519f7f59293af7
|
||||||
|
fdc05b3bfda3101bec7d66b5e0cfb589e06f1f4b4d6b2f9d84dac470b0abf53b
|
||||||
|
fdebde6f8bcdd5281c50bcb4669c45dc54e33563bec872f8d23d76d2fe3ebfdf
|
||||||
|
fe503d3e9a34b113d4c56b59c5986e13c44b6e401b6e6681eb62712d24368329
|
||||||
|
felhom-controller-data
|
||||||
|
filebrowser_filebrowser_data
|
||||||
|
immich_immich_ml_cache
|
||||||
|
immich_immich_postgres_data
|
||||||
|
immich_immich_redis_data
|
||||||
|
paperless-ngx_paperless_data
|
||||||
|
paperless-ngx_paperless_postgres_data
|
||||||
|
paperless-ngx_paperless_redis_data
|
||||||
|
privatebin_privatebin_data
|
||||||
|
ls: cannot access '/mnt/felhom-drives/scratch_hdd/felhom/backups/primary/': No such file or directory
|
||||||
|
Filesystem Size Used Avail Use% Mounted on
|
||||||
|
/dev/loop1 69G 11G 54G 17% /var/lib/felhom
|
||||||
|
/dev/nvme0n1 938G 210G 681G 24% /mnt/felhom-drives/scratch_hdd
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
10:10:26Z == 1 deploy
|
||||||
|
10:10:27Z deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||||
|
10:11:12Z state: running
|
||||||
|
10:11:42Z == 2 marker A + backup
|
||||||
|
10:11:46Z CREATE TABLE
|
||||||
|
INSERT 0 1
|
||||||
|
10:11:49Z [2] DB markers: 'A'
|
||||||
|
10:11:49Z POST /api/backup/run -> 200 {'ok': True, 'message': 'Mentés elindítva'}
|
||||||
|
10:12:59Z backup finished: {"db_dump": {"count": 3, "duration": "1m9.055678483s", "last_run": "2026-10-20T10:12:58.690358712Z", "success": true}, "enabled": true, "running": false}
|
||||||
|
10:13:04Z unit on disk:
|
||||||
|
UNIT /mnt/sys_drive/felhom-data/backups/primary/rallly
|
||||||
|
2026-10-20 10:12 487 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/app.yaml
|
||||||
|
2026-10-20 10:12 4071 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/docker-compose.yml
|
||||||
|
2026-10-20 10:12 11894 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/.felhom.yml
|
||||||
|
2026-10-20 10:12 52228608 /mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar
|
||||||
|
2026-10-20 10:11 111371 /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql
|
||||||
|
2026-10-20 10:12 3195 /mnt/sys_drive/felhom-data/backups/primary/rallly/manifest.json
|
||||||
|
2026-10-20 10:12 1191 /mnt/sys_drive/felhom-data/backups/primary/rallly/data-stamps.json
|
||||||
|
10:13:04Z == 3 POSITIVE CONTROL: marker B (live) then a normal unit restore
|
||||||
|
10:13:09Z INSERT 0 1
|
||||||
|
10:13:12Z [3-before] DB markers: 'A,B'
|
||||||
|
10:13:12Z [3] restore points offered: [{"time": "2026-10-20T10:11:50Z", "short_id": "helyi", "tier": 1, "drive_label": "Belső SSD (rendszer)"}]
|
||||||
|
10:13:12Z [3] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started']
|
||||||
|
10:13:12Z [3] restore-status: {"running": true, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:13:12.817369857Z", "last": {"op": "offbox-reconstitute", "stack": "rallly", "ok": true, "message": "A(z) rallly: 0 fájl és 1 adatkötet és az adatbázis visszaállítva (mentés: 2026. 10. 20. 09:30) — az alkalmazás újraindult.", "finished_at": "2026-10-20T07:36:57.852528534Z"}}
|
||||||
|
10:13:40Z [3] restore-status: {"running": false, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:13:12.817369857Z", "last": {"op": "restore", "stack": "rallly", "ok": true, "message": "A(z) rallly: 1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-10-20T10:13:39.161573115Z"}, "last_recent": true}
|
||||||
|
10:14:00Z [3] after: state=running hold=None
|
||||||
|
10:14:18Z [3-after] DB markers: 'A'
|
||||||
|
10:14:18Z == 4 marker C (the newest live row), then THE FAULT: the unit's DB volume tar truncated to half
|
||||||
|
10:14:22Z INSERT 0 1
|
||||||
|
10:14:25Z [4-live] DB markers: 'A,C'
|
||||||
|
10:14:29Z TRUNCATED /mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar 52228608 -> 26114304
|
||||||
|
10:14:29Z == 5 the unit restore over the damaged copy
|
||||||
|
10:14:29Z [5] restore points offered: [{"time": "2026-10-20T10:11:50Z", "short_id": "helyi", "tier": 1, "drive_label": "Belső SSD (rendszer)"}]
|
||||||
|
10:14:29Z [5] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started']
|
||||||
|
10:14:29Z [5] restore-status: {"running": true, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:14:29.288204925Z", "last": {"op": "restore", "stack": "rallly", "ok": true, "message": "A(z) rallly: 1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-10-20T10:13:39.161573115Z"}, "last_recent": true}
|
||||||
|
10:14:44Z [5] restore-status: {"running": false, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:14:29.288204925Z", "last": {"op": "restore", "stack": "rallly", "ok": false, "message": "Visszaállítás sikertelen: starting rallly after restore from unit: starting stack rallly: exit code 1\nstderr: ...twork rallly_rallly-internal Creating \n Network rallly_rallly-internal Created \n Network rallly_rallly-internal Created \n Container rallly-postgres Creating \n Container rallly-postgres Created \n Container rallly Creating \n Container rallly Created \n Container rallly-postgres Starting \n Container rallly-p
|
||||||
|
10:15:04Z [5] after: state=degraded hold=None
|
||||||
|
10:15:38Z [5-after] DB markers: 'Error response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running\nError response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running\nError response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running'
|
||||||
|
10:15:41Z containers:
|
||||||
|
rallly Created
|
||||||
|
rallly-postgres Restarting (1) 5 seconds ago
|
||||||
|
10:15:44Z db container log tail:
|
||||||
|
Data page checksums are enabled.
|
||||||
|
|
||||||
|
initdb: error: directory "/var/lib/postgresql/18/docker" exists but is not empty
|
||||||
|
initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/18/docker" or run initdb with an argument other than "/var/lib/postgresql/18/docker".
|
||||||
|
The files belonging to this database system will be owned by user "postgres".
|
||||||
|
This user must also own the server process.
|
||||||
|
|
||||||
|
The database cluster will be initialized with locale "en_US.utf8".
|
||||||
|
The default database encoding has accordingly been set to "UTF8".
|
||||||
|
The default text search configuration will be set to "english".
|
||||||
|
|
||||||
|
Data page checksums are enabled.
|
||||||
|
|
||||||
|
initdb: error: directory "/var/lib/postgresql/18/docker" exists but is not empty
|
||||||
|
initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/18/docker" or run initdb with an argument other than "/var/lib/postgresql/18/docker".
|
||||||
|
10:15:47Z any undo / safety copy of the live state on disk?
|
||||||
|
/mnt/sizing-bench/backups/secondary/paperless-ngx/recovery-unit/db-dumps/pre-restore-20260927T094411Z-paperless-ngx-postgres.sql
|
||||||
|
/mnt/sys_drive/felhom-data/backups/primary/rallly
|
||||||
|
/mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar
|
||||||
|
/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql
|
||||||
|
10:15:51Z controller log (restore lines):
|
||||||
|
Container rallly Created
|
||||||
|
Container rallly-postgres Starting
|
||||||
|
Container rallly-postgres Started
|
||||||
|
Container rallly-postgres Waiting
|
||||||
|
Container rallly-postgres Error dependency rallly-postgres failed to start
|
||||||
|
dependency failed to start: container rallly-postgres is unhealthy
|
||||||
|
2026/10/20 10:14:42 manager.go:1362: [ERROR] [stacks] Stack rallly start failed after 1.1s: exit code 1
|
||||||
|
stderr: ...twork rallly_rallly-internal Creating
|
||||||
|
Network rallly_rallly-internal Created
|
||||||
|
Network rallly_rallly-internal Created
|
||||||
|
Container rallly-postgres Creating
|
||||||
|
Container rallly-postgres Created
|
||||||
|
Container rallly Creating
|
||||||
|
Container rallly Created
|
||||||
|
Container rallly-postgres Starting
|
||||||
|
Container rallly-postgres Started
|
||||||
|
Container rallly-postgres Waiting
|
||||||
|
Container rallly-postgres Error dependency rallly-postgres failed to start
|
||||||
|
dependency failed to start: container rallly-postgres is unhealthy
|
||||||
|
2026/10/20 10:14:42 handlers.go:1881: [ERROR] [web] Restore failed (async): stack=rallly: starting rallly after restore from unit: starting stack rallly: exit code 1
|
||||||
|
stderr: ...twork rallly_rallly-internal Creating
|
||||||
|
Network rallly_rallly-internal Created
|
||||||
|
Network rallly_rallly-internal Created
|
||||||
|
Container rallly-postgres Creating
|
||||||
|
Container rallly-postgres Created
|
||||||
|
Container rallly Creating
|
||||||
|
Container rallly Created
|
||||||
|
Container rallly-postgres Starting
|
||||||
|
Container rallly-postgres Started
|
||||||
|
Container rallly-postgres Waiting
|
||||||
|
Container rallly-postgres Error dependency rallly-postgres failed to start
|
||||||
|
dependency failed to start: container rallly-postgres is unhealthy
|
||||||
|
2026/10/20 10:14:49 manager.go:920: [DEBUG] [stacks] restart-policy of down member "rallly" = "unless-stopped"
|
||||||
|
2026/10/20 10:14:59 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||||
|
2026/10/20 10:15:09 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||||
|
2026/10/20 10:15:09 main.go:4054: [INFO] [deadapp] rallly crossed the crash_loop threshold (8 in 10m0s) while a backup/quiesce/update holds it — NOT stopped
|
||||||
|
2026/10/20 10:15:19 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||||
|
2026/10/20 10:15:29 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||||
|
2026/10/20 10:15:39 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||||
|
2026/10/20 10:15:49 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||||
|
exit 0
|
||||||
+18
@@ -0,0 +1,18 @@
|
|||||||
|
2026/10/20 10:14:29 handlers.go:1873: [WARN] [web] Restore requested (async): stack=rallly, snapshot=helyi from 172.18.0.3:36322
|
||||||
|
2026/10/20 10:14:29 restore_dbcreds.go:114: [INFO] [backup] Restore rallly: database password(s) [DB_PASSWORD] come from the backup, [] stay as installed (older than the data) — values not logged (R-1106)
|
||||||
|
2026/10/20 10:14:29 restore_unit.go:393: [INFO] [backup] Restoring rallly from recovery unit /mnt/sys_drive/felhom-data/backups/primary/rallly: images=2, secrets recovered=2/2, data_keys=0
|
||||||
|
2026/10/20 10:14:29 manager.go:1440: [INFO] [stacks] Stopping stack: rallly
|
||||||
|
2026/10/20 10:14:40 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 10.8s)
|
||||||
|
2026/10/20 10:14:40 restore.go:215: [INFO] [backup] Restoring Docker volume rallly_rallly_postgres_data for rallly
|
||||||
|
2026/10/20 10:14:40 restore.go:246: [ERROR] [backup] Failed to populate volume rallly_rallly_postgres_data: tar: short read — exit status 1
|
||||||
|
2026/10/20 10:14:40 restore_unit.go:448: [ERROR] [backup] volume restore for rallly: failed to restore 1 volume(s): [rallly_rallly_postgres_data]
|
||||||
|
2026/10/20 10:14:40 [INFO] [stacks] SaveAppConfig: saved config for rallly
|
||||||
|
2026/10/20 10:14:40 deploy.go:883: [INFO] [stacks] Redeploying rallly from recovery unit with 4 env vars
|
||||||
|
2026/10/20 10:14:40 pin.go:93: [INFO] [stacks] pin rallly: rallly=lukevella/rallly:4.15.3, rallly-postgres=postgres:18-alpine
|
||||||
|
2026/10/20 10:14:40 [INFO] [stacks] SaveAppConfig: saved config for rallly
|
||||||
|
2026/10/20 10:14:40 restore_unit.go:467: [ERROR] [backup] rallly: NOT replaying the database copy — the volume restore failed, so the database is not known to be the copy's own
|
||||||
|
2026/10/20 10:14:40 manager.go:1351: [INFO] [stacks] Starting stack: rallly
|
||||||
|
2026/10/20 10:14:42 manager.go:1704: [ERROR] [stacks] Command failed: docker compose up -d (in /opt/docker/stacks/rallly) — exit code 1 (took 1.1s)
|
||||||
|
2026/10/20 10:14:42 manager.go:1712: [ERROR] [stacks] stderr (last part): ...twork rallly_rallly-internal Creating
|
||||||
|
2026/10/20 10:14:42 manager.go:1362: [ERROR] [stacks] Stack rallly start failed after 1.1s: exit code 1
|
||||||
|
2026/10/20 10:14:42 handlers.go:1881: [ERROR] [web] Restore failed (async): stack=rallly: starting rallly after restore from unit: starting stack rallly: exit code 1
|
||||||
+28
@@ -0,0 +1,28 @@
|
|||||||
|
200
|
||||||
|
{
|
||||||
|
"ok": true,
|
||||||
|
"message": "Stack rallly stop completed"
|
||||||
|
}
|
||||||
|
200
|
||||||
|
{
|
||||||
|
"ok": true,
|
||||||
|
"data": {
|
||||||
|
"removed": "rallly",
|
||||||
|
"volumes_removed": [
|
||||||
|
"rallly_rallly_postgres_data"
|
||||||
|
],
|
||||||
|
"hdd_paths_removed": [],
|
||||||
|
"hdd_paths_preserved": [],
|
||||||
|
"hdd_note": "Az alkalmazás nem tárolt saját adatot külső meghajtón, így ott nem volt mit törölni.",
|
||||||
|
"userdata_kept": [],
|
||||||
|
"backup_paths_removed": [
|
||||||
|
"/mnt/sys_drive/felhom-data/backups/primary/rallly (26M)"
|
||||||
|
],
|
||||||
|
"verified": true
|
||||||
|
},
|
||||||
|
"message": "Stack rallly removed"
|
||||||
|
}
|
||||||
|
0
|
||||||
|
0
|
||||||
|
ls: cannot access '/mnt/sys_drive/felhom-data/backups/primary/rallly': No such file or directory
|
||||||
|
/opt/docker/stacks/rallly
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# C3 — R1/D4 proven live: a failed LOCAL restore leaves a broken app and no copy of what it replaced
|
||||||
|
|
||||||
|
Scratch guest 9202 on demo-hp, controller **0.331.0** (the local-restore code is unchanged since v0.328.0 —
|
||||||
|
`git log` on `restore_unit.go`/`restore.go`; 0.336.0 runs the same lines). Hub off on 9202. The dashboard password was
|
||||||
|
reset with the controller's own escape hatch (`--print-reset-code` → `/claim`), held in `sec` only.
|
||||||
|
Architecture read: `07` §6.3 (the ladder „replay → rollback → hold"), §8; `09` §3 decision 154 (R-638 A), D20 (R-893 A).
|
||||||
|
|
||||||
|
| step (UTC) | act | observed |
|
||||||
|
|---|---|---|
|
||||||
|
| 10:10 | Rallly 4.15.3 + postgres:18 deployed through `POST /api/stacks/rallly/deploy` | running |
|
||||||
|
| 10:11 | marker A (psql in the app's own DB container); „Mentés most" (`POST /api/backup/run`) | unit written 10:12 (`rallly_rallly_postgres_data.tar` 52 MB + `rallly-postgres.sql`) |
|
||||||
|
| 10:13 | **positive control:** marker B, then the household's restore (`POST /backup/restore`, point „helyi") | „1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult"; markers **A** (B rewound, as designed) |
|
||||||
|
| 10:14:22 | marker C — the household's newest row | live: **A,C** |
|
||||||
|
| 10:14:29 | **fault:** the unit's DB volume tar truncated to half (a torn / damaged copy) | 52 228 608 → 26 114 304 B |
|
||||||
|
| 10:14:29 | the same restore | stop 10.8 s → `volume rm` → `tar: short read` → `[ERROR] volume restore … failed` → definition rewritten → „NOT replaying the database copy" → start → **postgres crash-loops** (`initdb: directory … exists but is not empty`) |
|
||||||
|
| 10:14:44 | what the household reads | „Visszaállítás sikertelen: starting rallly after restore from unit: starting stack rallly: exit code 1 stderr: ...twork rallly_rallly-internal Creating …" — **raw English Docker output** |
|
||||||
|
| 10:15:47 | any copy of the live state (marker C)? | **none** — no undo dump, no live-volume copy for Rallly (`find` of `*restore-undo*`/`pre-restore-*` finds only a September paperless file) |
|
||||||
|
| 10:15 | the dead-app guard | `rallly crossed the crash_loop threshold … while a backup/quiesce/update holds it — NOT stopped` |
|
||||||
|
|
||||||
|
**So:** the household's live data (marker C) was destroyed by the restore itself, the app does not start, and nothing
|
||||||
|
on the box can put the pre-restore state back. The off-site restore got exactly this protection in v0.336.0 (D20 = A:
|
||||||
|
„put back exactly as it was"); the LOCAL path — 07 §3's main customer restore — did not. Logs: `01-run.log`,
|
||||||
|
`02-controller-log-fault-window.txt`. Teardown through the product (`03-teardown.txt`): Rallly removed with its data
|
||||||
|
and backups; no container, no volume, no unit (the `/opt/docker/stacks/rallly` folder is the catalog template every
|
||||||
|
app has).
|
||||||
@@ -0,0 +1,153 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""r1_proof.py — C3 proof of candidate R1/D4 on scratch guest 9202 (data-loss hunt 2026-10-20).
|
||||||
|
|
||||||
|
Claim under test: a LOCAL (Tier-1 unit) restore that fails after it has started replacing the app's volumes
|
||||||
|
restarts the app on whatever is left, and keeps NO copy of the live state it replaced — so the household's newer
|
||||||
|
data is gone and nothing can put it back (unlike the off-site path, which got „put back exactly" in v0.336.0).
|
||||||
|
|
||||||
|
Method (mechanism proof — markers are written with psql inside the app's own database container, as the
|
||||||
|
2026-10-19 R-893 measurement did; the household-facing acts are the dashboard's own endpoints):
|
||||||
|
1 deploy Rallly (throwaway) through POST /api/stacks/rallly/deploy
|
||||||
|
2 marker A; „Mentés most" = POST /api/backup/run (whole box, scratch); wait
|
||||||
|
3 POSITIVE CONTROL: marker B (live), unit restore (POST /backup/restore) -> A present, B gone (expected rewind)
|
||||||
|
4 marker C (the household's newest live row); THE FAULT: truncate the unit's DB volume tar to half
|
||||||
|
5 unit restore again -> observe: the answer, the app's state, what the DB holds, any undo/safety copy on disk
|
||||||
|
6 teardown: remove Rallly with its data and backups through the product; verify nothing left
|
||||||
|
Env: CTL_BASE, CTL_DOMAIN, CTL_PW_ENV (see tools/ctl.py). Prints a timestamped log (no secrets).
|
||||||
|
"""
|
||||||
|
import json, os, re, subprocess, sys, time
|
||||||
|
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/rehearsal-2026-10-20/tools")
|
||||||
|
import ctl # noqa: E402
|
||||||
|
|
||||||
|
APP = "rallly"
|
||||||
|
DOMAIN = os.environ["CTL_DOMAIN"]
|
||||||
|
|
||||||
|
|
||||||
|
def say(*a):
|
||||||
|
print(time.strftime("%H:%M:%S", time.gmtime()) + "Z " + " ".join(str(x) for x in a), flush=True)
|
||||||
|
|
||||||
|
|
||||||
|
def guest(script, timeout=600):
|
||||||
|
r = subprocess.run(["ssh", "demo-hp", "export LC_ALL=C; cat > /tmp/r1p.sh; pct push 9202 /tmp/r1p.sh /tmp/r1p.sh;"
|
||||||
|
" pct exec 9202 -- bash /tmp/r1p.sh; pct exec 9202 -- rm -f /tmp/r1p.sh; rm -f /tmp/r1p.sh"],
|
||||||
|
input=script, capture_output=True, text=True, timeout=timeout)
|
||||||
|
return (r.stdout + r.stderr).strip()
|
||||||
|
|
||||||
|
|
||||||
|
def psql(sql):
|
||||||
|
return guest(f"""C=$(docker ps --format '{{{{.Names}}}}' | grep -E '^rallly.*(postgres|db)' | head -1)
|
||||||
|
U=$(docker exec $C printenv POSTGRES_USER); D=$(docker exec $C printenv POSTGRES_DB)
|
||||||
|
docker exec -i $C psql -v ON_ERROR_STOP=1 -U "$U" -d "$D" -tA <<'SQL'
|
||||||
|
{sql}
|
||||||
|
SQL""")
|
||||||
|
|
||||||
|
|
||||||
|
def stack():
|
||||||
|
code, d = ctl.call("GET", f"/api/stacks/{APP}")
|
||||||
|
return (d.get("data") or {}) if isinstance(d, dict) else {}
|
||||||
|
|
||||||
|
|
||||||
|
def wait_backup(t_cap=1200):
|
||||||
|
t0 = time.time()
|
||||||
|
time.sleep(5)
|
||||||
|
while time.time() - t0 < t_cap:
|
||||||
|
code, d = ctl.call("GET", "/api/backup/status")
|
||||||
|
dd = (d.get("data") or {}) if isinstance(d, dict) else {}
|
||||||
|
if not dd.get("running"):
|
||||||
|
return dd
|
||||||
|
time.sleep(5)
|
||||||
|
return {"timeout": True}
|
||||||
|
|
||||||
|
|
||||||
|
def restore(tag):
|
||||||
|
code, d = ctl.call("GET", f"/api/backup/snapshots?stack={APP}")
|
||||||
|
data = d.get("data") if isinstance(d, dict) else None
|
||||||
|
snaps = data if isinstance(data, list) else next((data[k] for k in ("snapshots", "items", "restore_points")
|
||||||
|
if isinstance(data, dict) and isinstance(data.get(k), list)), [])
|
||||||
|
say(f"[{tag}] restore points offered: {json.dumps(snaps, ensure_ascii=False)[:400]}")
|
||||||
|
sid = snaps[0].get("id") or snaps[0].get("snapshot_id") or snaps[0].get("short_id")
|
||||||
|
out = subprocess.run(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", ctl.HOST, "-H", "Cookie: " + ctl._SESS,
|
||||||
|
"-X", "POST", "--data-urlencode", "_csrf=" + ctl._CSRF, "--data-urlencode", "stack_name=" + APP,
|
||||||
|
"--data-urlencode", "snapshot_id=" + sid, ctl.BASE + "/backup/restore"],
|
||||||
|
capture_output=True, text=True).stdout
|
||||||
|
say(f"[{tag}] POST /backup/restore -> {out.splitlines()[0] if out else '?'} "
|
||||||
|
f"{[l for l in out.splitlines() if l.lower().startswith('location')][:1]}")
|
||||||
|
t0, last = time.time(), None
|
||||||
|
while time.time() - t0 < 900:
|
||||||
|
code, d = ctl.call("GET", "/api/backup/restore-status")
|
||||||
|
dd = (d.get("data") or {}) if isinstance(d, dict) else {}
|
||||||
|
cur = json.dumps(dd, ensure_ascii=False)[:600]
|
||||||
|
if cur != last:
|
||||||
|
say(f"[{tag}] restore-status: {cur}")
|
||||||
|
last = cur
|
||||||
|
if not dd.get("running") and time.time() - t0 > 8:
|
||||||
|
break
|
||||||
|
time.sleep(3)
|
||||||
|
time.sleep(20)
|
||||||
|
st = stack()
|
||||||
|
say(f"[{tag}] after: state={st.get('state')} hold={st.get('hold_reason')!r}")
|
||||||
|
|
||||||
|
|
||||||
|
def markers(tag):
|
||||||
|
say(f"[{tag}] DB markers: {psql('SELECT string_agg(m, \',\' ORDER BY m) FROM r1_marker;')!r}")
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
say("== 1 deploy")
|
||||||
|
if not stack().get("deployed"):
|
||||||
|
code, d = ctl.call("GET", f"/api/stacks/{APP}/deploy-fields")
|
||||||
|
fields = (((d.get("data") or {}).get("metadata") or {}).get("deploy_fields")) or []
|
||||||
|
vals = {"DOMAIN": DOMAIN, "SUBDOMAIN": "r1poll"}
|
||||||
|
for f in fields:
|
||||||
|
if f.get("env_var") in vals:
|
||||||
|
continue
|
||||||
|
if f.get("type") == "password":
|
||||||
|
import secrets
|
||||||
|
vals[f["env_var"]] = "R1-" + secrets.token_hex(10)
|
||||||
|
elif f.get("required") and f.get("default"):
|
||||||
|
vals[f["env_var"]] = f["default"]
|
||||||
|
code, d = ctl.call("POST", f"/api/stacks/{APP}/deploy", {"values": vals})
|
||||||
|
say("deploy ->", code, str(d)[:160])
|
||||||
|
for _ in range(90):
|
||||||
|
time.sleep(5)
|
||||||
|
st = stack()
|
||||||
|
if st.get("deployed") and (st.get("app_config") or {}).get("pinned_images") and st.get("state") in ("running", "unhealthy", "degraded"):
|
||||||
|
break
|
||||||
|
say("state:", stack().get("state"))
|
||||||
|
time.sleep(30)
|
||||||
|
say("== 2 marker A + backup")
|
||||||
|
say(psql("CREATE TABLE IF NOT EXISTS r1_marker(m text primary key); INSERT INTO r1_marker VALUES ('A') ON CONFLICT DO NOTHING;"))
|
||||||
|
markers("2")
|
||||||
|
code, d = ctl.call("POST", "/api/backup/run")
|
||||||
|
say("POST /api/backup/run ->", code, str(d)[:200])
|
||||||
|
say("backup finished:", json.dumps(wait_backup(), ensure_ascii=False)[:500])
|
||||||
|
unit = guest(f"for d in /mnt/felhom-drives/*/backups/primary/{APP} /mnt/sys_drive/felhom-data/backups/primary/{APP}; do "
|
||||||
|
f"[ -d $d ] && echo UNIT $d && find $d -maxdepth 3 -type f -printf '%TY-%Tm-%Td %TH:%TM %s %p\\n'; done")
|
||||||
|
say("unit on disk:\n" + unit)
|
||||||
|
say("== 3 POSITIVE CONTROL: marker B (live) then a normal unit restore")
|
||||||
|
say(psql("INSERT INTO r1_marker VALUES ('B');"))
|
||||||
|
markers("3-before")
|
||||||
|
restore("3")
|
||||||
|
time.sleep(15)
|
||||||
|
markers("3-after")
|
||||||
|
say("== 4 marker C (the newest live row), then THE FAULT: the unit's DB volume tar truncated to half")
|
||||||
|
say(psql("INSERT INTO r1_marker VALUES ('C');"))
|
||||||
|
markers("4-live")
|
||||||
|
say(guest(f"for d in /mnt/felhom-drives/*/backups/primary/{APP} /mnt/sys_drive/felhom-data/backups/primary/{APP}; do "
|
||||||
|
f"[ -d $d ] || continue; f=$(find $d -name '*.tar' -path '*volume*' | grep -iE 'postgres|db' | head -1); "
|
||||||
|
f"[ -n \"$f\" ] || f=$(find $d -name '*.tar' | head -1); s=$(stat -c %s \"$f\"); "
|
||||||
|
f"truncate -s $((s/2)) \"$f\"; echo \"TRUNCATED $f $s -> $(stat -c %s \"$f\")\"; done"))
|
||||||
|
say("== 5 the unit restore over the damaged copy")
|
||||||
|
restore("5")
|
||||||
|
time.sleep(30)
|
||||||
|
markers("5-after")
|
||||||
|
say("containers:\n" + guest("docker ps -a --format '{{.Names}} {{.Status}}' | grep -i rallly"))
|
||||||
|
say("db container log tail:\n" + guest("C=$(docker ps -a --format '{{.Names}}' | grep -E '^rallly.*(postgres|db)' | head -1); docker logs --tail 15 $C 2>&1"))
|
||||||
|
say("any undo / safety copy of the live state on disk?\n" + guest(
|
||||||
|
"find / -xdev \\( -path /proc -o -path /sys \\) -prune -o \\( -iname '*restore-undo*' -o -iname 'pre-restore-*' \\) -print 2>/dev/null | head; "
|
||||||
|
"for d in /mnt/felhom-drives/*/backups /mnt/sys_drive/felhom-data/backups; do find $d -maxdepth 4 -iname '*rallly*' -newer /etc/hostname 2>/dev/null | head; done"))
|
||||||
|
say("controller log (restore lines):\n" + guest("docker logs --since 30m felhom-controller 2>&1 | grep -iE 'restore|rallly' | grep -vi 'GET /api' | tail -40"))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
10:48:52Z == 1 deploy glance
|
||||||
|
10:48:52Z deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||||
|
10:49:12Z state: running family_gate in app_config: True
|
||||||
|
10:49:34Z [1 POSITIVE CONTROL after install] gate file: /opt/docker/stacks/traefik/dynamic/family-gate-glance.yml
|
||||||
|
10:49:34Z [1 POSITIVE CONTROL after install] a stranger's GET https://r1187glance.enkisfelhom.hu/ -> 401
|
||||||
|
10:49:34Z == 2 export to the data drive
|
||||||
|
10:49:34Z export start -> 200 {'ok': True}
|
||||||
|
10:49:34Z /api/export/status {"display_name": "Glance", "done": false, "error": "", "job_type": "export", "ok": true, "output_path": "", "output_size": "", "running": true, "stack_name": "glance", "steps": [{"label": "Előkészítés", "status": "running"}, {"label": "Konfiguráció mentése", "status": "pending"}, {"label": "Adatbázis mentése", "status": "pending"}, {"label": "Felhasználói adatok", "status": "pending"}, {"label": "
|
||||||
|
10:49:37Z /api/export/status {"display_name": "Glance", "done": true, "error": "", "job_type": "export", "ok": true, "output_path": "/mnt/felhom-drives/scratch_hdd/exports/glance_20261020-104935.fab", "output_size": "5.6 KB", "running": false, "stack_name": "glance", "steps": [{"label": "Előkészítés", "status": "done"}, {"label": "Konfiguráció mentése", "status": "done"}, {"label": "Adatbázis mentése", "status": "done"}, {"la
|
||||||
|
10:49:37Z bundles for glance: []
|
||||||
|
10:49:37Z == 3 remove glance (keep nothing)
|
||||||
|
10:50:09Z remove -> 200 {'ok': True, 'data': {'removed': 'glance', 'volumes_removed': ['glance_glance_config'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alka
|
||||||
|
10:50:21Z [3 after remove] gate file: /opt/docker/stacks/traefik/dynamic/family-gate-glance.yml
|
||||||
|
10:50:21Z [3 after remove] a stranger's GET https://r1187glance.enkisfelhom.hu/ -> 404
|
||||||
|
10:50:21Z == 4 import the bundle
|
||||||
|
10:50:21Z import start -> 400 {'error': 'Missing path', 'ok': False}
|
||||||
|
10:50:21Z /api/export/import/status {"display_name": "Glance", "done": true, "error": "", "job_type": "export", "ok": true, "output_path": "/mnt/felhom-drives/scratch_hdd/exports/glance_20261020-104935.fab", "output_size": "5.6 KB", "running": false, "stack_name": "glance", "steps": [{"label": "Előkészítés", "status": "done"}, {"label": "Konfiguráció mentése", "status": "done"}, {"label": "Adatbázis mentése", "status": "done"}, {"la
|
||||||
|
10:53:42Z state after import: not_deployed family_gate in app_config: False app_config keys: []
|
||||||
|
10:54:14Z [4 FAULT after import] gate file: (none)
|
||||||
|
10:54:14Z [4 FAULT after import] a stranger's GET https://r1187glance.enkisfelhom.hu/ -> 404
|
||||||
|
10:54:15Z controller log: 2026/10/20 10:48:52 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/20 10:48:58 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/20 10:49:34 family_gate.go:325: [DEBUG] [web] family gate glance: GET / without a pass — 401 (visitor 192.168.0.180)
|
||||||
|
2026/10/20 10:49:35 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/20 10:49:35 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/20 10:49:37 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/20 10:49:39 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/20 10:49:43 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/20 10:50:09 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
|
||||||
|
2026/10/20 10:50:09 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
|
||||||
|
2026/10/20 10:50:21 auth.go:144: [DEBUG] [web] auth: valid session for POST /api/export/import
|
||||||
|
2026/10/20 10:50:21 handler_export.go:328: [DEBUG] [web] apiImportStart: path="" hasPassword=false
|
||||||
|
2026/10/20 10:50:29 family_gate.go:190: [INFO] [stacks] glance: removed the family-gate file of an app that is not installed
|
||||||
|
2026/10/20 10:52:09 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
|
||||||
|
2026/10/20 10:54:09 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
|
||||||
|
exit 0
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
10:54:26Z == 4 import the bundle /mnt/felhom-drives/scratch_hdd/exports/glance_20261020-104935.fab
|
||||||
|
10:54:26Z import start -> 200 {'ok': True}
|
||||||
|
10:54:26Z /api/export/import/status {"display_name": "Glance", "done": false, "error": "", "job_type": "import", "ok": true, "output_path": "", "output_size": "", "running": true, "stack_name": "glance", "steps": [{"label": "Csomag megnyitása", "status": "done"}, {"label": "Alkalmazás előkészítése", "status": "done"}, {"label": "Konfiguráció visszaállítása", "status": "running"}, {"label": "Felhasználói adatok visszaállítása", "stat
|
||||||
|
10:54:29Z /api/export/import/status {"display_name": "Glance", "done": false, "error": "", "job_type": "import", "ok": true, "output_path": "", "output_size": "", "running": true, "stack_name": "glance", "steps": [{"label": "Csomag megnyitása", "status": "done"}, {"label": "Alkalmazás előkészítése", "status": "done"}, {"label": "Konfiguráció visszaállítása", "status": "done"}, {"label": "Felhasználói adatok visszaállítása", "status"
|
||||||
|
10:54:32Z /api/export/import/status {"display_name": "Glance", "done": true, "error": "", "job_type": "import", "ok": true, "output_path": "", "output_size": "", "running": false, "stack_name": "glance", "steps": [{"label": "Csomag megnyitása", "status": "done"}, {"label": "Alkalmazás előkészítése", "status": "done"}, {"label": "Konfiguráció visszaállítása", "status": "done"}, {"label": "Felhasználói adatok visszaállítása", "status"
|
||||||
|
10:56:13Z state after import: running family_gate in app_config: False app_config keys: ['deployed', 'deployed_at', 'desired_state', 'env', 'installed_images', 'locked_fields']
|
||||||
|
10:56:55Z [4 FAULT after import] gate file: (none)
|
||||||
|
10:56:55Z [4 FAULT after import] a stranger's GET https://r1187glance.enkisfelhom.hu/ -> 200
|
||||||
|
10:56:56Z controller log: 2026/10/20 10:54:33 manager.go:1814: [INFO] [stacks] glance glanceapp/glance:v0.8.5 running Up 3 seconds (health: starting)
|
||||||
|
2026/10/20 10:54:39 healthprobe.go:187: [DEBUG] Health probe glance: HTTP GET :8080/ → 200 (1ms)
|
||||||
|
2026/10/20 10:54:49 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 10s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:54:59 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 20s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:55:09 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 30s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:55:19 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 40s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:55:29 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 50s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:55:39 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 1m0s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:55:49 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 1m10s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:55:59 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 1m20s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:56:09 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 1m30s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:56:19 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 1m40s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:56:29 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 1m50s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:56:39 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 2m0s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:56:49 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 2m10s ago, effective interval 5m0s, healthy=true
|
||||||
|
exit 0
|
||||||
+24
@@ -0,0 +1,24 @@
|
|||||||
|
200
|
||||||
|
{
|
||||||
|
200
|
||||||
|
{
|
||||||
|
"ok": true,
|
||||||
|
Exception in thread Thread-1 (pump):
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "/usr/lib/python3.13/threading.py", line 1043, in _bootstrap_inner
|
||||||
|
self.run()
|
||||||
|
~~~~~~~~^^
|
||||||
|
File "/usr/lib/python3.13/threading.py", line 994, in run
|
||||||
|
self._target(*self._args, **self._kwargs)
|
||||||
|
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
File "/mnt/5_hdd/felhom.eu/git/cc-secrets/ccsec/cli.py", line 153, in pump
|
||||||
|
dst.write(m.close()); dst.flush()
|
||||||
|
~~~~~~~~~^^
|
||||||
|
BrokenPipeError: [Errno 32] Broken pipe
|
||||||
|
Exception ignored on flushing sys.stdout:
|
||||||
|
BrokenPipeError: [Errno 32] Broken pipe
|
||||||
|
removed '/mnt/felhom-drives/scratch_hdd/exports/glance_20261020-104935.fab'
|
||||||
|
0
|
||||||
|
0
|
||||||
|
0
|
||||||
|
stranger GET after teardown: 404
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# C3 — proven live: a `.fab` export → import makes a family-only app public
|
||||||
|
|
||||||
|
Scratch guest 9202, controller 0.331.0 (the `.fab` import code — `cmd/controller/main.go:3517-3530`
|
||||||
|
`SaveEncryptedAppConfig`, `stacks/family_gate.go:173-190` — read the same at 0.336.0, `7ca59e2`). Architecture: `09` §3
|
||||||
|
decisions 63/64 (the family gate), 221 (Glance only behind it), `07` §6.6 („never open by default").
|
||||||
|
|
||||||
|
| UTC | act (the dashboard's own endpoints) | observed (a stranger's GET through traefik; the gate file on disk) |
|
||||||
|
|---|---|---|
|
||||||
|
| 10:48:52 | `POST /api/stacks/glance/deploy` | running, `app_config.family_gate` set |
|
||||||
|
| 10:49:34 | **control** | gate file `traefik/dynamic/family-gate-glance.yml` present; stranger → **401** |
|
||||||
|
| 10:49:35 | `POST /api/export/start` (stop ticked, no password) | `glance_20261020-104935.fab`, 5.6 KB |
|
||||||
|
| 10:50:09 | stop + `POST /api/stacks/glance/remove` (data + backups) | removed; 10:50:29 „removed the family-gate file of an app that is not installed" |
|
||||||
|
| 10:54:26 | `POST /api/export/import` (the household's „Importálás") | import done in 6 s; app running |
|
||||||
|
| 10:56:13 | the app's record | `app_config` keys: deployed, deployed_at, desired_state, env, installed_images, locked_fields — **no family_gate** |
|
||||||
|
| 10:56:55 | **fault** | gate file **none**; stranger → **200** (Glance answers anyone who reaches the address) |
|
||||||
|
|
||||||
|
The first run's bundle-list parse was a harness slip (`01-run.log`, step 4 sent an empty path → 400, nothing done);
|
||||||
|
step 4 re-ran alone with the path the export's own status gave (`02-import.log`, `step4.py`). Teardown
|
||||||
|
(`03-teardown.txt`): Glance removed through the product, the bundle deleted (own file), no container, volume or gate
|
||||||
|
file left; the address answers 404.
|
||||||
|
|
||||||
|
On a household box the address is reachable from the internet through the tunnel. Family-gated apps in the catalog:
|
||||||
|
Glance, Grimmory, MeTube (the last two hold the family's books and videos).
|
||||||
+96
@@ -0,0 +1,96 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""fab_gate_proof.py — C3 for R-1187 (e) on scratch 9202: does a `.fab` export → remove → import drop the family gate?
|
||||||
|
|
||||||
|
Glance has no login of its own; the catalog puts it ONLY behind the family gate (decision 221). Observables, each
|
||||||
|
read from a DIFFERENT channel than the act: (1) the gate file `family-gate-glance.yml` on the guest's disk; (2) what a
|
||||||
|
stranger's request to the app's host gets back from traefik (302 to the family page vs 200 from Glance).
|
||||||
|
Acts are the dashboard's own endpoints: deploy, /api/export/start, stop+remove, /api/export/import.
|
||||||
|
Env: CTL_BASE, CTL_DOMAIN, CTL_PW_ENV (tools/ctl.py).
|
||||||
|
"""
|
||||||
|
import json, os, subprocess, sys, time
|
||||||
|
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/rehearsal-2026-10-20/tools")
|
||||||
|
import ctl # noqa: E402
|
||||||
|
|
||||||
|
APP, SUB = "glance", "r1187glance"
|
||||||
|
DOM = os.environ["CTL_DOMAIN"]
|
||||||
|
|
||||||
|
|
||||||
|
def say(*a):
|
||||||
|
print(time.strftime("%H:%M:%SZ", time.gmtime()), *a, flush=True)
|
||||||
|
|
||||||
|
|
||||||
|
def guest(cmd):
|
||||||
|
r = subprocess.run(["ssh", "demo-hp", f"export LC_ALL=C; pct exec 9202 -- bash -c {json.dumps(cmd)}"],
|
||||||
|
capture_output=True, text=True, timeout=300)
|
||||||
|
return (r.stdout + r.stderr).strip()
|
||||||
|
|
||||||
|
|
||||||
|
def observe(tag):
|
||||||
|
say(f"[{tag}] gate file:", guest("find / -xdev -name 'family-gate-glance.yml' 2>/dev/null | head -3; true") or "(none)")
|
||||||
|
out = subprocess.run(["curl", "-sk", "-o", "/dev/null", "-w", "%{http_code} %{redirect_url}",
|
||||||
|
"-H", f"Host: {SUB}.{DOM}", ctl.BASE + "/"], capture_output=True, text=True).stdout
|
||||||
|
say(f"[{tag}] a stranger's GET https://{SUB}.{DOM}/ ->", out)
|
||||||
|
|
||||||
|
|
||||||
|
def wait_stack(pred, n=60):
|
||||||
|
for _ in range(n):
|
||||||
|
code, d = ctl.call("GET", f"/api/stacks/{APP}")
|
||||||
|
st = (d.get("data") or {}) if isinstance(d, dict) else {}
|
||||||
|
if pred(st):
|
||||||
|
return st
|
||||||
|
time.sleep(5)
|
||||||
|
return st
|
||||||
|
|
||||||
|
|
||||||
|
def poll(path, n=120):
|
||||||
|
last = None
|
||||||
|
for _ in range(n):
|
||||||
|
code, d = ctl.call("GET", path)
|
||||||
|
dd = (d.get("data") or d) if isinstance(d, dict) else {}
|
||||||
|
s = json.dumps(dd, ensure_ascii=False)[:400]
|
||||||
|
if s != last:
|
||||||
|
say(" ", path, s)
|
||||||
|
last = s
|
||||||
|
if isinstance(dd, dict) and dd.get("status") in ("completed", "failed", "done", "error") or (
|
||||||
|
isinstance(dd, dict) and dd.get("running") is False and dd.get("status") not in ("running", "pending")):
|
||||||
|
return dd
|
||||||
|
time.sleep(3)
|
||||||
|
return dd
|
||||||
|
|
||||||
|
|
||||||
|
say("== 1 deploy glance")
|
||||||
|
code, d = ctl.call("POST", f"/api/stacks/{APP}/deploy", {"values": {"DOMAIN": DOM, "SUBDOMAIN": SUB}})
|
||||||
|
say("deploy ->", code, str(d)[:150])
|
||||||
|
st = wait_stack(lambda s: s.get("deployed") and s.get("state") in ("running", "unhealthy", "degraded"))
|
||||||
|
say("state:", st.get("state"), "family_gate in app_config:", bool((st.get("app_config") or {}).get("family_gate")))
|
||||||
|
time.sleep(20)
|
||||||
|
observe("1 POSITIVE CONTROL after install")
|
||||||
|
|
||||||
|
say("== 2 export to the data drive")
|
||||||
|
drive = "/mnt/felhom-drives/scratch_hdd"
|
||||||
|
code, d = ctl.call("POST", "/api/export/start", {"stack_name": APP, "dest_drive": drive, "password": "", "stop_app": True})
|
||||||
|
say("export start ->", code, str(d)[:200])
|
||||||
|
poll("/api/export/status")
|
||||||
|
code, d = ctl.call("GET", "/api/export/bundles")
|
||||||
|
bundles = [b for b in ((d.get("data") or {}).get("bundles") or d.get("data") or []) if isinstance(b, dict) and APP in json.dumps(b)]
|
||||||
|
say("bundles for glance:", json.dumps(bundles, ensure_ascii=False)[:500])
|
||||||
|
path = (bundles[-1].get("path") if bundles else None)
|
||||||
|
|
||||||
|
say("== 3 remove glance (keep nothing)")
|
||||||
|
ctl.call("POST", f"/api/stacks/{APP}/stop")
|
||||||
|
time.sleep(5)
|
||||||
|
code, d = ctl.call("POST", f"/api/stacks/{APP}/remove", {"remove_hdd_data": True, "remove_backups": True})
|
||||||
|
say("remove ->", code, str(d)[:160])
|
||||||
|
time.sleep(10)
|
||||||
|
observe("3 after remove")
|
||||||
|
|
||||||
|
say("== 4 import the bundle")
|
||||||
|
code, d = ctl.call("POST", "/api/export/import", {"path": path, "password": ""})
|
||||||
|
say("import start ->", code, str(d)[:200])
|
||||||
|
poll("/api/export/import/status")
|
||||||
|
st = wait_stack(lambda s: s.get("deployed") and s.get("state") in ("running", "unhealthy", "degraded"), n=40)
|
||||||
|
say("state after import:", st.get("state"), "family_gate in app_config:", bool((st.get("app_config") or {}).get("family_gate")),
|
||||||
|
"app_config keys:", sorted((st.get("app_config") or {}).keys()))
|
||||||
|
time.sleep(30)
|
||||||
|
observe("4 FAULT after import")
|
||||||
|
say("controller log:", guest("docker logs --since 15m felhom-controller 2>&1 | grep -iE 'family|import' | grep -v 'GET /api' | tail -15"))
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Step 4 of fab_gate_proof.py re-run alone: the first run's bundle-list parse found nothing (harness slip); the
|
||||||
|
export's own status gave the path."""
|
||||||
|
import sys
|
||||||
|
sys.argv = ["x"]
|
||||||
|
exec(open("fab_gate_proof.py").read().split('say("== 1 deploy glance")')[0])
|
||||||
|
path = "/mnt/felhom-drives/scratch_hdd/exports/glance_20261020-104935.fab"
|
||||||
|
say("== 4 import the bundle", path)
|
||||||
|
code, d = ctl.call("POST", "/api/export/import", {"path": path, "password": ""})
|
||||||
|
say("import start ->", code, str(d)[:200])
|
||||||
|
poll("/api/export/import/status")
|
||||||
|
st = wait_stack(lambda s: s.get("deployed") and s.get("state") in ("running", "unhealthy", "degraded"), n=40)
|
||||||
|
say("state after import:", st.get("state"), "family_gate in app_config:", bool((st.get("app_config") or {}).get("family_gate")),
|
||||||
|
"app_config keys:", sorted((st.get("app_config") or {}).keys()))
|
||||||
|
time.sleep(40)
|
||||||
|
observe("4 FAULT after import")
|
||||||
|
say("controller log:", guest("docker logs --since 6m felhom-controller 2>&1 | grep -iE 'family|import|glance' | grep -vE 'GET /api|ScanStacks|Env vars' | tail -15"))
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
# go test at controller 7ca59e2 (v0.336.0), scratch clone; 2026-10-20T11:11:31Z
|
||||||
|
=== RUN TestRehearsalR12_LeftoverTempBlocksEveryRetry
|
||||||
|
zz_rehearsal_r12_test.go:21: retry 1: err=fájlmásolás sikertelen: restoring appdata/app/p1.jpg: open /tmp/TestRehearsalR12_LeftoverTempBlocksEveryRetry3138873193/001/usb/appdata/app/p1.jpg.felhom-restoring: file exists; p1.jpg back=false; app restarted=true
|
||||||
|
zz_rehearsal_r12_test.go:21: retry 2: err=fájlmásolás sikertelen: restoring appdata/app/p1.jpg: open /tmp/TestRehearsalR12_LeftoverTempBlocksEveryRetry3138873193/001/usb/appdata/app/p1.jpg.felhom-restoring: file exists; p1.jpg back=false; app restarted=true
|
||||||
|
--- PASS: TestRehearsalR12_LeftoverTempBlocksEveryRetry (0.01s)
|
||||||
|
PASS
|
||||||
|
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.020s
|
||||||
+23
@@ -0,0 +1,23 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
// THROWAWAY proof for the 2026-10-20 data-loss hunt (R-1188 c). Never committed. A crash during the whole restore's
|
||||||
|
// file copy leaves `<file>.felhom-restoring` behind (the copy writes it with O_EXCL); the retry then meets it.
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRehearsalR12_LeftoverTempBlocksEveryRetry(t *testing.T) {
|
||||||
|
f, _, liveLeg := wholeFixture(t)
|
||||||
|
// what a power cut in the middle of copying p1.jpg leaves on the live drive
|
||||||
|
if err := os.MkdirAll(liveLeg, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
os.WriteFile(filepath.Join(liveLeg, "p1.jpg.felhom-restoring"), []byte("half"), 0o644)
|
||||||
|
for i := 1; i <= 2; i++ {
|
||||||
|
_, err := f.m.RestoreTier2Whole("app")
|
||||||
|
_, statErr := os.Stat(filepath.Join(liveLeg, "p1.jpg"))
|
||||||
|
t.Logf("retry %d: err=%v; p1.jpg back=%v; app restarted=%v", i, err, statErr == nil, f.fake.fullStarted)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
# go test at controller 7ca59e2 (v0.336.0), scratch clone, fake provider seams; 2026-10-20T10:13:05Z
|
||||||
|
=== RUN TestRehearsalR5_WholeRestoreUnitRefusalAfterFileHalf
|
||||||
|
zz_rehearsal_r5_test.go:22: live files BEFORE: map[p2.jpg:P2 live older]
|
||||||
|
zz_rehearsal_r5_test.go:25: RestoreTier2Whole err = A(z) app adatbázis-másolata csonka: nem ér véget, ahogy egy teljes másolat. A visszaállítás biztonsági okból nem indult el — az alkalmazás és az adatai érintetlenek.
|
||||||
|
zz_rehearsal_r5_test.go:26: file counts: {Restored:1 Replaced:1 KeptNewer:0 Unchanged:0 BytesTotal:17}
|
||||||
|
zz_rehearsal_r5_test.go:27: live files AFTER: map[p1.jpg:P1 p2.jpg:P2 mirror NEWER p2.jpg.felhom-20261020T101306Z:P2 live older]
|
||||||
|
zz_rehearsal_r5_test.go:28: app stopped=true, full start happened=false, provider calls=[stop]
|
||||||
|
--- PASS: TestRehearsalR5_WholeRestoreUnitRefusalAfterFileHalf (0.01s)
|
||||||
|
PASS
|
||||||
|
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.019s
|
||||||
+29
@@ -0,0 +1,29 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate R5). Never committed.
|
||||||
|
import (
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRehearsalR5_WholeRestoreUnitRefusalAfterFileHalf(t *testing.T) {
|
||||||
|
cut := strings.Replace(pgDump(1), "PostgreSQL database dump complete", "", 1) // a cut-off copy (R-640)
|
||||||
|
f := r102Tier2Fixture(t, []string{"app_data.tar"}, cut)
|
||||||
|
f.m.stackProvider = fileAppProvider{f.fake}
|
||||||
|
mid := time.Date(2026, 9, 22, 10, 0, 0, 0, time.UTC)
|
||||||
|
mirrorLeg := filepath.Join(f.destBase, "hdd", "appdata", "app")
|
||||||
|
liveLeg := filepath.Join(f.liveDrive, "appdata", "app")
|
||||||
|
writeAt(t, filepath.Join(mirrorLeg, "p1.jpg"), "P1", mid)
|
||||||
|
writeAt(t, filepath.Join(mirrorLeg, "p2.jpg"), "P2 mirror NEWER", mid.Add(72*time.Hour))
|
||||||
|
writeAt(t, filepath.Join(liveLeg, "p2.jpg"), "P2 live older", mid.Add(48*time.Hour))
|
||||||
|
before := fingerprint(t, liveLeg)
|
||||||
|
t.Logf("live files BEFORE: %v", before)
|
||||||
|
|
||||||
|
res, err := f.m.RestoreTier2Whole("app")
|
||||||
|
t.Logf("RestoreTier2Whole err = %v", err)
|
||||||
|
t.Logf("file counts: %+v", res.Files)
|
||||||
|
t.Logf("live files AFTER: %v", fingerprint(t, liveLeg))
|
||||||
|
t.Logf("app stopped=%v, full start happened=%v, provider calls=%v", f.fake.stopped, f.fake.fullStarted, f.fake.calls)
|
||||||
|
}
|
||||||
+8
@@ -0,0 +1,8 @@
|
|||||||
|
# go test at controller 7ca59e2 (v0.336.0), scratch clone; the PRODUCTION rsyncMirror (rsync -a --delete) on local temp dirs; 2026-10-20T11:02:49Z
|
||||||
|
=== RUN TestRehearsalT1_NightlyMirrorRunsDuringARestore
|
||||||
|
zz_rehearsal_t1_test.go:26: POSITIVE CONTROL: mirror after the night = [a.jpg b.jpg c.jpg d.jpg]
|
||||||
|
zz_rehearsal_t1_test.go:34: restore in progress: backup flag held = true
|
||||||
|
zz_rehearsal_t1_test.go:37: FAULT: mirror DURING the restore = [a.jpg b.jpg] (c.jpg, d.jpg — the files the restore is bringing back — are gone from the only copy)
|
||||||
|
--- PASS: TestRehearsalT1_NightlyMirrorRunsDuringARestore (0.22s)
|
||||||
|
PASS
|
||||||
|
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.230s
|
||||||
+39
@@ -0,0 +1,39 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate T1 / R-1180). Never committed.
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
type t1Provider struct{ *t2v2Provider }
|
||||||
|
|
||||||
|
func (p t1Provider) ListDeployedStacks() []StackSummary { return []StackSummary{{Name: "app"}} }
|
||||||
|
|
||||||
|
func TestRehearsalT1_NightlyMirrorRunsDuringARestore(t *testing.T) {
|
||||||
|
m, src, target, prov := newTier2V2(t, "app")
|
||||||
|
prov.has["app"] = true
|
||||||
|
prov.binds["app"] = []ClassifiedBind{mHDD("appdata/app/photos")}
|
||||||
|
m.stackProvider = t1Provider{prov}
|
||||||
|
m.tier2Mirror = rsyncMirror // the PRODUCTION mirror (rsync -a --delete), local temp dirs only
|
||||||
|
for _, n := range []string{"a.jpg", "b.jpg", "c.jpg", "d.jpg"} {
|
||||||
|
mustWrite(t, filepath.Join(src, "appdata", "app", "photos", n), n)
|
||||||
|
}
|
||||||
|
m.RunAllTier2()
|
||||||
|
mir := filepath.Join(target, "backups", "secondary", "app", "hdd", "appdata", "app", "photos")
|
||||||
|
ls := func(d string) []string { e, _ := os.ReadDir(d); var o []string; for _, x := range e { o = append(o, x.Name()) }; return o }
|
||||||
|
t.Logf("POSITIVE CONTROL: mirror after the night = %v", ls(mir))
|
||||||
|
|
||||||
|
// The household deleted half the photos and starts the second drive's restore; the restore holds the backup flag.
|
||||||
|
os.Remove(filepath.Join(src, "appdata", "app", "photos", "c.jpg"))
|
||||||
|
os.Remove(filepath.Join(src, "appdata", "app", "photos", "d.jpg"))
|
||||||
|
if err := m.acquireRunning(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Logf("restore in progress: backup flag held = %v", m.acquireRunning() != nil)
|
||||||
|
// 03:30 — the nightly second-drive leg fires while the restore is still copying.
|
||||||
|
m.RunAllTier2()
|
||||||
|
t.Logf("FAULT: mirror DURING the restore = %v (c.jpg, d.jpg — the files the restore is bringing back — are gone from the only copy)", ls(mir))
|
||||||
|
m.releaseRunning()
|
||||||
|
}
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
// Replays hub cmd/hub/main.go scheduleDaily's arithmetic (felhom.eu 5c48731, lines 1057-1062) over
|
||||||
|
// 2026-10-23..27 with a simulated clock, for each hub daily job. Prints every fire time.
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
bud, _ := time.LoadLocation("Europe/Budapest")
|
||||||
|
jobs := []struct{ name string; h, m int }{{"db-snapshot", 2, 0}, {"prune", 4, 30}, {"deadline-check", 5, 0}, {"offsite-key-audit", 7, 10}}
|
||||||
|
for _, j := range jobs {
|
||||||
|
now := time.Date(2026, 10, 23, 12, 0, 0, 0, bud)
|
||||||
|
end := time.Date(2026, 10, 27, 12, 0, 0, 0, bud)
|
||||||
|
fmt.Printf("%s (%02d:%02d):\n", j.name, j.h, j.m)
|
||||||
|
for now.Before(end) {
|
||||||
|
n := now.In(bud)
|
||||||
|
next := time.Date(n.Year(), n.Month(), n.Day(), j.h, j.m, 0, 0, bud)
|
||||||
|
if !next.After(n) {
|
||||||
|
next = next.Add(24 * time.Hour)
|
||||||
|
}
|
||||||
|
fmt.Printf(" fires %s (%s)\n", next.In(bud).Format("2006-01-02 15:04 MST"), next.UTC().Format("15:04Z"))
|
||||||
|
now = next // fn() runs instantly; loop re-reads the clock at the fire instant
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
--- go run, go version go1.26.0 linux/amd64
|
||||||
|
db-snapshot (02:00):
|
||||||
|
fires 2026-10-24 02:00 CEST (00:00Z)
|
||||||
|
fires 2026-10-25 02:00 CEST (00:00Z)
|
||||||
|
fires 2026-10-25 02:00 CET (01:00Z)
|
||||||
|
fires 2026-10-26 02:00 CET (01:00Z)
|
||||||
|
fires 2026-10-27 02:00 CET (01:00Z)
|
||||||
|
fires 2026-10-28 02:00 CET (01:00Z)
|
||||||
|
prune (04:30):
|
||||||
|
fires 2026-10-24 04:30 CEST (02:30Z)
|
||||||
|
fires 2026-10-25 03:30 CET (02:30Z)
|
||||||
|
fires 2026-10-25 04:30 CET (03:30Z)
|
||||||
|
fires 2026-10-26 04:30 CET (03:30Z)
|
||||||
|
fires 2026-10-27 04:30 CET (03:30Z)
|
||||||
|
fires 2026-10-28 04:30 CET (03:30Z)
|
||||||
|
deadline-check (05:00):
|
||||||
|
fires 2026-10-24 05:00 CEST (03:00Z)
|
||||||
|
fires 2026-10-25 04:00 CET (03:00Z)
|
||||||
|
fires 2026-10-25 05:00 CET (04:00Z)
|
||||||
|
fires 2026-10-26 05:00 CET (04:00Z)
|
||||||
|
fires 2026-10-27 05:00 CET (04:00Z)
|
||||||
|
fires 2026-10-28 05:00 CET (04:00Z)
|
||||||
|
offsite-key-audit (07:10):
|
||||||
|
fires 2026-10-24 07:10 CEST (05:10Z)
|
||||||
|
fires 2026-10-25 06:10 CET (05:10Z)
|
||||||
|
fires 2026-10-25 07:10 CET (06:10Z)
|
||||||
|
fires 2026-10-26 07:10 CET (06:10Z)
|
||||||
|
fires 2026-10-27 07:10 CET (06:10Z)
|
||||||
|
fires 2026-10-28 07:10 CET (06:10Z)
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
tester-1-d70be4 agent_update:
|
||||||
|
uploaded signed op to the hub jobs queue
|
||||||
|
tester-1-d70be4 agent_config_update:
|
||||||
|
uploaded signed op to the hub jobs queue
|
||||||
|
demo-felhom-8363b5 agent_update:
|
||||||
|
uploaded signed op to the hub jobs queue
|
||||||
|
demo-felhom-8363b5 agent_config_update:
|
||||||
|
uploaded signed op to the hub jobs queue
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
tester-1: felhom-agent 0.159.2 | demo-felhom: felhom-agent 0.159.2 | 12:25:17
|
||||||
|
Oct 18 14:23:46 felhom felhom-agent[3912599]: time=2026-10-18T14:23:46.774+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=c9b02a094671155e op=agent_update
|
||||||
|
Oct 18 14:23:48 felhom felhom-agent[3912599]: time=2026-10-18T14:23:48.727+02:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=75 total=75 degraded=""
|
||||||
|
Oct 18 14:23:48 felhom felhom-agent[3912599]: time=2026-10-18T14:23:48.728+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=fa2a707792f561ed op=agent_config_update
|
||||||
|
Oct 18 14:24:52 felhom felhom-agent[1371657]: time=2026-10-18T14:24:52.767+02:00 level=WARN msg="selfupdate: update committed" version=0.159.2 wrapper=""
|
||||||
|
Oct 18 14:25:00 demo-felhom felhom-agent[2419165]: time=2026-10-18T14:25:00.552+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=9d4397117eeed3b9 op=agent_update
|
||||||
|
Oct 18 14:25:01 demo-felhom felhom-agent[2419165]: time=2026-10-18T14:25:01.930+02:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=75 total=75 degraded=""
|
||||||
|
Oct 18 14:25:01 demo-felhom felhom-agent[2419165]: time=2026-10-18T14:25:01.930+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=63f8321055a65cb8 op=agent_config_update
|
||||||
|
|
||||||
|
[exited with code 0]
|
||||||
|
Oct 18 14:26:05 demo-felhom felhom-agent[3800522]: time=2026-10-18T14:26:05.216+02:00 level=WARN msg="selfupdate: update committed" version=0.159.2 wrapper=""
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
Oct 18 13:12:26 demo-hp felhom-agent[3566597]: time=2026-10-18T13:12:26.461+02:00 level=INFO msg="backup: space preflight passed" vmid=9201 target=local last_archive_bytes=6916890749 need_bytes=9719855260 avail_bytes=12123615232
|
||||||
|
Oct 18 13:12:27 demo-hp felhom-agent[3566597]: time=2026-10-18T13:12:27.526+02:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=local job=backup-9201-1792321945880394998
|
||||||
|
Oct 18 13:18:22 demo-hp felhom-agent[3566597]: time=2026-10-18T13:18:22.853+02:00 level=INFO msg="janitor: stale-lock sweep deferred — a heavy operation is in flight" busy=backup:local
|
||||||
|
Oct 18 13:19:59 demo-hp felhom-agent[3566597]: time=2026-10-18T13:19:59.260+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_10_18-13_12_26.tar.zst size_bytes=6938307162 uncovered_volumes=2
|
||||||
|
Oct 18 13:19:59 demo-hp felhom-agent[3566597]: time=2026-10-18T13:19:59.260+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1792321945880394998 archive=local:backup/vzdump-lxc-9201-2026_10_18-13_12_26.tar.zst
|
||||||
|
Oct 18 13:19:59 demo-hp felhom-agent[3566597]: time=2026-10-18T13:19:59.266+02:00 level=INFO msg="local-api: no OS leg after a manual backup — it follows the night's own backup (R-899)" vmid=9201 job=backup-9201-1792321945880394998
|
||||||
|
== sudo felhom-thin-check lines since the trigger:
|
||||||
|
0
|
||||||
|
== sudo lines since the trigger (any helper):
|
||||||
|
26 COMMAND=/usr/bin/chown
|
||||||
|
36 COMMAND=/usr/bin/lsblk
|
||||||
|
98 COMMAND=/usr/bin/lxc-info
|
||||||
|
78 COMMAND=/usr/bin/mkdir
|
||||||
|
8 COMMAND=/usr/bin/wg
|
||||||
|
36 COMMAND=/usr/sbin/blkid
|
||||||
|
75 COMMAND=/usr/sbin/lvs
|
||||||
|
16 COMMAND=/usr/sbin/nft
|
||||||
|
57 COMMAND=/usr/sbin/pct
|
||||||
|
150 COMMAND=/usr/sbin/smartctl
|
||||||
|
|
||||||
|
[exited with code 0]
|
||||||
|
== control: the night's backup on agent 0.159.1 (2026-10-18 04:33 local) ran the check:
|
||||||
|
Oct 18 04:33:15 demo-hp sudo[1664888]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-thin-check pve data
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
== trigger 2026-10-18T11:11:36Z (local 2026-10-18 13:11:36)
|
||||||
|
200
|
||||||
|
{"data": {"started": true}, "ok": true}
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
felhom-agent 0.159.1
|
||||||
|
Oct 18 04:43:52 demo-hp felhom-agent[3320002]: time=2026-10-18T04:43:52.498+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
|
||||||
|
Oct 18 04:43:53 demo-hp felhom-agent[3320002]: time=2026-10-18T04:43:53.613+02:00 level=INFO msg="osupdate: DONE" run=20261018T024228Z layer=pve vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=1 not_covered=0 restart_needed=11 reboot_needed=false wrapper_seconds=16.3
|
||||||
|
Oct 18 05:23:06 demo-hp felhom-agent[3320002]: time=2026-10-18T05:23:06.521+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-10-15T20:38:47Z) is already proven"
|
||||||
|
Oct 18 11:23:06 demo-hp felhom-agent[3320002]: time=2026-10-18T11:23:06.557+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-10-15T20:38:47Z) is already proven"
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
wrote envelope to /dev/null
|
||||||
|
uploaded signed op to the hub jobs queue
|
||||||
|
wrote envelope to /dev/null
|
||||||
|
uploaded signed op to the hub jobs queue
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
felhom-agent 0.159.2 11:08:30
|
||||||
|
Oct 18 13:08:17 demo-hp felhom-agent[3320002]: time=2026-10-18T13:08:17.765+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=763e0b8971dff83c op=agent_update
|
||||||
|
Oct 18 13:08:18 demo-hp felhom-os-apply[3566414]: os-apply: BUNDLE DONE agent=0.159.2 written=0 same=30 self-check=ok signers-created=False
|
||||||
|
Oct 18 13:08:18 demo-hp felhom-agent[3320002]: time=2026-10-18T13:08:18.862+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.159.2 written=0 same=30 self-check=ok signers-created=False"
|
||||||
|
Oct 18 13:08:19 demo-hp felhom-agent[3320002]: time=2026-10-18T13:08:19.883+02:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=75 total=75 degraded=""
|
||||||
|
Oct 18 13:08:19 demo-hp felhom-agent[3320002]: time=2026-10-18T13:08:19.884+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=268dfe26d949094f op=agent_config_update
|
||||||
|
|
||||||
|
[exited with code 0]
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
demo-hp-bb76ea last report 2026-10-18 12:38:33 agent 0.159.2 health None
|
||||||
|
tester-1-d70be4 last report 2026-10-18 12:39:04 agent 0.159.2 health None
|
||||||
|
demo-felhom-8363b5 last report 2026-10-18 12:25:17 agent 0.159.2 health None
|
||||||
|
report keys: ['addresses', 'agent_sha256', 'agent_version', 'audit_tail', 'backups', 'capabilities', 'cloudflared', 'controller_supervisor', 'dr_recipe', 'foreign_key_archives', 'guest_disk_trim', 'guest_net', 'guests', 'host', 'host_id', 'leaf_fingerprint', 'mgmt_plane', 'oob', 'pbs_dr', 'pbs_snapshots', 'reported_at', 'restore_tests', 'storage_targets', 'system', 'wireguard']
|
||||||
|
demo-hp-bb76ea 2026-10-18 12:38:33 agent_version 0.159.2 degraded None cap_ok/total None problems None
|
||||||
|
tester-1-d70be4 2026-10-18 12:39:04 agent_version 0.159.2 degraded None cap_ok/total None problems None
|
||||||
|
demo-felhom-8363b5 2026-10-18 12:25:17 agent_version 0.159.2 degraded None cap_ok/total None problems None
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
felhom-agent sha256 4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9
|
||||||
|
felhom-config-bundle.json sha256 477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9
|
||||||
@@ -0,0 +1,102 @@
|
|||||||
|
[release-agent] building 0.159.2 …
|
||||||
|
[release-agent] built ok: sha256 4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9
|
||||||
|
[release-agent] config bundle built: sha256 477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9
|
||||||
|
[release-agent] tagging v0.159.2 at 65c1994 …
|
||||||
|
[release-agent] publishing …
|
||||||
|
[publish-agent] publishing /tmp/felhom-agent-bMXfqp (16007752 bytes, sha256 4d98868e922d6c10…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/0.159.2/felhom-agent
|
||||||
|
[publish-agent] pre-delete existing artifact: HTTP 404 (404/204 expected)
|
||||||
|
[publish-agent] upload OK (HTTP 201)
|
||||||
|
[publish-agent] round-trip GET verified (sha256 matches)
|
||||||
|
[publish-agent] DONE. Record in the hub operator UI (Configs → Day-0 artifacts): agent 0.159.2 / 4d98868e922d6c10665a6c0da9323e862AGENT_VERSION=0.159.2
|
||||||
|
AGENT_SHA256=4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9
|
||||||
|
pre-push [felhom-agent]: running scripts/agent_gates.py --fast ...
|
||||||
|
agent_gates — 4 gate(s) [--fast]
|
||||||
|
--fast SKIPPED (deliberate periodic runs, never in a hook): published, gate-decoys
|
||||||
|
|
||||||
|
==============================================================================
|
||||||
|
== gate: reuse-refs (reuse_refs_check.py /mnt/5_hdd/felhom.eu/git/felhom-agent)
|
||||||
|
==============================================================================
|
||||||
|
note [felhom-agent] line 189: localapi/debuglogs_test.go (resolved by suffix → internal/localapi/debuglogs_test.go)
|
||||||
|
note [felhom-agent] line 205: hub/internal/store/dr_recipe.go (cross-repo → felhom.eu/hub/internal/store/dr_recipe.go)
|
||||||
|
OK [felhom-agent]: 117 cited paths — exact 115, suffix 1, ambiguous 0, cross-repo 1, FAILED 0 (siblings searched: app-catalog-felhom.eu, cc-secrets, felhom-controller, felhom.eu, felhom.eu-media, felhom.eu-sizing-push, felhom.eu-wt-sizing)
|
||||||
|
|
||||||
|
==============================================================================
|
||||||
|
== gate: instructions (instructions_gate.py /mnt/5_hdd/felhom.eu/git/felhom-agent)
|
||||||
|
==============================================================================
|
||||||
|
instructions_gate: /mnt/5_hdd/felhom.eu/git/felhom-agent
|
||||||
|
CLAUDE.md effective lines : 108 (ceiling 200)
|
||||||
|
version literals : 0
|
||||||
|
TEMPORARY blocks : 0
|
||||||
|
rule files : 6 (5 path-scoped)
|
||||||
|
workspace file : SYMLINK -> felhom.eu/documentation/runbooks/workspace-CLAUDE.md (resolves to the versioned copy)
|
||||||
|
memory index : 200 lines (ceiling 200), 24953 bytes (ceiling 25600)
|
||||||
|
memory index content : 28 version literal(s), 3 host address(es), 0 expired statement(s) [WARN only]
|
||||||
|
memory topic files : 181 indexed, 4 orphaned, 41 archived
|
||||||
|
register citations : 6 cited, 1109 register items known
|
||||||
|
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:31: version literal '0.14.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||||
|
- **[restic 0.14.0 `--verify` is size+mtime, NOT content](restic-0140-verify-and-lock-sema
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:89: version literal '0.142.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||||
|
- [Offsite continuity](offsite-continuity-2026-07-17.md) — ctrl 0.142.0 orphan guard + hub
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:89: version literal '0.60.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||||
|
- [Offsite continuity](offsite-continuity-2026-07-17.md) — ctrl 0.142.0 orphan guard + hub
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:96: version literal '0.152.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||||
|
- **[Hub operator UI: CC uses its token HUB_TOKEN, not the password](hub-operator-ui-basic
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:100: version literal '0.90.1' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||||
|
- [Agent 0.90.1 is wrapper-only](agent-0901-is-wrapper-only.md) — do NOT publish the binar
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:102: version literal '0.144.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||||
|
- [SMB „Megosztás" R-7 slice 1](smb-sharing-r7-slice1-2026-07-18.md) — ctrl 0.144.0 + felh
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md: ... and 22 more version literal(s) — full list from the tally counts above.
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:58: host address '192.168.0.192' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||||
|
- [hub.felhom.eu resolves to the LAN here](hub-resolves-to-lan-on-dooplex-network.md) — 19
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:77: host address '167.233.158.164' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||||
|
- [Offsite PBS box RAM ceiling](offsite-pbs-box-ram-ceiling-2026-07-27.md) — **root SSH =
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:185: host address '192.168.0.180' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||||
|
- **CC runs ON DooPlex** (192.168.0.180) — repos `/mnt/5_hdd/felhom.eu/git/felhom-*`, buil
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory: 4 top-level topic file(s) are not referenced by MEMORY.md, so nothing will ever read them: f10-demo-hp-offsite-resolved-2026-07-23.md, launcher-v0163-2026-07-24.md, peti-return-p1-stop-2026-07-13.md, stopped-not-fault-v0164-2026-07-24.md
|
||||||
|
|
||||||
|
instructions_gate: OK (11 warning(s))
|
||||||
|
|
||||||
|
==============================================================================
|
||||||
|
== gate: release-complete (check-release-complete.py)
|
||||||
|
==============================================================================
|
||||||
|
check-release-complete — the newest CHANGELOG version is a complete release
|
||||||
|
newest CHANGELOG version: v0.159.1
|
||||||
|
ok tag v0.159.1 -> 5c3b218822, an ancestor of HEAD
|
||||||
|
ok package 0.159.1 is downloadable
|
||||||
|
|
||||||
|
check-release-complete: v0.159.1 is tagged, placed and published.
|
||||||
|
|
||||||
|
==============================================================================
|
||||||
|
== gate: observations (observations_gate.py /mnt/5_hdd/felhom.eu/git/felhom-agent)
|
||||||
|
==============================================================================
|
||||||
|
observations gate OK — REPORT.md has no observations section (nothing to check)
|
||||||
|
|
||||||
|
==============================================================================
|
||||||
|
== summary
|
||||||
|
==============================================================================
|
||||||
|
reuse-refs OK (exit 0)
|
||||||
|
instructions OK (exit 0)
|
||||||
|
release-complete OK (exit 7ea9c589094e81d27d741e625d9d8f9
|
||||||
|
[release-agent] config bundle published (HTTP 201)
|
||||||
|
[release-agent] pushing v0.159.2 …
|
||||||
|
remote: . Processing 1 references
|
||||||
|
remote: Processed 1 references in total
|
||||||
|
To https://gitea.dooplex.hu0)
|
||||||
|
observations OK (exit 0)
|
||||||
|
|
||||||
|
all agent gates OK
|
||||||
|
pre-push [felhom-agent]: gates OK - push proceeding.
|
||||||
|
|
||||||
|
RELEASED — and installable, verified by download, not by this script's own say-so.
|
||||||
|
|
||||||
|
version : 0.159.2
|
||||||
|
tag : v0.159.2
|
||||||
|
sha256 : 4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9
|
||||||
|
bundle : 477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9 (felhom-config-bundle.json — vouch it with the agent)
|
||||||
|
|
||||||
|
NOT VOUCHED. Vouching is what points machines at this version and stays you/admin/felhom-agent.git
|
||||||
|
* [new tag] v0.159.2 -> v0.159.2
|
||||||
|
[release-agent] verifying by independent download …
|
||||||
|
r deliberate act:
|
||||||
|
hub operator UI → Configs → Day-0 artifacts. Until then boxes keep installing the previous one.
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
== published installer read back 2026-10-18T12:32:10Z
|
||||||
|
SCRIPT_VERSION="1.34.0" # the SINGLE version source (F-1): -h and the run banners follow it.
|
||||||
|
served sha256 df722ee49f38da595e7c7017aa932b45853696c5282bc408b67a5fb8f141b67a
|
||||||
|
tag sha256 df722ee49f38da595e7c7017aa932b45853696c5282bc408b67a5fb8f141b67a
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# Website calculator — Immich before/after (2026-10-18)
|
||||||
|
|
||||||
|
Immich peak: **3,290 MB → 3,983 MB** (the kernel in-use figure from the 4,000-photo import, all four containers; the heap alone peaked at 2,777 MB). Immich typical: **1,557 MB, unchanged** (rest after the import: 1,230 MB heap — the raise-only rule keeps the higher). Headroom ×1.5, guest rule and platform unchanged.
|
||||||
|
|
||||||
|
## Immich alone — peak with the platform: 3,760 → 4,453 MB
|
||||||
|
|
||||||
|
| Machine | guest memory | before | after |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 8 GB | 4,096 MB | fits, but tight | does not fit **(changes)** |
|
||||||
|
| 12 GB | 7,580 MB | fits well | fits well |
|
||||||
|
| 16 GB | 11,676 MB | fits well | fits well |
|
||||||
|
| 24 GB | 19,868 MB | fits well | fits well |
|
||||||
|
| 32 GB | 28,060 MB | fits well | fits well |
|
||||||
|
| 48 GB | 44,444 MB | fits well | fits well |
|
||||||
|
| 64 GB | 60,828 MB | fits well | fits well |
|
||||||
|
|
||||||
|
## Family starter set — peak with the platform: 6,021 → 6,714 MB
|
||||||
|
|
||||||
|
| Machine | guest memory | before | after |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 8 GB | 4,096 MB | does not fit | does not fit |
|
||||||
|
| 12 GB | 7,580 MB | fits, but tight | fits, but tight |
|
||||||
|
| 16 GB | 11,676 MB | fits well | fits well |
|
||||||
|
| 24 GB | 19,868 MB | fits well | fits well |
|
||||||
|
| 32 GB | 28,060 MB | fits well | fits well |
|
||||||
|
| 48 GB | 44,444 MB | fits well | fits well |
|
||||||
|
| 64 GB | 60,828 MB | fits well | fits well |
|
||||||
|
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
=== demo-hp (enkisfelhom.hu)
|
||||||
|
containers: 24 running; not healthy/running: traefik Up 2 days;
|
||||||
|
unhealthy: 0
|
||||||
|
2026/10/18 11:09:36 updater.go:855: [INFO] [selfupdate] Post-update startup: update successful (0.329.0 → 0.330.0)
|
||||||
|
allow-list refusals since start: 0
|
||||||
|
2026/10/18 11:09:36 sync.go:142: [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
|
||||||
|
installed apps (stacks with app.yaml): 12
|
||||||
|
public https://felhom.enkisfelhom.hu/login: 200
|
||||||
|
public https://files.enkisfelhom.hu/: 200
|
||||||
|
=== demo-felhom (demo-felhom.eu)
|
||||||
|
containers: 9 running; not healthy/running: traefik Up 2 days;
|
||||||
|
unhealthy: 0
|
||||||
|
2026/10/18 11:09:25 [INFO] [selfupdate] Post-update startup: update successful (0.329.0 → 0.330.0)
|
||||||
|
allow-list refusals since start: 0
|
||||||
|
2026/10/18 11:09:26 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
|
||||||
|
installed apps (stacks with app.yaml): 5
|
||||||
|
public https://felhom.demo-felhom.eu/login: 200
|
||||||
|
public https://files.demo-felhom.eu/: 200
|
||||||
|
=== tester-1 (enkicsifelhom.hu)
|
||||||
|
containers: 10 running; not healthy/running: traefik Up 2 days;
|
||||||
|
unhealthy: 0
|
||||||
|
2026/10/18 11:09:35 [INFO] [selfupdate] Post-update startup: update successful (0.329.0 → 0.330.0)
|
||||||
|
allow-list refusals since start: 0
|
||||||
|
2026/10/18 11:09:36 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
|
||||||
|
installed apps (stacks with app.yaml): 3
|
||||||
|
public https://felhom.enkicsifelhom.hu/login: 200
|
||||||
|
public https://files.enkicsifelhom.hu/: 200
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
== global floor 2026-10-18T11:10:59Z
|
||||||
|
HTTP/1.1 303 See Other
|
||||||
|
Location: /configuration?flash=floor_set
|
||||||
|
clear tester-1:
|
||||||
|
HTTP/1.1 303 See Other
|
||||||
|
Location: /customers/tester-1?flash=floor_set
|
||||||
|
clear demo-hp:
|
||||||
|
HTTP/1.1 303 See Other
|
||||||
|
Location: /customers/demo-hp?flash=floor_set
|
||||||
|
clear demo-felhom:
|
||||||
|
HTTP/1.1 303 See Other
|
||||||
|
Location: /customers/demo-felhom?flash=floor_set
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
set tester-1:
|
||||||
|
HTTP/1.1 303 See Other
|
||||||
|
Location: /customers/tester-1?flash=floor_set
|
||||||
|
set demo-hp:
|
||||||
|
HTTP/1.1 303 See Other
|
||||||
|
Location: /customers/demo-hp?flash=floor_set
|
||||||
|
set demo-felhom:
|
||||||
|
HTTP/1.1 303 See Other
|
||||||
|
Location: /customers/demo-felhom?flash=floor_set
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
('min_controller_version', '0.330.0')
|
||||||
|
('min_controller_version_declared_min_agent', '0.330.0=0.158.1')
|
||||||
|
per-box floors set: 0
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
|
||||||
|
== every public name redirects http -> https
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
# P8 — old-restic check: NOT RUN (the clean-up window has not opened). Read 2026-10-18T14:33:43Z.
|
||||||
|
|
||||||
|
## demo-hp's clean-up windows (hub DB with its WAL, one tar)
|
||||||
|
id|customer|opened_at|closes_by|closed_at|box_result
|
||||||
|
9|demo-hp|2026-10-12 02:16:53|2026-10-12 02:36:53|2026-10-12 02:17:06|pruned
|
||||||
|
6|demo-hp|2026-10-05 05:25:53|2026-10-05 05:45:53|2026-10-05 05:26:04|pruned
|
||||||
|
2|demo-hp|2026-10-04 05:36:14|2026-10-04 05:56:14|2026-10-04 05:36:17|nothing
|
||||||
|
|
||||||
|
## The rule (hub/internal/offsitekeys/service.go:255)
|
||||||
|
windowCadence = 6*24*time.Hour + 12*time.Hour // "weekly", with slack for the night chain's drift
|
||||||
|
Last window 2026-10-12 02:16:53 UTC + 6 d 12 h = 2026-10-18 14:16:53 UTC (the STATUS time).
|
||||||
|
The box asks for a window only AFTER its nightly off-site run (controller backup/offbox_window.go header).
|
||||||
|
|
||||||
|
## demo-hp guest 9201 controller (docker logs since 06:00 UTC, [offbox] lines + scheduler)
|
||||||
|
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.330.0 Up 3 hours (healthy)
|
||||||
|
2026/10/18 11:09:36 scheduler.go:150: [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-10-19 04:15 CEST
|
||||||
|
2026/10/18 11:09:36 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="offbox-backup" schedule="04:15" nextRun=2026-10-19T04:15:00+02:00 totalJobs=15
|
||||||
|
2026/10/18 11:09:36 scheduler.go:85: [DEBUG] [scheduler] daily job offbox-backup: next run at 2026-10-19 04:15:00 CEST (waiting 15h5m23s)
|
||||||
|
|
||||||
|
Next run: 2026-10-19 04:15 CEST = 02:15 UTC; the window opens right after it. No window was asked for or granted by CC.
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
Mon Oct 19 01:24:39 UTC 2026
|
||||||
|
200
|
||||||
|
{"data": {"started": true}, "ok": true}
|
||||||
|
200
|
||||||
|
{
|
||||||
|
"data": {
|
||||||
|
"backup": {
|
||||||
|
"target_id": "local",
|
||||||
|
"vmid": 9201,
|
||||||
|
"archive": "local:backup/vzdump-lxc-9201-2026_10_19-03_25_27.tar.zst",
|
||||||
|
"mode": "snapshot",
|
||||||
|
"crash_consistent": true,
|
||||||
|
"size_bytes": 6864786948,
|
||||||
|
"success": true,
|
||||||
|
"started_at": "2026-10-19T01:25:26Z",
|
||||||
|
"duration_seconds": 443.050964897
|
||||||
|
},
|
||||||
|
"error": "",
|
||||||
|
"job_id": "backup-9201-1792373126635435331",
|
||||||
|
"phase": "done"
|
||||||
|
},
|
||||||
|
"ok": true
|
||||||
|
}
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
Mon Oct 19 01:52:00 UTC 2026
|
||||||
|
tester-1-d70be4 agent_update:
|
||||||
|
signed: op=agent_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=3516434bfa0c5c47295e3de0a8bd20c7 expires=2026-10-19T02:37:00Z
|
||||||
|
uploaded signed op to the hub jobs queue
|
||||||
|
tester-1-d70be4 agent_config_update:
|
||||||
|
signed: op=agent_config_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=2dfc5b3e216030c82add18ba69899b41 expires=2026-10-19T02:37:00Z
|
||||||
|
uploaded signed op to the hub jobs queue
|
||||||
|
demo-felhom-8363b5 agent_update:
|
||||||
|
signed: op=agent_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=8f3a6c280cb124c6f85a4039c5b112dc expires=2026-10-19T02:37:00Z
|
||||||
|
uploaded signed op to the hub jobs queue
|
||||||
|
demo-felhom-8363b5 agent_config_update:
|
||||||
|
signed: op=agent_config_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=dc2ead6937a38d0a2d8acd4abf57bef9 expires=2026-10-19T02:37:00Z
|
||||||
|
uploaded signed op to the hub jobs queue
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
2026-10-19T01:55:42Z tester-1: felhom-agent 0.160.0 | demo-felhom: felhom-agent 0.160.0
|
||||||
|
Oct 19 03:54:05 felhom felhom-agent[1371657]: time=2026-10-19T03:54:05.642+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=a254c4ec38459c7c op=agent_update
|
||||||
|
Oct 19 03:54:06 felhom felhom-agent[1371657]: time=2026-10-19T03:54:06.786+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.160.0 written=0 same=30 self-check=ok signers-created=False"
|
||||||
|
Oct 19 03:54:07 felhom felhom-agent[1371657]: time=2026-10-19T03:54:07.652+02:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=75 total=75 degraded=""
|
||||||
|
Oct 19 03:54:07 felhom felhom-agent[1371657]: time=2026-10-19T03:54:07.653+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=a66f7fc0b9fd8fb6 op=agent_config_update
|
||||||
|
Oct 19 03:54:08 felhom felhom-agent[2443463]: time=2026-10-19T03:54:08.649+02:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true
|
||||||
|
Oct 19 03:54:08 felhom felhom-agent[2443463]: time=2026-10-19T03:54:08.649+02:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
|
||||||
|
Oct 19 03:55:10 felhom felhom-agent[2443463]: time=2026-10-19T03:55:10.315+02:00 level=WARN msg="selfupdate: update committed" version=0.160.0 wrapper=""
|
||||||
|
Oct 19 03:55:15 demo-felhom felhom-os-apply[498026]: os-apply: BUNDLE DONE agent=0.160.0 written=0 same=30 self-check=ok signers-created=False
|
||||||
|
Oct 19 03:55:15 demo-felhom felhom-agent[3800522]: time=2026-10-19T03:55:15.698+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.160.0 written=0 same=30 self-check=ok signers-created=False"
|
||||||
|
Oct 19 03:55:16 demo-felhom felhom-agent[3800522]: time=2026-10-19T03:55:16.340+02:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=75 total=75 degraded=""
|
||||||
|
Oct 19 03:55:16 demo-felhom felhom-agent[3800522]: time=2026-10-19T03:55:16.340+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=7dde89f299a0684c op=agent_config_update
|
||||||
|
Oct 19 03:55:17 demo-felhom felhom-agent[498196]: time=2026-10-19T03:55:17.698+02:00 level=INFO msg="backup tier armed" target=felhom-backup cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true
|
||||||
|
Oct 19 03:55:17 demo-felhom felhom-agent[498196]: time=2026-10-19T03:55:17.698+02:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
|
||||||
|
Oct 19 03:56:19 demo-felhom felhom-agent[498196]: time=2026-10-19T03:56:19.667+02:00 level=WARN msg="selfupdate: update committed" version=0.160.0 wrapper=""
|
||||||
|
|
||||||
|
[exited with code 0]
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
Volid Format Type Size VMID
|
||||||
|
local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst tar.zst backup 656970239 9100
|
||||||
|
local:backup/vzdump-lxc-9201-2026_10_19-03_25_27.tar.zst tar.zst backup 6864786948 9201
|
||||||
|
local:backup/felhom-golden-0.236.0.tar.zst tar.zst backup 654115664
|
||||||
|
Oct 19 03:23:37 demo-hp felhom-agent[2622137]: time=2026-10-19T03:23:37.628+02:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=1 prune="keep-daily=1" wait_timeout=30m0s prune_pbs_allowed=false primary=true
|
||||||
|
Oct 19 03:23:37 demo-hp felhom-agent[2622137]: time=2026-10-19T03:23:37.628+02:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
|
||||||
|
Oct 19 03:24:39 demo-hp felhom-agent[2622137]: time=2026-10-19T03:24:39.730+02:00 level=WARN msg="selfupdate: update committed" version=0.160.0 wrapper=""
|
||||||
|
Oct 19 03:32:49 demo-hp felhom-agent[2622137]: time=2026-10-19T03:32:49.686+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_10_19-03_25_27.tar.zst size_bytes=6864786948 uncovered_volumes=2
|
||||||
|
Oct 19 03:32:49 demo-hp felhom-agent[2622137]: time=2026-10-19T03:32:49.686+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1792373126635435331 archive=local:backup/vzdump-lxc-9201-2026_10_19-03_25_27.tar.zst
|
||||||
|
-rw------- 1 felhom-agent felhom-agent 323 Oct 19 03:32 /var/lib/felhom-agent/backup-success-state.json
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"target": "felhom-pbs",
|
||||||
|
"vmid": 9201,
|
||||||
|
"started_at": "2026-10-15T20:38:46Z",
|
||||||
|
"recent": [
|
||||||
|
"2026-10-15T20:38:46Z"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"target": "local",
|
||||||
|
"vmid": 9201,
|
||||||
|
"started_at": "2026-10-19T01:25:26Z",
|
||||||
|
"recent": [
|
||||||
|
"2026-10-18T11:12:25Z",
|
||||||
|
"2026-10-19T01:25:26Z"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
Mon Oct 19 01:22:11 UTC 2026
|
||||||
|
felhom-agent 0.159.2
|
||||||
|
demo-hp-bb76ea
|
||||||
|
Volid Format Type Size VMID
|
||||||
|
local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst tar.zst backup 656970239 9100
|
||||||
|
local:backup/vzdump-lxc-9201-2026_10_18-13_12_26.tar.zst tar.zst backup 6938307162 9201
|
||||||
|
local:backup/felhom-golden-0.236.0.tar.zst tar.zst backup 654115664
|
||||||
|
"local_backup_retention": 1
|
||||||
|
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
|
||||||
|
341 night1004-tester1 running 8192 200.00 1851
|
||||||
|
perl: warning: Setting locale failed.
|
||||||
|
perl: warning: Please check that your locale settings:
|
||||||
|
LANGUAGE = (unset),
|
||||||
|
LC_ALL = (unset),
|
||||||
|
LC_CTYPE = "UTF-8",
|
||||||
|
LC_NUMERIC = (unset),
|
||||||
|
LC_COLLATE = (unset),
|
||||||
|
LC_TIME = (unset),
|
||||||
|
LC_MESSAGES = (unset),
|
||||||
|
LC_MONETARY = (unset),
|
||||||
|
LC_ADDRESS = (unset),
|
||||||
|
LC_IDENTIFICATION = (unset),
|
||||||
|
LC_MEASUREMENT = (unset),
|
||||||
|
LC_PAPER = (unset),
|
||||||
|
LC_TELEPHONE = (unset),
|
||||||
|
LC_NAME = (unset),
|
||||||
|
LANG = "en_US.UTF-8"
|
||||||
|
are supported and installed on your system.
|
||||||
|
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
|
||||||
|
VMID Status Lock Name
|
||||||
|
9201 running demo-hp
|
||||||
|
9202 running demo-hp-scratch
|
||||||
|
9401 stopped upgrade-harness
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
Mon Oct 19 01:22:47 UTC 2026
|
||||||
|
demo-hp-bb76ea agent_update:
|
||||||
|
signed: op=agent_update host=demo-hp-bb76ea guest="" key_id=felhom-op-1 nonce=71396ac328c9ea3192d027b830ec0990 expires=2026-10-19T02:07:47Z
|
||||||
|
wrote envelope to /dev/null
|
||||||
|
uploaded signed op to the hub jobs queue
|
||||||
|
demo-hp-bb76ea agent_config_update:
|
||||||
|
signed: op=agent_config_update host=demo-hp-bb76ea guest="" key_id=felhom-op-1 nonce=603f1d8ed0937492cf01a05146fc9e48 expires=2026-10-19T02:07:47Z
|
||||||
|
wrote envelope to /dev/null
|
||||||
|
uploaded signed op to the hub jobs queue
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
Mon Oct 19 01:47:33 UTC 2026
|
||||||
|
code 0 (the stored demo-hp code, before):
|
||||||
|
HTTP 200 opens; restic_pw_sha256 8a9e33aa
|
||||||
|
control, a wrong code:
|
||||||
|
HTTP 400 the recovery code did not open the sealed bundle — nothing was written
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
01:47:43Z start 200 {"data":{"job_id":"escrow-1792374463410689900","phase":"running"},"error":"","ok":true}
|
||||||
|
|
||||||
|
status 200 {"phase": "done", "claimable": true, "uploaded": false, "restic_pw_sealed": true, "detail": ""}
|
||||||
|
claim 200 code words: 10 {"confirm_required": true, "verify_positions": [3, 10]}
|
||||||
|
status after claim {"phase": "done", "claimed": true, "uploaded": false}
|
||||||
|
confirm with one wrong word 422 A megadott szavak nem egyeznek — ellenőrizd a felírt kódot.
|
||||||
|
01:47:47Z confirm with the right words 200 {"uploaded": true}
|
||||||
|
code 1 after the confirm:
|
||||||
|
HTTP 200 opens; restic_pw_sha256 8a9e33aa
|
||||||
|
code 0 after the confirm:
|
||||||
|
HTTP 422 the recovery code is correct, but it belongs to an EARLIER sealed package (superseded 2026-10-19T01:47:47Z), not the one
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
01:48:09Z start 200 {"data":{"job_id":"escrow-1792374489199785658","phase":"running"},"error":"","ok":true}
|
||||||
|
|
||||||
|
status 200 {"phase": "done", "claimable": true, "uploaded": false, "restic_pw_sealed": true, "detail": ""}
|
||||||
|
claim 200 code words: 10 {"confirm_required": true, "verify_positions": [5, 10]}
|
||||||
|
status after claim {"phase": "done", "claimed": true, "uploaded": false}
|
||||||
|
after the abandon — code 1 (stored):
|
||||||
|
HTTP 200 opens; restic_pw_sha256 8a9e33aa
|
||||||
|
code 2 (shown, never typed back):
|
||||||
|
HTTP 400 the recovery code did not open the sealed bundle — nothing was written
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
time=2026-10-19T03:47:33.706+02:00 level=INFO msg="local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)" vmid=9201
|
||||||
|
time=2026-10-19T03:47:34.786+02:00 level=INFO msg="local-api: offsite repository password RECOVERED from the sealed escrow — returning that field ONLY (the tunnel token, the PBS token and the WG key stay inside the agent and are not returned)" vmid=9201 rest
|
||||||
|
time=2026-10-19T03:47:35.044+02:00 level=INFO msg="local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)" vmid=9201
|
||||||
|
time=2026-10-19T03:47:36.053+02:00 level=WARN msg="local-api: offsite key recovery: the fetched bundle did not open with the supplied recovery code" vmid=9201 err="escrow: the recovery code did not unwrap the identity escrow (wrong recovery code, or a corrupt
|
||||||
|
time=2026-10-19T03:47:43.409+02:00 level=INFO msg="local-api: staged offsite restic repo password for escrow (field name only)" vmid=9201
|
||||||
|
time=2026-10-19T03:47:43.410+02:00 level=INFO msg="local-api: escrow ceremony started (controller-driven)" vmid=9201 job_id=escrow-1792374463410689900
|
||||||
|
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=escrow-create --upload --output=json
|
||||||
|
time=2026-10-19T03:47:45.992+02:00 level=INFO msg="local-api: escrow ceremony complete — R claimable (in-memory, one-shot); package uploads at the claim" job_id=escrow-1792374463410689900 restic_pw_sealed=true uploaded=false claim_ttl_s=600 duration_ms=2581
|
||||||
|
time=2026-10-19T03:47:47.507+02:00 level=INFO msg="local-api: escrow ceremony R claimed — the package waits for the typed-back words (R-1151)" vmid=9201 job_id=escrow-1792374463410689900 confirm_ttl_s=1800
|
||||||
|
time=2026-10-19T03:47:47.550+02:00 level=INFO msg="local-api: escrow confirm — the typed words do not match; still waiting (R-1151)" vmid=9201
|
||||||
|
time=2026-10-19T03:47:47.619+02:00 level=INFO msg="local-api: escrow confirm — words typed back; the new package is on the hub (R-1151)" vmid=9201 job_id=escrow-1792374463410689900 duration_ms=48
|
||||||
|
time=2026-10-19T03:47:48.247+02:00 level=INFO msg="local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)" vmid=9201
|
||||||
|
time=2026-10-19T03:47:49.287+02:00 level=INFO msg="local-api: offsite repository password RECOVERED from the sealed escrow — returning that field ONLY (the tunnel token, the PBS token and the WG key stay inside the agent and are not returned)" vmid=9201 rest
|
||||||
|
time=2026-10-19T03:47:49.635+02:00 level=INFO msg="local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)" vmid=9201
|
||||||
|
time=2026-10-19T03:48:09.199+02:00 level=INFO msg="local-api: staged offsite restic repo password for escrow (field name only)" vmid=9201
|
||||||
|
time=2026-10-19T03:48:09.199+02:00 level=INFO msg="local-api: escrow ceremony started (controller-driven)" vmid=9201 job_id=escrow-1792374489199785658
|
||||||
|
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=escrow-create --upload --output=json
|
||||||
|
time=2026-10-19T03:48:11.881+02:00 level=INFO msg="local-api: escrow ceremony complete — R claimable (in-memory, one-shot); package uploads at the claim" job_id=escrow-1792374489199785658 restic_pw_sealed=true uploaded=false claim_ttl_s=600 duration_ms=2681
|
||||||
|
time=2026-10-19T03:48:13.294+02:00 level=INFO msg="local-api: escrow ceremony R claimed — the package waits for the typed-back words (R-1151)" vmid=9201 job_id=escrow-1792374489199785658 confirm_ttl_s=1800
|
||||||
|
time=2026-10-19T03:48:14.059+02:00 level=INFO msg="local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)" vmid=9201
|
||||||
|
time=2026-10-19T03:48:15.141+02:00 level=INFO msg="local-api: offsite repository password RECOVERED from the sealed escrow — returning that field ONLY (the tunnel token, the PBS token and the WG key stay inside the agent and are not returned)" vmid=9201 rest
|
||||||
|
time=2026-10-19T03:48:15.389+02:00 level=INFO msg="local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)" vmid=9201
|
||||||
|
time=2026-10-19T03:48:17.435+02:00 level=WARN msg="local-api: offsite key recovery: the fetched bundle did not open with the supplied recovery code" vmid=9201 err="escrow: the recovery code did not unwrap the identity escrow (wrong recovery code, or a corrupt
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
Mon Oct 19 02:08:15 UTC 2026
|
||||||
|
demo-felhom-8363b5|0.160.0|2026-10-19 01:55:31
|
||||||
|
demo-hp-bb76ea|0.160.0|2026-10-19 01:53:50
|
||||||
|
tester-1-d70be4|0.160.0|2026-10-19 01:54:15
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
== root@192.168.0.154
|
||||||
|
Mon Oct 19 01:51:20 UTC 2026
|
||||||
|
felhom-agent 0.159.2
|
||||||
|
"host_id": "tester-1-d70be4"
|
||||||
|
"local_backup_retention": 3
|
||||||
|
Volid Format Type Size VMID
|
||||||
|
local:backup/vzdump-lxc-9100-2026_10_04-21_40_45.tar.zst tar.zst backup 648135998 9100
|
||||||
|
local:backup/vzdump-lxc-9201-2026_10_15-22_43_40.tar.zst tar.zst backup 3364084456 9201
|
||||||
|
local:backup/vzdump-lxc-9201-2026_10_17-04_32_43.tar.zst tar.zst backup 3479976346 9201
|
||||||
|
local:backup/vzdump-lxc-9201-2026_10_18-04_37_41.tar.zst tar.zst backup 1867626596 9201
|
||||||
|
== felhom-pve
|
||||||
|
Mon Oct 19 01:51:13 UTC 2026
|
||||||
|
felhom-agent 0.159.2
|
||||||
|
"host_id": "demo-felhom-8363b5"
|
||||||
|
"local_backup_retention": 3
|
||||||
|
Volid Format Type Size VMID
|
||||||
|
local:backup/vzdump-lxc-9100-2026_07_18-18_30_52.tar.zst tar.zst backup 612397450 9100
|
||||||
|
local:backup/vzdump-lxc-9100-2026_07_20-17_50_57.tar.zst tar.zst backup 648549366 9100
|
||||||
|
local:backup/vzdump-lxc-9100-2026_08_03-06_58_00.tar.zst tar.zst backup 649547835 9100
|
||||||
|
local:backup/vzdump-lxc-9100-2026_08_03-09_15_34.tar.zst tar.zst backup 649547835 9100
|
||||||
|
local:backup/vzdump-lxc-9201-2026_07_28-01_27_56.tar.zst tar.zst backup 5820882781 9201
|
||||||
|
local:backup/vzdump-lxc-9201-2026_07_28-06_44_22.tar.zst tar.zst backup 5840583923 9201
|
||||||
|
local:backup/vzdump-lxc-9201-2026_07_28-17_53_26.tar.zst tar.zst backup 5929975285 9201
|
||||||
|
{'local_backup_target': 'felhom-backup', 'local_backup_retention': 3, 'restore_test_cadence_seconds': 0, 'backup_cadence_seconds': 0, 'backup_targets': [{'target_id': 'felhom-pbs', 'cadence_seconds': 604800, 'keep_last': 0}]}
|
||||||
|
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
|
||||||
|
felhom-backup dir active 960303848 7518604 903930732 0.78%
|
||||||
|
felhom-pbs pbs active 0 0 0 0.00%
|
||||||
|
local dir active 98497780 33631712 59816520 34.14%
|
||||||
|
local-lvm lvmthin active 365760512 11082543 354677968 3.03%
|
||||||
|
time=2026-10-16T05:03:53.319+02:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-backup archive=felhom-backup:backup/vzdump-lxc-9201-2026_10_16-05_03_12.tar.zst size_bytes=1300947312 uncovered_volumes=2
|
||||||
|
time=2026-10-17T05:07:43.382+02:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-backup archive=felhom-backup:backup/vzdump-lxc-9201-2026_10_17-05_07_02.tar.zst size_bytes=1334976334 uncovered_volumes=2
|
||||||
|
time=2026-10-18T05:08:13.307+02:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-backup archive=felhom-backup:backup/vzdump-lxc-9201-2026_10_18-05_07_32.tar.zst size_bytes=1296570428 uncovered_volumes=2
|
||||||
|
Volid Format Type Size VMID
|
||||||
|
felhom-backup:backup/vzdump-lxc-9201-2026_10_16-05_03_12.tar.zst tar.zst backup 1300947312 9201
|
||||||
|
felhom-backup:backup/vzdump-lxc-9201-2026_10_17-05_07_02.tar.zst tar.zst backup 1334976334 9201
|
||||||
|
felhom-backup:backup/vzdump-lxc-9201-2026_10_18-05_07_32.tar.zst tar.zst backup 1296570428 9201
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
time=2026-10-19T03:49:56.342+02:00 level=INFO msg="audit: gate decision" class=storage_wipe host=demo-hp-bb76ea guest="" source=one_shot_job disposition=customer_confirmable allowed=false reason=pending_confirmation key_id="" nonce="" durab
|
||||||
|
time=2026-10-19T03:49:56.342+02:00 level=INFO msg="gate decision (storage wipe)" role=user-data disposition=customer_confirmable allowed=false reason=pending_confirmation durable_id=byid:wwn-0x3333333000000fa0-part1+fsuuid:ef8fc766-1e12-491
|
||||||
|
time=2026-10-19T03:49:56.342+02:00 level=WARN msg="local-api: user-data data-bearing format refused" vmid=9201 device=/dev/sdb1 durable_id=byid:wwn-0x3333333000000fa0-part1+fsuuid:ef8fc766-1e12-491e-b6f2-31433e1989c8 reason=pending_confirma
|
||||||
|
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||||
|
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||||
|
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||||
|
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||||
|
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||||
|
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||||
|
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||||
|
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||||
|
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||||
|
backup-success-state.json
|
||||||
|
controller-swap-9201.json
|
||||||
|
drive-intents.json
|
||||||
|
escrow-stage
|
||||||
|
felhom-sshd
|
||||||
|
format-job.json
|
||||||
|
guest-binds.json
|
||||||
|
guest-disk-trim.json
|
||||||
|
guests
|
||||||
|
journal.log
|
||||||
|
local-api.crt
|
||||||
|
local-api.key
|
||||||
|
local-tokens.log
|
||||||
|
nonces.log
|
||||||
|
os
|
||||||
|
pbsdr
|
||||||
|
restore-test-state.json
|
||||||
|
selfupdate
|
||||||
|
units
|
||||||
|
wg
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
Mon Oct 19 01:49:49 UTC 2026
|
||||||
|
new test disk: /dev/sdb
|
||||||
|
NAME SIZE MODEL
|
||||||
|
sdb 64M scsi_debug
|
||||||
|
--- the empty partition (the R-1093 input):
|
||||||
|
{
|
||||||
|
"blockdevices": [
|
||||||
|
{
|
||||||
|
"name": "sdb1",
|
||||||
|
"fstype": null,
|
||||||
|
"pttype": "gpt"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
DEVNAME=/dev/sdb1
|
||||||
|
PART_ENTRY_SCHEME=gpt
|
||||||
|
PART_ENTRY_UUID=aec56a69-b9a0-46de-9c8e-911ca8558717
|
||||||
|
PART_ENTRY_TYPE=0fc63daf-8483-4772-8e79-3d69d8477de4
|
||||||
|
PART_ENTRY_NUMBER=1
|
||||||
|
PART_ENTRY_OFFSET=2048
|
||||||
|
PART_ENTRY_SIZE=126976
|
||||||
|
PART_ENTRY_DISK=8:16
|
||||||
|
blkid rc=0
|
||||||
|
lrwxrwxrwx 1 root root 9 Oct 19 03:49 scsi-33333333000000fa0 -> ../../sdb
|
||||||
|
lrwxrwxrwx 1 root root 10 Oct 19 03:49 scsi-33333333000000fa0-part1 -> ../../sdb1
|
||||||
|
lrwxrwxrwx 1 root root 9 Oct 19 03:49 wwn-0x3333333000000fa0 -> ../../sdb
|
||||||
|
--- POST /disks/format on the EMPTY partition (no confirmation sent):
|
||||||
|
HTTP 200 {"ok":true,"data":{"vmid":9201,"device":"/dev/sdb1","formatted":true,"data_bearing":false,"reason":"blank device formatted ext4","durable_id":"byid:wwn-0x3333333000000fa0-part1","fs_uuid":"ef8fc766-1e12-491e-b6f2-31433e1989c8"}}
|
||||||
|
|
||||||
|
--- the partition now:
|
||||||
|
TYPE=ext4
|
||||||
|
PART_ENTRY_SCHEME=gpt
|
||||||
|
--- CONTROL: the same call again, on the now-ext4 partition:
|
||||||
|
HTTP 403 {"ok":false,"data":{"vmid":9201,"device":"/dev/sdb1","formatted":false,"data_bearing":true,"reason":"device has a ext4 filesystem","role":"user-data","needs_confirmation":true,"durable_id":"byid:wwn-0x3333333000000fa0-part1+fsuuid:ef8fc766-1e12-491e-b6f2-31433e1989c8"},"error":"device is data-bearing — customer confirmation required"}
|
||||||
|
|
||||||
|
--- teardown
|
||||||
|
not mounted
|
||||||
|
scsi_debug removed
|
||||||
|
0
|
||||||
@@ -0,0 +1,102 @@
|
|||||||
|
[release-agent] building 0.160.0 …
|
||||||
|
[release-agent] built ok: sha256 7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1
|
||||||
|
[release-agent] config bundle built: sha256 6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f
|
||||||
|
[release-agent] tagging v0.160.0 at c11e3e2 …
|
||||||
|
[release-agent] publishing …
|
||||||
|
[publish-agent] publishing /tmp/felhom-agent-YBD0P9 (16071264 bytes, sha256 7ebb547f2b1ab935…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/0.160.0/felhom-agent
|
||||||
|
[publish-agent] pre-delete existing artifact: HTTP 404 (404/204 expected)
|
||||||
|
[publish-agent] upload OK (HTTP 201)
|
||||||
|
[publish-agent] round-trip GET verified (sha256 matches)
|
||||||
|
[publish-agent] DONE. Record in the hub operator UI (Configs → Day-0 artifacts): agent 0.160.0 / 7ebb547f2b1ab93544ba828d3040351b6AGENT_VERSION=0.160.0
|
||||||
|
AGENT_SHA256=7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1
|
||||||
|
pre-push [felhom-agent]: running scripts/agent_gates.py --fast ...
|
||||||
|
agent_gates — 4 gate(s) [--fast]
|
||||||
|
--fast SKIPPED (deliberate periodic runs, never in a hook): published, gate-decoys
|
||||||
|
|
||||||
|
==============================================================================
|
||||||
|
== gate: reuse-refs (reuse_refs_check.py /mnt/5_hdd/felhom.eu/git/felhom-agent)
|
||||||
|
==============================================================================
|
||||||
|
note [felhom-agent] line 190: localapi/debuglogs_test.go (resolved by suffix → internal/localapi/debuglogs_test.go)
|
||||||
|
note [felhom-agent] line 206: hub/internal/store/dr_recipe.go (cross-repo → felhom.eu/hub/internal/store/dr_recipe.go)
|
||||||
|
OK [felhom-agent]: 118 cited paths — exact 116, suffix 1, ambiguous 0, cross-repo 1, FAILED 0 (siblings searched: app-catalog-felhom.eu, cc-secrets, felhom-controller, felhom.eu, felhom.eu-media, felhom.eu-sizing-push, felhom.eu-wt-sizing)
|
||||||
|
|
||||||
|
==============================================================================
|
||||||
|
== gate: instructions (instructions_gate.py /mnt/5_hdd/felhom.eu/git/felhom-agent)
|
||||||
|
==============================================================================
|
||||||
|
instructions_gate: /mnt/5_hdd/felhom.eu/git/felhom-agent
|
||||||
|
CLAUDE.md effective lines : 108 (ceiling 200)
|
||||||
|
version literals : 0
|
||||||
|
TEMPORARY blocks : 0
|
||||||
|
rule files : 6 (5 path-scoped)
|
||||||
|
workspace file : SYMLINK -> felhom.eu/documentation/runbooks/workspace-CLAUDE.md (resolves to the versioned copy)
|
||||||
|
memory index : 200 lines (ceiling 200), 25109 bytes (ceiling 25600)
|
||||||
|
memory index content : 28 version literal(s), 3 host address(es), 0 expired statement(s) [WARN only]
|
||||||
|
memory topic files : 183 indexed, 4 orphaned, 41 archived
|
||||||
|
register citations : 6 cited, 1116 register items known
|
||||||
|
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:31: version literal '0.14.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||||
|
- **[restic 0.14.0 `--verify` is size+mtime, NOT content](restic-0140-verify-and-lock-sema
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:90: version literal '0.142.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||||
|
- [Offsite continuity](offsite-continuity-2026-07-17.md) — ctrl 0.142.0 orphan guard + hub
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:90: version literal '0.60.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||||
|
- [Offsite continuity](offsite-continuity-2026-07-17.md) — ctrl 0.142.0 orphan guard + hub
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:97: version literal '0.152.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||||
|
- **[Hub operator UI: CC uses its token HUB_TOKEN, not the password](hub-operator-ui-basic
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:101: version literal '0.90.1' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||||
|
- [Agent 0.90.1 is wrapper-only](agent-0901-is-wrapper-only.md) — do NOT publish the binar
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:103: version literal '0.144.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||||
|
- [SMB „Megosztás" R-7 slice 1](smb-sharing-r7-slice1-2026-07-18.md) — ctrl 0.144.0 + felh
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md: ... and 22 more version literal(s) — full list from the tally counts above.
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:59: host address '192.168.0.192' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||||
|
- [hub.felhom.eu resolves to the LAN here](hub-resolves-to-lan-on-dooplex-network.md) — 19
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:78: host address '167.233.158.164' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||||
|
- [Offsite PBS box RAM ceiling](offsite-pbs-box-ram-ceiling-2026-07-27.md) — **root SSH =
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:186: host address '192.168.0.180' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
|
||||||
|
- **CC runs ON DooPlex** (192.168.0.180) — repos `/mnt/5_hdd/felhom.eu/git/felhom-*`, buil
|
||||||
|
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory: 4 top-level topic file(s) are not referenced by MEMORY.md, so nothing will ever read them: f10-demo-hp-offsite-resolved-2026-07-23.md, launcher-v0163-2026-07-24.md, peti-return-p1-stop-2026-07-13.md, stopped-not-fault-v0164-2026-07-24.md
|
||||||
|
|
||||||
|
instructions_gate: OK (11 warning(s))
|
||||||
|
|
||||||
|
==============================================================================
|
||||||
|
== gate: release-complete (check-release-complete.py)
|
||||||
|
==============================================================================
|
||||||
|
check-release-complete — the newest CHANGELOG version is a complete release
|
||||||
|
newest CHANGELOG version: v0.159.2
|
||||||
|
ok tag v0.159.2 -> 65c1994d3b, an ancestor of HEAD
|
||||||
|
ok package 0.159.2 is downloadable
|
||||||
|
|
||||||
|
check-release-complete: v0.159.2 is tagged, placed and published.
|
||||||
|
|
||||||
|
==============================================================================
|
||||||
|
== gate: observations (observations_gate.py /mnt/5_hdd/felhom.eu/git/felhom-agent)
|
||||||
|
==============================================================================
|
||||||
|
observations gate OK — REPORT.md has no observations section (nothing to check)
|
||||||
|
|
||||||
|
==============================================================================
|
||||||
|
== summary
|
||||||
|
==============================================================================
|
||||||
|
reuse-refs OK (exit 0)
|
||||||
|
instructions OK (exit 0)
|
||||||
|
release-complete OK (exit 0c4126327a5b85bbcb33f8f2b3dfca1
|
||||||
|
[release-agent] config bundle published (HTTP 201)
|
||||||
|
[release-agent] pushing v0.160.0 …
|
||||||
|
remote: . Processing 1 references
|
||||||
|
remote: Processed 1 references in total
|
||||||
|
To https://gitea.dooplex.hu0)
|
||||||
|
observations OK (exit 0)
|
||||||
|
|
||||||
|
all agent gates OK
|
||||||
|
pre-push [felhom-agent]: gates OK - push proceeding.
|
||||||
|
|
||||||
|
RELEASED — and installable, verified by download, not by this script's own say-so.
|
||||||
|
|
||||||
|
version : 0.160.0
|
||||||
|
tag : v0.160.0
|
||||||
|
sha256 : 7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1
|
||||||
|
bundle : 6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f (felhom-config-bundle.json — vouch it with the agent)
|
||||||
|
|
||||||
|
NOT VOUCHED. Vouching is what points machines at this version and stays you/admin/felhom-agent.git
|
||||||
|
* [new tag] v0.160.0 -> v0.160.0
|
||||||
|
[release-agent] verifying by independent download …
|
||||||
|
r deliberate act:
|
||||||
|
hub operator UI → Configs → Day-0 artifacts. Until then boxes keep installing the previous one.
|
||||||
Binary file not shown.
|
After Width: | Height: | Size: 99 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 79 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 36 KiB |
@@ -0,0 +1,17 @@
|
|||||||
|
03:05:34 goto https://felhom.enkisfelhom.hu/login -> https://felhom.enkisfelhom.hu/login
|
||||||
|
03:05:39 eval -> "focused"
|
||||||
|
03:05:43 eval -> "https://felhom.enkisfelhom.hu/launcher"
|
||||||
|
03:05:43 goto https://felhom.enkisfelhom.hu/apps/calibre-web -> https://felhom.enkisfelhom.hu/apps/calibre-web
|
||||||
|
03:05:48 eval -> "https://felhom.enkisfelhom.hu/__files/open?to=… | button: Fájlok ↗ | folder: /files/NVME 1TB/media/books"
|
||||||
|
03:05:48 shot 01-app-page-files-button
|
||||||
|
03:05:49 eval -> "pressed Fájlok"
|
||||||
|
03:05:57 eval -> "files.enkisfelhom.hu/files/NVME 1TB/media/books | login form: no | title: FileBrowser Quantum - Files - books"
|
||||||
|
03:05:57 eval -> "FileBrowser user: felhom-household | admin: false | modify: true | loginMethod: proxy"
|
||||||
|
03:05:59 shot 02-filebrowser-opened-signed-in
|
||||||
|
03:05:59 goto https://felhom.enkisfelhom.hu/launcher -> https://felhom.enkisfelhom.hu/launcher
|
||||||
|
03:06:04 eval -> "signed out on the dashboard"
|
||||||
|
03:06:08 eval -> "https://felhom.enkisfelhom.hu/login"
|
||||||
|
03:06:08 goto https://files.enkisfelhom.hu/files/ -> https://felhom.enkisfelhom.hu/login
|
||||||
|
03:06:15 eval -> "felhom.enkisfelhom.hu/login | login form: YES"
|
||||||
|
03:06:15 shot 03-after-sign-out-asks-again
|
||||||
|
LOG ('netfail', 'net::ERR_ABORTED', 'Fetch')
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user